- anonymous RSS allows about one request per minute per IP, so fetching each
subreddit separately hit a 429 and a 60s back-off on nearly every run
- search the subreddits as one combined feed at Reddit's full page size and
group posts by the subreddit each entry names; each subreddit keeps its own
limit, and a full page is not taken as evidence of absence
- drop the unused JSON search path, still blocked with a 403, and the
per-subreddit pacing; arguments after subreddits are keyword-only
- the sentiment prompt no longer asks for vote and comment counts, which the
RSS feed does not carry
- a failed fetch and an empty search both returned [], so a 429 rendered as
'no posts found' and the sentiment analyst read throttling as real silence;
when every subreddit was throttled the summary asserted it outright
- a failed fetch now returns None and renders as unavailable, and the summary
only claims silence for subreddits actually searched
- raise the headerless-429 back-off to 60s, which is where a retry starts
succeeding; pay it at most once per run so three throttled subreddits do not
stall the analysis, and match the Retry-After cap to it #1295
- ElementTree does not resolve external entities, so the reported XXE flag
doesn't apply; the real residual is an unbounded read of untrusted network XML
- cap both the RSS and JSON reads at 5 MiB; overflow degrades to empty / RSS
fallback through the existing failure paths #1206#1276
- a valid Retry-After: 0 means retry at once but was treated as absent
(`or 5.0`) and waited 5s; honour it exactly now
- jitter our own headerless fallback and the inter-subreddit pacing so several
analyses sharing an IP don't retry in lockstep and re-collide on the limit;
keep the single-retry ceiling (more retries can't fix an exhausted IP budget) #1193
- crypto reached StockTwits as Yahoo's BTC-USD (404) instead of BTC.X, and Reddit
searched the dashed pair that barely matches; both now resolve the base via a
shared crypto_base() helper, restoring crypto sentiment
- also fixes a StockTwits resilience test class that pytest never collected #1113
The JSON search endpoint is reliably WAF-blocked (403) for public clients, so
probing it on every call doubled request volume against Reddit's per-IP rate
limit and tripped 429 on the RSS fallback, blanking the sentiment feed. Fetch
the Atom/RSS feed directly (JSON kept as an opt-in path that still degrades to
RSS on 403), back off once on a 429 honouring Retry-After, and pace requests a
little wider. Also broaden the error handling to catch http.client chunked
transfer errors (IncompleteRead/BadStatusLine) alongside OSError, which on their
own slipped through and crashed the pipeline.
Reddit blocks the anonymous JSON search endpoint, which silently emptied
the sentiment analyst's Reddit source. Fall back to the public RSS search
feed when JSON fails. RSS lacks score/comment counts, so those posts are
marked "via RSS feed" rather than shown with fake zeros.
#862
