- a failed fetch and an empty search both returned [], so a 429 rendered as
'no posts found' and the sentiment analyst read throttling as real silence;
when every subreddit was throttled the summary asserted it outright
- a failed fetch now returns None and renders as unavailable, and the summary
only claims silence for subreddits actually searched
- raise the headerless-429 back-off to 60s, which is where a retry starts
succeeding; pay it at most once per run so three throttled subreddits do not
stall the analysis, and match the Retry-After cap to it #1295
- ElementTree does not resolve external entities, so the reported XXE flag
doesn't apply; the real residual is an unbounded read of untrusted network XML
- cap both the RSS and JSON reads at 5 MiB; overflow degrades to empty / RSS
fallback through the existing failure paths #1206#1276
- a valid Retry-After: 0 means retry at once but was treated as absent
(`or 5.0`) and waited 5s; honour it exactly now
- jitter our own headerless fallback and the inter-subreddit pacing so several
analyses sharing an IP don't retry in lockstep and re-collide on the limit;
keep the single-retry ceiling (more retries can't fix an exhausted IP budget) #1193
- crypto reached StockTwits as Yahoo's BTC-USD (404) instead of BTC.X, and Reddit
searched the dashed pair that barely matches; both now resolve the base via a
shared crypto_base() helper, restoring crypto sentiment
- also fixes a StockTwits resilience test class that pytest never collected #1113
The JSON search endpoint is reliably WAF-blocked (403) for public clients, so
probing it on every call doubled request volume against Reddit's per-IP rate
limit and tripped 429 on the RSS fallback, blanking the sentiment feed. Fetch
the Atom/RSS feed directly (JSON kept as an opt-in path that still degrades to
RSS on 403), back off once on a 429 honouring Retry-After, and pace requests a
little wider. Also broaden the error handling to catch http.client chunked
transfer errors (IncompleteRead/BadStatusLine) alongside OSError, which on their
own slipped through and crashed the pipeline.
Reddit blocks the anonymous JSON search endpoint, which silently emptied
the sentiment analyst's Reddit source. Fall back to the public RSS search
feed when JSON fails. RSS lacks score/comment counts, so those posts are
marked "via RSS feed" rather than shown with fake zeros.
#862
