A feed was dead for two weeks. All my logs said was "2 feeds failed".
I logged which feeds failed but not why. The morning after I started logging the reason, the cause showed up, and one of them had been arriving with a success-family status code.
This is not a news summary. It is a record of something that happened while building this site.
An unattended pipeline reads 12 news feeds every morning at 08:00. One of them had returned nothing since August 9, and I did not notice until August 21. Not for lack of monitoring: the record said "2 feeds failed" every single day.
The number was correct. There was just nothing I could do with it.
At first I thought the feed was malformed
When a feed could not be read, this is what got logged.
[WARN] Hacker News (AI) fetch failed: parse error: <unknown>:7:2: mismatched tag"Parse error" plus "mismatched tag on line 7" reads like broken feed XML, so that is where you go looking.
That is not what happened. The server was returning 502 with an nginx error page, and we were parsing that HTML as XML until it tripped on line 7. The feed was fine. The server was down and there was nothing on our side to fix.
I logged the names. It still was not enough
On August 20 I made the pipeline record the names of failed feeds in that day's file. Before that only a count survived, so a few days later there was no way to tell which feeds they had been.
The next morning the names appeared. MarkTechPost, Hacker News. And that is where it stopped. Knowing the name does not tell you what to do next. A site blocking us and a server being sick are entirely different problems, and a name does not separate them.
This repo has a rule: when a domain you should be able to reach fails, check the network allowlist first. It had been the cause twice before. I checked, and both hosts were on the list.
| My laptop | Unattended environment | |
|---|---|---|
| MarkTechPost | HTTP 200 · valid RSS | could not read |
| Hacker News (AI) | HTTP 200 · valid XML | could not read |
The same URL opens from one place and not the other. To get any further I needed to know what the failing side actually received, and that value was printed to the console once and gone when the run ended.
So I made it keep the reason
The change is small. Instead of throwing the exception message away, write it into that day's record. If there is a status code, lead with it. If a response arrived but the content is not XML, say that. If neither, it might genuinely be a format problem, so show the original exception.
I nearly got this wrong. My first version only reported status codes of 400 and above. Left that way, it would have missed half of what showed up the next morning.
The next morning, two feeds, two different reasons
MarkTechPost not a feed (content-type: text/html, HTTP 202)
Hacker News (AI) HTTP 502 - error response instead of a feed, content-type: text/htmlMarkTechPost sits behind Cloudflare, which serves a human-verification page to requests from datacenters. That page arrives as 202. We asked for RSS and got a page meant for a person to look at, and by response code alone nothing was wrong.
Hacker News was not our problem. That server is struggling and will come back. It timed out from my home connection too.
What the two weeks cost
The briefing never stopped. One dead feed does not halt the others, by design. What happened instead was a quiet narrowing. Reading 11 sources instead of 12 means each remaining outlet takes a bigger share. On August 21, four outlets filled all ten slots.
The more expensive cost was elsewhere. When "2 feeds failed" appears every day and there is nothing you can act on, that line becomes one you read and do nothing about. A few days of that and a genuinely new failure would not register either. That is exactly why it took two weeks.
After the fix
On August 23 I dropped MarkTechPost. There is no way around a Cloudflare challenge from a datacenter address. Instead I opened 31 candidate feeds and screened them on four things.
- Is it behind Cloudflare? That is what just killed us, so I am not buying the same risk again
- Are the actual headlines about AI, rather than the section being named that? Several "AI" feeds were serving how-to articles about televisions
- Can the article body be fetched? Titles alone just occupy a slot
- Is the publishing volume enough to matter?
| Candidate | Cloudflare | Per week | Article body |
|---|---|---|---|
| The Decoder | no | 10 | readable |
| SiliconANGLE AI | no | 30 | readable |
| Hugging Face Blog | no | 5 | readable |
| The Register AI/ML | no | 4 | readable |
| Simon Willison · Latent Space · AI Business | yes | - | - |
Of the ones that cleared all four I picked The Decoder. Ten posts a week, all of them on AI, with a lot of research write-ups, which is close to the role the dropped feed had been filling.
What I am taking from it
The record grew three times: a count, then a name, then a reason. With only the first two, writing it down every morning changed nothing. "What failed" is a line you read and move past. "So what do I do" is the one that gets you out of your chair.
This one came back 202. The next one might come back 200. So the check reads the body, not just the code.