Vasyl’s Dev Notes

I had not opened the TextStack repo properly in a while. I have been working through Claude Code for weeks, task by task, and each task looked small. Yesterday I opened the whole thing and looked at it as one project. It has grown into something I cannot hold in my head anymore. That is a separate problem and I will get to it at the end.

While I was looking around I opened .github/workflows/health-check.yml. It is the file that tells me my site is alive.

on:
  schedule:
    - cron: '*/5 * * * *'

Every 5 minutes. It curls /health on textstack.app and textstack.dev, then does two smoke checks, book listing and search. If any of them fails, the run goes red and I get a notification.

288 runs a day. That is what the file says.

Then I counted the actual runs.

The number

The workflow was created on 6 January 2026. Here is what it did, month by month.

Month	Runs per day	Percent of 288
January (from the 6th)	107	37%
February	74	26%
March	73	25%
April	53	18%
May	35	12%
June	26	9%
July	15	5%
August (first 11 days)	20	7%

It went down every single month from January to July.

In July my health check ran 15 times a day. I had asked for 288. That is one check every 96 minutes instead of every 5.

And it never started at 100 percent either. Even in the first month it was already at about a third.

I did not notice any of this for seven months.

What made me look

On 6 August GitHub Actions went down. The incident opened at 15:22 UTC and was fully resolved at 02:04 UTC the next day, so almost 11 hours. At the peak, 71 percent of workflow runs were failing with infrastructure errors. My own work stopped for about four hours that afternoon, which is why I remembered the date.

So I checked what my health check did in that window. Four runs. Two of them failed. The largest gap that day was 5 hours 25 minutes, from 18:33 to 23:59, and that gap sits inside the incident.

At first I thought I had a clean story. GitHub went down, my monitor went down with it, blind for five hours.

Then I looked at the same day before the incident started. There was already a gap of 3 hours 22 minutes, from 00:08 to 03:30, hours before anything broke. And on 3, 4 and 5 August the workflow ran 12, 13 and 13 times.

So no. The outage did not blind me. The outage was just the day I happened to look. It had been like this since spring.

Why

This is documented. It is in the GitHub docs, in the part about the schedule event. Scheduled events can be delayed during periods of high load of GitHub Actions workflow runs, and if the load is high enough, some queued jobs may be dropped.

Dropped. Not delayed. Dropped.

I knew that sentence existed. I read it as “sometimes it runs a bit late.” It does not mean that. It means the run never happens and nothing tells you.

The part I did not expect

Same repository, same seven months, same schedule trigger. My daily database backup:

on:
  schedule:
    - cron: '0 3 * * *'
jobs:
  backup:
    runs-on: self-hosted

224 runs since January. Zero missing days. Not one. Every month it ran on exactly as many days as the month has, and in May and June it ran a couple of extra times because I triggered it by hand.

So in the same repo one scheduled workflow degraded to 5 percent and the other stayed at 100 percent.

I want to say the difference is runs-on: self-hosted, because that is the obvious difference and it is the one I like. But the two workflows also differ in how often they ask for something. One asks 288 times a day, the other asks once. High frequency schedules are exactly what a queue sheds first. I cannot separate those two causes with the data I have, so I am not going to pretend I can.

I did get one thing wrong before checking, and it is worth saying. I assumed self-hosted would not protect me on 6 August, because the root cause that day was invalid job assignments to runners, and job assignment is GitHub’s side, not mine. That sounded right. The backup ran normally on 6 and 7 August anyway. My reasoning was fine and the answer was still wrong.

Three layers, all built by me, none of them checking anything

Once I started pulling on this I found it was not one problem.

My README says the project has uptime monitoring with UptimeRobot probes. There is a whole file for it, docs/03-ops/uptime-monitoring.md. It is a good document. It has a table of three monitors with URLs, expected responses and intervals. It has numbered setup steps. It has an alert-response runbook with the exact commands to run for each kind of alert, down to which containers to check logs for.

I do not have an UptimeRobot account. I never signed up. I wrote the whole document for the setup I was going to build, and then did not build it.

The last section of that same file is the part that got me. It describes the GitHub Actions health check as internal belt-and-braces, useful when UptimeRobot itself has an outage, but redundant otherwise.

Redundant. My documentation called it the backup to the real monitoring. It was the only monitoring. And it was running at 5 percent.

Second layer: the cron says every 5 minutes. It runs every 96 minutes.

Third layer: the Actions tab is green. A green tab means every run that happened passed. It looks exactly the same when almost no runs happened at all. There is no colour for “did not run.”

Three sources of confidence. All three mine. None of them lying on purpose, and none of them checking anything.

This is the same shape as a bug I wrote about a few weeks ago, where my backup verifier filled the disk of the machine it was verifying. Someone in the comments gave me the sentence for it: a verifier that shares a resource pool with the thing it verifies eventually becomes the largest consumer of it. This is one level up. A monitor that shares a platform with the thing it monitors inherits that platform’s bad days, and stays silent about it.

What I am not going to do

The obvious fix is to sign up for an external monitoring service. Free tier, five minute checks, runs on somebody else’s machines, done in twenty minutes.

I am not going to do that, and I want to explain why, because the reason is the thing I actually learned this week.

I opened this project yesterday and could not hold it in my head. It is one person’s side project and it has a backend, a web app, an admin app, a mobile app, a worker, an MCP server, a browser extension, an SSG pipeline, and now eight workflows. Every one of those was a reasonable decision on the day I made it. Together they are more than I can carry.

Every new service is not just the service. It is another account, another set of credentials, another page in the docs, another thing that is quietly broken in six months while I believe it is running. I know that last one is real now, because that is exactly what my UptimeRobot documentation was: a doc for a thing that did not exist.

Adding a monitoring service to fix bad monitoring is how the pile got this big.

So the rule I am taking out of this is not “add a monitor.” It is narrower. Do not add a second thing where the first thing was never checked. And when you do keep something, keep it where you can see it.

Why I am leaving it exactly where it is

The obvious fix is to move the health check to runs-on: self-hosted, next to the backup that has not missed a day in seven months. One line changed. 100 percent instead of 5 percent. I spent a day thinking that was the answer.

It is the wrong answer, and working out why is the part of this I am actually glad about.

My self-hosted runner is a machine in my flat. If that machine dies, a monitor running on it dies at the same moment and tells me nothing. I would find out on my next login, and I log in maybe once a week.

GitHub is the opposite. When GitHub breaks, somebody whose job it is fixes it, usually within hours, without me. The 6 August outage lasted 11 hours and I did nothing about it.

So the two options are not “5 percent reliable” versus “100 percent reliable.” They are:

• A check that runs every 96 minutes and is not on my machine.
• A check that runs every 5 minutes and dies silently with my machine.

The second one has perfect uptime for every failure except the one failure I have no other way of seeing. My server dying is the thing I am most exposed to, because I am not looking at it. A monitor that shares its fate is not a monitor. It is a second thing that will be dead at the same time.

So the degraded one is the one worth keeping. Not because 15 runs a day is good, but because it runs somewhere I do not control and do not have to maintain, and that is the whole property I need from it.

What I give up is resolution. If the site goes down I might hear about it 96 minutes later instead of 5. For a project with my traffic that is fine, and I would rather know late than not know.

The mistake I made for seven months was not putting the check on GitHub. That was right. The mistake was reading */5 * * * * as a promise and never checking whether it was kept. I optimised the number in the file and never looked at the number in reality.

A monitor has two properties. How often it runs, and whether it can survive what it is watching. I spent all my attention on the first one. The second one is the one that decides whether it is a monitor at all.

The trade used to be easier

I want to be honest that I am less comfortable with this than I would have been a year ago.

Leaning on GitHub used to be a decision you made once and forgot. It was background. It was there. That is what made “just put it on Actions” the obvious call for a solo project.

It is not that obvious now, and the numbers say why. My own schedule went from 37 percent of what I asked for in January to 5 percent in July, and nothing about my repo changed in that time. GitHub had nine service-degrading incidents in May alone. On 6 August, Actions was broken for almost 11 hours with 71 percent of runs failing at the peak. In June, Microsoft started renting capacity from AWS to keep GitHub standing up, which is not a thing a company does when the situation is comfortable.

The load underneath is not a mystery either. Commits went from roughly 1 billion in all of 2025 to 275 million per week. Actions compute hit 2.1 billion minutes in one week. Pull requests opened by AI agents went from about 4 million to over 17 million in six months. The platform is absorbing a change in how code gets written, and the parts that get squeezed first are the low priority background ones. A 5 minute cron on a small public repo is exactly that. I should be careful here though. I have one repo, so I cannot prove my curve and their curve are the same story. What I can say is that the shape matches and that dropping runs under load is the documented behaviour, not my theory.

So I am keeping the trade, but I am not calling it free anymore. It has a cost that is going up, and the cost is paid in silence, which is the worst currency for a monitor.

What I will do is check the number again in three months. If 5 percent has become 2 percent, then the check has stopped being a check and I will have to pay for something. I would rather find that out by counting than by an outage.

The one thing I am changing now is the documentation. docs/03-ops/uptime-monitoring.md still describes an UptimeRobot setup I never built, and calls the GitHub check redundant. It is not redundant. It is the whole thing. That file now says what actually exists, roughly how often it really runs, and what it does not catch.

That edit does not improve my uptime by one second. It just means that in six months I will not believe something that is not true.

What I keep from all this is the number. 15 runs a day out of 288, for a month, without noticing. And the thing that finally told me was not any of my monitoring. It was me opening the repo and counting by hand.

If you have a scheduled workflow you rely on, go count its runs. Not the failures. The runs. It takes two minutes and I would like to know whether my curve is mine alone.