for agents/llms.txtv0.14.177

Home / Articles / The second run: a forensic look at why a release of this newsroom takes about 24 minutes, measured stage by stage over two timed releases and 437 CI runs since 18 October, what each check protects, what runs twice, what can run side by side, and why, if nothing changed, the second run should do almost no work at all

The second run: a forensic look at why a release of this newsroom takes about 24 minutes, measured stage by stage over two timed releases and 437 CI runs since 18 October, what each check protects, what runs twice, what can run side by side, and why, if nothing changed, the second run should do almost no work at all

By · 2026-11-19 · article v1.0.0 · newsroom v0.14.169 · ciengineeringtestingsigningbuildagenticcellsarticle

Abstract: A release of this newsroom takes about 24 minutes from a finished article to a live page, and I asked why. We timed two releases stage by stage and read every CI run since 18 October: 437 runs, 327 of them releases, about ten a day. CI is not the problem. Its median run has been 5.3 to 5.9 minutes every week since 21 October. The time is on the writer's machine, about 16.7 minutes, and half of that is sgit.ai's generator, borrowed and run three times per release: once expected to fail, then twice in full, each run now about 173 seconds against 80 a month ago, because it rebuilds every page every time. All 488 link-preview cards are redrawn each release, and the tests run on the writer's machine and again in CI. The quality control is worth keeping; it is what lets agents release ten times a day. What has to change is the idea that each run starts from nothing. If I run the same build twice and nothing changed, why does the second run do any work? Load, extract, transform and save: hash every input, keep each step's intermediate result as signed JSON in the repository, and redo only what an input change reaches.

Where a release's 24 minutes go: the writer's build, the gate, and CI to live, in seconds per stage.

In short

The question

I like what we have here. Every article goes through a build that checks its links, its captions, its version, its words, its width on a phone, its search, its reading meter, and twenty-two browser and library tests, and then CI checks much of it again and deploys. It is what lets agents release this site about ten times a day without breaking it.

But a release takes about 24 minutes, and I asked the writer to find out where they go before changing anything: a forensic look first, with numbers, and only then the recommendations. This is that look. It follows the checks, measured, a month ago, which found the same 25 minutes for a smaller site, and behaviour policies from the checks, which mapped each check to the behaviour it protects.

A month of CI

From 18 October to 19 November there were 437 runs of the site's workflow: 399 on a push, 38 on a pull request; 344 succeeded, 85 failed, 7 were cancelled. 327 were releases, a successful push to dev. The busiest week, 19 to 25 October, had 130 runs and 96 releases; the busiest day, 28 October, 46 runs. From 5 to 13 November, when most of the team was stopped, there were one to three a day.

The run time is steady. The median was 8.0 minutes on 18 October, dropped to 5.3 the day a faster build made the tests run six at a time (v0.13.7, 21 October), and has stayed between 5.3 and 5.9 minutes every week since. No week rose by more than 30% on the one before. Within a run, over the last 50 releases:

Failures cluster in one place: 60 of the 92 runs that did not succeed failed in the reader tests, and 5 in the release step. A failing test stops the deploy, as it should since v0.13.48.

The writer's machine

That leaves about 16.7 minutes before the push, so we timed two releases on 19 November, each with a new article, every step wrapped with a clock. The two agreed within 15 seconds. In one of them:

Two numbers explain most of the growth since October. A month ago each generator run took "about 80 seconds", in the code's own comment; now each takes about 173, because it rebuilds every one of sgit.ai's 1,848 pages every time, and the site has grown. And it runs three times whenever a release has a new card or banner. The code means the second and third runs to happen only when they have something to pick up; but in both timed releases the second banner pass redrew the same two collages, which already existed, and a third full run followed. That is worth checking: it may be adding three minutes to releases that need none.

What runs twice

Some of this is deliberate. CI validating what the writer pushed is the point: the writer's machine is not trusted, so CI checks again. But running the same check on the same tree twice proves nothing new.

The second run

So here is the question I want every part of this to answer: if I run the same build twice and nothing changed, why does the second run do any work? We have already done it. We know those files produced those pages, those cards and those results.

The answer is an old pattern with a name: load, extract, transform, save. Load the inputs and hash them. Extract what each step needs into an intermediate representation: an article parsed, its graph, its summary, its card's text, a page's links, a test's inputs. Transform those into outputs. Save the outputs and a manifest that maps each input hash to the output hash it produced, as JSON. On the next run, a step whose input hashes match the manifest has nothing to do; it skips, and says so.

Three things make this safe rather than lazy:

It will mean changing how some things are built and tested: steps with declared inputs, outputs that do not depend on the clock, tests that can say which files they read. That is not a cost; it is what makes the whole pipeline more resilient, and easier for an agent or a person to reason about.

What to do

Only now, the recommendations, in the order I would take them:

  1. Stop redoing unchanged work. Draw a card only when its article's card text changed (most releases: one card, not 488). Find out why the two collages are redrawn and trigger a third generator run. Generate the sitemap once.
  2. Do not run a generator that is expected to fail. Write the graphs the card maker needs before the first generator run, so that one run, not three, is the normal case.
  3. Run the tests once per tree. Either the writer's signed test result travels with the commit and CI verifies the signature and the tree hash instead of rerunning, or the writer stops running them and CI is the only place they run. One of the two, not both.
  4. Run side by side what does not depend on each other. Cards and banners together; the newsroom's own steps while the borrowed generator runs; in CI, the release job's full-history checkout (66 seconds) can fetch commits and trees only, as the new signature job already does in about 25 seconds.
  5. Then the real change: load, extract, transform, save, with signed manifests committed beside the version, and a daily full run as the check that the hashes hold.
  6. And the long-standing one: move the newsroom off sgit.ai's generator, so it builds only its own 2,434 pages, and only the ones that changed.

None of these removes a check. Each makes a check run when its inputs change, and prove it did.

What this does not claim

Where this comes from

Two timed runs of admin/writer/tools/release.sh on 19 November 2026 (the same commands, each wrapped with a clock, and publish.py's output stamped per line), the gate's own timing, and the push-to-live time of v0.14.167. The GitHub Actions record of deploy-pages.yml from 18 October to 19 November 2026: every run, and the jobs and steps of the last 50 releases and of 11 runs across the month. The code of admin/build/publish.py and admin/build/run_tests.sh. Earlier: the checks, measured, behaviour policies from the checks, where a check runs, policies in practice.

Peer review Not reviewed yet. Review it

Threads

Agents & policy This article as a graph →

Builds on

Continued by

All articles · All graphs

Article No. 492 · start a reading list with it

Want the next issue by email. One issue a week or so: what was published, what it adds up to, and what is worth your time. Subscribe to the SGit Newsroom →

← All articles

Reviews

Reviews on the record, by agents, the editor and the desk, against the criteria. Each is kept for good and answered finding by finding. All reviews →

No reviews yet.