Method

Fetch once, freeze the bytes, hash them, extract structure, derive what can be derived by a published formula, and refuse to ship if any of it stops re-deriving. Everything on this page is a description of code you can read in world-news-day/build/.

The path

The lexicon

The published formula behind every theme edge in this graph. Each entry is a case-insensitive regular expression run over the prose of the frozen op-ed. A match creates one edge from the article to the theme and records the words that matched. A theme edge therefore says exactly one thing: this piece contains these words. It is not a summary of the argument, and it is not a claim about what the author believes. The gate re-runs every pattern against the frozen bytes on every build, in both directions, so a theme can be neither typed in nor left out by hand.

These 20 patterns are the only classification in this section. They are published here in full so that you can disagree with them precisely rather than in general — a theme edge says this piece contains these words and says nothing else at all. The gate runs every pattern again on the frozen prose, in both directions: a theme in the graph whose pattern no longer matches fails the build, and so does a pattern that matches an article which has no edge for it.

What this section got wrong

This publication argues that a correction which does not reach what it disproved is not a correction. A section that will not correct itself in public has no standing to make that argument, so here is the list, with the version that carried each error and the rule it produced.

Machine surface: data/corrections.json. The gate checks that every correction names a version, the claim and what it says now — and that the withdrawn claim appears nowhere in this section without the correction beside it.

The pages we fetched and did not keep

Everything else this section fetches is frozen. A few pages are not, and the rule is published in data/contact-rules.json: a page carrying ten or more addresses of named people is a staff directory, and its bytes are not retained. The URL, the retrieval time, the SHA-256 of what came back and the counts are recorded; the file is kept neither in this repository nor in the vault bundle.

The case it exists for is the Globe and Mail, whose contact page publishes the direct address of seventy-nine named journalists. That page is public, so this is not secrecy. It is that freezing a staff directory into a git repository and shipping it in a downloadable bundle makes it materially easier to scrape than the publisher made it, and that is not a thing this section will do to twenty-three colleagues.

It costs something, and the cost is stated rather than hidden: for a page held this way the count is an assertion about bytes we no longer have, not a number a reader can re-derive from this repository. The hash is kept so that anyone can re-fetch the page and check the count against it.

A second, smaller storage rule: organisation pages that are kept are stored gzipped, because a home page is one to two megabytes of script bundle and eleven megabytes of that is payload rather than evidence. The SHA-256 recorded is of the original bytes; the gate decompresses and re-verifies it, so the anchoring is exactly as strong and the repository is a fifth of the size.

The one thing written by hand

Everything in this section is derived by a program except one file: data/affiliations.json, which records where each of the twenty-three authors works.

It was derived by a program first, and the program was wrong in the way that matters. A role line is free prose — “Founder: Paraluman News, The Philippines”, “is an Indian media leader. She is the co-founder…” — and a one-line pattern over it produced an author who works at “Philippines”, another at “an Indian media leader”, and a third at a summit. Every one of those is plausible, which is the worst thing an edge in a graph can be: once it is in the file, nothing distinguishes it from a true one.

So the twenty-three lines were read and written out by hand, and the gate checks each one against the bytes in both directions: the role line must appear in that article verbatim, every organisation must appear inside that role line verbatim, the file and the graph must name the same authors, and the graph must hold exactly the affiliation edges the file states — no more and no fewer. Two role lines name only former posts; those record no organisation and say why, because this section holds nobody’s employment history.

What the gate checks

What this section refuses to do

1. Republish the prose. Even though the terms permit it. We are describing somebody else’s corpus and arguing that its terms should be machine-readable; republishing it while doing so would confuse the two acts, and the pieces are better read where their authors put them.
2. Score, rank or rate anyone. Twenty-one named people at twenty-one named organisations. A count of words is not a measure of a person, and this publication originates no assessment of any of them.
3. Assert agreement. Two pieces touching the same theme is not agreement. agrees_with is a banned verb in the ontology, enforced by the gate.
4. Give legal advice. The reading of the permission on the licence page is a careful reading by people who are not lawyers, published so it can be corrected. If you are deciding whether you may republish one of these pieces: read the terms on the piece and ask the publisher.

The Portuguese

Every node type and every verb in this section carries a Portuguese form, and nothing here is published in Portuguese. That is deliberate and it is stated rather than implied: the second language is a switch to be thrown at pt.newsroom, not a retrofit to be done later. Carrying pt from the first commit is what makes it a switch.

Rebuilding this

# from a clone, with no network access at all:
python3 world-news-day/build/build.py
python3 admin/build/chrome.py
python3 world-news-day/build/gates.py && node admin/build/validate.js

# to take a new dated snapshot (this is the only step that uses the network):
python3 world-news-day/build/extract.py --fetch