Home / Articles / The augmented book: one source in layers, projected to the web, the iPad and print, so the reader gets the right information at the right time, and what fifty years of attempts teach

The augmented book: one source in layers, projected to the web, the iPad and print, so the reader gets the right information at the right time, and what fifty years of attempts teach

By Dinis Cruz · 2026-10-31 · article v1.0.0 · newsroom v0.14.14 · bookpublishingreadingattentionprovenancedesignipadarticle

Abstract: Every article here ends with a section called Where this comes from, and the book of The Ultimate Insider will have them too. It is important: it is the provenance, the evidence that what we say is true. But it is the right information at the wrong time. At the end of a chapter it is too much, a list of words the reader has no connection to, and too little, because on paper none of it links to anything. That made me see a concept I think we are arriving at: the augmented book. The book is one source, in layers: the text everyone reads; what opens on demand, a definition, a source, a step expanded where it is; the provenance of each claim; and the record behind it, the graph, the versions and the reviews. Each medium projects it differently. The website is hyperlinked and lets the reader choose how much shows. The iPad app shows layers on the page, with the review's comments where they belong. The PDF and print keep the page clean and put a QR code where the layer would open. Like augmented reality, the extra is there when you look for it and not before, and one day it may reach smart glasses. I asked who has done this before. The mechanisms are old: Ted Nelson's stretchtext, Voyager's expanded books in 1992, the MagicBook in 2001, Kindle's X-Ray, Wikipedia's page previews. What failed, again and again, was the platform: books built for one company's engine died with it. And the research on cognitive load says what the editor in me wanted to hear: notes next to the claim, few, signalled and under the reader's control.

In short

• Where this comes from is the right information at the wrong time. Too much at the end of a chapter, too little because nothing links.
• The augmented book is one source in layers: the text, what opens on demand, the provenance, and the record behind it.
• Each medium projects it differently. The web links everything, the iPad shows layers on the page, print puts a QR code where a layer would open.
• The reader controls how much shows. The extra is there when you look for it, and not before.
• The mechanisms are fifty years old. Stretchtext, expanded books, the MagicBook, X-Ray, page previews.
• What failed was the platform. Books built for one company's engine went when the engine did.
• The research agrees: notes next to the claim, few, signalled and under the reader's control.

The right information at the wrong time

Every article in this newsroom ends with a section called Where this comes from, and so does this one. It matters: it is the provenance, the evidence that what we say is true, with the date each source was read. Our standards require it.

But reading the book of The Ultimate Insider, I noticed that this section is the right information at the wrong time. At the end of a chapter it is too much: a list of repositories, commits and dates the reader has no connection to by then, because the claim it supports was pages ago. And it is too little: on paper none of it links to anything, so it is only words. The reader pays the attention and gets almost nothing back.

The problem is not the provenance. It is where and when it appears. The evidence for a claim is most useful at the moment the reader doubts the claim, next to it, and nowhere else.

One source, in layers

So I think the book should be written as one source with layers, not as one page with everything on it:

• The text. What everyone reads, and what has to work on its own.
• On demand. A definition, a source, a step of the argument expanded where it is, opened by the reader and closed again.
• Provenance. For each claim, its evidence: the commit, the memo, the page, the date it was read.
• The record. The graph of the book, its versions, its reviews and its corrections.

This newsroom already does part of it. Every section of every article has a line and a paragraph written for it, and a reader who comes back can fold what they have read to its line or its paragraph. Every article has a graph, every version stays public, and every claim's source is in the record. What we have not done is put that record where the reader needs it.

Three projections

The layers are the same; each medium shows what it can.

• The web is hyperlinked. Everything connects: a claim opens its source in place, a term its definition, a section its place in the graph. The reader chooses how much shows, from the text alone to everything.
• The iPad is the reader and the review app. The iPad app I have briefed shows layers on the page, on demand, with the review's comments and cards where they belong, and works offline from its vault. It is the digital edition of the book, something like an EPUB but with the record behind it.
• The PDF and print are fixed, so they carry less on the page: the text, and little else. Where a layer would open, a small QR code leads to that claim's page, with its evidence. Provenance stops being a list at the end and becomes a door next to the claim.

I think of it as augmented reality for the book. The extra is there when you look for it, and only then. One day the same layers could appear through smart glasses, over the printed page, if the page carries the cues.

Who has done this before

I asked a research agent who has done this and what we can learn. Its sources are listed at the end; it found them through search, and the pages could not be opened from here, so what follows is what those results report.

• Stretchtext, Ted Nelson. Nelson's idea, from around 1967 and named in print by 1970: text that expands or contracts in place to show more or less detail, rather than a link that takes you away. It is the closest ancestor of what I want, and it never became a product.
• Expanded Books, Voyager, 1992. Bob Stein's Voyager put books on the first PowerBooks with search, bookmarks, margin notes and, in The Complete Annotated Alice, pop-up annotations. One review at the time called them little more than footnotes and cross-references. Voyager ended when its market moved to the web.
• CommentPress, the Institute for the Future of the Book, 2007. Comments anchored to the paragraph and shown beside it, not below the page.
• The MagicBook, 2001. A research prototype by Billinghurst, Kato and Poupyrev: a printed book whose pages, seen through a handheld display, show 3D scenes. The printed page as the anchor for a digital layer, which is what a QR code is.
• Kindle X-Ray, 2011, and Word Wise, 2014. X-Ray shows characters and terms from a file prepared beside the book; Word Wise puts short definitions above hard words, and the reader sets how many.
• Our Choice, Push Pop Press, 2011. An iPad book of Al Gore's that won an Apple Design Award. Facebook bought the company that August and its book plans ended.
• Wikipedia's page previews, 2017 and 2018. A card with a summary when you hover over a link, so you can check it without leaving the page. Wikimedia's own analysis found pages per session fell by a few per cent, which fits readers staying where they were.
• The W3C's Web Annotation standard, 2017, and Hypothesis, which annotates web pages and PDFs with it.
• Quantum Country, 2019. Andy Matuschak and Michael Nielsen's primer with review questions in the text, timed over days: supporting material placed in time as well as on the page.
• iBooks Author, 2012 to 2020. Apple's tool for interactive books, withdrawn from sale in July 2020.
• Gwern.net and LiquidText. Notes in the margin on a wide screen and as footnotes on a narrow one, with a switch to turn them off; and an iPad app for reviewing PDFs by pulling out and linking excerpts.

What they teach us

• The mechanisms are not new. Pop-up notes, notes in the margin, text that expands and a printed page with a digital layer have existed for decades. What makes them valued is the quality of the layer, not the trick.
• The platform is what fails. Voyager, Push Pop Press and iBooks Author each built the book into one engine, and the books went when the engine did. Our source is markdown and JSON in a public repository, and each projection is generated from it; no projection owns the book.
• The cost of the layer kills it. Hand-building an interactive layer for every chapter is expensive. Ours are generated by agents from the record we already keep, which is what makes them affordable.
• Print needs a door that can move. A QR code should point at a page we control, so the layer behind it can be corrected after the book is printed.
• Standards exist for the notes. EPUB 3 marks a note so that each reading system can show it as a pop-up, and the W3C annotation model anchors a comment to text. One layer can serve every edition.

Attention is the budget

The part I care about most is attention. A reader has only so much, and every footnote, link and box spends some of it. The research the agent found points the same way:

• Keep the note next to the claim. The split-attention effect, from Chandler and Sweller's work in 1991, is that material the reader has to join up in their head, a diagram here and its text there, costs more than material placed together. An endnote is the extreme case.
• Fewer links, and say what they are. DeStefano and LeFevre's 2007 review of hypertext reading found that deciding whether to follow a link adds load, most for readers with less working memory or less prior knowledge, and that fewer links, with their type visible, helped.
• Signal, do not repeat. Cues that show how the material is organised help; the same material said twice hurts.
• Let the reader decide. Progressive disclosure, Kindle's adjustable Word Wise and Gwern.net's switch all leave the amount to the reader.

So the rule for the augmented book is simple to say and hard to keep: the right information at the right time, and nothing else on the page.

What we will try

1. Provenance next to the claim on the web: each claim in the book's website opens its evidence in place, and the end-of-chapter list becomes an index.
2. QR codes in the print edition, a few per chapter, each to a claim's page at an address we control.
3. Layers in the iPad app, on demand, with the reader's choice remembered.
4. Typed links: source, definition, graph, version, so the reader knows where a link goes before they follow it.
5. A measure of attention: whether readers open the layers, and which ones, from the reading meter the site already keeps.

What this does not claim

• It is a concept. None of the book's projections exists yet; the site's folds and graphs do.
• The precedents were found through search, on 31 October 2026, and their pages were not opened from here. The dates and facts are as those results report them; no quotation is used.
• The research on attention is mixed in places. No controlled study of footnotes against endnotes was found, and one small study found no difference between hypertext and footnotes for recall.
• Smart glasses are a direction, not a plan.
• My interest: I write the book and build the newsroom.

Where this comes from

Yes, at the end, for now.

• My voice memo of 31 October 2026, word for word in admin/notes/2026-10-31__augmented-book/memo-01.md in this site's repository.
• Found through search on 31 October 2026, pages not opened: Stretchtext; Expanded Books and TidBITS on the birth of the ebook; the Institute for the Future of the Book; the MagicBook paper; X-Ray) and Amazon on Word Wise; AppleInsider on Push Pop Press; Wikimedia's page previews analysis; the W3C annotation recommendations; Quantum Country notes; 9to5Mac on iBooks Author; Gwern.net on sidenotes; LiquidText; Apple's EPUB footnotes; the split-attention effect; Nielsen Norman Group on progressive disclosure; DeStefano and LeFevre, "Cognitive load in hypertext reading: A review", Computers in Human Behavior 23(3), 2007.
• Related: an iPad to review the book, the book of the talk and its companion, a book that looks like the team that made it, your reading, your controls.

Peer review Not reviewed yet. Review it

Want the next issue by email. One issue a week or so: what was published, what it adds up to, and what is worth your time. Subscribe to the SGit Newsroom →

← All articles