Data engineering
I built an append-only news ledger
2026-08-26 / Marcos Defina
I wanted a news record that could show what changed without quietly rewriting what it had already observed. That led to an append-only Node.js and SQLite ledger built around provenance, version fingerprints, and a strict public evidence boundary.
Why current text wasn't enough
Most news pages answer one question: what does the publisher show now? I needed a second answer too: what did an approved feed show when the collector saw it, and how did that observation differ from the one before it? News Ledger stores every retrieval with its timestamp, request and final URLs, response metadata, compressed payload, parser result, collector version, and SHA-256 fingerprint. An article points to its current observation, but older observations remain linked instead of being overwritten.
The database enforces the promise
The evidence model uses separate tables for feed captures, articles, article versions, classifications, and reception snapshots. A changed normalized fingerprint creates a new article version with a predecessor ID and an explicit list of changed fields. An unchanged content fingerprint creates no version; it advances the article's last-seen timestamp and can fill a missing external ID. SQLite triggers reject updates and deletes on feed captures, article versions, classifications, and reception snapshots. The database test directly checks a version update, a capture delete, and a reception update. Append-only is therefore a database property, not a comment asking future code to behave.
Facts and classifications stay separate
A source title, byline, timestamp, and excerpt are observations. A label is an interpretation. The schema records whether each classification came from the publisher, a machine rule, or a human review, together with the method and creation time. That distinction matters because a language signal is not an allegation or verdict. A later review can add context without erasing the original record.
The public boundary matters
An official feed is a collection source, not permission to republish an article. Public pages expose source-provided metadata, short excerpts, revision fingerprints, and capture provenance. Full payloads and body text stay in private evidence storage. The service can show that it observed a change; it cannot claim to preserve every edit a publisher ever made, prove intent, or make the record legally tamper-proof.
What I learned
The useful part of a ledger isn't the word immutable. It's the chain of attributable decisions around each record: where it came from, when it arrived, what changed, who classified it, and what remained private. Once those questions shape the schema, the interface becomes simpler because it no longer has to pretend that every field carries the same kind of truth.