NewsIntel

A self-hosted news-intelligence platform that turns a firehose of RSS feeds into a searchable, clustered, investigable archive — no LLM or cloud API required.

_{Figure 1. The Events view. Independent reports on the same happening, grouped by a deterministic engine that can tell you why. The screenshots in this README come from a small sample archive of fictional outlets.}

Abstract

News arrives as a stream of near-duplicates: the same event, rewritten by a dozen outlets, each with its own headline and its own idea of what matters. NewsIntel is a system for reading that stream at archive scale. It continuously collects articles from RSS feeds, deduplicates them, extracts full text, annotates entities and keywords, clusters independent reporting on the same story, associates stories into events, and serves the result through full-text search, an entity relationship graph, a map, and per-source and per-event dossiers. An authority file, in the tradition of library cataloguing, gives every person, place and organisation one established name, gathers its other spellings under it, and records the links you state between entities; each established name can also be tied to its Wikidata item, whose names and identifiers are kept in the archive's own database. Every derived object — a cluster, an event, a graph edge — remains traceable to the bounded set of articles that produced it. The architecture targets roughly five million archived articles without a rewrite, and the whole system runs self-hosted, auditable, and free of API keys. No large language models were consulted in the making of any conclusion.

Quickstart

You need Docker with Compose. From the repository root:

cp .env.example .env
# A secret key of your own; compose will not start without one:
sed -i "s|^NEWSINTEL_SECRET_KEY=.*|NEWSINTEL_SECRET_KEY=$(python3 -c 'import secrets; print(secrets.token_urlsafe(48))')|" .env
# Edit .env: set POSTGRES_PASSWORD, and NEWSINTEL_ADMIN_USERNAME / NEWSINTEL_ADMIN_PASSWORD
# (12+ characters) for your first account.
alias dc='docker compose --env-file .env -f docker/compose.yaml'
dc up -d --build    # the setup service applies migrations and creates that account; the search index builds itself

Open http://127.0.0.1:8080 (or your NEWSINTEL_PORT), sign in, and add a feed under Sources. The Today page keeps a short checklist until the archive has sources, articles and a search index.

• Entities are off by default. The default image leaves out spaCy, so the entity charts, the graph and the entity dossiers stay empty, and story clustering and event association work without their entity signal. Add -f docker/compose.ner.yaml to every dc command (and rebuild) to turn named-entity recognition on.
• Sign-in limit. Ten failed sign-ins for one name within 15 minutes hold that name for 15 minutes, after which the hold lifts on its own.
• More accounts, or a first account without the .env variables: dc run --rm api python -m app.cli create-user <name> asks for a password.

Contents

1. Design principles
2. System architecture
3. The pipeline, stage by stage
4. A tour in figures
5. Tech stack
6. Operations
7. Development checks
8. Limitations

1. Design principles

P1. PostgreSQL is the single source of truth; Elasticsearch is disposable. Every write lands in Postgres and is committed before any indexing work exists. The search index can be wiped and rebuilt from scratch at any time, and ingestion keeps working while Elasticsearch is down.

P2. Derived data stays traceable. Clusters, events and graph edges are computed from bounded, identifiable article sets. Every edge in the graph is explainable co-occurrence: open it and you get the exact articles and stories behind it, under the same filters as the graph. Every event stores why each cluster joined it.

P3. Deterministic and versioned over clever and opaque. NLP annotations (spaCy NER, keywords, language, country) are versioned per processor. Story clustering and event association are rule-based engines behind swappable interfaces, so the same input yields the same output, and a result can be argued with.

P4. Descriptive, not judgemental. Source dossiers report health, coverage and timing, each metric shown against its denominator. There is no "quality score": the system describes what a source did, not what it is worth.

P5. One domain, one module. The backend is not a god-object API. feeds, articles, nlp, search, clustering, analytics, entities, graph, investigations, monitors, events, sources, compare, geo, operations, processes, wikidata, auth and jobs are separate modules under backend/app, each owning its models, service layer and routes.

P6. Nothing slow happens in a request. Feed polling, extraction, annotation, clustering and indexing run as background jobs; the API reads what they have committed. When you change the authority file, the API records the decision at once, and the scheduler moves the affected articles over in batches.

P7. A decision about a name can be undone. Merging two spellings never deletes either: the other name stays a variant that can be split back out, every mention keeps the spelling it was found under, and each change is written to the entity's history. Names you mark as different are not suggested again, and links between entities are stated by you, never inferred, so they never count as co-occurrence.

2. System architecture

Five application services and three data services run under Docker Compose, each doing one job. A sixth, one-shot setup service applies the migrations and creates the first account before the others start; it is left out of the figure.

flowchart TB
    feeds(["RSS / Atom feeds"])
    scheduler["scheduler<br/>claims due work ·<br/>advances authority runs"]
    redis[("Redis<br/>Dramatiq broker")]
    worker["worker<br/>ingestion · extraction ·<br/>search indexing · monitors ·<br/>Wikidata runs"]
    nlp["nlp-worker<br/>NER · keywords ·<br/>clustering · events"]
    pg[("PostgreSQL<br/>source of truth")]
    es[("Elasticsearch<br/>rebuildable index")]
    api["api<br/>FastAPI, read-mostly ·<br/>records authority decisions"]
    frontend["frontend<br/>Next.js static export"]
    wikidata(["Wikidata<br/>api.php"])

    scheduler -- "sends actors" --> redis
    redis --> worker & nlp
    feeds --> worker
    worker -. "one request at a time,<br/>cached in Postgres" .-> wikidata
    worker & nlp -- "commit results +<br/>next job rows" --> pg
    worker -- "indexes" --> es
    pg -. "due job rows" .-> scheduler
    scheduler -- "authority runs:<br/>move articles in batches" --> pg
    frontend --> api
    api --> pg & es

Loading

_{Figure 2. Service topology. Arrows show who talks to whom; note that no worker talks to another worker. The scheduler also carries out authority runs itself: a merge or split moves a batch of articles each cycle and asks for them to be reindexed. Wikidata is the one outside service, and it is optional: the worker asks it one request at a time, everything it learns is stored in PostgreSQL, and the rest of the system never waits for it.}

3. The pipeline, stage by stage

The pipeline has one idea, and it is worth stating plainly: no stage ever calls the next one. Instead, each stage writes the next stage's job row in the same database transaction as its own results. A single scheduler loop (backend/app/scheduler.py) wakes every ten seconds, claims due rows under a time-limited lease, and dispatches the matching Dramatiq actor. If a worker dies mid-job, the lease expires and the work simply becomes visible again. Crashes, retries and Elasticsearch outages therefore cost latency, never data: the only thing that can be lost is work that was never committed, and that work never happened.

flowchart LR
    poll["Poll<br/>due feeds"] --> fetch["Fetch + dedup<br/>per-host pacing"]
    fetch -- "full-text feeds" --> extract["Extract<br/>Trafilatura"]
    fetch -- "title + summary" --> nlp["Annotate<br/>NER · keywords ·<br/>language · country"]
    extract -- "body text" --> nlp
    authority[("Authority file<br/>roots · variants ·<br/>see-also links")]
    authority -. "a known variant<br/>counts under its root" .-> nlp
    nlp -- "entities" --> cluster["Cluster<br/>stories"]
    cluster -. "every 30 s" .-> events["Associate<br/>events"]

    fetch --> index[("Index<br/>Elasticsearch")]
    extract --> index
    nlp --> index
    cluster --> index

    index -. "on a timer" .-> monitors["Evaluate<br/>monitors"]

Loading

_{Figure 3. Article flow. Solid arrows are job rows written in the producing stage's transaction; dotted arrows are timer-driven, except the one from the authority file, which annotation reads. Annotation runs once on the feed item and again when the body text arrives.}

Stage	Runs on	Writes	Triggers next
Poll	scheduler	feed claim, feed_fetches	an ingest_feed actor per due feed
Fetch + dedup	worker	articles (deduplicated on normalised URL via INSERT … ON CONFLICT DO NOTHING), feed_articles	extraction job (full-text feeds), NLP jobs and an index delivery when the input is new or changed
Extract	worker	article_contents with a content hash, so unchanged pages are not reprocessed	NLP jobs and an index delivery
Annotate	nlp-worker	entities, keywords, language and country annotations, one versioned run per processor	a clustering job (entities processor only); an index delivery (every processor)
Cluster	nlp-worker	story_clusters, members, each article's opening-wording terms	index deliveries for every article whose cluster changed
Associate events	nlp-worker, every 30 s plus a 5-minute sweep	events, event_clusters with join scores and signals, a record of each run	—
Index	worker	per-article search state and deliveries per index target	—
Monitors	worker, on each monitor's own interval	cursors and unseen counts on monitors	—
Wikidata runs	worker, one run at a time, handed out by the scheduler (your buttons first, then refreshes, then the daily sweep)	cached items, suggestions for unlinked names, refreshed links and their identifiers	—
Authority runs	scheduler, one batch per cycle	after a merge or split, each article's entity rows under the root or the variant; a rename reindexes the root's articles	an index delivery per article; the events those articles belong to are recomputed

Two details make the index trustworthy. Each article carries a revision bumped on every change, and a delivery only lands if its revision is still current, so a slow worker cannot overwrite a newer document with an older one. And indices are versioned behind an alias: a reindex builds a new index alongside the live one and swaps the alias, with no downtime.

4. A tour in figures

_{Figure 4. Search. Full-text queries with field filters (the Query syntax panel has examples you can click), bounded facets in a column beside the results, a matching-articles timeline above them, and highlighted hits that name each article's three main entities. Save and Watch beside Search archive, and Export (CSV or JSON, up to 5,000 articles with their sources and main entities, also on events, entities and comparisons): any search can be saved as a running case file or turned into a monitor that the scheduler re-evaluates in the background. A watch counts an article once, when its processing has finished (full text, language, entities, keywords and countries, or a step that failed for good), so a watch on a name still sees an article whose entities arrive late. Investigations lists both in one place, with a Watch switch on each row (watching a saved search keeps it one row, and switching it off pauses the monitor without losing its history); a watched one opens on its new articles and stories, with a deterministic "What changed" summary (new sources, entities and stories, and stories that gained sources) linked to the evidence.}

Search is the investigation hub. The query and filters live in the URL, and every view reads that same state: bounded facets beside the results (10 values per group, 25 at most), the Graph and the Map. A word matches whatever its accents, capitals or ending in Greek and English (Αθήνα finds ΑΘΗΝΑ, Τσίπρας finds Τσίπρα, flood finds floods); a quoted phrase matches exactly as written. (Today stays on its own recent-window scope; the API's scope=investigation mode exists but the only caller is Map.) An article's detail page adds Related coverage: other articles with similar wording from outside the article's own story, found by Elasticsearch more_like_this, where Greek and English stop words do not count. Similar wording is not a confirmed connection, and the panel says so. A story page is named after its representative article and shows the story's main entities, the event it belongs to, its articles by hour, and who reported it first and how far behind the others followed; brushing its timeline searches those hours within the story.

_{Figure 5. The relationship graph: who and what keeps showing up together. Deliberately bounded — "narrow the filters to see more" is a feature, not an apology. Each edge opens the articles and stories behind it, and each entity has a dossier in four tabs: Overview (mentions per day and the events it is part of), Articles & stories, Relationships (see-also links and closest neighbours) and Catalogue (its authority entry and Wikidata link). The header names the entity with its Wikidata description and its first other names. Stated links draws the see-also links you recorded (Attica is part of Greece, Ingrid Halvorsen leads Norvane Systems) as dotted purple lines and lists them; they never count as co-occurrence. Click one, on the graph or in the list, to see each period it was stated for, with its note and source article. Dashed yellow lines are new: all their articles date from the last week. The graph takes the same filters as Search (sources, tags, countries, detected language, entity types), and Download PNG saves the picture as drawn.}

_{Figure 6. An event dossier: a UTC-day timeline, the member stories with their join scores and signals, the articles, and every entity involved. Events are grouped by time, shared entities, headline overlap and story country, and served by the read-only /api/v1/events API. The Events list can be searched by words in a story's headline or an entity's name; a dossier offers Search within this event (an event_id filter that works in Search, saved searches and Map) and Watch event, a monitor that reports new articles in the event's stories.}

_{Figure 7. The map, by story country: the one country an article is about. Location roles are never summed together, and the page tells you how many articles have no country at all, because a choropleth that hides its denominator is just a very confident guess. Opened from Search, the map covers the whole investigation from Elasticsearch, and its story and source counts are estimates, shown as ≈N (estimated). Opened on its own, it keeps the recent-window PostgreSQL mode, with exact counts, which works while Elasticsearch is down.}

The authority file

NER finds names, not entities: "WHO" and "the World Health Organization", or "A. Okafor" and "Amara Okafor", arrive as different entities, and a spelling that splits an entity in two also splits its counts, its graph edges and its search results. The authority file is how you join them, following library cataloguing practice: one established name per entity (the root), the other spellings as its variants, and see-also links between entities that are related but not the same.

_{Figure 8. The Authority file page. Names is a table, most used first, filtered by any of an entity's names, language, type, status, Wikidata link or hidden names; click a name to edit it in the side panel, or tick several rows to merge them into one (the name with the most articles is kept unless you pick another, a warning says when their types differ, and Bring every type here also claims each merged name), give them a type, mark them established or provisional, or hide them. The language is picked from the languages the file has. Maybe the same? lists likely duplicates (the same name under two types, an acronym, initials, a surname on its own, a similar spelling) with both types and the articles both names share, 25 at a time; nothing merges until you answer, and Same entity opens the name kept. Wikidata holds the suggested items and Changes the history, which shows every change or one kind (merges, names, different entities, see-also links, claims or Wikidata). The tab, the filters and the open name are in the URL.}

_{Figure 9. The side panel, the same on the Authority file page and behind Edit on a dossier: give the entity its type (Topic included, for subjects such as artificial intelligence), rename it, mark it established, provisional or ambiguous (a name that may stand for several people), keep a note, merge it with another name or record that two names are different. Same name, other types lists the entities spelled the same under another type; Bring every type of “AI” here merges them now and makes every "AI" the articles find later, whatever its type, join this entity. Hide as not a name leaves a word NER took for a name (a weekday) out of search, the graph and the suggestions without deleting it. When the linked Wikidata item points to another type, the panel offers it. Each other name shows how many articles use it and the first of them, the evidence for it. Merging into an entity of another type says which type the result is shown as; a merge a see-also link stops offers Remove the link first. Every merged name can be split back out; splitting one a claim brought in asks whether to stop the claim too. The history lists every change.}

A dossier names the entity under its preferred form and lists the other names it is known by (its spellings, its merged names, and its Wikidata label and aliases in the languages you read) under Also known as. Opening a merged name redirects to its root and says from which name. The entity picker and entity:"…" in search also find a root by its preferred form, such as "Brandt, Hugo", even when no article wrote it that way.

_{Figure 10. See-also links on a dossier: earlier and later names (an organisation that renamed itself), parts, members and leaders, or a plain "related" link, which needs a note saying how. Each link can carry dates, a note and the article it came from, and shows from both sides.}

_{Figure 11. Linking two entities while reading: the article's own entities fill the form, and the link records the article as its source.}

_{Figure 12. Searches and monitors can follow see-also links: filtering by an entity can also take in its earlier and later names, or its parts, so a search for a country can include its regions, and their towns in turn. The filter's chip says how many entities each kind of link adds.}

Searching, the graph, the dossiers, monitors and saved searches all work on roots: a filter on a variant finds its root's articles, and ids saved before a merge keep working. Merging and splitting move each article's mentions in batches (Figure 2), so a large merge never blocks a request. python -m app.cli authority export writes the whole file as JSON, by name rather than by id, and authority import applies it to another database (version 2 of the file also carries each root's Wikidata link and where each name came from, version 3 the type the user gave a root, whether it is hidden, and the names it takes every type of) or to the same one after a rebuild, before NLP runs again; authority seed-countries ties the usual spellings of each country ("USA", "U.K.") to its entity. A country renamed with the same territory (Swaziland, now Eswatini) is one entity: merge the old name into the new one rather than linking them.

Linking to Wikidata

An established name can be tied to one Wikidata item (MARC 024 with $2 wikidata), so "Αλέξης Τσίπρας" and "Alexis Tsipras" become one entity and a search for either finds both the Greek and the English articles. Nothing is ever linked on its own: the system suggests, you decide.

• On a dossier's Catalogue tab, Find on Wikidata searches for the name; the Wikidata section then shows the suggested items with their descriptions and the reasons for each (an exact label, a name the entity already has, a type that fits), with Link and Not this one; tick any of a suggestion's aliases to add them as names as you link it. If you know the item, type its QID and press Link item. Linking brings the item's labels in Greek and English in as variants; its other names (nicknames, a bare surname) are listed under Names on Wikidata and added only if you tick them. A linked dossier shows the description, when the item was last checked, and the VIAF, ISNI and Library of Congress identifiers the item carries, with Refresh from Wikidata and Unlink; an entity without a preferred name offers Use the Wikidata name. An item already linked to another entity cannot be linked twice: the refusal offers Merge into that entity, or says why the two cannot merge (marked as different, for instance).
• On the Authority file page, Wikidata suggestions lists likely items for names not yet linked, best first, and Approve all exact links every name whose one exact label has the right type; each suggestion's aliases can be ticked before Link. The authority file shows each root's QID and filters by linked or unlinked, and "Maybe the same?" gains two reasons: two names that share a Wikidata name, and two linked names whose items Wikidata says are different ("different from"), which offer only Mark as different.
• One item, one root. A QID already linked elsewhere is refused, with the offer to merge the two names, and a name marked ambiguous cannot be linked at all. A name that already belongs to another root is never moved: the pair turns up in "Maybe the same?" instead.
• Kept current, gently. Each linked item is checked every 30 days, 50 to a request, and fetched again only if it changed. A redirect on Wikidata (two items merged) moves the link, with history; a deleted item keeps its link and is marked missing. The worker sends one request at a time, at most one every 3 seconds and 2,000 a day, and pauses when Wikidata asks it to.
• Offline is fine. Everything is read from PostgreSQL, so the archive works without the internet; only new suggestions wait. Nothing is sent until you set NEWSINTEL_WIKIDATA_CONTACT (an email address or a URL, which Wikimedia's User-Agent policy requires), and NEWSINTEL_WIKIDATA_ENABLED=false stops all traffic (the test suite sets it, so no test can reach Wikidata). python -m app.cli wikidata status shows the state and the day's budget, and wikidata refresh --all (or --qid Q…) refreshes now. Data from Wikidata is CC0.

The remaining routes follow the same design language:

• Today (the home page): watched investigations with something new, the biggest events of the last day, and Rising names: entities named at least three times in the last day and twice as often as on an average day of the week before. One line appears only when a service is down or a process is failing or stalled, with a link to Processes. The 30-day ingestion, entity and country charts sit below.
• Sources: a list you can search by name or URL, filter by state (collecting, failing, disabled, retired) or tag, and sort by name; each source is edited in place (name, URL, country, language, tags, collection mode, interval). A source that keeps failing backs off: after the first failure it waits twice as long each time, up to a day (or its own interval if longer), and its row says when it is tried next; one success puts it back on its interval. A 4xx answer is recorded as a permanent HTTP error, and changing a source's URL forgets the old address's cache headers. Test feed reads an address before you add it (title, entries, language) and finds the feed a site's page links to; Import OPML previews a feed reader's export and adds only the new sources, with its folders as tags, and Export OPML downloads them all. A tag is also a filter on Search, Graph and the Map (Source tag under Advanced filters), and saved searches and monitors keep it. Per-feed dossiers have health, fetch history and coverage, and where the source sits in story timing (first to publish in N of M shared stories, or the median minutes behind the first article).
• Compare: two entities, two sources or two countries side by side, with the articles and stories only one has and those both share. Each side also counts the window of the same length just before, so a subject can be read then and now. Entities and sources are picked by searching their names (every source, retired ones included).
• Clusters and Articles. Opening an article marks it read; s stars it and a note stays with it. ? on any page (or the ? button beside Sign out) lists the keyboard shortcuts: / for the quick search everywhere, and in the reader j/k, f, s and o. The list filters by source, unread or starred, language, full text and the days it was found, and Search results show the star, the note and whether you have read the article. Every other list of articles (entity, source, event and comparison pages, the Map, graph evidence, a monitor's results and related coverage) shows the star and the read state the same way.
• Processes: every background process on one page (it replaces Jobs and Operations): service health, a card per process, one activity list with Retry and Stop, feed health and storage.
• Settings: Appearance (a dark theme, chosen per browser or following the system, the default, with the charts and the article reader's page dark too; and the time zone dates are shown in, this browser's by default), Account (change the password, which signs out every other browser, or sign the other browsers out), the Wikidata status and how to change it, and Notifications: news from watched investigations is pushed to your phone or desktop through ntfy (ntfy.sh or a self-hosted server, with an access token for a protected topic). Each investigation announces only articles it has not announced before; up to three get a message each and more are put together in one. Quiet hours hold messages until they end, an optional daily summary at a chosen hour lists every watched investigation with news, and Notify me on a monitor's page mutes just that one. Send test message checks the address; a failed send is tried again after five minutes, and the panel shows the last delivery and the last error.

5. Tech stack

Layer	Technology
Backend	Python, FastAPI, SQLAlchemy 2, Alembic, Pydantic, uv
Background jobs	Dramatiq, Redis, PostgreSQL job tables with leases
Canonical storage	PostgreSQL
Search	Elasticsearch (versioned indices behind an alias, zero-downtime reindexing): full text, facets, investigation analytics and map, related coverage
NLP	spaCy NER for English and Greek (optional image, off by default; enable with docker/compose.ner.yaml, then switch Greek on in Settings), YAKE keywords and country detection for English and Greek, Lingua language detection, pluggable/versioned processors
Extraction	Trafilatura, behind a replaceable extractor interface
Reference data	Wikidata Action API (optional), cached in PostgreSQL
Frontend	Next.js (App Router, static export), React, TypeScript, TanStack Query, Apache ECharts
Deployment	Docker Compose

6. Operations

First run. See Quickstart. Migrations run on every up through the one-shot setup service, which the application services wait for. Elasticsearch starts empty; when there is no search index the scheduler starts a rebuild on its own, and Search or the Today page's rising names and charts have something to show once it has cut over. The Processes page's Search index rebuild card shows its progress, and python -m app.cli search-index-status lists rebuilds.

What needs Elasticsearch. Search, facets, the Today page's rising names and charts, the Graph, a watched investigation's results, "What changed" and evaluation, the investigation Map and Related coverage read the index. While it is down they return an error or say they are unavailable, and ingestion, processing and the recent-window Map keep working. When an older version built the live index, Search and every view that reads it say the index needs a rebuild (the Processes page's rebuild card says so too) instead of reading fields it lacks; press Rebuild search on that card (or run python -m app.cli rebuild-search), which builds the new index beside the live one and moves the alias once it has caught up. The previous index is kept for rollback.

Estimated counts. Three numbers come from Elasticsearch's cardinality estimate (precision 3000). Two are labelled: the distinct stories behind a Graph edge ("about N stories (estimated)"), and the story and source counts of the investigation Map (≈N in the table, "estimated" to a screen reader). The "N new stories" badge on Investigations is not marked; it is a notification count and near-exact below 3000 distinct stories. Every other count is exact for the indexed snapshot.

The Processes page (/processes/; the old /jobs/ and /operations/ addresses lead there) shows what the archive does in the background, refreshed every 5 seconds while the tab is open:

• Services: PostgreSQL, Redis, Elasticsearch, the NLP models, the scheduler heartbeat and a live worker on every queue.
• A card per process, in three groups. Per item: feed fetching, article download, NLP, story clustering, search indexing, watchlist monitors. Bulk runs: NLP reprocessing, name changes, search index rebuild, source reindex. Scheduled: event linking, history cleanup, Wikidata refresh and suggestions, and notifications (the investigations waiting to be announced, the last message sent and the last error). A card turns red only for failures inside the chosen window (1 h, 24 h or 7 d), and the Processes link in the menu shows in red how many cards are failing.
• Activity: one list across all of them, by default what needs attention (running, retrying, failed in the window). Click a card to see only that process.
• Actions: Retry on a failed row, Retry N failed on a card (200 at a time), Run now on the scheduled processes, and Stop on NLP reprocessing and Wikidata runs. Rebuild search on the index rebuild card starts a rebuild (one at a time). Name changes, source reindexing and the index rebuild cannot be stopped: half a merge or reindex would leave the index inconsistent. Every bulk run goes to a worker: the scheduler hands it out a turn at a time under a lease, so a slow batch holds up no other queue and a worker that dies is taken over.
• Feeds (with Fetch now), Storage and Wikidata: the requests sent today against the daily budget, any pause (a 429, 503 or maxlag answer pauses them), and the runs waiting.

Two things it does not show:

• Article file size. Retained article HTML lives on the worker's article-data volume, which only the worker mounts. Measure it from the host: docker compose --env-file .env -f docker/compose.yaml exec worker du -sh /var/lib/newsintel/articles.
• History retention. Every hour (or on Run now in the History cleanup card), a worker deletes succeeded job rows older than 30 days that a newer row replaces, feed fetches older than 30 days except each feed's newest, sessions that expired or were revoked more than 30 days ago, and temporary article HTML past its expiry that nothing references. It deletes batch after batch until the backlog is gone or five minutes have passed (NEWSINTEL_RETENTION_SECONDS). Failed job rows are kept for diagnosis, and so are the feed failures of the last 30 days.

Settings and sizing

Every setting has one default, in backend/app/core/config.py; .env changes one by naming it (NEWSINTEL_ plus the field name in capitals), and docker/compose.yaml passes it to the containers only when .env sets it. .env.example lists the ones an install usually tunes, commented out at their defaults: the host names the app answers to, batch sizes, worker processes and threads, the Elasticsearch heap and Postgres memory. Two limits protect the database: a page's query is cancelled after 60 seconds (NEWSINTEL_API_STATEMENT_TIMEOUT_SECONDS), and a transaction left idle for 5 minutes is ended (NEWSINTEL_IDLE_TRANSACTION_TIMEOUT_SECONDS). The scheduler keeps a small connection pool (NEWSINTEL_SCHEDULER_POOL_SIZE, default 2) instead of opening a connection per item.

Backup and restore

PostgreSQL holds everything that matters (P1). Elasticsearch is rebuilt from it, so it needs no backup. Retained article HTML, on the worker's article-data volume, is optional. The full Docker test gate rehearses the dump and restore (infra/test-restore.sh).

Back up:

alias dc='docker compose --env-file .env -f docker/compose.yaml'
dc exec -T postgres pg_dump -U newsintel -Fc newsintel > newsintel-$(date +%F).dump
dc run --rm --no-deps -T worker tar czf - -C /var/lib/newsintel articles > articles-$(date +%F).tgz  # optional

Restore:

dc stop api worker nlp-worker scheduler
dc exec -T postgres pg_restore -U newsintel -d newsintel --clean --if-exists --no-owner --exit-on-error < newsintel-DATE.dump
dc run --rm --no-deps -T worker tar xzf - -C /var/lib/newsintel < articles-DATE.tgz  # if backed up
dc run --rm api alembic upgrade head    # a dump from an older release needs the newer migrations
dc up -d
dc run --rm api python -m app.cli rebuild-search --wait   # the index no longer matches the restored rows

A dump taken before the migrations were squashed into one baseline (October 2026) is at revision 0031, which no longer exists. Its schema is the baseline's, so tell Alembic so before upgrading: dc run --rm api alembic stamp 0001 --purge.

The dump carries the authority file, with its Wikidata links and cached items, with everything else. To keep your naming decisions across a fresh database instead, export them first and apply them to the new database before the feeds are ingested again:

dc run --rm -T api python -m app.cli authority export > authority.json
# … the new database is up …
dc run --rm -v "$PWD/authority.json:/authority.json:ro" api python -m app.cli authority import --file /authority.json          # dry run
dc run --rm -v "$PWD/authority.json:/authority.json:ro" api python -m app.cli authority import --file /authority.json --apply

Starting over with an empty database

To rebuild the archive from nothing (a new schema, or data you no longer trust), keep what you want to carry over, remove both volumes and let the stack build itself again:

1. Make sure main is what you run and its full Docker gate is green.

2. Optional: export the authority file (see above) and the sources (Export OPML on the Sources page).

3. Stop the stack and delete the PostgreSQL and Elasticsearch volumes together. An index left over from the old database would describe rows that no longer exist.

   dc down
   docker volume rm newsintel_postgres-data newsintel_elasticsearch-data

4. Check .env: NEWSINTEL_SECRET_KEY must be set (compose refuses to start without it; the Quickstart shows the command that makes one), and so must NEWSINTEL_ADMIN_USERNAME / NEWSINTEL_ADMIN_PASSWORD for the first account.

5. dc up -d --build (with -f docker/compose.ner.yaml if you use entities). The setup service applies the migrations and creates the account, and the scheduler starts the search index rebuild on its own.

6. Before adding feeds, apply the authority file you exported (authority import, dry run then --apply) and tie the country spellings to their entities: dc run --rm api python -m app.cli authority seed-countries --apply.

7. Add the sources again (Import OPML on the Sources page).

8. On the Processes page every card should be green and the Search index rebuild card done; then run one Greek search to check that the index answers.

7. Development checks

The system is typed and tested end to end, and everything runs in containers: development and validation need only Docker with Compose, Git and ordinary POSIX shell utilities. The backend is checked with ruff and strict mypy; the frontend is TypeScript, using API types generated from the checked-in OpenAPI spec.

• ./infra/test-quick.sh is the fast loop: unit tests, linting, type checks and the OpenAPI/TypeScript contract, with no service containers.
• ./infra/test-docker.sh is the full gate: it builds dedicated test images, runs backend and frontend checks, rejects skipped tests and stale generated contracts, rehearses backup/restore, and runs every browser workflow against disposable Compose stacks.

GitHub Actions runs the quick loop, the frontend build and a workflow lint on every pull request and every push to main (.github/workflows/ci.yml). It runs the full gate after every merge to main, nightly, on demand, and on pull requests that change infra/, docker/, the Dockerfiles, .github/actions/ or the workflow itself (.github/workflows/full-gate.yml). A change to nothing but the Markdown docs, the images in assets/ or the licence starts neither. No test talks to Wikidata: the browser workflows use a stand-in for its API on the fixture server, and a monthly workflow (.github/workflows/wikidata-contract.yml) checks the client against the real one. A green pull request therefore usually covers the quick loop only, so run the full gate before merging a change to application behaviour.

CONTRIBUTING.md covers running the stack locally, the integration and browser loops, regenerating the OpenAPI spec and TypeScript types, writing migrations, and the test harness's options and reports.

8. Limitations

In the tradition of papers that are honest about their methods:

• No summarisation or "insight" generation. NewsIntel groups, counts and links; it does not paraphrase. This is a choice: every output can be traced to its articles (P2), which a generated summary cannot promise.
• Entity quality is spaCy's quality. NER mislabels things: the small models used here call a city a person now and then, and tag one name with two types. Dates, times, amounts and time phrases it tags as names are filtered out, but other mislabels get through. The authority file (Figures 8 and 9) joins the spellings and types of one entity and lets you correct its type, but only after the fact, and its suggestions only look within one language. Annotations are versioned, so a better model can be rerun over the archive without losing the old results.
• Story country is conservative. It is only assigned when a country is named alone in the title and repeated in the text, and only for English and Greek articles, so most articles have none. The map says so rather than guessing.
• Rule-based clustering and events. They are deterministic and explainable. Story clustering also matches articles whose opening words share rare specifics (stemmed, and weighted by how rare each term is in the 48-hour window), so a reworded headline no longer hides a story. A true paraphrase in different vocabulary is still missed, and event association still relies on entities and headline terms. Articles clustered before the wording rule have no terms until they are reclustered (python -m app.cli recluster --from-date … --to-date …).
• Wikidata needs the internet and your judgement. The daily sweep only looks for names with at least three articles (Find on Wikidata works for any), and it searches by name, so an entity Wikidata knows under a quite different name may get no suggestion; you can still link it by its QID. A link is only as good as the item: Wikidata is edited by anyone, and a refresh brings in changed descriptions and identifiers, though never new names without you.
• Related coverage is wording, not meaning. It needs at least five shared terms, so a paraphrase in different words is missed, and a short or text-poor article gets no related coverage rather than a guess.
• Scale is designed, not unlimited. The target is on the order of five million articles on a single Compose host. Beyond that, the ceilings are named in the code as they are met.

License

MIT