Candor Memory Engine & Dry-Run Action System

I built this project to implement the Candor take-home interface without PostgreSQL, pgvector, Docker, or an external vector database.

My goal was to keep the system deterministic, reproducible, temporally correct, and easy to run from a clean Python 3.10+ environment.

What this is

A deterministic memory engine and dry-run action planner over the mock Brightline data. No database or vector store; Python 3.10+ standard library (optional rank_bm25, python-dateutil).

• src/indexer.py: ingestion for meetings, dictation, Slack, Gmail, Calendar, Codex, ChatGPT. Records are visible only when delivery_time <= as_of; Slack edits/deletes are state transitions; credential-like strings and HTML-comment injections are redacted. Also mines recurring proper nouns (customers, products) as entities.
• src/retrieval.py: BM25 + entity/alias, phrase, date, source-intent and recency signals. Nothing is keyed to a specific question.
• src/answer.py: generic answer step. Uses an Anthropic or OpenAI-compatible endpoint if configured in .env; otherwise a generic extractive answerer with an evidence-support check for abstention. Cited sources are the records the answer was built from.
• src/action_runner.py, contacts.py, timeparse.py: general parsing of commands. People, channels and events are resolved from connector data; ambiguity gives clarify, destructive requests give confirm.
• src/textutil.py: tokenizing, stemming, date and value extraction shared by the modules above.
• src/memory_runner.py: the retrieve -> support check -> answer pipeline, with as_of enforced at every step.
• run.py: entrypoint (memory, actions, verify).
• tests/test_engine.py: unit tests (standard library unittest).

Run

python3 run.py verify                        # regenerate outputs and score all four splits
python3 -m unittest discover -s tests -v     # 40 unit tests, ~3 s

out/ is generated by verify and is gitignored; it is not shipped.

Results (offline mode, no LLM)

Split	What it is	Retrieval	Memory (strict)	Actions
Train (27 / 12)	development	92.0%	92.6%	100%
Held-out (14 / 14)	development	100%	100%	100%
Fresh (14 / 12)	development (see below)	100%	100%	100%
Blind (12 / 10)	never tuned on	100%	83.3%	100%

How to read this. Only the blind split is close to an honest test, and with 12 and 10 items its confidence interval is wide (memory 58% to 100%). Train and held-out were both used during development, so their numbers are upper bounds, not estimates of generalization. The fresh split was written and scored before the answer-composer fixes, then read afterwards, so it became a development set too. Its baseline before any fix was 78.6% memory and 75% actions, which showed that the earlier 100% held-out action score was partly overfit.

The earlier history matters too. A first version of this repo reported 100% on train via corpus-specific answer strings, record IDs and per-question branches (benchmark_rules.py). That code was removed. A second revision scored 48.1% train and 78.6% held-out memory. The jump to the figures above came from general mechanisms (see "What changed in the answer step"), not from per-question rules, but note that the failures were read to decide which mechanisms to build.

Known failures

• MEM-TR-12 ("how many days after the call did I send the proposal"): needs two-event date arithmetic. Lexical retrieval anchors "send the proposal" to the promise rather than the actual send.
• MEM-TR-26 ("has Acme signed the contract?"): needs a negative answer and a retrieval hit that has no word overlap with the question. TR-21 also misses retrieval (its answer passes).
• MEM-BL-04: answers "Tuesday" without resolving it to a date from the message timestamp.
• MEM-BL-12: fails to abstain on "Dana Lee's favorite restaurant" because unrelated sentences share a word or two with the question.
• Cited-source recall is low (train 0.59, blind 0.80): answers are right more often than they cite every supporting record.

What changed in the answer step

All of these are general mechanisms driven by the question and the evidence, not by IDs or expected answers:

• Dictations drop the speech-to-text transcript; emails drop greetings and sign-offs.
• List items inherit their header sentence, and short email replies inherit their thread subject.
• "Date", "time" and "where" questions are detected anywhere in the question and prefer sentences with an absolute date or a place.
• Answers name the speaker or sender, so reported speech is attributed to the person who said it; "who set it up" adds a calendar event's organizer.
• Self-corrections ("800, sorry, I misread that, it's 1.8") are trimmed to the corrected value.
• "Current state" questions let the latest value of the same attribute win; yes/no status questions lean on the latest statement.
• Calendar metadata is down-weighted unless the question is about the calendar; cancelled events are not evidence.
• Abstention distinguishes a missing name or number (abstain) from a missing everyday word (usually just different phrasing).
• Retrieval: single-word entities ("Acme", "Pinecrest") are mined from the data, and "what's on my calendar on " boosts events on that date.

Design decisions

1. Temporal correctness

I wanted time to be a first-class part of retrieval rather than an afterthought.

Every source is normalized to a delivery timestamp. For each query, I rebuild the visible index at the requested as_of time so future records cannot leak into retrieval.

For Slack, I treat edits and deletions as state transitions:

• An edit changes the visible message text from the edit timestamp onward.
• A deletion removes the target message from the visible corpus from the deletion timestamp onward.

2. Granular retrieval

I kept meeting transcripts chunked by segment ID such as MTG-...#xxxx rather than collapsing an entire meeting into one document.

I also kept ChatGPT conversations addressable by message ID such as CGPT-...#mX.

This matches the benchmark's unit-scoring rules and makes the retrieved evidence precise and traceable.

3. Safety

I treat connected data as data, not as instructions.

Before retrieval and answer composition, I remove HTML-comment prompt injections and credential-like strings. This prevents content embedded inside source records from silently influencing the system's behavior.

4. Actions are dry-run only

I intentionally keep actions non-destructive.

For example, a delete request produces a confirm result instead of performing the deletion. When a request is ambiguous, I return clarify rather than guessing.

Name references are resolved from the contact directory built from the data (restricted to the channel the command names, e.g. Slack); if more than one person matches, the planner asks which one.

Known limitations

I made a few deliberate trade-offs to keep the implementation small and deterministic:

• Offline answers are extractive: they quote and splice source sentences, with the speaker's name prepended where it matters. They are accurate on this corpus but terse and not fluent prose. Configure an LLM in .env for synthesized answers. The LLM path has not been evaluated.
• The extractive answerer does no arithmetic across events and cannot say "no" on its own; those cases need the LLM path or a semantic retriever (see Known failures).
• Several of the heuristics (mined entities, speaker attribution, correction trimming) were developed against this one small synthetic corpus. Expect them to need re-tuning on real data.
• My entity resolution is lightweight and local. A production system would benefit from a richer identity graph across people, aliases, accounts, and sources.
• Natural-language scheduling covers common phrasings (relative days, am/pm, durations, "N minutes before X") but not recurrence.
• The optional LLM path supports the Anthropic Messages API and OpenAI-compatible /chat/completions endpoints through the Python standard library.

Tools and models

I intentionally kept the project lightweight:

• Python 3.10+ standard library
• Optional rank_bm25
• Optional python-dateutil
• Optional Anthropic or OpenAI-compatible LLM endpoint

Repository layout

.
├── run.py
├── src/
│   ├── indexer.py        ingestion, visibility at as_of, redaction, entity mining
│   ├── retrieval.py      time-aware hybrid retrieval
│   ├── answer.py         extractive answerer, support check, optional LLM back end
│   ├── memory_runner.py  retrieve -> answer pipeline
│   ├── action_runner.py  dry-run action planner
│   ├── contacts.py       people directory
│   ├── timeparse.py      natural-language dates, times, durations
│   └── textutil.py       tokenizing, stemming, date/value extraction
├── tests/
│   └── test_engine.py
├── data/
├── evals/                train, heldout, fresh, blind (memory + actions)
├── eval_harness/
├── examples/
├── requirements.txt
└── README.md

Reproducibility

I designed the project so that a fresh checkout can regenerate the benchmark outputs instead of relying on pre-generated answer files.

The main reproducibility path is:

python3 run.py verify

That command rebuilds the outputs first and then runs all supplied scorers, which is the same path used for every number above.