Structured extraction
The model reads the guides and picks out the facts, but it isn't allowed to write any of them.
A pipeline that turns community game guides into data a handheld can browse offline, with every line on the device copied from a real guide.
The gap
The data I wanted doesn't exist anywhere
I wanted to browse a big game library the way you browse anything else: by mood, by how long a session takes, by how hard it is. None of that exists in any catalog you can download. Catalogs carry genre, year, and publisher, which tell you almost nothing about whether you want to play something tonight.
The information does exist, in the guides people write about these games. It is buried in prose, phrased differently every time, and there is no field to read it out of. A model can infer it, which is the obvious move and also where the trouble starts.
The trust model
The model never writes the text
A model asked to describe a game will make things up. Not often enough to notice while you are testing it, and often enough to matter once it has run over tens of thousands of titles. The handheld has no network, so a wrong line sits on the device with nothing to check it against and no way to correct it.
So the model never writes fact text at all. It returns a field, the sentence it came from, and a verbatim quote. Python then searches the source guide for that quote and throws out anything that isn't there character for character. Every line on the device is copied from a real guide, with the source one tap away, and anything the model can't ground is skipped.
I tried having it return character positions instead, which would have been cheaper and saved a search. It miscounted them, so I went back to searching for the quote.
Achievements checked against guides
37,002
Across 2,215 games. Grounded text kept for 11,077, the rest skipped rather than guess
3,448 quotes across the run were found verbatim and still dropped, judged unhelpful for their achievement.
The check every kept line has to pass
def extract_spans(source_text, fields_desc, system_hint="", model=MODEL, base=PROXY):
"""The VALIDATED faithful-by-construction primitive.
Sentence-segments `source_text`, asks the model for {field, sentence_id, verbatim
quote}, then DETERMINISTICALLY resolves each quote back to a precise (start,end)
offset and REJECTS any quote not found verbatim. The model never emits fact text
that we keep -- Python copies the located span. Returns list of dicts:
{field, sentence_id, quote, start, end, ok}
Only rows with ok=True are trustworthy; ok=False rows are mispoints to drop.
"""
sents = _sentences(source_text)
numbered = "\n".join(f"[{i}] {s}" for i, s in enumerate(sents))
messages = [
{"role": "system", "content":
"You extract from retro game guides. You NEVER invent facts. For each "
"requested field return the sentence id it lives in and a 'quote' that is "
"an EXACT verbatim substring copied character-for-character from that "
"sentence (no paraphrase, no added words). If a field is not present in "
"the guide, omit it. " + system_hint},
{"role": "user", "content":
"GUIDE (numbered sentences):\n" + numbered + "\n\nExtract: " + fields_desc},
]
content = chat(messages, schema=schema, model=model, base=base, max_tokens=1600)
try:
spans = json.loads(content).get("spans", [])
except (json.JSONDecodeError, AttributeError):
return []
out = []
for s in spans:
q, sid = s.get("quote", ""), s.get("sentence_id", -1)
# FAITHFULNESS = the quote is verbatim-present in the source.
start = source_text.find(q) if q else -1
ok = bool(q and len(q) >= 3 and start >= 0)
out.append({"field": s.get("field"), "sentence_id": sid, "quote": q,
"start": start, "end": start + len(q) if start >= 0 else -1,
"ok": ok})
return out
The database
One canonical record, many feeds
Once more than one source has an opinion about a game, the obvious thing to do is let each feature read whichever source it prefers. That is how you end up with two screens on the same device disagreeing about what a game is called.
Every external source is a feed into one database, never a parallel path to the device. Anything a new feature needs becomes a column in that database, filled by its own resumable backfill and exported through the same single file the device reads. Series names come from a metadata API first and a model only fills what the API missed, snapped back onto the canonical spelling so the two layers agree.
Where it stands
Still filling. More than 50,000 titles are classified and exported, driving the browse screen on the device, and the catalog pass runs continuously. So far it has kept grounded text for 11,077 of the 37,002 achievements checked, and skipped the rest rather than guess.
every line verified against its source · abstains rather than guess · runs offline