Lexical data carpentry · reproducible sample
Clean the data.
Keep the evidence.
A dictionary rarely arrives as one tidy table. Fields drift, glosses mix languages, records repeat, and one harmless-looking cleanup can erase where a reading came from. This small proof takes mixed lexical files into SQLite while keeping every accepted value traceable.
The sample is synthetic and project-owned. It demonstrates workflow behavior, not compatibility with a particular database, linguistic accuracy, or a customer result.
The useful distinction
Source data and display data are not the same thing.
Normalizing a gloss can make a browser easier to use. Overwriting the original gloss makes later review harder. The proof stores the cleaned English display value separately and points the transformation back to the exact raw field.
Keep
- the source file hash and record ID;
- a stable locator for every field;
- the exact raw value as received;
- the canonical field name and transformation note;
- rejected records and the reason they stopped.
Do not silently invent
- a missing lemma;
- a pronunciation that was never supplied;
- a translation with no source trail;
- a successful record after partial failure;
- compatibility before the real field map is reviewed.
One small pipeline
Dry-run first. Commit once.
The builder parses and validates both sources before opening the output database. Four complete records become lexemes. Two records with no lemma stay visible in the source ledger but never become canonical entries.
Failure stays boring. A deliberate interruption after three writes leaves every ingestion table empty. Running the accepted data a second time makes no new writes and leaves the database bytes unchanged.
A reviewable path
Each step leaves something a person can inspect.
Toolbox-style text + CSV→dry run→field map + validation→one SQLite transaction→manifest + report
Trace one value back
A canonical gloss can resolve to its original record, field locator, raw text, and source hash without relying on memory or a parallel spreadsheet.
Preserve what was supplied
IPA is copied into a clearly named raw field. The sample does not infer, convert, or validate pronunciation.
Rerun without duplicates
Stable keys turn an identical replay into zero writes. A later source change can be compared against a known prior state.
Run it locally
The builder uses the Python standard library. The repository includes both synthetic inputs, the generated SQLite database, validation and rollback ledgers, a manifest, and tests.
python examples/lexical-ingest-proof/build.py --check
python -m unittest tests.test_lexical_ingest_proofBring the source before choosing the system
Does your glossary or dictionary fit?
Start with a free fit check for one rights-cleared collection, one language goal, and one existing machine. If it fits, the USD 250 Local Knowledge Terminal sprint maps the data and provenance, tests an agreed sample, and builds up to two cited browser cards. Custom adapters, bulk conversion, OCR, production deployment, and hardware are separate work.
Start the free fit check
See the sprint deliverable →
No source upload or payment at the fit-check stage.