Any text format
Send documents the way you already have them. AgentLex turns each one into pages before anything else happens.
- Text and Markdown, split into pages at form feeds or
--- page 12 --- markers, which also carry the printed page number.
- HTML, with navigation, scripts and footers dropped, headings and lists kept, tables turned into Markdown tables, and the title and author read from the page.
- CSV and TSV, quoted fields handled, 40 rows per page with the header repeated, and the column names kept as metadata.
- JSON records, one page per record, nested fields flattened; or printed pages with their number, volume and heading.
POST /v1/collections/:id/documents · format detected, or set with format
request
POST /v1/collections/col_8hQ/documents
{
"title": "Data Processing Agreement",
"metadata": { "jurisdiction": "Arcadia" },
"format": "html",
"html": "<h1>Article 1</h1>..."
}
response
{ "pages": 9, "chunks": 61, "chunker": "legal",
"format": "html", "identifiers": { "total": 0 } }
Structure-aware chunking
A passage should be a unit its readers would recognize, so documents are cut where their authors cut them. Four chunkers, chosen by detection or by name:
- Clinical notes: one passage per section (chief complaint, medications, labs, assessment, plan), headers like HPI and A/P mapped to canonical names, ICD-10 codes, dates and drug names kept as metadata.
- Legal texts: the numbering hierarchy (Article, Section, §, (a), (1), (i)) with a breadcrumb on every passage, each defined term a passage of its own, citations collected.
- Classical Arabic print: body and editor's footnotes apart and linked, a hadith kept whole with its chain, a verse with the commentary after it, a poem's lines together.
- Prose: headings and paragraphs, packed to about 700 characters, long paragraphs cut at sentence ends.
POST /v1/chunk previews the cut without a key and without storing anything
one clinical note, cut into passages
Chief Complaintheading
Medications, 4 items in one passagemedication · list
Laboratory Resultslab · table
Assessment (E11.9, I10)diagnosis
Planplan · date
one contract, cut by its hierarchy
Definitions › “Personal Data”definition
Article 4 › 4.2 › (b)clause
Article 9 › 9.1clause · citation
Tables kept whole
A table row is a fact, and a row cut off from its header is a fact without its meaning. Markdown, CSV and space-aligned tables are found in every document and never split mid-row.
- A table that fits becomes one passage, flagged table.
- A long table is split between rows, and the header row is repeated at the top of every piece.
- Text before and after a table stays in its own passages; code blocks are never mistaken for tables.
a 60-row lab table, piece 2 of 2
| Test | 03/02 | 06/01 | header repeated
|------------|--------|--------|
| Potassium | 4.6 | 4.4 |
| eGFR | 44 | 47 |
| ...
Pages, passages and offsets
Two units do two jobs. Whole pages are what get found and cited; passages are what get read and shown. Every passage records exactly where it sits on its page.
- Each passage carries start and end offsets:
page.text.slice(start, end) is the passage, letter for letter.
- Each passage carries its kinds (medication, lab, definition, clause, footnote, hadith, verse and more), its heading or breadcrumb, and its metadata.
- Any page can be fetched whole with its passages, so an interface can open the source and highlight the line.
GET /v1/collections/:id/documents/:doc/pages/:n
a passage
{ "pos": 3,
"text": "Metformin increased from 500 mg to 1000 mg BID",
"kinds": ["plan", "medication"],
"heading": "Assessment and Plan",
"start": 1184, "end": 1231,
"meta": { "drugs": ["Metformin"] } }