Spec sheet / Documents
Business document dataset spec
A business document dataset holds company files (reports, proposals, spreadsheets, policies and wiki pages) converted to Markdown with headings and tables intact, plus a JSONL index of metadata. The strongest sets keep versions or pair working inputs with the finished deliverable. This spec covers conversion, metadata scrubbing, size drivers, quality checks and long-context uses.
Common sources
SharePoint
OneDriveGoogle DriveNotionConfluenceOne line holds
Format
Volume
Record structure
How this type uses the shared conversation schema. Fields not listed here follow the general definition.
| Field | What it holds |
|---|---|
| docs/<id>.md | Converted text with headings, lists and tables |
| index.jsonl: type | policy, proposal, report, minutes, specification, wiki page, spreadsheet |
| index.jsonl: version_of | Earlier version of the same document, where history exists |
| index.jsonl: pairs_with | Inputs that fed a finished deliverable |
| index.jsonl: dates | Created and modified, with the dataset's date offset |
| sheets/<id>.csv | Spreadsheet tabs, with formulas in a sidecar file |
Conversion decides usefulness
Documents arrive as Word, PowerPoint, Excel, PDF and wiki pages. A model reads text, so everything is converted, and conversion is where most document datasets lose value. Headings flattened into plain lines, tables turned into runs of numbers and footnotes moved into the body all produce text that no longer resembles the original. Ask for Markdown that keeps heading levels, lists and tables, and check a sample against the source format.
Spreadsheets need their own treatment. A tab rendered as a table loses the formulas that made it a model rather than a list of numbers. Keeping formulas in a sidecar file lets you train or test on the logic as well as the values.
Metadata hides identity
Office files carry more than their text: author and last editor fields, comments with names, tracked changes, hidden sheets, speaker notes, headers and footers, and file paths that contain a person's or client's name. Each of these must be scrubbed or removed. The checklist has the full list.
Versions and pairs
The most valuable document sets show work in progress. A draft and its final version show how an editor improves text. A set of inputs (notes, figures, an earlier report) paired with the deliverable built from them shows how a professional turns material into a document. These pairs exist in accounting, consulting, legal and engineering work, but only if the source system kept them. Ask whether versions and pairs are available before you size the request.
Writing the spec
Name the document types, the professions behind them, the languages and the period. Say whether scanned files are acceptable and at what OCR quality. If you want pairs, describe what counts as an input and what counts as the finished document. The index file should then record the pairing so you can build tasks from it.
Size, sample checks and questions to ask
What drives size
- File count by type after duplicates are removed
- Near-duplicates, which can be a large share of a shared drive
- Scanned PDFs that need OCR
- Version history kept by the source system
- Folders excluded at source
Checks on a sample
- Tables and lists survive conversion on a sample
- OCR error rate measured on scanned files
- File properties scrubbed: author, last editor, comments, tracked changes
- Exact and near-duplicates removed, with the threshold stated
- No names in titles, file paths or headers and footers
- Embedded images removed or replaced by a short description
Ask the supplier
- Which systems and folders were exported, and which were left out
- How tracked changes, comments and hidden sheets were treated
- Whether version history exists and how versions are linked
- The conversion tool and its known weak spots
- How spreadsheets with formulas were represented
Where documents data fits in training
Long-context SFT
Drafting from inputs
Extraction evals
Retrieval test sets
FAQ
Documents datasets: questions
What format should business documents be delivered in?
Markdown that keeps headings, lists and tables, plus a JSONL index with type, dates, versions and pairings. Spreadsheets work best as CSV per tab with formulas in a sidecar file.
Which document metadata needs scrubbing?
Author and last editor, comments, tracked changes, hidden sheets, speaker notes, headers and footers, and file names or paths that contain names.
Why do document versions matter for training?
A draft next to its final version shows how text is improved. Inputs next to a finished deliverable show how material becomes a document. Both are rare in public data.
How should duplicates be handled in document datasets?
Remove exact duplicates and near-duplicates above a stated similarity threshold. Shared drives often hold many copies of the same file with small changes.
Are scanned documents usable?
Yes, after OCR, if the error rate is acceptable for your use. Ask for a measured error rate on a sample and a flag on every scanned file.
Keep reading
Next step
Need documents data built to this spec?
Send the languages, period, functions and intended use. We reply by email with what can be assembled.