Spec sheet / Source code
Source code dataset spec
A source code dataset is a set of private repositories delivered with their commit history, plus pull requests, review comments and linked issues as JSONL. History is what makes it valuable: each change pairs a before state, an intent and an after state. This spec covers structure, secret and licence scanning, size drivers and uses for coding agents and evals.
Common sources
One line holds
Format
Volume
Record structure
How this type uses the shared conversation schema. Fields not listed here follow the general definition.
| Field | What it holds |
|---|---|
| repos/*.bundle | Each repository as a git bundle, author identities rewritten to tokens |
| commits.jsonl | sha, parents, author token, timestamp, message, files changed, line stats |
| source.kind = pull_request | Review threads in the conversation schema |
| turns[].anchor | File path and line range a review comment refers to |
| source.kind = issue | Issues and bug reports in the conversation schema |
| links | Issue to pull request to merge commit, by id |
History is the product
A snapshot of a codebase teaches a model what finished code looks like. History teaches how code changes: a bug report, the discussion that narrowed it down, the commit that fixed it and the review that asked for one more test. That sequence is what coding agents need to learn, and it exists only in private repositories that were used for real work over years. So the first thing to specify is depth of history, not repository count.
The delivery has two layers. Repositories travel as git bundles you can clone locally, with author names and emails rewritten to tokens. Pull requests, review comments and issues travel as conversations in the published schema, with a file and line anchor on each review comment and links joining an issue to the change that closed it.
Ownership comes before anything else
Code is the data type where rights are most often unclear. Contractors may hold copyright unless their contracts assigned it. Open-source files may be copied in under licences that impose conditions. Some repositories contain code a client paid for and owns. Ask the supplier to state who owns each repository, to list the third-party code it found, and to exclude anything it cannot clear. The procurement page covers how to put that in the licence.
Scanning that actually covers history
Secrets deleted from the latest commit still sit in older ones. A scan that only checks the current tree misses them. The same goes for personal data in test fixtures and seed files that were later replaced. Ask for a scan report across every revision, and for history to be rewritten where something was found.
Code for RL environments
A repository becomes an environment when a task can be checked by running something. The public pattern, made familiar by SWE-bench, is an issue, the repository before the fix, and tests that fail before and pass after. Private code makes those tasks uncontaminated, but only if the build still works. If you plan RL use, put "tests run at sampled commits" in the spec.
Size, sample checks and questions to ask
What drives size
- Number of repositories and years of active development
- Commit count after generated and vendored files are removed
- Share of changes that went through review
- Monorepo or many small repositories
- Binary assets, which are excluded and listed
Checks on a sample
- Secret scan across the full history, not only the latest commit
- Third-party and open-source files identified with their licences
- Generated, minified and vendored files excluded
- One person maps to one token across commits, reviews and issues
- No customer data in test fixtures, seeds or logs
- For RL use: builds and tests run at sampled commits
Ask the supplier
- Proof that the licensor owns the code, including work by contractors
- How history was rewritten and whether commit hashes changed
- Which secret scanner ran and on which revisions
- Which third-party code is present and under which licences
- Whether the test suite runs, and with which toolchain
Where source code data fits in training
SFT on intent and change
Review generation
RL for coding agents
Private evals
FAQ
Source code datasets: questions
Why does commit history matter in a code dataset?
Because it records change: what was wrong, what the engineer intended and what they did. A snapshot shows only the result. Coding agents need the before, the intent and the after.
How should secrets in old commits be handled?
Scan every revision, not just the latest, then rewrite history to remove anything found. The licensor should rotate those credentials too. Ask for the scan report with counts per type.
Who owns code written by contractors?
It depends on the contract. Without a written assignment, a contractor may hold copyright. The supplier should confirm ownership per repository and exclude what it cannot confirm.
Can private repositories be used for coding RL?
Yes, when tasks can be checked. Pair an issue with the code before the fix and tests that pass after it. The build and test suite must still run, so check that on a sample.
What format should a code dataset be delivered in?
Git bundles for the repositories, so you keep full history, plus JSONL for pull requests, reviews and issues in a conversation schema with links between them.
Should open-source code inside private repositories be removed?
Identify it first. Some licences allow use with attribution, some impose conditions you may not accept. Ask for a list with licences and decide per case.
Keep reading
Next step
Need source code data built to this spec?
Send the languages, period, functions and intended use. We reply by email with what can be assembled.