Skip to content
aidataset.broker home

Open schema

A JSONL schema for enterprise conversation and ticket data

This schema stores one enterprise conversation per JSONL line: a chat thread, an email thread or a helpdesk ticket, with ordered turns, typed pseudonym tokens, labels, the outcome and a provenance block. It is called aidb.conversation/1.0, it is free to use, and the example file below holds four records that are entirely fictional.

Example file (.jsonl)
A data engineer reviewing lines of structured records on a large monitor at night while a colleague beside her points at one field

Conversation object

Top-level fields

13 fields. 10 are required in every record; the rest appear when the source has the information.
Top-level fields of a conversation record
FieldTypeRequiredMeaning
schemastringyesVersion tag, "aidb.conversation/1.0"
idstringyesConversation id, stable within the dataset
datasetstringyesDataset id, matches the datasheet
sourceobjectyessystem, kind and channel the records came from
languagestringyesMain language as a BCP 47 tag (nl, de-AT, sv)
periodobjectyesstart and end, ISO 8601, after the date offset
participantsarrayyesPseudonym token, side and generalised role
labelsobjectnoSource labels kept as found: priority, category, channel type
turnsarrayyesOrdered messages, notes and events
outcomeobject | nullnoHow it ended: status, code, reopened, closed_at
linksarraynoCross-references to other records in the same delivery
deidobjectyesPolicy version, token counts, date offset rule, review type
provenanceobjectyesLicensor reference, export route, licence id

Turns

Fields inside each turn

Turns are ordered by n. Messages, replies and internal notes carry text; events carry a field change instead.
Fields of one turn
FieldTypeRequiredMeaning
nintegeryesPosition in the conversation, from 1
atstringyesTimestamp, ISO 8601 UTC, offset applied
fromstringyesPseudonym token of the author
toarraynoTokens of addressees, where the source records them
kindstringyesmessage, reply, note or event
visibilitystringnopublic, internal or private, as in the source
textstringnoDe-identified body; absent on events
eventobjectnoFor kind = event: field changed and new value
langstringnoOnly when it differs from the conversation language
reply_tointegernon of the turn being answered (chat, reviews)
directionstringnoEmail only: inbound, outbound or internal
anchorobjectnoCode review only: file path and line range
attachmentsarraynoFile type and size per attachment, never content
reactionsobjectnoChat only: count per emoji, no reactors
botbooleannotrue for integration and workflow posts

Example

One record, pretty-printed

The first line of the example file, a Dutch helpdesk ticket about a wrong quantity on an invoice. Written for illustration only.

Pseudonym tokens

  • <PER_n>A person: name, nickname, handle, initials or display name
  • <ORG_n>A customer, prospect or the licensing company itself
  • <EMAIL_n>An email address
  • <PHONE_n>A phone number
  • <ADDR_n>A street address
  • <ID_n>Account, order, invoice or customer number
  • <URL_n>A link that points to a person or an internal system
  • <SECRET>A credential or key, never numbered
{
  "schema": "aidb.conversation/1.0",
  "id": "c-0001",
  "dataset": "fictional-example",
  "example": "Fictional record written for illustration. No real company, person or ticket.",
  "source": {
    "system": "zendesk",
    "kind": "support_ticket",
    "channel": "email"
  },
  "language": "nl",
  "period": {
    "start": "2023-03-06T08:14:00Z",
    "end": "2023-03-07T10:02:00Z"
  },
  "participants": [
    {
      "ref": "<PER_1>",
      "side": "customer",
      "role": "office manager"
    },
    {
      "ref": "<PER_2>",
      "side": "internal",
      "role": "support agent"
    }
  ],
  "labels": {
    "priority": "normal",
    "category": "invoice"
  },
  "turns": [
    {
      "n": 1,
      "at": "2023-03-06T08:14:00Z",
      "from": "<PER_1>",
      "kind": "message",
      "visibility": "public",
      "text": "Goedemorgen, op factuur <ID_1> staan 40 dozen printpapier, wij hebben er 4 besteld. Kunnen jullie dit corrigeren?"
    },
    {
      "n": 2,
      "at": "2023-03-06T09:40:00Z",
      "from": "<PER_2>",
      "kind": "note",
      "visibility": "internal",
      "text": "Orderregel handmatig ingevoerd, typefout. Creditnota aanmaken."
    },
    {
      "n": 3,
      "at": "2023-03-07T10:02:00Z",
      "from": "<PER_2>",
      "kind": "event",
      "event": {
        "field": "status",
        "value": "solved"
      }
    }
  ],
  "outcome": {
    "status": "solved",
    "code": "credit_note",
    "reopened": false,
    "closed_at": "2023-03-07T10:02:00Z"
  },
  "links": [],
  "deid": {
    "policy": "v1",
    "tokens": {
      "PER": 2,
      "ID": 1
    },
    "date_offset": "constant per dataset",
    "review": "automated + human sample"
  },
  "provenance": {
    "licensor": "L-EXAMPLE",
    "route": "admin export",
    "licence": "LIC-EXAMPLE"
  }
}

Design choices

The conversation is the record

Enterprise sources store messages, but models learn from exchanges. A record that holds a whole thread or ticket can be passed straight to a chat template, an evaluation harness or an environment. It also makes sampling honest: drawing a thousand records gives a thousand complete conversations, not fragments.

Typed tokens, numbered per conversation

Replacing a name with a blank or a generic word destroys the structure a model needs: who asked, who answered, who was copied. Typed tokens keep that structure. Numbering restarts in every record, so <PER_1> in one ticket and <PER_1> in the next are unrelated. That limits how much anyone can piece together across records. If you need stable identities across a dataset, for example to model one agent's style, say so in the spec: it raises re-identification risk and needs explicit agreement.

Dates shifted, intervals kept

Timestamps carry one constant offset per dataset. Response times, weekdays and the order of events survive; the real calendar dates do not. The datasheet states the rule but never the offset.

Outcomes and provenance travel with the record

The outcome block turns a conversation into a task with a result, which evaluation and RL depend on. The provenance block ties every record to a licensor reference and a licence id, so a record found in a training mix years later can still be traced to the agreement that allows its use.

Loading and validating

Files follow the JSON Lines convention: UTF-8, one object per line, no trailing commas. A first validation pass should check that required fields are present, that turn numbers start at 1 and increase without gaps, that every reply_to points backwards, that timestamps rise, and that no text contains a raw email address or phone number. The same layout can be written to Apache Parquet for large deliveries, and the set-level description can be mirrored in a Hugging Face dataset card if your tooling expects one.

Versioning

The schema field carries a version tag. Adding an optional field is a minor change; changing what a field means is a major one and gets a new tag. Keep the tag in your loaders and refuse records you do not recognise. Spec sheets for each data type show which optional fields they use: chat, email, tickets and code reviews.

FAQ

Schema questions

Why one conversation per line instead of one message per line?

Because training and evaluation work on whole exchanges. A conversation object can be fed to a chat template or an environment without a join, and a loader never sees half a thread.

What are the pseudonym tokens in the text?

Typed placeholders such as <PER_1> or <ORG_2> that replace people, organisations, contact details and identifiers. They are numbered within one conversation, so a model can follow who said what without learning who anyone is.

Is the example file real data?

No. Every record in the example file was written for illustration and carries a field saying so. No real company, person or ticket is described.

Can I use this schema for my own datasets?

Yes. The schema and the example file are free to use and adapt. If you change field meanings, change the version tag too, so records from different versions are never mixed by accident.

How do I turn a record into SFT training examples?

Pick the role the model should play, map that side's turns to assistant messages and the rest to user messages, drop or summarise internal notes as your task requires, and keep the outcome for filtering or rewards.

Does the schema work for source code?

Pull requests, review comments and issues fit as conversations, with an anchor field for file and line. The repositories themselves are delivered as git bundles alongside.

Should large deliveries use Parquet instead of JSONL?

Either works. JSONL is easy to inspect and stream. Parquet is smaller and faster to scan. The field layout should be identical so the same validation applies to both.

Keep reading

Datasheet templateThe set-level document that goes with the files.De-identification checklistHow to test the tokens in a sample.Spec sheets by data typeWhich optional fields each type uses.

Next step

Want a dataset delivered in this schema?

Name the data type, languages, period, volume and intended use. We answer by email with what can be assembled and how it would be documented.

Request a datasetBrowse spec sheets