Filter what you crawl or train on

Check document hashes 1,000 at a time. Keep signed, labeled text; drop what is sealed or corrected.

Who this is for

Crawlers, dataset builders and training or retrieval pipelines.

The problem

Scraped text arrives with no reliable signal of who made it, whether a model wrote it, or whether its author later corrected or withdrew it. Records give each document a signed origin and a public correction history that a pipeline can check in bulk.

How it works

• The pipeline hashes each document and checks the hashes in batches.
• Documents with a valid signature and no seal keep their publisher, model and record digest as metadata.
• Sealed documents are dropped. Corrected ones are flagged for review.
• Source terms tell the pipeline whether the creator denies use, wants attribution or asks a royalty.

Routes it uses

• POST /v1/verify-batch: signature, inclusion proof, errata count and judge for up to 1000 digests in one call.
• GET /a/{sha256}?format=json: every record about a file, its lineage and its errata.
• GET /v1/sources/{source_id}/terms: a creator's signed standing terms: deny, attribute or royalty.

Set it up

Step 1. Hash every document

Hash the raw bytes you stored, not a cleaned copy, so the hash matches what the publisher signed.

Check: Re-hashing a stored document gives the same value.

Step 2. Map hashes to records

Look each hash up as an asset. Keep the newest record digest per asset.

curl -s -H 'Accept: application/json' 'https://aisourceoftruth.com/a/{sha256}?format=json'

Check: A known asset returns at least one record.

Step 3. Check in batches

Send up to 1000 record digests per call. Keep documents whose signature verifies and that are not sealed. Flag any with errata_count above zero.

curl -s -X POST 'https://aisourceoftruth.com/v1/verify-batch' \
  -H 'Content-Type: application/json' \
  --data '{"digests":["{digest}"]}'

Check: A batch with one sealed test record returns it with sealed: true.

Step 4. Read the terms

For a source with claimed terms, honor AISOT-DENY before any use, credit AISOT-ATTRIB, and record what you owe under AISOT-ROYALTY-{rate}.

curl -s -H 'Accept: application/json' 'https://aisourceoftruth.com/v1/sources/{source_id}/terms'

Check: A denied source never reaches the training set.

Step 5. Keep the provenance

Write publisher, model, record digest and checkpoint into the dataset manifest so every training example traces back to its record.

Check: The manifest has a record digest for every attested document.

Wire it into your agent

Run the check as a pipeline stage after fetch and before dedupe. It is a read: no token, no account, and the registry logs no reader identity.

MCP: add https://aisourceoftruth.com/mcp to the agent's MCP servers. Tools used: verify_batch, lookup. Read tools need no token.

With the PromptReports products, or alone

• PromptReports: PromptReports research is published with signed records, so a pipeline can include it with its provenance intact.
• Alone: every step above uses only AI Source of Truth.

Expected outcomes

• Every kept document has a known publisher and record, or is marked unattested.
• Sealed content never enters the dataset, and corrected content is reviewed.
• The dataset manifest shows which sources asked for credit or payment.

What it does not prove

• Most of the web has no record yet. Treat a missing record as unknown, not as a fault.
• A model field is the publisher's statement about how a document was made. The registry does not detect AI text.

The OpenSpec plan for your agent

Your coding agent fetches the plan and writes it into your repo as openspec/changes/integrate-aisot-dataset-provenance/:

curl -H 'Accept: application/json' 'https://aisourceoftruth.com/use-cases/crawl-and-train/openspec?lang=python&with=promptreports'

• proposal.md: why, what changes in your product, impact.
• design.md: where the calls sit in your agent, with code for each call.
• tasks.md: checkable tasks, each step paired with a check against the live registry.
• specs/dataset-provenance/spec.md: the requirements and scenarios.

Read it here: OpenSpec plan. Back to all use cases or the documentation library.