Filter what you crawl or train on

# Proposal: Filter what you crawl or train on

## Why
Scraped text arrives with no reliable signal of who made it, whether a model wrote it, or whether its author later corrected or withdrew it. Records give each document a signed origin and a public correction history that a pipeline can check in bulk.

## What Changes
- The pipeline hashes each document and checks the hashes in batches.
- Documents with a valid signature and no seal keep their publisher, model and record digest as metadata.
- Sealed documents are dropped. Corrected ones are flagged for review.
- Source terms tell the pipeline whether the creator denies use, wants attribution or asks a royalty.


## Capabilities
### New Capabilities
- `dataset-provenance`: Check document hashes 1,000 at a time. Keep signed, labeled text; drop what is sealed or corrected.

## Impact
- External service: AI Source of Truth at https://aisourceoftruth.com. Every call here is an anonymous read: no account, no token, and the registry logs no reader identity.
- Routes used: `POST /v1/verify-batch`, `GET /a/{sha256}?format=json`, `GET /v1/sources/{source_id}/terms`.
- No PromptReports product is required.
- Source of this plan: https://aisourceoftruth.com/use-cases/crawl-and-train (CC0). Regenerate it with `curl -H 'Accept: application/json' 'https://aisourceoftruth.com/use-cases/crawl-and-train/openspec?lang=http'`.

# Design: Filter what you crawl or train on

## Context
Crawlers, dataset builders and training or retrieval pipelines. Scraped text arrives with no reliable signal of who made it, whether a model wrote it, or whether its author later corrected or withdrew it. Records give each document a signed origin and a public correction history that a pipeline can check in bulk.

## Where it sits in your agent
Run the check as a pipeline stage after fetch and before dedupe. It is a read: no token, no account, and the registry logs no reader identity.

Over MCP, point the agent at `https://aisourceoftruth.com/mcp` (Streamable HTTP, JSON-RPC 2.0; read tools need no token). The tools this change uses: `verify_batch`, `lookup`.

## Decisions
- Treat the registry as a source of facts about records, never about the world. Show what it says, with the digest, and do not turn a verdict into a quality score.
- If the registry cannot be reached, treat the answer as unknown (the same as `check`), never as `rely`, and retry later.
- Keep content private: send hashes. Never put personal data in a record; it is public and permanent.

## Calls
### POST /v1/verify-batch
signature, inclusion proof, errata count and judge for up to 1000 digests in one call.

```sh
curl -s -X POST 'https://aisourceoftruth.com/v1/verify-batch' \
  -H 'Content-Type: application/json' \
  --data '{"digests":["{digest}"]}'
```

### GET /a/{sha256}?format=json
every record about a file, its lineage and its errata.

```sh
curl -s -H 'Accept: application/json' 'https://aisourceoftruth.com/a/{sha256}?format=json'
```

### GET /v1/sources/{source_id}/terms
a creator's signed standing terms: deny, attribute or royalty.

```sh
curl -s -H 'Accept: application/json' 'https://aisourceoftruth.com/v1/sources/{source_id}/terms'
```

## What this does not prove
- Most of the web has no record yet. Treat a missing record as unknown, not as a fault.
- A model field is the publisher's statement about how a document was made. The registry does not detect AI text.

# Tasks: Filter what you crawl or train on

## 1. Configure
- [ ] 1.1 Set `AISOT_ORIGIN=https://aisourceoftruth.com` in the agent's environment.
- [ ] 1.2 Optional: add the MCP server `https://aisourceoftruth.com/mcp` to the agent's MCP config.

## 2. Build
- [ ] 2.1 Hash every document: Hash the raw bytes you stored, not a cleaned copy, so the hash matches what the publisher signed.
- [ ] 2.2 Map hashes to records: Look each hash up as an asset. Keep the newest record digest per asset. (`GET /a/{sha256}?format=json`)
- [ ] 2.3 Check in batches: Send up to 1000 record digests per call. Keep documents whose signature verifies and that are not sealed. Flag any with `errata_count` above zero. (`POST /v1/verify-batch`)
- [ ] 2.4 Read the terms: For a source with claimed terms, honor `AISOT-DENY` before any use, credit `AISOT-ATTRIB`, and record what you owe under `AISOT-ROYALTY-{rate}`. (`GET /v1/sources/{source_id}/terms`)
- [ ] 2.5 Keep the provenance: Write publisher, model, record digest and checkpoint into the dataset manifest so every training example traces back to its record.

## 3. Verify against the live registry
- [ ] 3.1 Hash every document: Re-hashing a stored document gives the same value.
- [ ] 3.2 Map hashes to records: A known asset returns at least one record.
- [ ] 3.3 Check in batches: A batch with one sealed test record returns it with `sealed: true`.
- [ ] 3.4 Read the terms: A denied source never reaches the training set.
- [ ] 3.5 Keep the provenance: The manifest has a record digest for every attested document.

## 4. Done when
- [ ] 4.1 Every kept document has a known publisher and record, or is marked unattested.
- [ ] 4.2 Sealed content never enters the dataset, and corrected content is reviewed.
- [ ] 4.3 The dataset manifest shows which sources asked for credit or payment.

# dataset-provenance Specification

## ADDED Requirements

### Requirement: Bulk provenance check
The pipeline SHALL check document records in batches of at most 1000 digests and SHALL drop every document whose record is sealed.

#### Scenario: Sealed document
- **WHEN** verify-batch returns `sealed: true` for a document's record
- **THEN** the document is excluded and the exclusion is logged with the digest

#### Scenario: Corrected document
- **WHEN** `errata_count` is above zero
- **THEN** the document is held for review instead of kept silently

### Requirement: Terms are honored
The pipeline SHALL read a claimed source's standing terms and SHALL exclude sources under `AISOT-DENY`.

#### Scenario: Denied source
- **WHEN** a source's terms are `AISOT-DENY`
- **THEN** none of its documents reach the dataset