athlon-schema

Spec version 0.1.0 (draft) · Based on NIST AI 200-2 initial public draft (August 2026) · Apache-2.0

A draft machine-readable file format for assessments conducted under the TEVV-Athlon Framework, with JSON Schemas and a reference validator. The Python package and command-line tool are named tevv-athlon.

Purpose

NIST AI 200-2 ipd describes what each stage of a TEVV-Athlon should produce but does not specify a format for recording it. This project proposes one, so that plans and results from different assessments can be compared, checked automatically and reviewed by others. It is meant as a reference point for discussion, not as a proposed standard. It records the outputs of the framework's four stages in a structured form and does not modify or reinterpret the framework.

Mapping to the framework

Stage (NIST AI 200-2 ipd)	File	Contents
1. Articulate & Organize §2.1	`athlon.yaml`	System description, answers to the seven goal questions, trustworthiness characteristics, lifecycle stage
2. Define & Construct §2.2	`athlon.yaml`	Blocks, each with a definition, the evidence it requires, and measures (unit, direction, optional target)
3. Apply & Measure §2.3	`athlon.yaml` `evidence/*.jsonl`	Events (type, setting, condition factors) and the Toolbox in the plan; one evidence record per observation
4. Synthesize & Interrogate §2.4	`athlon.yaml` `review.yaml`	Interrogation questions registered in the plan before testing; responses, limitations and validity boundary in the review afterwards

athlon.yaml is written before any testing takes place. The controlled vocabulary (the seven AI RMF trustworthiness characteristics, Event types, settings and lifecycle stages) follows the draft and is listed in section 4 of the specification.

Design choices

No composite score or pass/fail verdict. Measures are reported individually. A target is optional, set per measure, and reported beside the estimate rather than used as a threshold.
Pre-registered interrogation. Stage 4 questions are recorded in the plan before any results exist and answered in the review afterwards. The validator warns (W201) when none are registered.
Provenance on every evidence record. Each record names its Event, Tool, Block and measure. It also carries the SHA-256 digest of its raw source, the adapter and adapter version that produced it, the run environment (model, version, configuration hash) and a collection timestamp.
Conditions carried through to Stage 4. Events declare condition factors and their levels, and each evidence record states the levels under which it was observed, so results can be compared across them.
Strict, versioned and extensible. Unknown fields are rejected everywhere except inside an extensions object. Each schema has a versioned $id.

Schemas

JSON Schema draft 2020-12, generated from the project's data models. Each URL below is also the schema's $id.

Plan

athlon.yaml: goals, characteristics, Blocks, Events, Toolbox and interrogation questions.

https://neems.github.io/athlon-schema/schema/0.1/athlon.schema.json

Evidence record

One line of evidence/*.jsonl: a single observation with its provenance.

https://neems.github.io/athlon-schema/schema/0.1/evidence.schema.json

Review

review.yaml: interrogation responses, limitations and validity boundary.

https://neems.github.io/athlon-schema/schema/0.1/review.schema.json

Using the schemas

You do not need this project's Python package to check a file. Any JSON Schema validator that supports draft 2020-12 will do, for example check-jsonschema:

check-jsonschema --schemafile https://neems.github.io/athlon-schema/schema/0.1/athlon.schema.json athlon.yaml

Evidence files hold one JSON object per line, so each line is checked against the evidence schema on its own.

To get field completion and inline errors while editing a plan, put this line at the top of athlon.yaml. Editors that use the YAML language server, such as VS Code with the Red Hat YAML extension, can then check the file as you type.

# yaml-language-server: $schema=https://neems.github.io/athlon-schema/schema/0.1/athlon.schema.json

The URLs under /schema/0.1/ will not change. A future spec version will be published under its own path, and earlier versions will stay available.

Quickstart

The reference validator also checks cross-references between goals, characteristics, Blocks, Events and Tools, which JSON Schema alone cannot express. It requires Python 3.11 or later and uv.

git clone https://github.com/neems/athlon-schema.git
cd athlon-schema
uv sync
uv run tevv-athlon validate spec/examples/ambient-scribe/athlon.yaml

validate exits 0 when there are no errors, 1 when there are errors and 2 on a usage error. Add --format json for machine-readable output, and run uv run tevv-athlon codes to list every issue code with a fix hint.

Examples

The draft's worked example (§3, the Query-Violation problem) encoded as a plan.
A plan for an agentic system: an ambient clinical scribe that drafts a visit note and proposes medical-record writes for clinician approval (an Action-Violation problem). Plan only; no Event has been run.
An illustrated walkthrough of the same plan populated with synthetic evidence and a review. The results are fabricated for illustration and are not measurements.

All examples use synthetic data. None contains protected health information, and none describes a medical device.

Status

Draft, spec version 0.1.0. The format will change while NIST AI 200-2 remains a draft; version 1.0 is intended to align with the final publication. The package is not published to PyPI.

In this version the command-line tool validates athlon.yaml plans. Evidence and review files are defined by their schemas and can be checked with any JSON Schema validator. Evidence adapters, report generation and migration tooling are planned but not yet implemented.

Feedback

Issues and pull requests are welcome on GitHub. Corrections to how the format represents the draft are particularly useful.