repro-evidence-kit

repro-evidence-kit is a maintainer-focused CLI for reviewing generated artifacts with hash manifests, evidence bundles, and CI-friendly sandbox-output checks.

It helps maintainers review artifact-heavy pull requests, release automation, and research outputs without committing private datasets, proprietary samples, or noisy execution logs.

Why this matters

Generated artifacts are hard to review when the only proof is a large log, a private input tree, or a verbal claim that "nothing important changed." repro-evidence-kit keeps the review surface small: it records byte hashes, separates expected output changes from unexpected ones, and stores enough command context for another maintainer to rerun or challenge the evidence.

The project is intentionally target-neutral. It should help maintainers in CI, release automation, data-processing, security research, and binary-analysis workflows without requiring them to publish proprietary samples or project-specific case files.

Use cases

• Review what changed during artifact-heavy CI or release automation.
• Verify that a sandboxed experiment only produced explicitly allowed outputs.
• Attach compact, hash-backed evidence bundles to pull requests or research notes.
• Keep generated reports reviewable without publishing private input data.

What this proves

• File manifests prove byte identity for the files they list.
• Manifest diffs separate expected artifact changes from unexpected ones.
• Sandbox verification proves the observed output set stayed inside an explicit allowlist.
• Evidence bundles preserve command context, inputs, outputs, and hashes for review.
• Local HMAC sidecars provide tamper detection for exact evidence-bundle bytes.

What this does not prove

• Hashes do not prove that generated outputs are semantically correct.
• A passing sandbox check does not prove that a command was safe.
• Local HMAC sidecars do not prove signer identity, public trust, command execution, or artifact semantics.
• Private or proprietary inputs still require reviewer judgment outside this repository.

Features

• Create SHA-256 manifests with deterministic file ordering and per-file digests for files or filtered directory trees.
• Diff two manifests to identify added, removed, changed, and unchanged artifacts.
• Verify sandbox/experiment outputs against explicit allowlists, with optional JUnit XML for CI report consumers.
• Validate simple YAML or JSON evidence bundles, with optional JSON Schema checks.
• Create and verify local HMAC sidecars for exact-bundle tamper detection.
• Includes only synthetic public examples.

Install

Install the latest release from PyPI:

pip install repro-evidence-kit

For local development:

python -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"
pytest -q

Quick start

repro-evidence manifest create examples/dummy-binary -o before.json
repro-evidence manifest diff before.json before.json
repro-evidence evidence validate examples/evidence-bundle.yaml

For larger artifact trees, filter manifests with explicit include/exclude patterns:

repro-evidence manifest create artifacts --include reports --exclude "*.tmp" -o manifest.json

Manifest file ordering and per-file digests are deterministic for identical inputs. The manifest document itself includes created_at, so its complete JSON bytes are not identical across runs. Generated manifests also disclose the built-in .git and __pycache__ directory exclusions in implicit_excluded_directories.

For stricter evidence-bundle checks, install the optional schema extra and validate against the checked-in JSON Schema:

pip install "repro-evidence-kit[schema]"
repro-evidence evidence validate examples/evidence-bundle.yaml --schema

Signed bundle sidecars are optional. For a local tamper-detection prototype, create or provide local trust material and keep it out of git:

printf 'synthetic local test key only\n' > local-test.key
repro-evidence evidence sign examples/evidence-bundle.yaml --key local-test.key -o evidence-bundle.yaml.sig.json
repro-evidence evidence verify-signature examples/evidence-bundle.yaml --signature evidence-bundle.yaml.sig.json --key local-test.key

Sandbox verification compares a baseline manifest with an after-run manifest:

repro-evidence verify sandbox-run before.json after.json --allow-added report.json

The command exits 0 when all changes are allowed and 1 when unexpected changes are present.

Documentation

• CLI reference
• CLI exit codes
• Tutorial
• Evidence bundle format
• Use cases
• Signed evidence bundles design note
• Signer trust, rotation, and revocation policy
• Maintainer workflow
• Release checklist
• PyPI publishing
• GitHub Actions cookbook — CI recipes for validation, manifests, sandbox checks, and schema-backed filtered workflows.
• Design principles
• Why not just ...?
• Claim boundaries
• Roadmap

Data policy

This repository is for generic reproducibility tooling. Do not add proprietary binaries, private datasets, copyrighted samples, live credentials, forensic case data, or project-specific reverse-engineering targets. Public examples must be synthetic or clearly redistributable.

Status

0.4.x is an early maintainer-tooling release series. The CLI and schema stay intentionally small, conservative, and synthetic-example-only.