import os
import subprocess
import sys
import tempfile
from pathlib import Path
from spotregen import make_sample_repo
tmp = tempfile.TemporaryDirectory()
root = make_sample_repo(Path(tmp.name) / "demo")
def spotregen(*args):
"""Run `spotregen` in the sample repository; show the call and its output."""
done = subprocess.run(
[sys.executable, "-m", "spotregen", *args],
cwd=root,
capture_output=True,
text=True,
check=False,
)
print("$ spotregen", *args)
print(done.stdout + done.stderr, end="")
print(f"[exit {done.returncode}]")Vectors as durable evaluations
Definition 1 (Vector files) A durable evaluation judges any implementation of a unit through its public names. Most evaluations are written as code and therefore bind to one language. A vector file is an evaluation written as data: a list of worked examples, each with the inputs and the output every implementation must reproduce. The file is generated once by a reference implementation, reviewed by a person, committed next to the records, and replayed by one pytest evaluation per implementation. A second implementation in another language reads the same file. Fowler’s freeq project keeps such a file for its encrypted direct messages: spec/e2ee-dm-vectors.json, generated by the Rust SDK and replayed byte for byte by the JavaScript SDK, which closed months of silent disagreement between the two.
spotregen needs no new field for this. The pytest function that replays the file is an ordinary evaluation: it is named by an intent item, it imports only public names (EVAL03), and it carries the marker regen (EVAL02). The vector file is a fixture of the evaluation, so it travels with the evaluation file in the regeneration bundle only if the evaluation reads it from a path a generator can see. Keep it under tests/ next to the evaluation.
One vector file for the sample unit
The sample repository has one unit, samplepkg.core.calc, with one promise, add(a, b) == a + b. A vector file for it lists cases that a rewrite must reproduce, including the edge cases a competent stranger would not think of:
import json
vectors = [
{"a": 0, "b": 0, "sum": 0},
{"a": -1, "b": 1, "sum": 0},
{"a": 2**63, "b": 1, "sum": 2**63 + 1},
{"a": -(2**31), "b": -(2**31), "sum": -(2**32)},
]
(root / "tests/calc-vectors.json").write_text(json.dumps(vectors, indent=2) + "\n")
print((root / "tests/calc-vectors.json").read_text())[
{
"a": 0,
"b": 0,
"sum": 0
},
{
"a": -1,
"b": 1,
"sum": 0
},
{
"a": 9223372036854775808,
"b": 1,
"sum": 9223372036854775809
},
{
"a": -2147483648,
"b": -2147483648,
"sum": -4294967296
}
]
The evaluation replays every case. It is appended to the evaluation file of the unit, which already carries the marker:
evaluation = '''
def test_add_reproduces_the_vectors():
"""Every worked example of tests/calc-vectors.json holds."""
import json
from pathlib import Path
for case in json.loads((Path(__file__).parent / "calc-vectors.json").read_text()):
assert add(case["a"], case["b"]) == case["sum"], case
'''
path = root / "tests/test_regen_eval_calc.py"
path.write_text(path.read_text() + evaluation)
print(path.read_text())import pytest
from samplepkg.core.calc import add
pytestmark = pytest.mark.regen
def test_add():
assert add(2, 3) == 5
def test_add_reproduces_the_vectors():
"""Every worked example of tests/calc-vectors.json holds."""
import json
from pathlib import Path
for case in json.loads((Path(__file__).parent / "calc-vectors.json").read_text()):
assert add(case["a"], case["b"]) == case["sum"], case
The intent item names the new evaluation, so that EVAL05 does not report it as unlinked:
unit = root / "regen/units/core.calc.toml"
unit.write_text(
unit.read_text().replace(
'evaluations = ["test_add"]',
'evaluations = ["test_add", "test_add_reproduces_the_vectors"]',
)
)
spotregen("check")$ spotregen check
regen check: 1 units, 0 errors, 0 warnings
[exit 0]
The evaluations pass on the implementation:
done = subprocess.run(
[sys.executable, "-m", "pytest", "-q", "-m", "regen", "tests"],
cwd=root,
capture_output=True,
text=True,
env={**os.environ, "PYTHONPATH": "src"},
)
print(done.stdout.strip().splitlines()[-1])2 passed, 1 warning in 0.00s
When to write one
A vector file earns its place where an implementation in another language is plausible, where the output is a byte sequence or a number that must match exactly, and where the reference implementation exists and can generate the cases. A metamorphic relation stays the better choice when the right answer cannot be stated in advance. Regenerate the file only from the reference implementation, never from a candidate: the file is the oracle, and a candidate that rewrites its own oracle has judged itself.