BT411/context/test-harness.md

id, title, status, source_sections, related_topics, key_terms, open_questions

id	title	status	source_sections	related_topics	key_terms	open_questions
test-harness	The Test Harness — how benches run, and what counts as verified	living	scratchpad/night6/bench_common.sh (the contract); build-and-run.md §bench-parity; nights 6-9 bench scripts; the #86/#110 verification failure	build-and-run reconstruction-method multiplayer reconstruction-gotchas experience-levels	bench harness field-composition scalpel

The Test Harness — how benches run, and what counts as verified

Two subjects, deliberately in one file: the MACHINERY (how to launch and read a bench) and the DOCTRINE (what a bench must show before a fix may be called fixed). The doctrine exists because the machinery was used wrong more than once.

THE DOCTRINE: verify the FIELD COMPOSITION, not a constructed proxy

The archetype failure (#86 → #110). "Weapons on damaged arms keep firing" was fixed and verified by destroying the WEAPON directly (BT_KILL_SUBSYS). The field scenario was the ARM ZONE dying while the weapon stayed healthy — a path the bench never took. The claim "fixed" stood for two playtests while the players kept hitting the bug; the actual defect (a stubbed cascade walk, gotcha #26) sat exactly in the gap between the proxy and the field. A fix verified only against a reproduction you constructed is not verified.

Rules, in order of application:

1. Reproduce the reported scenario first — the player's composition, not a convenient adjacent one. If the report is "peer's arm shot off in MP", the proof is a two-node run where PEER FIRE kills the arm (mp_armgrind.sh), not a solo self-damage run.
2. Scalpels are for isolation, not for verdicts. Deterministic hooks (BT_SELF_DAMAGE_ZONE, BT_KILL_SUBSYS, BT_FORCE_SEEK, BT_ARMOR_FORCE) are the right way to LOCATE a defect. Before a scalpel result supports a "fixed" claim, prove PATH IDENTITY to the field composition (e.g. the self-damage harness dispatches a real Entity::TakeDamageMessage — the same message + handler a peer round takes — so the scalpel shares every line from the handler down; verified 2026-08-02, and then the field composition was STILL run).
3. If the symptom is multiplayer, the proof is two nodes. Master-side correctness says nothing about what a peer sees; replication is its own layer (records, edge-detects, instance gates). #94, #84 and #110 each had a peer-side half invisible to any solo bench.
4. If the symptom is visual, the proof is pixels — BT_SHOT_EVERY frames diffed against a control region, not a log line saying the value changed (gotcha #23: a colour change that logs perfectly and renders nothing).
5. Coverage claims need a sweep, not an instance. "Works for every weapon type / chassis" = enumerate the axis and measure each point (all 3 fire gates through a zone death; all 8 chassis' authored cascade flags — zonesweep.sh). Per-chassis behaviour usually lives in AUTHORED DATA, so fixed code on one chassis proves nothing about the others.
6. Chase every anomaly in a passing run to ground. The Owens rack cascade also killing the sensor suite looked like a walk bug; it was authored skeleton data (the mast is a child of the rack). A pass with an unexplained extra effect is not yet a pass.

THE MACHINERY

The contract: scratchpad/night6/bench_common.sh

The single source of truth for launching a node THE WAY A PLAYER'S SHORTCUT DOES — read the file, its comments carry the incident history. Summary:

• bt_player_env — BT_PLATFORM=glass, BT_START_INSIDE=1 (cockpit, not chase cam), BT_DEV_GAUGES=1. Copied verbatim from the shipped .bats; bt_assert_player_env warns on drift.
• bt_launch <log> <egg> <affinity> [args] — one node, env-scoped, PID tracked in /tmp/bt_bench_pids.$$.
• bt_expert_egg — shipped eggs are ALL experience=expert; a novice egg silences heat/crits/jams and is only legal for the gimp bench (which must say so out loud). See experience-levels.
• bt_kill_ours — teardown kills ONLY this run's PIDs (a blanket taskkill /IM once shot down two live sessions).

Single-node bench skeleton

. /c/git/bt411/scratchpad/night6/bench_common.sh
cd /c/git/bt411/content || exit 1
taskkill //F //IM btl4.exe > /dev/null 2>&1; sleep 2      # stale-node clear
sed "s/^map=.*/map=grass/; s/^time=.*/time=day/; 0,/^vehicle=.*/s//vehicle=$V/" MP.EGG > X.EGG
export BT_<gates>=...                                     # diagnostics + scalpels
bt_launch x.log X.EGG 0x03
sleep <duration>; taskkill //F //IM btl4.exe; sleep 2      # then grep x.log

The vehicle sed must replace the WHOLE line (s/^vehicle=.*/), or you mint vehicle=thr1bhk1 and the spawn FATALs. Chassis codes: ava1 bhk1 lok1/2 mad1/2 own1 snd1 thr1 vul1 (own1 has NO arm zones — rack zones instead).

Weapon/combat benches need map=grass time=day. MP.EGG authors map=cavern time=night; a bench that copies it without the map sed (e.g. via bt_expert_egg alone — mp_double.sh has this gap) parks the GOTO-driven mechs against cavern rock and they shoot terrain for the whole window. Always sed the map/time on the COPY, single-node and two-node alike.

THE ZONE-WALK MATRIX (scratchpad/night10/zonewalk.sh) — systematic

per-panel hit-location testing, operator-watchable (built for #124)

The precision-targeting rig: two nodes. Node B (launched first, back window) is the TARGET — BT_SPIN_SELF=15 rotates its own viewpoint mech in place (deg/s; the BT_SPAWN_AT write pattern, applied per frame) so every aspect passes the shooter's boresight; its window shows its own paper doll taking the hits (BT_DMG_LOG=1 prints every [dmghit] with zone + level). Node A (foreground — it owns the pick focus) runs BT_ZONE_WALK=<secs/zone>: walks the target's damage zones in index order, resolves each zone's CARRIER SEGMENT's live world position (BTResolveSegmentWorld — the same resolver the attached smoke rides; zone→segment via the BTMechZoneSegAndName bridge), SERVOS the torso twist + aim elevation until the centered reticle's pick ray (BTGetAimRay(0,0)) points at the segment, then pulses the laser trigger 3× and advances. [walk] ZONE/FIRE/HOLD lines on A pair with [dmghit] on B. Launch: bash scratchpad/night10/zonewalk.sh (relay included; NO kill timer — the session stays up for observation; teardown = taskkill btl4). Node A also carries BT_ARMOR_LOG=1 + BT_SHOT_EVERY=300 BT_SHOT_PREFIX=zwA (added for #87): the armour watcher's [armor] ... level P -> Q lines on A are the OBSERVER-side receipt that B's zone levels replicated (the walk is the standing zone-replication bench — A-side peaks must match B's [dmghit] finals), and the numbered captures are the pixel record of the enemy hull darkening. Digest: python scratchpad/night10/digest_walk.py (per-zone hits, amt/hit, level first→last on B; replicant peaks seen by A; zone-repl lines). Companion scalpels: BT_ASPECT_TEST=1 (the zero-premise frame probe — 4 world-cardinal self-impacts + one at each weapon's physical MUZZLE, the left/right anchors that caught the #124 reflection-vs-rotation error); BT_AIM_SWEEP=<period> (blind elevation sweep — superseded by the walker). Hard-won constraints baked into the walker (do not relearn these):

• Engage gate: servo only when the aim ray is live AND range < 150 AND |bearing| < 1.1 rad; otherwise RELAX the twist toward 0 and hold the zone clock. Without it the servo winds the twist to its limit while the goto marches (limit-clamp fight = the mech visibly SHAKES) — and the gate must sit well above the goto's resting bearing error (~0.55 rad at STOP=90) or it deadlocks in HOLD.
• Yaw polarity: the twist cell's angular sense is OPPOSITE atan2(x,−z) world yaw (operator-observed live: += turned the torso AWAY — the second witness for the #124 SelectSlice twist-sign flip). The servo starts at −1 and carries a growth WATCHDOG that self-flips if the error diverges.
• Damped servo: gain 0.40, step cap 0.025 rad/frame, deadband — the aim ray lags the twist write by a frame; full-error correction oscillates.
• Approach port: dropzones spawn cross-map; the walker one-shot teleports A to 100 u off the target (BT_SPAWN_AT pattern incl. the +500 y ground-snap lift) instead of minutes of marching.

Two-node bench skeleton (the MP pattern)

bt_assert_player_env
bt_expert_egg MP.EGG X.EGG
BT_MP_LOG=1 <gates> bt_launch x_b.log X.EGG 0x0C -net 1601   # node B first
sleep 2
BT_MP_LOG=1 <gates> bt_launch x_a.log X.EGG 0x03 -net 1501   # node A
sleep 5
python ../tools/btconsole.py X.EGG 127.0.0.1:1501 127.0.0.1:1601 &   # the relay STARTS the mission
sleep <duration>; kill $relay; bt_kill_ours

• A -net node renders NOTHING until btconsole.py (the headless relay) starts the mission — a black window is pre-mission, not a hang.
• Affinity: TWO logical processors per node, disjoint sets (0x03/0x0C); one LP starves the gauge executive (multiplayer peer-shakiness fix).
• Give the SHOOTER dense fire (BT_AF_PERIOD=3 + BT_AF_MISSILE=1) and the victim sparse (=9) when one side must lose a grind. Unthrottled autofire trips the FailureHeat all-weapons brick and combat dies.
• Combat drive: BT_GOTO=enemy BT_GOTO_STOP=<units> — 100 ends in a visual ram scrum; 180 gives a standing exchange.
• Read each claim on the node that can see it: damage on the VICTIM's log, replication on the OBSERVER's ([zone-repl], [mlrec], mirrored DETs).

Process hygiene (every one of these cost a session)

• Stale-node taskkill FIRST. A leftover node joins the next lobby as a third instance and poisons the run.
• Never double-background. run_in_background + an inner & orphans the script mid-startup: nodes launch, the script's own cleanup never runs.
• Teardown kill order closes windows one at a time — a "crashed" window at the end of a timed run is usually the bench's own taskkill.
• Stale exe: LNK1104 / a bench ignoring new diagnostics = the previous process still holds btl4.exe, or the build silently didn't run — check the btl4.vcxproj -> line printed, then rerun.
• Editing source through bash-heredoc python collapses \ one level even single-quoted -- a "\n" you meant as backslash-n arrives as a real newline, the "fix" silently no-ops (good == bad), and the C2001 hunt repeats. Build backslashes as bytes([92])/chr(92), and verify a replace by LENGTH DELTA or a re-read, never by the script printing "fixed".
• Bench artifacts (*.EGG copies, *_NNN.png, bench logs) must be deleted from content/ before any dist cut; field logs are never committed.

Reading logs without fooling yourself

• A capped/throttled diagnostic that is silent is NOT evidence of absence — the #99 [seqrun] shared counter "proved" a sequence never ran; it was throttle starvation. Per-object throttles for per-object questions.
• Alarm-line counters are not trend lines — free=0 printed AT the failure is true by definition; the 30s census carries the trend (#32, three separate times).
• Log the actor's NAME at every refusal/decision ('PPC' fire REFUSED); anonymous counters (FIRED #20) cannot support attribution claims.
• Field logs have NO diagnostic gates set — a fix whose acceptance evidence sits behind an env var is unverifiable in the field. Spawn-time summaries (one line, ungated) answer questions retroactively; per-frame traces stay gated.

Bench-script gotchas (each has burned a session)

• Write benches CLEAN, never sed-derive a chain — sed-derived copies silently dropped envs twice (the [replgimp] silence); export envs INSIDE the per-node subshell.
• Absolute paths everywhere; cd /c/git/bt411 first — the cwd trap (heredoc python + cd scratchpad/nightN) has broken later builds/appends with fatal: pathspec/cannot stat.
• No apostrophes/backticks through bash heredocs — write a .py file and run it (quoting has mangled posted tracker comments and killed scripts mid-parse).
• Drain stale watchdogs before relaunching — a leftover waiter's taskkill has killed a live bench mid-run.
• Scripted input goes through the REAL seam: BT_BTNTEST/BT_BTNTEST2 (RIO queue → mode-mask drain). In 2-node runs press at poll ≥900 — round-start jitter can eat earlier presses.
• Capture the viewpoint that can see the change (gotcha #13 bullet): cockpit view can't show your own legs; pair screenshots with a state probe and READ a frame before concluding.
• Build ritual first — see build-and-run §Build ritual (stale exe = phantom results).

Key Relationships

• Uses: build-and-run (parity, env gates, BT_SHOT capture) · experience-levels (expert vs novice gating)
• Informs: reconstruction-method (step 4 "verify honestly" — this file is the how)
• Incident sources: reconstruction-gotchas §23 (pixels), §26 (silent stubs); multiplayer (replication layers)