Add MW4COMPARE: .mw4 decompiler toolchain and V4H comparison harness

Tooling built to recover editable source for six 'Mech chassis that exist in the parallel FS_Build_V4H build but not in this repo. Reverse-engineers every compiled record type in the .mw4 package format back to the .data / .instance / .subsystems / .damage / .contents / .torso / .engine / .armature sources the content pipeline consumes. Nothing here is wired into the game build. It is a standalone analysis harness run from Linux. Package format -------------- "#VBD" container. Directory records are [len][name][FILETIME][origSize] [storedSize][offset], payload base at dword 0x0C. A record is stored raw when storedSize == origSize, otherwise LZW (9->12-bit LSB-first codes, 256=clear, 257=EOF, dict from 258), per Database.cpp:451. GameModel records are flat /Zp4 structs following the C++ inheritance chain Entity(0) -> Mover(28) -> MWObject(80) -> Vehicle(664) -> Mech(756), 1636 bytes total. CreateMessage records follow Replicator -> Entity -> Mover -> MWMover -> MWObject -> Vehicle -> Mech from start=16 (the undeclared Connection__Message header), ending at 341 and padded to 344. tools/decompile/ ---------------- datamap.py header-driven layout engine; CHAIN + ANCHORS {Vehicle:664, Mech:756} assert the struct offsets mw4msg.py CreateMessage reader/walker data.py .data constants.py define/table symbol resolution damage.py .damage contents.py .contents smallmodel.py .torso + .engine instance.py .instance armature.py / armature_parts.py .armature + armaturedata/armaturevideo assembly.py joint hierarchy renderer make_generic_doll.py builds generic MFD/Radar damage dolls verify_*.py per-type round-trip verifiers Verified round-trip across all 64 shared chassis: .armature 2938/2976 pages .subsystems 7579/7585 keys .data map 6071/6071 values .data trip 8291/8306 keys .damage 6605/6605 keys .contents 7480/7480 keys .torso+.engine 1280/1280 keys .instance 896/896 keys, 64/64 pages armature_parts 1202/1202 .data, 1149/1202 .video Layout-discovery lessons (documented in DECOMPILING.md) ------------------------------------------------------- - Never let a field map be discovered by the values that verify it. A value-matching pass reported 4288/4288 while mis-assigning 34 keys. The map was rebuilt from header declaration order, anchored on uniquely resolved fields. - Read the factory, not the data. 12 .data fields and 5 Torso fields are declared plain Stuff::Scalar but multiplied by Radians_Per_Degree in Mech_Tool.cpp:889 / Torso_Tool.cpp. - Strip typedefs before walking a header. A stray `typedef int AttributeID;` masked a missing ClassID - two 4-byte errors cancelling out, caught only by the ANCHORS assertion. - A verifier that silently narrows its own input reports success. Braced blocks must be hidden before splitting pages, replacing both CR and LF, because a `Shadow={...}` block contains a line reading `[shadow]` and splitlines() also splits on bare CR. - NSWIZZLE is undefined, so the #else branch is live and orders members differently. bool is 1 byte; char x[MaxStringLength] is 256. - V4H carries stale Mech IDs (their Atlas is 5, ours 6), so 64 of 65 shared chassis are off by one; --retarget-ids emits $(M_<Chassis>)/$(IDS_<Chassis>). reports/ holds generated diffs. The two ~5 MB manifest-*.tsv intermediates are gitignored; regenerate everything with run-comparison.sh. Co-authored-by: Claude Opus 5 (Anthropic) <noreply@anthropic.com> Co-authored-by: GitHub Copilot <copilot@github.com>
2026-08-08 16:47:57 -05:00
parent e088555b96
commit 83478b7666
60 changed files with 13442 additions and 0 deletions
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+"""Verification harness for the mech .data{GameModel} field map.
+
+Reads every mapped value through its header-derived offset and compares against
+the source .data, across all 64 chassis we hold both forms for.
+
+Unlike a value-discovered map this one can genuinely fail: the offsets come from
+declaration order, so a mis-parsed header shows up at once as a whole column of
+wrong values. Keys that miss consistently by a constant factor are reported with
+their decoded/source ratio, which is how unit conversions (deg -> rad, kph ->
+m/s) announce themselves.
+
+    python3 verify_data.py
+"""
+import sys, os, re, collections
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+import datamap
+
+
+def close(a, b):
+    return abs(a - b) <= max(1e-4, abs(b) * 1e-5)
+
+
+def matches(got, want_text, typ):
+    """Compare a decoded value against the raw source text."""
+    if typ in datamap.VECTORS:
+        nums = [float(x) for x in re.findall(r'-?\d+\.?\d*(?:[eE][-+]?\d+)?', want_text)]
+        return len(nums) == len(got) and all(close(a, b) for a, b in zip(got, nums))
+    if typ == "char":
+        return got.lower() == want_text.strip().lower()
+    if typ in ("bool", "BYTE"):
+        w = want_text.strip().lower()
+        if w in ("true", "yes", "1"):
+            return got == 1
+        if w in ("false", "no", "0"):
+            return got == 0
+        return None
+    try:
+        return close(float(got), float(want_text))
+    except ValueError:
+        return None
+
+
+def main():
+    pairs, field_map = datamap.build()
+    t = collections.Counter()
+    bad = collections.Counter()
+    examples = []
+
+    for ch, kv, blob in pairs:
+        for key, (off, typ, size, angle) in field_map.items():
+            if key not in kv:
+                continue
+            got = datamap.read(blob, off, typ, size, angle)
+            ok = matches(got, kv[key], typ)
+            if ok is None:
+                t["skipped"] += 1
+                continue
+            t["values"] += 1
+            if ok:
+                t["ok"] += 1
+            else:
+                bad[key] += 1
+                if len(examples) < 12:
+                    examples.append((ch, key, typ, kv[key], got, off))
+
+    print(f"chassis         : {len(pairs)}")
+    print(f"mapped keys     : {len(field_map)}")
+    print(f"values compared : {t['values']}   exact: {t['ok']}   wrong: {t['values'] - t['ok']}")
+    print(f"not comparable  : {t['skipped']} (enum/resource text, decoded separately)")
+    if bad:
+        print("\nkeys not matching:")
+        for k, v in bad.most_common(25):
+            print(f"   {k:34s} x{v}")
+        print("\nexamples (chassis, key, type, source, decoded, offset):")
+        for ch, key, typ, src, got, off in examples:
+            print(f"   {ch:14s} {key:28s} {typ:10s} src={src!r:24s} got={got!r} @{off}")
+
+
+if __name__ == "__main__":
+    main()