Files
firestorm/MW4COMPARE/tools/decompile/verify_armature.py
T
83478b7666 Add MW4COMPARE: .mw4 decompiler toolchain and V4H comparison harness
Tooling built to recover editable source for six 'Mech chassis that exist
in the parallel FS_Build_V4H build but not in this repo. Reverse-engineers
every compiled record type in the .mw4 package format back to the .data /
.instance / .subsystems / .damage / .contents / .torso / .engine /
.armature sources the content pipeline consumes.

Nothing here is wired into the game build. It is a standalone analysis
harness run from Linux.

Package format
--------------
"#VBD" container. Directory records are [len][name][FILETIME][origSize]
[storedSize][offset], payload base at dword 0x0C. A record is stored raw
when storedSize == origSize, otherwise LZW (9->12-bit LSB-first codes,
256=clear, 257=EOF, dict from 258), per Database.cpp:451.

GameModel records are flat /Zp4 structs following the C++ inheritance
chain Entity(0) -> Mover(28) -> MWObject(80) -> Vehicle(664) -> Mech(756),
1636 bytes total. CreateMessage records follow Replicator -> Entity ->
Mover -> MWMover -> MWObject -> Vehicle -> Mech from start=16 (the
undeclared Connection__Message header), ending at 341 and padded to 344.

tools/decompile/
----------------
  datamap.py        header-driven layout engine; CHAIN + ANCHORS
                    {Vehicle:664, Mech:756} assert the struct offsets
  mw4msg.py         CreateMessage reader/walker
  data.py           .data      constants.py  define/table symbol resolution
  damage.py         .damage    contents.py   .contents
  smallmodel.py     .torso + .engine         instance.py  .instance
  armature.py / armature_parts.py  .armature + armaturedata/armaturevideo
  assembly.py       joint hierarchy renderer
  make_generic_doll.py  builds generic MFD/Radar damage dolls
  verify_*.py       per-type round-trip verifiers

Verified round-trip across all 64 shared chassis:
  .armature      2938/2976 pages     .subsystems  7579/7585 keys
  .data map      6071/6071 values    .data trip   8291/8306 keys
  .damage        6605/6605 keys      .contents    7480/7480 keys
  .torso+.engine 1280/1280 keys      .instance     896/896 keys, 64/64 pages
  armature_parts 1202/1202 .data, 1149/1202 .video

Layout-discovery lessons (documented in DECOMPILING.md)
-------------------------------------------------------
- Never let a field map be discovered by the values that verify it. A
  value-matching pass reported 4288/4288 while mis-assigning 34 keys. The
  map was rebuilt from header declaration order, anchored on uniquely
  resolved fields.
- Read the factory, not the data. 12 .data fields and 5 Torso fields are
  declared plain Stuff::Scalar but multiplied by Radians_Per_Degree in
  Mech_Tool.cpp:889 / Torso_Tool.cpp.
- Strip typedefs before walking a header. A stray `typedef int AttributeID;`
  masked a missing ClassID - two 4-byte errors cancelling out, caught only
  by the ANCHORS assertion.
- A verifier that silently narrows its own input reports success. Braced
  blocks must be hidden before splitting pages, replacing both CR and LF,
  because a `Shadow={...}` block contains a line reading `[shadow]` and
  splitlines() also splits on bare CR.
- NSWIZZLE is undefined, so the #else branch is live and orders members
  differently. bool is 1 byte; char x[MaxStringLength] is 256.
- V4H carries stale Mech IDs (their Atlas is 5, ours 6), so 64 of 65 shared
  chassis are off by one; --retarget-ids emits $(M_<Chassis>)/$(IDS_<Chassis>).

reports/ holds generated diffs. The two ~5 MB manifest-*.tsv intermediates
are gitignored; regenerate everything with run-comparison.sh.

Co-authored-by: Claude Opus 5 (Anthropic) <noreply@anthropic.com>
Co-authored-by: GitHub Copilot <copilot@github.com>
2026-08-08 16:47:57 -05:00

97 lines
3.7 KiB
Python

#!/usr/bin/env python3
"""Verification harness for the .armature decompiler.
Rebuilds every chassis from our own packed records and compares against the
known source .armature. Compares page transforms as a multiset keyed on
(name, transform) so duplicate page names - Victor has two [site_lshellport]
pages under different joints - are handled.
python3 verify_armature.py
"""
import sys, os, glob, re, collections
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import armature
def source_entries(path):
"""-> (list of (name, rot, trans), {parent: [child, ...]})"""
txt = open(path, "rb").read().decode("latin-1")
entries, children = [], {}
for m in re.finditer(r'^\[([^\]]+)\]\s*\r?\n(.*?)(?=^\[|\Z)', txt, re.M | re.S):
name, body = m.group(1).lower(), m.group(2)
r = re.search(r'Rotation=([-\d.eE ]+)', body)
t = re.search(r'Translation=([-\d.eE ]+)', body)
kids = [c.lower() for c in re.findall(r'Child=(\S+)', body)]
if kids:
children[name] = kids
if r and t:
entries.append((name,
tuple(float(x) for x in r.group(1).split()),
tuple(float(x) for x in t.group(1).split())))
return entries, children
def close(a, b, tol):
return a and b and max(abs(x - y) for x, y in zip(a, b)) <= tol
def angles_close(a, b, tol=0.01):
return a and b and max(abs(((x - y + 180) % 360) - 180) for x, y in zip(a, b)) <= tol
def main():
src_paths = {os.path.basename(p)[:-len(".armature")].lower(): p
for p in glob.glob(armature.OUR_MECH_SOURCE + "/*/*.armature")}
t = collections.Counter()
residual = collections.Counter()
for mech_dir in sorted(glob.glob(armature.OUR_RECORDS + "/*")):
if not glob.glob(mech_dir + "/*{armature}"):
continue
chassis, entries, children = armature.rebuild(mech_dir)
if not chassis or chassis.lower() not in src_paths:
continue
t["chassis"] += 1
src_list, src_children = source_entries(src_paths[chassis.lower()])
got = [(e["name"].lower(), e["rot"], e["trans"]) for e in entries]
pool = list(got)
for name, rot, tr in src_list:
t["pages"] += 1
hit = next((g for g in pool if g[0] == name
and close(g[2], tr, 2e-3) and angles_close(g[1], rot)), None)
if hit:
pool.remove(hit)
t["exact"] += 1
continue
near = next((g for g in pool if g[0] == name and close(g[2], tr, 2e-3)), None)
if near:
pool.remove(near)
t["rot_only"] += 1
residual[name] += 1
else:
t["missing"] += 1
residual["MISSING " + name] += 1
for parent, kids in src_children.items():
t["childlists"] += 1
if sorted(kids) == sorted(c.lower() for c in children.get(parent, ())):
t["childlists_ok"] += 1
else:
residual["CHILDREN " + parent] += 1
print(f"chassis : {t['chassis']}")
print(f"pages compared : {t['pages']}")
print(f" fully exact : {t['exact']}")
print(f" translation ok, rotation lost by the packer : {t['rot_only']}")
print(f" not recovered : {t['missing']}")
print(f"child lists : {t['childlists_ok']}/{t['childlists']} exact")
if residual:
print("\nresidual, by page:")
for k, v in residual.most_common(10):
print(f" {k:22s} x{v}")
if __name__ == "__main__":
main()