Files
firestorm/MW4COMPARE/tools/variants-report.py
T
83478b7666 Add MW4COMPARE: .mw4 decompiler toolchain and V4H comparison harness
Tooling built to recover editable source for six 'Mech chassis that exist
in the parallel FS_Build_V4H build but not in this repo. Reverse-engineers
every compiled record type in the .mw4 package format back to the .data /
.instance / .subsystems / .damage / .contents / .torso / .engine /
.armature sources the content pipeline consumes.

Nothing here is wired into the game build. It is a standalone analysis
harness run from Linux.

Package format
--------------
"#VBD" container. Directory records are [len][name][FILETIME][origSize]
[storedSize][offset], payload base at dword 0x0C. A record is stored raw
when storedSize == origSize, otherwise LZW (9->12-bit LSB-first codes,
256=clear, 257=EOF, dict from 258), per Database.cpp:451.

GameModel records are flat /Zp4 structs following the C++ inheritance
chain Entity(0) -> Mover(28) -> MWObject(80) -> Vehicle(664) -> Mech(756),
1636 bytes total. CreateMessage records follow Replicator -> Entity ->
Mover -> MWMover -> MWObject -> Vehicle -> Mech from start=16 (the
undeclared Connection__Message header), ending at 341 and padded to 344.

tools/decompile/
----------------
  datamap.py        header-driven layout engine; CHAIN + ANCHORS
                    {Vehicle:664, Mech:756} assert the struct offsets
  mw4msg.py         CreateMessage reader/walker
  data.py           .data      constants.py  define/table symbol resolution
  damage.py         .damage    contents.py   .contents
  smallmodel.py     .torso + .engine         instance.py  .instance
  armature.py / armature_parts.py  .armature + armaturedata/armaturevideo
  assembly.py       joint hierarchy renderer
  make_generic_doll.py  builds generic MFD/Radar damage dolls
  verify_*.py       per-type round-trip verifiers

Verified round-trip across all 64 shared chassis:
  .armature      2938/2976 pages     .subsystems  7579/7585 keys
  .data map      6071/6071 values    .data trip   8291/8306 keys
  .damage        6605/6605 keys      .contents    7480/7480 keys
  .torso+.engine 1280/1280 keys      .instance     896/896 keys, 64/64 pages
  armature_parts 1202/1202 .data, 1149/1202 .video

Layout-discovery lessons (documented in DECOMPILING.md)
-------------------------------------------------------
- Never let a field map be discovered by the values that verify it. A
  value-matching pass reported 4288/4288 while mis-assigning 34 keys. The
  map was rebuilt from header declaration order, anchored on uniquely
  resolved fields.
- Read the factory, not the data. 12 .data fields and 5 Torso fields are
  declared plain Stuff::Scalar but multiplied by Radians_Per_Degree in
  Mech_Tool.cpp:889 / Torso_Tool.cpp.
- Strip typedefs before walking a header. A stray `typedef int AttributeID;`
  masked a missing ClassID - two 4-byte errors cancelling out, caught only
  by the ANCHORS assertion.
- A verifier that silently narrows its own input reports success. Braced
  blocks must be hidden before splitting pages, replacing both CR and LF,
  because a `Shadow={...}` block contains a line reading `[shadow]` and
  splitlines() also splits on bare CR.
- NSWIZZLE is undefined, so the #else branch is live and orders members
  differently. bool is 1 byte; char x[MaxStringLength] is 256.
- V4H carries stale Mech IDs (their Atlas is 5, ours 6), so 64 of 65 shared
  chassis are off by one; --retarget-ids emits $(M_<Chassis>)/$(IDS_<Chassis>).

reports/ holds generated diffs. The two ~5 MB manifest-*.tsv intermediates
are gitignored; regenerate everything with run-comparison.sh.

Co-authored-by: Claude Opus 5 (Anthropic) <noreply@anthropic.com>
Co-authored-by: GitHub Copilot <copilot@github.com>
2026-08-08 16:47:57 -05:00

94 lines
3.4 KiB
Python
Executable File

#!/usr/bin/env python3
"""
variants-report.py - inventory of resource/Variants/*.mw4 (saved mechlab loadouts).
python3 variants-report.py <variantsDir> <v4hManifest.tsv> <oursManifest.tsv>
A variant package holds 4 records:
[0] 'Resource\\Variants\\<name>' 4 B (raw id)
[1] '{Mech}' ~350 B
[2] '<chassis>{Subsystem}' ~4 KB <- names the chassis it needs
[3] '<8-char token>' 16 B
So the chassis a variant depends on is readable straight from the manifest,
with no decompression: the record whose name ends in '{Subsystem}'.
The report groups variants by chassis and flags any whose chassis is absent
from the target build's core.mw4 (those will not load).
"""
import sys, os, collections
def chassis_in_core(manifest):
"""Set of chassis names that core.mw4 defines (one *.subsystems per chassis)."""
out = set()
with open(manifest, encoding="latin-1") as fh:
for line in fh:
pkg, _rid, name, *_ = line.rstrip("\n").split("\t")
if pkg.lower() != "core.mw4":
continue
n = name.lower().replace("\\", "/")
if n.endswith(".subsystems"):
out.add(n.rsplit("/", 1)[-1][:-len(".subsystems")])
return out
def variant_chassis(manifest):
"""variantPkgRelPath -> chassis name it requires."""
out = {}
with open(manifest, encoding="latin-1") as fh:
for line in fh:
pkg, _rid, name, *_ = line.rstrip("\n").split("\t")
if not pkg.lower().startswith("variants/"):
continue
if name.endswith("{Subsystem}"):
out[pkg] = name[:-len("{Subsystem}")].lower()
return out
def main():
if len(sys.argv) != 4:
sys.exit(__doc__)
vdir, v4h_manifest, ours_manifest = sys.argv[1:4]
have_v4h = chassis_in_core(v4h_manifest)
have_ours = chassis_in_core(ours_manifest)
vc = variant_chassis(v4h_manifest)
print(f"# variant packages: {len(vc)} (files on disk: "
f"{len([f for f in os.listdir(vdir) if f.lower().endswith('.mw4')])})")
print(f"# chassis defined in V4H core.mw4 : {len(have_v4h)}")
print(f"# chassis defined in OUR core.mw4 : {len(have_ours)}")
print(f"# chassis only V4H defines : {sorted(have_v4h - have_ours)}")
print()
by = collections.defaultdict(list)
for pkg, ch in vc.items():
by[ch].append(pkg)
loadable_now = broken = needs_new_mech = 0
print(f"{'chassis':22s} {'count':>5s} status")
for ch in sorted(by, key=lambda c: (-len(by[c]), c)):
n = len(by[ch])
if ch in have_ours:
status = "loadable in OUR build today"
loadable_now += n
elif ch in have_v4h:
status = "needs the new chassis ported"
needs_new_mech += n
else:
status = "BROKEN - chassis missing from V4H too"
broken += n
print(f"{ch:22s} {n:5d} {status}")
print()
print(f"loadable in our build today : {loadable_now}")
print(f"blocked on new chassis : {needs_new_mech}")
print(f"broken / orphaned : {broken}")
for ch in sorted(by):
if ch not in have_v4h:
print(f" orphan chassis '{ch}': " + ", ".join(sorted(by[ch])))
if __name__ == "__main__":
main()