Add MW4COMPARE: .mw4 decompiler toolchain and V4H comparison harness
Tooling built to recover editable source for six 'Mech chassis that exist
in the parallel FS_Build_V4H build but not in this repo. Reverse-engineers
every compiled record type in the .mw4 package format back to the .data /
.instance / .subsystems / .damage / .contents / .torso / .engine /
.armature sources the content pipeline consumes.
Nothing here is wired into the game build. It is a standalone analysis
harness run from Linux.
Package format
--------------
"#VBD" container. Directory records are [len][name][FILETIME][origSize]
[storedSize][offset], payload base at dword 0x0C. A record is stored raw
when storedSize == origSize, otherwise LZW (9->12-bit LSB-first codes,
256=clear, 257=EOF, dict from 258), per Database.cpp:451.
GameModel records are flat /Zp4 structs following the C++ inheritance
chain Entity(0) -> Mover(28) -> MWObject(80) -> Vehicle(664) -> Mech(756),
1636 bytes total. CreateMessage records follow Replicator -> Entity ->
Mover -> MWMover -> MWObject -> Vehicle -> Mech from start=16 (the
undeclared Connection__Message header), ending at 341 and padded to 344.
tools/decompile/
----------------
datamap.py header-driven layout engine; CHAIN + ANCHORS
{Vehicle:664, Mech:756} assert the struct offsets
mw4msg.py CreateMessage reader/walker
data.py .data constants.py define/table symbol resolution
damage.py .damage contents.py .contents
smallmodel.py .torso + .engine instance.py .instance
armature.py / armature_parts.py .armature + armaturedata/armaturevideo
assembly.py joint hierarchy renderer
make_generic_doll.py builds generic MFD/Radar damage dolls
verify_*.py per-type round-trip verifiers
Verified round-trip across all 64 shared chassis:
.armature 2938/2976 pages .subsystems 7579/7585 keys
.data map 6071/6071 values .data trip 8291/8306 keys
.damage 6605/6605 keys .contents 7480/7480 keys
.torso+.engine 1280/1280 keys .instance 896/896 keys, 64/64 pages
armature_parts 1202/1202 .data, 1149/1202 .video
Layout-discovery lessons (documented in DECOMPILING.md)
-------------------------------------------------------
- Never let a field map be discovered by the values that verify it. A
value-matching pass reported 4288/4288 while mis-assigning 34 keys. The
map was rebuilt from header declaration order, anchored on uniquely
resolved fields.
- Read the factory, not the data. 12 .data fields and 5 Torso fields are
declared plain Stuff::Scalar but multiplied by Radians_Per_Degree in
Mech_Tool.cpp:889 / Torso_Tool.cpp.
- Strip typedefs before walking a header. A stray `typedef int AttributeID;`
masked a missing ClassID - two 4-byte errors cancelling out, caught only
by the ANCHORS assertion.
- A verifier that silently narrows its own input reports success. Braced
blocks must be hidden before splitting pages, replacing both CR and LF,
because a `Shadow={...}` block contains a line reading `[shadow]` and
splitlines() also splits on bare CR.
- NSWIZZLE is undefined, so the #else branch is live and orders members
differently. bool is 1 byte; char x[MaxStringLength] is 256.
- V4H carries stale Mech IDs (their Atlas is 5, ours 6), so 64 of 65 shared
chassis are off by one; --retarget-ids emits $(M_<Chassis>)/$(IDS_<Chassis>).
reports/ holds generated diffs. The two ~5 MB manifest-*.tsv intermediates
are gitignored; regenerate everything with run-comparison.sh.
Co-authored-by: Claude Opus 5 (Anthropic) <noreply@anthropic.com>
Co-authored-by: GitHub Copilot <copilot@github.com>
This commit is contained in:
co-authored by
Claude Opus 5
GitHub Copilot
Claude Opus 5
GitHub Copilot
parent
e088555b96
commit
83478b7666
@@ -0,0 +1,278 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
data.py - decompiler for a mech `<chassis>.data` file.
|
||||
|
||||
Rebuilds the `[GameData]` page from the compiled records:
|
||||
|
||||
<chassis>.data{GameModel} 1636-byte flat struct (datamap.py)
|
||||
<chassis>.data{FootSteps} foot-step texture list
|
||||
<chassis>.data[shadow] inline Shadow notation block
|
||||
|
||||
Value sources, in order of preference:
|
||||
|
||||
* a struct member - typed read through datamap.chain_layout()
|
||||
* a ResourceID member - record id (HIGH word) resolved via the manifest
|
||||
* a symbolic constant - int reversed through constants.py
|
||||
* an explicit factory key - handled below, because SaveGameModel writes these
|
||||
outside the attribute table
|
||||
|
||||
Four keys cannot be recovered and are deliberately omitted: BattleDamageRatio,
|
||||
BattleKillBonus, DragoonValue and VehicleTradeValue. They appear in every source
|
||||
.data but are read by NOTHING in the engine - no factory, no attribute
|
||||
registration, no runtime reference - so they never enter the package. They are
|
||||
authoring metadata; omitting them changes no behaviour.
|
||||
|
||||
python3 data.py <chassis-record-dir> [-o out.data]
|
||||
python3 data.py --verify
|
||||
"""
|
||||
import argparse, collections, os, glob, re, struct, subprocess, sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import datamap, constants, subsystems
|
||||
|
||||
MANIFEST = "/home/rich/Repositories/FS_Ours_extracted/_manifest.tsv"
|
||||
|
||||
# Adept.hpp:212. NoMaterial is 0.
|
||||
MATERIALS = [
|
||||
"NoMaterial", "Grass", "Water", "Concrete", "GreyDirt", "BrownDirt", "Rock",
|
||||
"DarkConcrete", "DarkGreyDirt", "DarkBrownDirt", "DarkRock", "Blacktop",
|
||||
"Snow", "Wood", "Lava", "Glass", "Steel", "Us", "Them", "LightMineral",
|
||||
"DarkMineral", "Ash", "CrackedLava", "OpenLava",
|
||||
]
|
||||
|
||||
# Identical in all 64 chassis; SaveGameModel writes them outside the attribute table.
|
||||
CONSTANTS = {
|
||||
"Class": "MechWarrior4::Mech",
|
||||
"FaceLighting": "yes",
|
||||
"LookupLighting": "yes",
|
||||
"VertexLighting": "yes",
|
||||
"LightMapLighting": "no",
|
||||
# byte-identical in all 89 mech .data files, destroyed variants included
|
||||
"Shadow": "{\n[shadow]\nLightType=Shadow\nInnerRadius=4.0\nOuterRadius=10.0\n"
|
||||
"BlobDistance=200.0\nShadowMap=ShadowMask\nIntensity=0.4\n}",
|
||||
}
|
||||
|
||||
# Keys whose source name differs from the struct member name.
|
||||
ALIASES = {
|
||||
"AnimationScript": "animScriptName",
|
||||
"HeatManager": "heatManagerResource",
|
||||
"FootEffectsFile": "footFallEffectsTable",
|
||||
}
|
||||
|
||||
# CraterName is stored as MString::GetHashValue (DeathEntity_Tool.cpp:34), which
|
||||
# is one-way. All 64 chassis hold the same hash, so the single authored value is
|
||||
# recoverable by constancy rather than by inversion.
|
||||
CRATER_HASH = {217688981: "crater01"}
|
||||
|
||||
# Read by nothing in the engine - see module docstring.
|
||||
UNRECOVERABLE = ["BattleDamageRatio", "BattleKillBonus", "DragoonValue", "VehicleTradeValue"]
|
||||
|
||||
SYMBOLIC = ("MechID", "TechType", "NameIndex", "MoveTypeFlag")
|
||||
|
||||
|
||||
def stem(mech_dir):
|
||||
"""File stem used inside a mech folder; it need not match the folder name.
|
||||
|
||||
jenner2c/ holds jenner_2c.*, the same trap Black Hawk/nova sprang on
|
||||
.subsystems -- never assume the folder name.
|
||||
"""
|
||||
names = [f for f in os.listdir(mech_dir) if f.lower().endswith(".data")]
|
||||
if names:
|
||||
return names[0][:-len(".data")]
|
||||
for f in os.listdir(mech_dir):
|
||||
m = re.match(r'(.+)\.data[\{\[]', f, re.I)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return os.path.basename(mech_dir.rstrip("/"))
|
||||
|
||||
|
||||
def record(mech_dir, suffix):
|
||||
"""Read one qualified record. Not glob -- '[shadow]' is a character class."""
|
||||
want = (stem(mech_dir) + suffix).lower()
|
||||
for fn in os.listdir(mech_dir):
|
||||
if fn.lower() == want:
|
||||
return open(os.path.join(mech_dir, fn), "rb").read()
|
||||
return None
|
||||
|
||||
|
||||
def bool_words():
|
||||
"""-> {key: (trueWord, falseWord)}; sources spell these inconsistently.
|
||||
|
||||
Collider and CanBeShot are written true/false, the CanLoad* flags Yes/No.
|
||||
"""
|
||||
seen = collections.defaultdict(collections.Counter)
|
||||
for _ch, kv, _b in datamap.corpus():
|
||||
for k, v in kv.items():
|
||||
w = v.strip().lower()
|
||||
if w in ("true", "false", "yes", "no"):
|
||||
seen[k][w] += 1
|
||||
out = {}
|
||||
for k, c in seen.items():
|
||||
plain = c["true"] + c["false"] >= c["yes"] + c["no"]
|
||||
out[k] = ("true", "false") if plain else ("Yes", "No")
|
||||
return out
|
||||
|
||||
|
||||
def foot_steps(blob):
|
||||
"""-> (defaultTexture, [(texture, materialName)]).
|
||||
|
||||
Stream written by Mech_Tool.cpp:196: int material (-1 = default), int length
|
||||
NOT counting the terminator, the characters, a NUL, then a one-byte
|
||||
isDefault flag.
|
||||
"""
|
||||
default, rows, off = None, [], 0
|
||||
while off + 8 <= len(blob):
|
||||
material, length = struct.unpack_from("<iI", blob, off)
|
||||
off += 8
|
||||
text = blob[off:off + length].decode("latin-1")
|
||||
off += length + 1 # skip the terminator
|
||||
is_default = blob[off] if off < len(blob) else 0
|
||||
off += 1
|
||||
if is_default or material < 0:
|
||||
default = text
|
||||
else:
|
||||
name = MATERIALS[material] if 0 <= material < len(MATERIALS) else str(material)
|
||||
rows.append((text, name))
|
||||
return default, rows
|
||||
|
||||
|
||||
def decompile(mech_dir, manifest=None, retarget_ids=False, obb_dir=None):
|
||||
"""-> OrderedDict key -> value or [values]."""
|
||||
manifest = manifest if manifest is not None else subsystems.load_manifest(MANIFEST)
|
||||
ch = os.path.basename(mech_dir.rstrip("/"))
|
||||
gm = record(mech_dir, ".data{GameModel}")
|
||||
if gm is None:
|
||||
raise SystemExit(f"no GameModel record in {mech_dir}")
|
||||
|
||||
layout = datamap.chain_layout()[0]
|
||||
angles = datamap.angle_fields()
|
||||
tables = constants.tables()
|
||||
words = bool_words()
|
||||
by_member = {datamap.norm(n): n for n in layout}
|
||||
out = collections.OrderedDict()
|
||||
|
||||
def member(key):
|
||||
n = ALIASES.get(key) or by_member.get(datamap.norm(key))
|
||||
return (n, *layout[n]) if n in layout else None
|
||||
|
||||
# every key the corpus knows about, so output matches the authored shape.
|
||||
# Keys only a handful of mechs author (VehicleBattleValue: 1 of 64) are left
|
||||
# out rather than emitted at their default.
|
||||
corpus = datamap.corpus()
|
||||
common = collections.Counter(k for _c, kv, _b in corpus for k in kv)
|
||||
for key in sorted(common):
|
||||
if key in UNRECOVERABLE or common[key] * 2 < len(corpus):
|
||||
continue
|
||||
if key in CONSTANTS:
|
||||
out[key] = CONSTANTS[key]
|
||||
continue
|
||||
if key == "CraterName":
|
||||
name = CRATER_HASH.get(datamap.read(gm, *layout["m_craterID"]))
|
||||
if name:
|
||||
out[key] = name
|
||||
continue
|
||||
m = member(key)
|
||||
if key in SYMBOLIC and m:
|
||||
_n, off, typ, size = m
|
||||
raw = datamap.read(gm, off, typ, size)
|
||||
sym = constants.symbol(tables, key, raw, hint=ch)
|
||||
# MechID/NameIndex are authored as $(M_Chassis)/$(IDS_Chassis). A
|
||||
# foreign package may store an int from a different roster -- V4H's
|
||||
# are shifted by one against ours -- so --retarget-ids emits the
|
||||
# chassis's own symbol and lets the build resolve it.
|
||||
if retarget_ids and key in ("MechID", "NameIndex"):
|
||||
prefix = "M_" if key == "MechID" else "IDS_"
|
||||
if not sym or constants.norm(ch) not in constants.norm(sym):
|
||||
out[key] = f"$({prefix}{ch[:1].upper() + ch[1:]})"
|
||||
continue
|
||||
if sym:
|
||||
out[key] = sym
|
||||
continue
|
||||
if m:
|
||||
name, off, typ, size = m
|
||||
val = datamap.read(gm, off, typ, size, datamap.norm(name) in angles
|
||||
or typ == "Radian")
|
||||
if typ == "ResourceID":
|
||||
path = manifest.get(val >> 16)
|
||||
if path:
|
||||
out[key] = path
|
||||
elif typ in datamap.VECTORS:
|
||||
out[key] = " ".join(f"{v:g}" for v in val)
|
||||
elif typ in ("bool", "BYTE"):
|
||||
yes, no = words.get(key, ("true", "false"))
|
||||
out[key] = yes if val else no
|
||||
elif typ in datamap.FLOATS:
|
||||
out[key] = f"{val:g}"
|
||||
elif typ == "char":
|
||||
if val:
|
||||
out[key] = val
|
||||
else:
|
||||
out[key] = str(val)
|
||||
|
||||
# The OBB filenames are source-side names the package never stores. Prefer the
|
||||
# .obb files actually shipped next to the output so the keys cannot disagree
|
||||
# with them; fall back to the usual convention when none are present.
|
||||
solid = hier = None
|
||||
if obb_dir and os.path.isdir(obb_dir):
|
||||
for fn in sorted(os.listdir(obb_dir)):
|
||||
if not fn.lower().endswith(".obb"):
|
||||
continue
|
||||
if fn.lower().endswith("_solid.obb"):
|
||||
solid = fn
|
||||
else:
|
||||
hier = fn
|
||||
out["SolidOBB"] = solid or f"{ch}_Skeleton_SOLID.obb"
|
||||
out["HierarchicalOBB"] = hier or f"{ch}_Skeleton.obb"
|
||||
|
||||
fs = record(mech_dir, ".data{FootSteps}")
|
||||
if fs:
|
||||
default, rows = foot_steps(fs)
|
||||
if default:
|
||||
out["DefaultFootStepTexture"] = default
|
||||
if rows:
|
||||
out["FootStepTexture"] = [f"{t},{m}" for t, m in rows]
|
||||
|
||||
sh = record(mech_dir, ".data[shadow]")
|
||||
if sh is None:
|
||||
out.pop("Shadow", None)
|
||||
return out
|
||||
|
||||
|
||||
def emit(kv):
|
||||
lines = ["[GameData]"]
|
||||
for key, val in kv.items():
|
||||
for v in (val if isinstance(val, list) else [val]):
|
||||
lines.append(f"{key}={v}")
|
||||
return "\r\n".join(lines) + "\r\n"
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("mech_dir", nargs="?")
|
||||
ap.add_argument("-o", "--output")
|
||||
ap.add_argument("-m", "--manifest", default=MANIFEST,
|
||||
help="package manifest for resolving ResourceIDs")
|
||||
ap.add_argument("--retarget-ids", action="store_true",
|
||||
help="emit the chassis's own MechID/NameIndex symbol when the "
|
||||
"stored int belongs to a different roster")
|
||||
ap.add_argument("--verify", action="store_true")
|
||||
args = ap.parse_args()
|
||||
if args.verify:
|
||||
here = os.path.dirname(os.path.abspath(__file__))
|
||||
raise SystemExit(subprocess.call([sys.executable,
|
||||
os.path.join(here, "verify_roundtrip.py")]))
|
||||
if not args.mech_dir:
|
||||
ap.error("mech_dir required")
|
||||
text = emit(decompile(args.mech_dir, subsystems.load_manifest(args.manifest),
|
||||
retarget_ids=args.retarget_ids,
|
||||
obb_dir=os.path.dirname(args.output) if args.output else None))
|
||||
if args.output:
|
||||
with open(args.output, "wb") as fh:
|
||||
fh.write(text.encode("latin-1"))
|
||||
print(f"wrote {args.output}")
|
||||
else:
|
||||
sys.stdout.write(text)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user