# MW4COMPARE -- FireStorm source tree vs. the `FS_Build_V4H` binary build Comparison of our source repository against **FireStorm Battletech SMT Build Four (v4h)**, a parallel, independently-maintained *binary-only* build of the same game, located at `/home/rich/Repositories/FS_Build_V4H` (1.7 GB, dated Oct/Nov 2021). Goal: work out what each side has that the other does not, so the two can eventually be merged into one build containing all the mechs, maps, game types and variants. **Status: Part 1 (survey) complete. Part 2 (extraction + pruning) complete.** Nothing has been merged into the repo. Every V4H asset is unpacked to a parallel tree outside the repo at `/home/rich/Repositories/FS_Build_V4H_extracted`, reduced to **differences only** -- 5,385 files / 65 MB out of an original 46,707 files / 1.24 GB -- and reshaped to mirror `Gameleap/mw4`. See section 5. --- ## 0. TL;DR | | V4H has, we don't | We have, V4H doesn't | |---|---|---| | **Mechs** | **6 new chassis** (IDs 65-70): Champion, Jenner IIC, Dasher, Marauder, Thunderbolt, Griffin -- full geometry, textures, HUD/MFD/radar art, footsteps | Source for ~25 unregistered/experimental chassis (Blackheart, Bowman, Canis, Crab, Gargoyle, koto, locust2c, marauder2, pitbull, privateer, razorback, reaper, reaver, shadowhawk, strider, urbanmech_iic, ursus, vulture2, vulturec, ...) plus 4 Elite\_\* pilot variants | | **Variants** | **797 saved mechlab variants** (`resource/Variants/*.mw4`); we ship 1 | -- | | **Maps** | none | **19 maps** (conroe01-03, desert, doneg01, gage, minerl01, mountn01/03, ngoth, ruin03, scrub01, scrub06, volcan01/03, ddc_msl, freezer, ...) | | **Missions** | none | **17 missions** + 5 UserMissions (aspen, canyon, conroe01-03, gagetown, gladiatorpit, lakeside, mechworks, minehq, newgothem, rubble, sanddunes, spaceport, vbase, editortemplate) | | **Skies** | *missing entirely* -- no `skies.mw4` | `skies.mw4` (40 MB, 2975 entries) | | **Shell scripts** | reworked MechLab + 11 loose script/header files we don't have at all | our own console/lobby work (RookieMission, `-tmfds`, etc.) | | **Engine** | binary only -- cannot be recovered | full source | The single most important structural finding: > **Mech IDs 0-64 are byte-for-byte identical in both forks.** V4H *appended* its six new > mechs as IDs 65-70. There is no re-indexing to do -- a merge is additive on both sides. --- ## 1. What `FS_Build_V4H` is A deployed game folder, not a source tree. Notable contents: ``` FS_Build_V4H/ MW4.exe 3,633,152 B 2021-09-08 (their fork; no source available) ScriptStrings.dll 163,840 B 2021-02-03 (has the 6 new mech display names) MissionLang.dll 188,416 B 2021-09-01 content/ 14 loose files (shell scripts + 1 ABL header) hsh/ 279 files -- the "block HUD" set introduced in v4f hshoriginal/ 447 files -- the pre-v4f HUD set, archived resource/ core.mw4 12,709 entries props.mw4 10,779 entries textures.mw4 10,419 entries maps/ 27 maps Missions/ 29 missions Variants/ 797 saved mechlab loadouts Pilots/Tesla/options.mw4 (no skies.mw4) stats/zWorldStats/ stock 2001-era MS stats uploader, logs end 2001 -- dead weight uploadbuildv4h/ 777 MB self-extracting installer of this same build firestorm build V4h.txt changelog for v4 -> v4h (also in uploadbuildv4h/) ``` Their own changelog (`firestorm build V4h.txt`) is worth reading; it documents the six new mechs, the 3-ton-per-weapon ammo cap, the MechLab save crash, and the fact that corrupt variants can corrupt the whole build. --- ## 2. Method The `.mw4` files are GameOS `Stuff::Database` packages (magic `#VBD`) -- a flat directory of named records, each optionally LZW-compressed. See `tools/mw4db.py` for the format and a faithful port of `gos_LZDecompress`. Two comparison levels were used: 1. **Index-only** (`mw4index.py` + `diffindex.py`) -- parses the directory, no decompression. ~1 s for the whole 780 MB tree. Gives an exact **entry-name** diff. 2. **Decoded** (`pkgcmp.py`) -- decompresses every record and compares plaintext bytes. Slower (~2 MB/s) but is the only trustworthy content test. ### Two traps that cost time -- do not repeat them * **Case.** V4H's packer lowercased every entry name (`ablscripts\bots\0-bot.abl` vs our `ablscripts\bots\0-Bot.abl`). A case-sensitive comparison reported 5,909 "new" and 5,155 "missing" entries in `core.mw4` that were the *same files*. Always compare through `mw4db.norm()`. * **Stored-byte md5 is not an equality test.** Comparing the compressed payloads over-reports massively: `maps/peaks.mw4` showed 9 differing blobs by stored bytes but only **1** after decompression. Use `pkgcmp.py` before believing any difference. --- ## 3. Findings ### 3.1 The six new mechs (the headline) `core.mw4` in V4H is a **strict superset** of ours: 754 extra entries, **zero** entries we have that they lack. All 754 belong to six new chassis and their destroyed models: | chassis | core entries | textures | props (animation) | |---|---|---|---| | marauder | 63 armaturedata + 49 root + 21 armaturevideo + 5 destroyed | 37 + 1 | reuses existing | | griffin | 57 + 54 + 19 + 5 | 38 + 1 | reuses existing | | champion | 57 + 51 + 19 + 5 | 36 + 1 | reuses existing | | thunderbolt | 54 + 48 + 18 + 5 | 33 + 1 | reuses existing | | dasher | 45 + 48 + 15 + 5 | 29 + 1 | reuses existing | | jenner2c | 45 + 46 + 15 + 5 | 30 + 1 | **155 new entries** under `mechs/jenner2c/animation` | Each chassis brings the full set: `.contents` (+ per-joint `{armature}` / `{sites}` pages), `.damage`, `.data` (+ `{element}` `{footsteps}` `{gamemodel}` `{hierarchicalobb}` `{solidobb}`), `.engine`, `.instance`, `.subsystems`, `.torso`. `textures.mw4` adds 299 entries: the six mechs' skins, six sets of `textures/@a{chp,grf,jec,mar,thu,...}0-5.tga` cockpit/pilot art, and `textures/footsteps/_{default,dirt,snow}.tga`. **Nothing else is new in `core.mw4`.** No new weapons, no new subsystem types, no new tables. Whatever balance changes V4H made live in the per-mech `.subsystems` records and in their `MW4.exe`. ### 3.2 Mech ID space -- identical for 0-64 `content/ABLscripts/mwconst.abi` differs from ours by exactly this: ```diff -LastMechID = 64 ; -NoMechID = 65 ; +M_Champion = 65 ; +M_Jenner2c = 66 ; +M_Dasher = 67 ; +M_Marauder = 68 ; +M_Thunderbolt = 69 ; +M_Griffin = 70 ; +LastMechID = 70 ; +NoMechID = 71 ; +LastMP1MechID = M_Solitaire; ``` Corroborated three ways: * `content/shellscripts/mechbay/mechnames.script` and `buildmechnames.script` list `mech_name[0..64]` exactly matching our `mechnames[]` in `CoreTech/Libraries/GameOS/render.cpp`, then append 65-70. * `core.mw4` defines 65 `*.subsystems` on our side, 71 on theirs; the extra six are `champion, dasher, griffin, jenner_2c, marauder, thunderbolt`. * `ScriptStrings.dll` (UTF-16) gains display strings `Champion`, `Jenner2c`, `Dasher`, `Marauder`, `Thunderbolt`, `Griffin`. Note the naming inconsistency in their own build: the chassis directory is `jenner2c` but the subsystems record is `jenner_2c`, while three of their variants still reference `jenner_iic1` (see section 3.3). ### 3.3 797 mech variants `resource/Variants/*.mw4` -- saved MechLab loadouts. We ship exactly one (`MW4/Resource/Variants/Argus Variant1.mw4`). Each variant package is 4 records: ``` [0] 'Resource\Variants\' 4 B raw [1] '{Mech}' ~350 B [2] '{Subsystem}' ~4 KB <- names the chassis it needs [3] '<8-char token>' 16 B ``` Because the chassis is named in record 2, dependency analysis needs no decompression (`tools/variants-report.py`). Result: ``` loadable in our build TODAY : 727 blocked on the 6 new chassis: 67 (thunderbolt 22, griffin 15, marauder 15, dasher 10, champion 5) broken / orphaned : 3 (chassis 'jenner_iic1' does not exist even in V4H -- Jenner2c 1 / Jenner2c 4 / Jenner2c Pharaoh) ``` Top chassis by variant count: warhammer 36, rifleman 30, nova 24, madcat 23, atlas 23, thunderbolt 22, hunchback 21, battlemaster 21, archer 21, thor 20. Full table: `reports/variants-inventory.txt`. ### 3.4 Maps and missions -- we are ahead, they changed a little * **Maps**: V4H has 27, we have 46. **Zero** V4H-only maps. Common maps are effectively identical -- file sizes differ by 0-2 bytes and `peaks.mw4` decodes to exactly **one** differing record (`peaks.data{gamemodel}`, an internal reference blob). * **Missions**: V4H has 29, we have 46 + 5 UserMissions. **Zero** V4H-only missions. Every common mission decodes to 2-4 differing records, always the same shape: * `.lights` and `night.lights` -- **3 bytes** differ out of 33 KB. Noise. * `.contents` -- **real**. Coliseum: 11,588 of 52,954 bytes differ, and the differing region contains new ASCII entity names. Their mission layouts have been edited. Worth a proper look before merging either direction. * `Missions/freezer.mw4` additionally has **30 V4H-only records**: stadium crowd VO (`audio/vo/generic/mp/{cheer1-9,boo1-3,jeers,roar,ahh,aww,stadamb}.wav`) and a rewritten `scripts/freezer_teamattrition.abl` (10,710 B vs our 832 B). * All 71 loose `.nfo` / `.tga` / `.txt` files in `resource/` (mission metadata, game-type definitions, thumbnails) are **byte-identical**. No game-type divergence. * **`skies.mw4` is absent from V4H entirely.** 40 MB / 2,975 entries that only we have. ### 3.5 Shell scripts -- V4H has a diverged MechLab and 11 files we don't have at all `props.mw4` decodes to 419 differing records, only 3 entries missing on their side (`lobbydecals/decal_46/47/49.tga`) and 155 extra (jenner2c animation). The interesting differences (V4H size vs ours): | script | V4H | ours | delta | |---|---|---|---| | `mechbay/mechbay_main.script` | 94,517 | 61,432 | **+33 KB** | | `mechbay/weapons.script` | 102,025 | 83,653 | **+18 KB** | | `mechbay/armor.script` | 64,539 | 61,807 | +2.7 KB | | `mechbay/infobox.script` | 43,875 | 41,595 | +2.3 KB | | `listboxes.script` | 30,785 | 29,269 | +1.5 KB | | `scriptstrings.h` | 71,608 | 70,811 | +797 B | | `netlobby.script` | 163,740 | 163,088 | +652 B | | `multiplayer/buildrestriction.script` | 16,443 | 15,713 | +730 B | | `mechbay/chassis.script` | 82,596 | 81,881 | +715 B | | `conlobby.script` | 141,044 | 140,532 | +512 B | | ...plus ~15 more with small deltas | | | | Also: 24 `mechbay/graphics/installed/weapon_-1_*.tga` slot images are **much smaller** in V4H (1,036 B vs 6,540 B etc.) -- this is the "weapon box sizes set to minimum" change from their v4 changelog. **Eleven loose files in `content/shellscripts/` exist in V4H and nowhere in our repo:** ``` clock.script wall-clock overlay cm_listboxes.script console mech list box firestorm.h 20 console callsigns firestorm64.h 64 Dragon-MP callsigns globals.script shared script globals options.h USE_ALLOWED_MECHS, Battle_Ammo, USE_OBSERVER, ... mechbay/buildmechnames.script 71-entry mech name table mechbay/mechnames.script 71-entry mech name table mechbay/mechdescriptions.script 74 KB of mech description text mechbay/old mechdescriptions.script graphics/multiplayer/lobbydecals/decal_.tga ``` These are loaded loose from disk (they are in neither `core.mw4` nor `props.mw4` on either side). Their `computerplayer.script` `#include`s `firestorm.h` where ours `#include`s `callsign.h` -- so this is a genuinely diverged shell-script codebase, not just extra data. ### 3.6 HUD art (`hsh/`) V4H maintains **two** HUD sets: * `hshoriginal/` (447 files) -- the detailed paper-doll set, close to ours. * `hsh/` (279 files) -- the "block HUD" compromise introduced in v4f so the new mechs would have *something* to display. Their changelog admits HUDs do not always match the mech actually in use. Against our `hsh/` (456 files): | | identical | differ | V4H-only | ours-only | |---|---|---|---|---| | `hsh/` | 182 | 68 | 29 | 206 | | `hshoriginal/` | 275 | 148 | 24 | 33 | V4H-only files are the six new mechs' `hud/`, `MFD/`, `radar/hud/` and `Mechs/` art, `decals/DDC.bmp`, plus `Mechs/battlemasteriic.bmp` and `Mechs/mad cat mk.ii.bmp` (pre-rename spellings of files we renamed). Colour depth differs systematically: their `Mechs/*.bmp` are 24-bpp 500x400 (600,054 B), ours are 8-bpp palettised (201,080 B). Same for `MFD/rifleman.bmp` (43,256 vs 15,476). `hsh/hud/Convert.txt` and `hsh/radar/hud/Convert.txt` are **the authoring recipe for the paper-doll art** and explicitly say the coordinates go into `coord.cpp` under GameOS/External dependencies. Independent corroboration of `MFD-RADAR-MAPPINGS.md`. They are reproduced verbatim in `reference/Convert-hud.txt` / `reference/Convert-radar.txt`. ### 3.7 Per-mech `.subsystems` differ for every single chassis Every one of the 65 shared chassis has a differing `.subsystems` record -- same length, different content (Atlas: 1,830 of 6,082 bytes). This is the compiled form, not the source text, so it is not directly readable. It is consistent with V4H's documented changes: 3-ton ammo cap per weapon, revised default weapon groups, restricted hardpoints on the new mechs. **This is the one area where merging is genuinely risky** -- see the open questions. Also differing in `core.mw4` for essentially every mech: `.data{gamemodel}`, `.torso{gamemodel}`, and the `contents[joint_torsoabove]{armature}` / `contents[joint_*belowankle]{armature}` records. Byte inspection of `atlas.data{gamemodel}` shows one side stores an inline path string where the other stores a numeric reference -- i.e. a packer/format difference, most likely benign. Confirm before treating any of these as content changes. ### 3.8 Root config and binaries Identical: 37 root files, including all three `ctcl-*.ini`. Differing binaries (theirs older/forked, ours newer): `MW4.exe`, `autoconfig.exe`, `Launcher.exe`, `mw4print.exe`, `ctcls.dll`, `Language.dll`, `MissionLang.dll`, `ScriptStrings.dll`. V4H-only root files: `Console launcher.bat` (`mw4.exe -window`), `optionssmt.ini`, `optionssmtv4.ini`, `options.ini.rp411old`, `pre-uninstall.bat`, `RMTSHARE.EXE`, `SetNetworkAccess.reg`, `firestorm build V4h.txt`. `options.ini` differences worth flagging: | key | V4H | ours | |---|---|---| | `bitdepth` | **32** | 16 | | `ruletype` | 17 | 2 | | `unlimitedammo` | 0 | 1 | | `friendlyfirepercentage` | 100 | 0 | | `heaton` | 1 | 0 | | `allowedbeam1` | `fffffffd` | `ffffffff` | | `allowedmissile1` | `ff3bffff` | `ffffffff` | | `allowedprojectile1` | `ffffdfff` | `ffffffff` | | `playericon` / `teamicon` | decal\_49 / decal\_50 | decal\_48 / decal\_48 | | `[RookieMission]` section | absent | present (ours) | | `RuleBook`/`DawnWar`/`BiggieSizeIt`/`CanYouHearTheFootSteps` | absent | present (ours) | `bitdepth=32` is interesting given all the `DWM8And16BitMitigation` work recorded in the repo memory -- V4H apparently runs the console at 32 bpp. `optionssmtv4.ini` additionally carries a full `[cameraship params]` block (over-shoulder offsets, death-sequence camera, HUD name/chat toggles) that ours does not. ### 3.9 What *we* have that V4H does not Besides the maps/missions/skies above, `Gameleap/mw4/Content/Mechs` contains 93 non-destroyed chassis directories, of which **27 are not in V4H's 71-mech roster**: ``` Blackheart Bowman Canis Crab Duangung Gargoyle Gesu Grimreaper_DNU koto locust2c marauder2 pitbull privateer razorback reaper reaver shadowhawk strider urbanmech_iic ursus vulture2 vulturec Elite_Daishi_Castro Elite_Daishi_William Elite_Madcat_James Elite_Thor_Duncan jenner2c (chassis 'jenner2c' -- V4H spells theirs 'jenner_2c') ``` Two of these matter immediately: * **`Dasher_DNU/`** -- chassis `dasher`, 71 source files including `.subsystems`, `.damage`, `.animscript`, geometry. We already have Dasher source; it is simply not registered in the roster or packed into `core.mw4`. * **`jenner2c/`** -- 80 source files, plus `jenner2c_destroyed/`. Likewise unregistered. So of V4H's six new mechs we may already hold source for two. `marauder2` is *Marauder II*, a different chassis from their `marauder`. --- ## 4. Tools All under `tools/`. Python 3, no dependencies. | file | what it does | |---|---| | `mw4db.py` | `#VBD` package reader + `gos_LZDecompress` port. Import this; everything else builds on it. Documents the on-disk format. | | `mw4index.py` | `mw4index.py > manifest.tsv` -- directory-only manifest of every `*.mw4`. ~1 s for 780 MB. | | `diffindex.py` | `diffindex.py A.tsv B.tsv [labelA] [labelB]` -- case-insensitive package/entry-name diff. | | `pkgcmp.py` | `pkgcmp.py A.mw4 B.mw4` -- **authoritative** decoded content diff of one package. | | `dumprec.py` | `dumprec.py pkg.mw4 "entry/name" [out]` -- extract one decoded record; `--list` lists entries. | | `treediff.py` | `treediff.py dirA dirB [labelA] [labelB]` -- case-insensitive loose-file tree diff by md5. | | `variants-report.py` | chassis-dependency inventory of `resource/Variants`, flags variants that cannot load. | | `extract-all.py` | `extract-all.py ` -- unpack every package into a real directory tree. See section 5. | | `prune-identical.py` | `prune-identical.py [--apply]` -- delete everything we already have byte-for-byte. Dry-run by default. | | `classify-survivors.py` | `classify-survivors.py ` -- split what is left into NEW vs DIFFERS. | | `restructure.py` | `restructure.py [--apply]` -- reshape a pruned tree to the repo's layout and casing. Refuses to run if any two packages would collide. | | `split-source.py` | `split-source.py [--apply]` -- separate repackable source from compiled records and rename the `.obb`s. | `run-comparison.sh` regenerates every report: ```bash cd MW4COMPARE ./run-comparison.sh # uses the default paths ./run-comparison.sh /path/to/OTHER_BUILD /path/to/our/mw4 ``` Runtime is a few minutes; `textures.mw4` dominates. There is an older, standalone version of the same decompressor at `build-env/extract-mw4.py`. `tools/mw4db.py` supersedes it and is the one to extend. ## 5. The extracted asset tree Every record of every V4H package, unpacked to real files at **`/home/rich/Repositories/FS_Build_V4H_extracted`** -- deliberately *outside* the repo. Nothing has been merged; this is derived data and is regenerable: ```bash python3 MW4COMPARE/tools/extract-all.py \ /home/rich/Repositories/FS_Build_V4H/resource \ /home/rich/Repositories/FS_Build_V4H_extracted ``` 857 packages -> **46,707 files, 1.24 GB**, about 3 minutes. | directory | files | size | from | |---|---:|---:|---| | `core/` | 12,709 | 3.4 MB | `core.mw4` | | `props/` | 10,779 | 178 MB | `props.mw4` | | `textures/` | 10,419 | 358 MB | `textures.mw4` | | `maps//` | 7,219 | 576 MB | 27 map packages | | `Missions//` | 2,391 | 60 MB | 29 mission packages | | `Variants//` | 3,188 | 4.8 MB | 797 saved loadouts | | `Pilots/Tesla/options/` | 2 | 5 KB | | | `_merged/` | 42,100 | *hardlinks* | flattened Content-shaped view | Extraction is **per package**, because 842 entry paths are claimed by more than one package and 70 of those hold genuinely different content (mostly a mission packing its own copy of a global `props.mw4` asset -- 29 are Coliseum/Freezer crowd VO). A single flat tree would silently lose data. `_merged/` is a convenience view built with **hardlinks** (no extra disk) that mirrors `Gameleap/mw4/Content`, so it diffs directly against our source tree. Conflicts resolve `props > core > textures > maps > missions` and every one is listed in `_conflicts.tsv`. Variants and Pilots are excluded from it -- their records have no directory part (`{Mech}`, `{Subsystem}`) so all 797 variants would collide. Treat `_merged/` as read-only; hardlinks mean editing there edits the per-package copy. Full details, including the duplicate-entry handling, are in the tree's own `_README.txt`. ### How close packed records are to real source Verified against `mechs/jenner2c`: * **Recovered as-is** -- `.contents`, `.damage`, `.data`, `.engine`, `.instance`, `.subsystems`, `.torso`, `.animscript`, every `*.erf` geometry part, `*_cage.erf`, `runninglights.erf`, `armaturedata/`, `armaturevideo/`. Decoded TGAs are valid (`Targa image data - RGB 512 x 512 x 32`); `core/ablscripts/mwconst.abi` extracts byte-identical to V4H's loose copy, which independently validates the decompressor. * **Recoverable by renaming a qualified record** -- `X.data{solidobb}` -> `X_skeleton_SOLID.obb` and `X.data{hierarchicalobb}` -> `X_skeleton.obb`. Both carry the `#BBO` magic and byte-identical headers to our source `.obb` files. * **Not directly present** -- `X.armature`. The packer splits it into per-joint `X.contents[joint_*]{armature}` records. Also worth knowing before any merge: **V4H's Jenner IIC is not a rename of ours.** Their part prefix is `jec_` and chassis files are `jenner_2c.*`; ours are `j2c_` and `jenner2c.*`. Zero files in common. They are separately authored assets. ### Pruning to differences only Our own packages were extracted the same way to **`/home/rich/Repositories/FS_Ours_extracted`** (102 packages, 51,382 files, 1.76 GB), then everything V4H has that we already hold byte-for-byte was deleted: ```bash python3 MW4COMPARE/tools/prune-identical.py /home/rich/Repositories/FS_Build_V4H_extracted --apply python3 MW4COMPARE/tools/classify-survivors.py /home/rich/Repositories/FS_Build_V4H_extracted ``` **46,707 files / 1.24 GB -> 5,385 files / 66 MB.** 41,322 deleted (88.5%), 1,912 empty directories removed. `_pruned.tsv` logs every deletion and which rule matched. | package group | NEW | DIFFERS | | |---|---:|---:|---| | `Variants` | 3,188 | 0 | 797 loadouts x 4 records; we ship 1 | | `core` | 746 | 440 | 6 new mechs + per-chassis changes | | `props` | 155 | 419 | jenner2c animation + reworked MechLab | | `textures` | 297 | 2 | new mechs' skins / footsteps | | `Missions` | 1 | 114 | `.contents`, `.lights`, freezer crowd VO | | `maps` | 0 | 21 | | | `Pilots` | 0 | 2 | | | **total** | **4,387** | **998** | | #### Why our source tree alone is not a sufficient baseline A `.mw4` is not a zip of `Content/`. Measured: * **17,683 records (42%)** are qualified derivatives the packer *generates* and that have no source file at all -- `foo.data{gamemodel}`, `foo.contents[joint_hip]{armature}`, `bar.tga{hint}`. * **~7,400 more** have a source path but the packer **rewrites the bytes**. Proven by comparing *our own* `Content/` against *our own* `.mw4` output: 3,238 `.data`, 2,320 `.video`, 583 `.instance`, 361 `.contents`, 345 `.audio`, 235 `.damage`, 117 `.torso`, 103 `.subsystems`, 65 `.engine`, 58 `.lights`. Matching against `Content/` alone would have "kept" roughly 24,000 files we already have. So each candidate is tested in order: (1) same package + entry path in our extracted packages, (2) same entry path in any of our packages, (3) unqualified name at the same path in `Content/`. In practice 41,320 matched rule 1 and 2 matched rule 3. > **Case warning.** V4H's packer lowercased every entry name; ours did not. The same record > is `core/mechs/longbow/longbow.data{element}` in V4H and > `core/Mechs/Longbow/longbow.data{Element}` in ours. Every comparison here is > case-insensitive -- a plain `diff -r` between the two trees is meaningless. #### Reading the 998 DIFFERS By extension: 352 `data{gamemodel}`, 96 `torso{gamemodel}`, 93 `subsystems`, 114 `contents[joint_*]{armature}`, 55 `lights`, 54 `audio`, 38 `instance`, 37 `contents`, 29 `script`, 28 `tga`, 15 `wav`. * **Probably packer noise** -- the `{gamemodel}` records encode cross-package references (one side stores an inline path string where the other stores a numeric id), and the `.lights` differences measured 3 bytes out of 33 KB. Verify before treating as content. * **Real, needs a design decision** -- the 93 `.subsystems` (V4H's balance pass) and the 37 mission `.contents` (substantial layout edits). * **Real, straightforward** -- the 29 `.script` files (their MechLab rework) and the freezer crowd VO. ### Reshaped to the repo layout Once pruned, the V4H tree has **zero cross-package collisions** -- 5,385 files map to 5,385 distinct destinations -- so it was flattened to the repo's shape: ```bash python3 MW4COMPARE/tools/restructure.py /home/rich/Repositories/FS_Build_V4H_extracted --apply ``` ``` FS_Build_V4H_extracted/ Content/ same shape and casing as Gameleap/mw4/Content Resource/Variants// 797 saved loadouts (records have no path) Resource/Pilots/Tesla/options/ ``` Path components are re-cased against the repo wherever a counterpart exists, so `mechs/longbow/longbow.data{element}` became `Content/Mechs/Longbow/longbow.data{Element}`. Components with no counterpart keep the packer's spelling -- which is why the six new chassis sit at `Content/Mechs/champion`, `dasher`, `griffin`, `jenner2c`, `marauder`, `thunderbolt` in lower case. `_layout.tsv` records the originating package and entry name for every file, so the per-package view can be reconstructed at any time. **`FS_Ours_extracted` is deliberately left in per-package form.** Unpruned it still has 60 genuine collisions (mission-local copies of shared `Culturals/` props, e.g. `arena_debris_large01.data` differing across Coliseum / Reduex / StormCanyon) plus 2,617 harmless duplicate paths. Flattening it would lose data, and it is regenerable in ~4 minutes anyway. `restructure.py` refuses to run on a tree with collisions for exactly this reason. ### Source vs compiled -- only 502 of the 2,195 files are repackable **A `.mw4` does not store the source tree.** Some records are the source file byte for byte; others are what the packer produced from it. Which is which was established empirically by hash-matching every record of *our own* packages against `Gameleap/mw4/Content` (22,138 matched a source file, 28,728 did not): | | | |---|---| | **repackable source** | `.tga` `.erf` `.mw4anim` `.bid` `.wav` `.bsp` `.material` `.abl` `.obb` `.ebf` `.bounds` `.animscript` `.fgd` `.tcf` `.mlr` `.d3f` `.gaf` `.script` `.h` `.abi` | | **compiled** | `.data` `.instance` `.subsystems` `.damage` `.torso` `.engine` `.audio` `.video` `.lights` `.mw4`, and every `{hint}` `{handle}` `{gamemodel}` `{element}` `{footsteps}` `{zones}` `{nametable}` `[shadow]` `[joint_*]{armature}` `[joint_*]{sites}` record | The decisive evidence: our source `Content/Mechs/Atlas/atlas.data` is **4,891 bytes of text**; the record packed under that name is a **12-byte binary stub**. The real payload went into the `{gamemodel}` / `{element}` records. So a mech's `.data`, `.damage`, `.subsystems`, `.instance`, `.contents`, `.torso` and `.engine` **cannot be recovered from any package** -- they must be re-authored. `.armature` does not exist in a package either; the packer splits it into per-joint `X.contents[joint_*]{armature}` records. ```bash python3 MW4COMPARE/tools/split-source.py /home/rich/Repositories/FS_Build_V4H_extracted --apply ``` ``` FS_Build_V4H_extracted/ Content/ 502 files repackable source, repo shape + casing - drop straight in _compiled/ 1693 files packer output, reference only Resource/ variants + pilot options (use the original .mw4 files) ``` Renames applied -- these genuinely are `.obb` files (`#BBO` magic, headers byte-identical to ours): | record | becomes | |---|---| | `champion.data{solidobb}` | `champion_skeleton_SOLID.obb` | | `champion.data{hierarchicalobb}` | `champion_skeleton.obb` | (`_SOLID.obb` / `.obb` outside `Content/Mechs`.) The real source spelling is declared inside the `.data` text we don't have, so a convention was applied and logged in `_source-vs-compiled.tsv` -- whatever `.data` gets authored later must reference these names. What survives in `Content/`: 208 `.erf`, 154 `.mw4anim`, 73 `.tga`, 29 `.script`, 17 `.obb`, 15 `.wav`, 3 `.h`, 1 each `.animscript` `.abl` `.abi`. The resulting `Content/Mechs/dasher/` now matches our own `Content/Mechs/Dasher_DNU/` naming exactly (`das_hip.erf`, `dasher_cage.erf`, `dasher_skeleton.obb`, `dasher_skeleton_SOLID.obb`). > For the six new chassis we therefore have **all geometry, animation and art**, but their > definition files must be written by hand. Note we already hold complete source for two of > them in our own repo: `Content/Mechs/Dasher_DNU` and `Content/Mechs/jenner2c`. --- ## 6. Reports Written to `reports/` by `run-comparison.sh`. The two `manifest-*.tsv` files are ~5 MB intermediates and are gitignored. | file | contents | |---|---| | `packages-entrydiff.txt` | per-package entry-name diff, all 857 V4H packages vs our 102 | | `core-decoded-diff.txt` | 433 real differences + 754 V4H-only entries in `core.mw4` | | `props-decoded-diff.txt` | 419 real differences + 155 V4H-only in `props.mw4` | | `textures-decoded-diff.txt` | 1 real difference + 299 V4H-only in `textures.mw4` | | `maps-decoded-diff.txt` | per-map decoded diff (all <=1 real difference) | | `missions-decoded-diff.txt` | per-mission decoded diff (`.contents` / `.lights` / freezer) | | `hsh-diff.txt`, `hshoriginal-diff.txt` | HUD art trees | | `content-diff.txt` | the 11 loose scripts + `mwconst.abi` | | `resource-files-diff.txt` | package-file and loose-file level diff of `resource/` | | `variants-inventory.txt` | 797 variants grouped by chassis with load status | `reference/` holds verbatim copies of small, high-value V4H text files that are not in our tree, so they survive independently of the `FS_Build_V4H` folder: | file | source | |---|---| | `v4h-changelog.txt` | `firestorm build V4h.txt` -- their v4 -> v4h release notes | | `Convert-hud.txt`, `Convert-radar.txt` | `hsh/hud/Convert.txt`, `hsh/radar/hud/Convert.txt` -- paper-doll authoring recipe, points at `coord.cpp` | | `mechnames.script`, `buildmechnames.script` | the 71-entry mech name tables | | `v4h-firestorm.h`, `v4h-firestorm64.h` | 20 console / 64 Dragon-MP callsigns | | `v4h-options.h` | `USE_ALLOWED_MECHS`, `Battle_Ammo`, `USE_OBSERVER`, ... | | `mwconst.abi.diff` | the `M_Champion..M_Griffin` / `LastMechID=70` delta | | `options.ini.diff` | ours -> theirs | Not copied (too large, read them from `FS_Build_V4H/content/shellscripts/mechbay/`): `mechdescriptions.script` (74,883 B) and `old mechdescriptions.script` (74,438 B). --- ## 6b. Decompiling the compiled records **Full write-up: [DECOMPILING.md](DECOMPILING.md).** Read that before touching any of this. The compiled records are not a dead end. The engine source contains both the writer (`MWTool::BuildResource` -> `MWObject::CreateSubsystemStream` -> `Tool::ConstructCreateMessage` -> a per-class factory in one of 51 `mw4/Code/MW4/*_Tool.cpp`) and the reader, and the message structs are plain PODs declared in our own headers. We also have a closed-loop test harness: 65 chassis where we hold both the source text and the compiled record. | record type | status | |---|---| | `.armature` | **done and verified** -- 2,938 of 2,976 pages exact across 65 chassis, 1,849/1,849 child lists exact. All 38 residuals are data the packer itself discarded, so the game never sees them either. | | `.subsystems` | **done and verified** -- 7,579 of 7,585 keys exact across 63 chassis. The 6 residuals are Behemoth/Behemoth II `GroupIndex`, changed in source by `e45a67a8` after the package was built. | | `.data` `.damage` `.contents` `.instance` `.torso` `.engine` | same mechanism, not started | ```bash python3 MW4COMPARE/tools/decompile/armature.py --verify # 65-chassis harness python3 MW4COMPARE/tools/decompile/armature.py -o out.armature ``` `.armature` files have been generated for all six new chassis into `FS_Build_V4H_extracted/Content/Mechs/*/`. Two things found along the way that matter beyond this section: * The extracted baseline reflects the **packages**, not `Content/` at HEAD. `core.mw4` and `textures.mw4` have never been repacked since the initial mirror (`2b8ca921`); `props.mw4` is current as of `8bfaf9b9`. Check this before concluding anything is missing. See DECOMPILING.md section 8. * `ResourceID` packs the package record id in its **high word**, which is how `Model=` references resolve back to a path. --- ## 7. Merge assessment Ranked by value/risk. **None of this has been done.** ### Low risk, high value -- take from V4H 1. **727 mech variants** that reference chassis we already have. Drop-in: they are self-contained `.mw4` files in `Resource/Variants/`, keyed by chassis *name*, and our chassis names match. Sanity-check a sample in the MechLab first -- their changelog warns that a corrupt variant can corrupt a build. 2. **The 11 loose shell scripts** -- at minimum `mechdescriptions.script` (74 KB of mech text we simply do not have) and the two mech-name tables. `firestorm.h`/`firestorm64.h` callsign lists are a straight content lift. 3. **Freezer stadium crowd VO** (17 wavs) and the expanded `freezer_teamattrition.abl`. 4. **HUD/MFD/radar art for the six new mechs**, and the 24-bpp `Mechs/*.bmp` masters if we want higher-quality portraits than our 8-bpp copies. ### Medium risk -- the six new mechs The assets are all extractable from `core.mw4`/`textures.mw4`/`props.mw4`. The blockers are on *our* side and are all code, not data: * `render.cpp` `mechnames[]` -- append 6 entries (currently 65, `"dasher"` is commented out). * `coord.cpp` -- MFD/radar paper-doll coordinate tables must gain 6 entries. * `mwconst.abi` -- apply the `M_Champion..M_Griffin` / `LastMechID=70` delta. * `ScriptStrings.dll` (`StringResource.rc`) -- add `DNL_CHAMPION`, `DNL_JENNER2C`, `DNL_DASHER`, `DNL_MARAUDER`, `DNL_THUNDERBOLT`, `DNL_GRIFFIN`. * MechLab scripts must learn about IDs 65-70 (this is what their +33 KB `mechbay_main.script` is doing). * `MW4Shell.cpp` / anything else with a hard-coded 65. Because IDs 0-64 are identical, this is purely additive. Extracting the mech *source* (rather than the packed records) is the open question -- the records in `core.mw4` are the compiled forms, and we need `.erf`/`.obb`/`.data` sources to rebuild `core.mw4` with our own packer. We already hold source for `dasher` and `jenner2c`. ### High risk -- do not merge blind * **`.subsystems` for the 65 shared chassis.** All 65 differ. Taking theirs imports their entire balance pass (3-ton ammo cap, revised weapon groups) wholesale; taking ours keeps our balance. This is a *design* decision, not a merge decision. * **Mission `.contents`.** Real, substantial edits on their side for every shared mission. Needs a proper structural diff before any decision. * **MechLab scripts.** Their MechLab is documented (by them) as crash-prone on save. Do not import their `mechbay_*.script` wholesale. ### Not worth taking `stats/zWorldStats` (dead 2001 telemetry), `RMTSHARE.EXE`, `uploadbuildv4h/` (777 MB installer of the same content), `options.ini.rp411old`. --- ## 8. Open questions for the project owner 1. **Balance.** Do we adopt V4H's `.subsystems` (3-ton ammo cap, their weapon groups) for the 65 shared mechs, keep ours, or merge selectively per mech-- 2. **Mission layouts.** Their `.contents` differ for every shared mission. Is that deliberate content work on their side (more drop zones? different props?) or drift we should ignore-- 3. **Mech source.** Is there a source tree for V4H anywhere, or is `core.mw4` the only copy of the six new mechs? That decides whether we re-pack from source or have to write an unpacker that reconstructs `.erf`/`.obb` from packed records. 4. **`bitdepth=32`.** V4H runs the console at 32 bpp; we run 16. Given the `DWM8And16BitMitigation` dependency documented in the repo memory, is their build actually rendering at 32 bpp, and should we try it-- 5. **The 25 unregistered chassis on our side** (Canis, Gargoyle, razorback, reaver, ...) -- are these finished assets that were never wired up, or abandoned work? If usable, the combined roster could be far larger than 71. 6. **Variant hygiene.** Their changelog warns that corrupt variants can corrupt the build. Do we import all 727 loadable variants, or curate-- 7. **`jenner_iic1`.** Three variants reference a chassis that exists in neither build. Drop them, or re-point at `jenner_2c`-- --- *Generated 2026-08-08 against `FS_Build_V4H` (Oct/Nov 2021) and branch `5.1.0b-in-progress`.*