vwemirror/firestorm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9092b78dbfe3409d63f4432fc493df3a84f4785a
firestorm/MW4COMPARE/README.md
T
dicionClaude Opus 5GitHub Copilot
 83478b7666 Add MW4COMPARE: .mw4 decompiler toolchain and V4H comparison harness 
Tooling built to recover editable source for six 'Mech chassis that exist
in the parallel FS_Build_V4H build but not in this repo. Reverse-engineers
every compiled record type in the .mw4 package format back to the .data /
.instance / .subsystems / .damage / .contents / .torso / .engine /
.armature sources the content pipeline consumes.

Nothing here is wired into the game build. It is a standalone analysis
harness run from Linux.

Package format
--------------
"#VBD" container. Directory records are [len][name][FILETIME][origSize]
[storedSize][offset], payload base at dword 0x0C. A record is stored raw
when storedSize == origSize, otherwise LZW (9->12-bit LSB-first codes,
256=clear, 257=EOF, dict from 258), per Database.cpp:451.

GameModel records are flat /Zp4 structs following the C++ inheritance
chain Entity(0) -> Mover(28) -> MWObject(80) -> Vehicle(664) -> Mech(756),
1636 bytes total. CreateMessage records follow Replicator -> Entity ->
Mover -> MWMover -> MWObject -> Vehicle -> Mech from start=16 (the
undeclared Connection__Message header), ending at 341 and padded to 344.

tools/decompile/
----------------
  datamap.py        header-driven layout engine; CHAIN + ANCHORS
                    {Vehicle:664, Mech:756} assert the struct offsets
  mw4msg.py         CreateMessage reader/walker
  data.py           .data      constants.py  define/table symbol resolution
  damage.py         .damage    contents.py   .contents
  smallmodel.py     .torso + .engine         instance.py  .instance
  armature.py / armature_parts.py  .armature + armaturedata/armaturevideo
  assembly.py       joint hierarchy renderer
  make_generic_doll.py  builds generic MFD/Radar damage dolls
  verify_*.py       per-type round-trip verifiers

Verified round-trip across all 64 shared chassis:
  .armature      2938/2976 pages     .subsystems  7579/7585 keys
  .data map      6071/6071 values    .data trip   8291/8306 keys
  .damage        6605/6605 keys      .contents    7480/7480 keys
  .torso+.engine 1280/1280 keys      .instance     896/896 keys, 64/64 pages
  armature_parts 1202/1202 .data, 1149/1202 .video

Layout-discovery lessons (documented in DECOMPILING.md)
-------------------------------------------------------
- Never let a field map be discovered by the values that verify it. A
  value-matching pass reported 4288/4288 while mis-assigning 34 keys. The
  map was rebuilt from header declaration order, anchored on uniquely
  resolved fields.
- Read the factory, not the data. 12 .data fields and 5 Torso fields are
  declared plain Stuff::Scalar but multiplied by Radians_Per_Degree in
  Mech_Tool.cpp:889 / Torso_Tool.cpp.
- Strip typedefs before walking a header. A stray `typedef int AttributeID;`
  masked a missing ClassID - two 4-byte errors cancelling out, caught only
  by the ANCHORS assertion.
- A verifier that silently narrows its own input reports success. Braced
  blocks must be hidden before splitting pages, replacing both CR and LF,
  because a `Shadow={...}` block contains a line reading `[shadow]` and
  splitlines() also splits on bare CR.
- NSWIZZLE is undefined, so the #else branch is live and orders members
  differently. bool is 1 byte; char x[MaxStringLength] is 256.
- V4H carries stale Mech IDs (their Atlas is 5, ours 6), so 64 of 65 shared
  chassis are off by one; --retarget-ids emits $(M_<Chassis>)/$(IDS_<Chassis>).

reports/ holds generated diffs. The two ~5 MB manifest-*.tsv intermediates
are gitignored; regenerate everything with run-comparison.sh.

Co-authored-by: Claude Opus 5 (Anthropic) <noreply@anthropic.com>
Co-authored-by: GitHub Copilot <copilot@github.com>
2026-08-08 16:47:57 -05:00

733 lines
36 KiB
Markdown
Raw Blame History

# MW4COMPARE -- FireStorm source tree vs. the `FS_Build_V4H` binary build
Comparison of our source repository against **FireStorm Battletech SMT Build Four (v4h)**,
a parallel, independently-maintained *binary-only* build of the same game, located at
`/home/rich/Repositories/FS_Build_V4H` (1.7 GB, dated Oct/Nov 2021).
Goal: work out what each side has that the other does not, so the two can eventually be
merged into one build containing all the mechs, maps, game types and variants.
**Status: Part 1 (survey) complete. Part 2 (extraction + pruning) complete.** Nothing has
been merged into the repo. Every V4H asset is unpacked to a parallel tree outside the repo
at `/home/rich/Repositories/FS_Build_V4H_extracted`, reduced to **differences only** --
5,385 files / 65 MB out of an original 46,707 files / 1.24 GB -- and reshaped to mirror
`Gameleap/mw4`. See section 5.
---
## 0. TL;DR
| | V4H has, we don't | We have, V4H doesn't |
|---|---|---|
| **Mechs** | **6 new chassis** (IDs 65-70): Champion, Jenner IIC, Dasher, Marauder, Thunderbolt, Griffin -- full geometry, textures, HUD/MFD/radar art, footsteps | Source for ~25 unregistered/experimental chassis (Blackheart, Bowman, Canis, Crab, Gargoyle, koto, locust2c, marauder2, pitbull, privateer, razorback, reaper, reaver, shadowhawk, strider, urbanmech_iic, ursus, vulture2, vulturec, ...) plus 4 Elite\_\* pilot variants |
| **Variants** | **797 saved mechlab variants** (`resource/Variants/*.mw4`); we ship 1 | -- |
| **Maps** | none | **19 maps** (conroe01-03, desert, doneg01, gage, minerl01, mountn01/03, ngoth, ruin03, scrub01, scrub06, volcan01/03, ddc_msl, freezer, ...) |
| **Missions** | none | **17 missions** + 5 UserMissions (aspen, canyon, conroe01-03, gagetown, gladiatorpit, lakeside, mechworks, minehq, newgothem, rubble, sanddunes, spaceport, vbase, editortemplate) |
| **Skies** | *missing entirely* -- no `skies.mw4` | `skies.mw4` (40 MB, 2975 entries) |
| **Shell scripts** | reworked MechLab + 11 loose script/header files we don't have at all | our own console/lobby work (RookieMission, `-tmfds`, etc.) |
| **Engine** | binary only -- cannot be recovered | full source |
The single most important structural finding:
> **Mech IDs 0-64 are byte-for-byte identical in both forks.** V4H *appended* its six new
> mechs as IDs 65-70. There is no re-indexing to do -- a merge is additive on both sides.
---
## 1. What `FS_Build_V4H` is
A deployed game folder, not a source tree. Notable contents:
```
FS_Build_V4H/
MW4.exe 3,633,152 B 2021-09-08 (their fork; no source available)
ScriptStrings.dll 163,840 B 2021-02-03 (has the 6 new mech display names)
MissionLang.dll 188,416 B 2021-09-01
content/ 14 loose files (shell scripts + 1 ABL header)
hsh/ 279 files -- the "block HUD" set introduced in v4f
hshoriginal/ 447 files -- the pre-v4f HUD set, archived
resource/
core.mw4 12,709 entries
props.mw4 10,779 entries
textures.mw4 10,419 entries
maps/ 27 maps Missions/ 29 missions
Variants/ 797 saved mechlab loadouts
Pilots/Tesla/options.mw4
(no skies.mw4)
stats/zWorldStats/ stock 2001-era MS stats uploader, logs end 2001 -- dead weight
uploadbuildv4h/ 777 MB self-extracting installer of this same build
firestorm build V4h.txt changelog for v4 -> v4h (also in uploadbuildv4h/)
```
Their own changelog (`firestorm build V4h.txt`) is worth reading; it documents the
six new mechs, the 3-ton-per-weapon ammo cap, the MechLab save crash, and the fact that
corrupt variants can corrupt the whole build.
---
## 2. Method
The `.mw4` files are GameOS `Stuff::Database` packages (magic `#VBD`) -- a flat directory of
named records, each optionally LZW-compressed. See `tools/mw4db.py` for the format and a
faithful port of `gos_LZDecompress`.
Two comparison levels were used:
1. **Index-only** (`mw4index.py` + `diffindex.py`) -- parses the directory, no decompression.
~1 s for the whole 780 MB tree. Gives an exact **entry-name** diff.
2. **Decoded** (`pkgcmp.py`) -- decompresses every record and compares plaintext bytes.
Slower (~2 MB/s) but is the only trustworthy content test.
### Two traps that cost time -- do not repeat them
* **Case.** V4H's packer lowercased every entry name (`ablscripts\bots\0-bot.abl` vs our
`ablscripts\bots\0-Bot.abl`). A case-sensitive comparison reported 5,909 "new" and 5,155
"missing" entries in `core.mw4` that were the *same files*. Always compare through
`mw4db.norm()`.
* **Stored-byte md5 is not an equality test.** Comparing the compressed payloads
over-reports massively: `maps/peaks.mw4` showed 9 differing blobs by stored bytes but
only **1** after decompression. Use `pkgcmp.py` before believing any difference.
---
## 3. Findings
### 3.1 The six new mechs (the headline)
`core.mw4` in V4H is a **strict superset** of ours: 754 extra entries, **zero** entries we
have that they lack. All 754 belong to six new chassis and their destroyed models:
| chassis | core entries | textures | props (animation) |
|---|---|---|---|
| marauder | 63 armaturedata + 49 root + 21 armaturevideo + 5 destroyed | 37 + 1 | reuses existing |
| griffin | 57 + 54 + 19 + 5 | 38 + 1 | reuses existing |
| champion | 57 + 51 + 19 + 5 | 36 + 1 | reuses existing |
| thunderbolt | 54 + 48 + 18 + 5 | 33 + 1 | reuses existing |
| dasher | 45 + 48 + 15 + 5 | 29 + 1 | reuses existing |
| jenner2c | 45 + 46 + 15 + 5 | 30 + 1 | **155 new entries** under `mechs/jenner2c/animation` |
Each chassis brings the full set: `.contents` (+ per-joint `{armature}` / `{sites}` pages),
`.damage`, `.data` (+ `{element}` `{footsteps}` `{gamemodel}` `{hierarchicalobb}` `{solidobb}`),
`.engine`, `.instance`, `.subsystems`, `.torso`.
`textures.mw4` adds 299 entries: the six mechs' skins, six sets of
`textures/@a{chp,grf,jec,mar,thu,...}0-5.tga` cockpit/pilot art, and
`textures/footsteps/<mech>_{default,dirt,snow}.tga`.
**Nothing else is new in `core.mw4`.** No new weapons, no new subsystem types, no new
tables. Whatever balance changes V4H made live in the per-mech `.subsystems` records and
in their `MW4.exe`.
### 3.2 Mech ID space -- identical for 0-64
`content/ABLscripts/mwconst.abi` differs from ours by exactly this:
```diff
-LastMechID = 64 ;
-NoMechID = 65 ;
+M_Champion = 65 ;
+M_Jenner2c = 66 ;
+M_Dasher = 67 ;
+M_Marauder = 68 ;
+M_Thunderbolt = 69 ;
+M_Griffin = 70 ;
+LastMechID = 70 ;
+NoMechID = 71 ;
+LastMP1MechID = M_Solitaire;
```
Corroborated three ways:
* `content/shellscripts/mechbay/mechnames.script` and `buildmechnames.script` list
`mech_name[0..64]` exactly matching our `mechnames[]` in
`CoreTech/Libraries/GameOS/render.cpp`, then append 65-70.
* `core.mw4` defines 65 `*.subsystems` on our side, 71 on theirs; the extra six are
`champion, dasher, griffin, jenner_2c, marauder, thunderbolt`.
* `ScriptStrings.dll` (UTF-16) gains display strings `Champion`, `Jenner2c`, `Dasher`,
`Marauder`, `Thunderbolt`, `Griffin`.
Note the naming inconsistency in their own build: the chassis directory is `jenner2c`
but the subsystems record is `jenner_2c`, while three of their variants still reference
`jenner_iic1` (see section 3.3).
### 3.3 797 mech variants
`resource/Variants/*.mw4` -- saved MechLab loadouts. We ship exactly one
(`MW4/Resource/Variants/Argus Variant1.mw4`).
Each variant package is 4 records:
```
[0] 'Resource\Variants\<name>' 4 B raw
[1] '{Mech}' ~350 B
[2] '<chassis>{Subsystem}' ~4 KB <- names the chassis it needs
[3] '<8-char token>' 16 B
```
Because the chassis is named in record 2, dependency analysis needs no decompression
(`tools/variants-report.py`). Result:
```
loadable in our build TODAY : 727
blocked on the 6 new chassis: 67 (thunderbolt 22, griffin 15, marauder 15,
dasher 10, champion 5)
broken / orphaned : 3 (chassis 'jenner_iic1' does not exist even in V4H --
Jenner2c 1 / Jenner2c 4 / Jenner2c Pharaoh)
```
Top chassis by variant count: warhammer 36, rifleman 30, nova 24, madcat 23, atlas 23,
thunderbolt 22, hunchback 21, battlemaster 21, archer 21, thor 20.
Full table: `reports/variants-inventory.txt`.
### 3.4 Maps and missions -- we are ahead, they changed a little
* **Maps**: V4H has 27, we have 46. **Zero** V4H-only maps. Common maps are effectively
identical -- file sizes differ by 0-2 bytes and `peaks.mw4` decodes to exactly **one**
differing record (`peaks.data{gamemodel}`, an internal reference blob).
* **Missions**: V4H has 29, we have 46 + 5 UserMissions. **Zero** V4H-only missions.
Every common mission decodes to 2-4 differing records, always the same shape:
* `<name>.lights` and `<name>night.lights` -- **3 bytes** differ out of 33 KB. Noise.
* `<name>.contents` -- **real**. Coliseum: 11,588 of 52,954 bytes differ, and the
differing region contains new ASCII entity names. Their mission layouts have been
edited. Worth a proper look before merging either direction.
* `Missions/freezer.mw4` additionally has **30 V4H-only records**: stadium crowd VO
(`audio/vo/generic/mp/{cheer1-9,boo1-3,jeers,roar,ahh,aww,stadamb}.wav`) and a
rewritten `scripts/freezer_teamattrition.abl` (10,710 B vs our 832 B).
* All 71 loose `.nfo` / `.tga` / `.txt` files in `resource/` (mission metadata, game-type
definitions, thumbnails) are **byte-identical**. No game-type divergence.
* **`skies.mw4` is absent from V4H entirely.** 40 MB / 2,975 entries that only we have.
### 3.5 Shell scripts -- V4H has a diverged MechLab and 11 files we don't have at all
`props.mw4` decodes to 419 differing records, only 3 entries missing on their side
(`lobbydecals/decal_46/47/49.tga`) and 155 extra (jenner2c animation).
The interesting differences (V4H size vs ours):
| script | V4H | ours | delta |
|---|---|---|---|
| `mechbay/mechbay_main.script` | 94,517 | 61,432 | **+33 KB** |
| `mechbay/weapons.script` | 102,025 | 83,653 | **+18 KB** |
| `mechbay/armor.script` | 64,539 | 61,807 | +2.7 KB |
| `mechbay/infobox.script` | 43,875 | 41,595 | +2.3 KB |
| `listboxes.script` | 30,785 | 29,269 | +1.5 KB |
| `scriptstrings.h` | 71,608 | 70,811 | +797 B |
| `netlobby.script` | 163,740 | 163,088 | +652 B |
| `multiplayer/buildrestriction.script` | 16,443 | 15,713 | +730 B |
| `mechbay/chassis.script` | 82,596 | 81,881 | +715 B |
| `conlobby.script` | 141,044 | 140,532 | +512 B |
| ...plus ~15 more with small deltas | | | |
Also: 24 `mechbay/graphics/installed/weapon_-1_*.tga` slot images are **much smaller** in
V4H (1,036 B vs 6,540 B etc.) -- this is the "weapon box sizes set to minimum" change from
their v4 changelog.
**Eleven loose files in `content/shellscripts/` exist in V4H and nowhere in our repo:**
```
clock.script wall-clock overlay
cm_listboxes.script console mech list box
firestorm.h 20 console callsigns
firestorm64.h 64 Dragon-MP callsigns
globals.script shared script globals
options.h USE_ALLOWED_MECHS, Battle_Ammo, USE_OBSERVER, ...
mechbay/buildmechnames.script 71-entry mech name table
mechbay/mechnames.script 71-entry mech name table
mechbay/mechdescriptions.script 74 KB of mech description text
mechbay/old mechdescriptions.script
graphics/multiplayer/lobbydecals/decal_.tga
```
These are loaded loose from disk (they are in neither `core.mw4` nor `props.mw4` on either
side). Their `computerplayer.script` `#include`s `firestorm.h` where ours `#include`s
`callsign.h` -- so this is a genuinely diverged shell-script codebase, not just extra data.
### 3.6 HUD art (`hsh/`)
V4H maintains **two** HUD sets:
* `hshoriginal/` (447 files) -- the detailed paper-doll set, close to ours.
* `hsh/` (279 files) -- the "block HUD" compromise introduced in v4f so the new mechs
would have *something* to display. Their changelog admits HUDs do not always match the
mech actually in use.
Against our `hsh/` (456 files):
| | identical | differ | V4H-only | ours-only |
|---|---|---|---|---|
| `hsh/` | 182 | 68 | 29 | 206 |
| `hshoriginal/` | 275 | 148 | 24 | 33 |
V4H-only files are the six new mechs' `hud/`, `MFD/`, `radar/hud/` and `Mechs/` art,
`decals/DDC.bmp`, plus `Mechs/battlemasteriic.bmp` and `Mechs/mad cat mk.ii.bmp`
(pre-rename spellings of files we renamed).
Colour depth differs systematically: their `Mechs/*.bmp` are 24-bpp 500x400 (600,054 B),
ours are 8-bpp palettised (201,080 B). Same for `MFD/rifleman.bmp` (43,256 vs 15,476).
`hsh/hud/Convert.txt` and `hsh/radar/hud/Convert.txt` are **the authoring recipe for the
paper-doll art** and explicitly say the coordinates go into `coord.cpp` under
GameOS/External dependencies. Independent corroboration of `MFD-RADAR-MAPPINGS.md`.
They are reproduced verbatim in `reference/Convert-hud.txt` / `reference/Convert-radar.txt`.
### 3.7 Per-mech `.subsystems` differ for every single chassis
Every one of the 65 shared chassis has a differing `.subsystems` record -- same length,
different content (Atlas: 1,830 of 6,082 bytes). This is the compiled form, not the source
text, so it is not directly readable. It is consistent with V4H's documented changes:
3-ton ammo cap per weapon, revised default weapon groups, restricted hardpoints on the new
mechs.
**This is the one area where merging is genuinely risky** -- see the open questions.
Also differing in `core.mw4` for essentially every mech: `.data{gamemodel}`,
`.torso{gamemodel}`, and the `contents[joint_torsoabove]{armature}` /
`contents[joint_*belowankle]{armature}` records. Byte inspection of
`atlas.data{gamemodel}` shows one side stores an inline path string where the other
stores a numeric reference -- i.e. a packer/format difference, most likely benign.
Confirm before treating any of these as content changes.
### 3.8 Root config and binaries
Identical: 37 root files, including all three `ctcl-*.ini`.
Differing binaries (theirs older/forked, ours newer): `MW4.exe`, `autoconfig.exe`,
`Launcher.exe`, `mw4print.exe`, `ctcls.dll`, `Language.dll`, `MissionLang.dll`,
`ScriptStrings.dll`.
V4H-only root files: `Console launcher.bat` (`mw4.exe -window`), `optionssmt.ini`,
`optionssmtv4.ini`, `options.ini.rp411old`, `pre-uninstall.bat`, `RMTSHARE.EXE`,
`SetNetworkAccess.reg`, `firestorm build V4h.txt`.
`options.ini` differences worth flagging:
| key | V4H | ours |
|---|---|---|
| `bitdepth` | **32** | 16 |
| `ruletype` | 17 | 2 |
| `unlimitedammo` | 0 | 1 |
| `friendlyfirepercentage` | 100 | 0 |
| `heaton` | 1 | 0 |
| `allowedbeam1` | `fffffffd` | `ffffffff` |
| `allowedmissile1` | `ff3bffff` | `ffffffff` |
| `allowedprojectile1` | `ffffdfff` | `ffffffff` |
| `playericon` / `teamicon` | decal\_49 / decal\_50 | decal\_48 / decal\_48 |
| `[RookieMission]` section | absent | present (ours) |
| `RuleBook`/`DawnWar`/`BiggieSizeIt`/`CanYouHearTheFootSteps` | absent | present (ours) |
`bitdepth=32` is interesting given all the `DWM8And16BitMitigation` work recorded in the
repo memory -- V4H apparently runs the console at 32 bpp.
`optionssmtv4.ini` additionally carries a full `[cameraship params]` block (over-shoulder
offsets, death-sequence camera, HUD name/chat toggles) that ours does not.
### 3.9 What *we* have that V4H does not
Besides the maps/missions/skies above, `Gameleap/mw4/Content/Mechs` contains 93
non-destroyed chassis directories, of which **27 are not in V4H's 71-mech roster**:
```
Blackheart Bowman Canis Crab Duangung Gargoyle Gesu Grimreaper_DNU
koto locust2c marauder2 pitbull privateer razorback reaper reaver
shadowhawk strider urbanmech_iic ursus vulture2 vulturec
Elite_Daishi_Castro Elite_Daishi_William Elite_Madcat_James Elite_Thor_Duncan
jenner2c (chassis 'jenner2c' -- V4H spells theirs 'jenner_2c')
```
Two of these matter immediately:
* **`Dasher_DNU/`** -- chassis `dasher`, 71 source files including `.subsystems`,
`.damage`, `.animscript`, geometry. We already have Dasher source; it is simply not
registered in the roster or packed into `core.mw4`.
* **`jenner2c/`** -- 80 source files, plus `jenner2c_destroyed/`. Likewise unregistered.
So of V4H's six new mechs we may already hold source for two. `marauder2` is *Marauder II*,
a different chassis from their `marauder`.
---
## 4. Tools
All under `tools/`. Python 3, no dependencies.
| file | what it does |
|---|---|
| `mw4db.py` | `#VBD` package reader + `gos_LZDecompress` port. Import this; everything else builds on it. Documents the on-disk format. |
| `mw4index.py` | `mw4index.py <resourceRoot> > manifest.tsv` -- directory-only manifest of every `*.mw4`. ~1 s for 780 MB. |
| `diffindex.py` | `diffindex.py A.tsv B.tsv [labelA] [labelB]` -- case-insensitive package/entry-name diff. |
| `pkgcmp.py` | `pkgcmp.py A.mw4 B.mw4` -- **authoritative** decoded content diff of one package. |
| `dumprec.py` | `dumprec.py pkg.mw4 "entry/name" [out]` -- extract one decoded record; `--list` lists entries. |
| `treediff.py` | `treediff.py dirA dirB [labelA] [labelB]` -- case-insensitive loose-file tree diff by md5. |
| `variants-report.py` | chassis-dependency inventory of `resource/Variants`, flags variants that cannot load. |
| `extract-all.py` | `extract-all.py <resourceRoot> <outDir>` -- unpack every package into a real directory tree. See section 5. |
| `prune-identical.py` | `prune-identical.py <extractedDir> [--apply]` -- delete everything we already have byte-for-byte. Dry-run by default. |
| `classify-survivors.py` | `classify-survivors.py <prunedDir>` -- split what is left into NEW vs DIFFERS. |
| `restructure.py` | `restructure.py <extractedDir> [--apply]` -- reshape a pruned tree to the repo's layout and casing. Refuses to run if any two packages would collide. |
| `split-source.py` | `split-source.py <extractedDir> [--apply]` -- separate repackable source from compiled records and rename the `.obb`s. |
`run-comparison.sh` regenerates every report:
```bash
cd MW4COMPARE
./run-comparison.sh # uses the default paths
./run-comparison.sh /path/to/OTHER_BUILD /path/to/our/mw4
```
Runtime is a few minutes; `textures.mw4` dominates.
There is an older, standalone version of the same decompressor at
`build-env/extract-mw4.py`. `tools/mw4db.py` supersedes it and is the one to extend.
## 5. The extracted asset tree
Every record of every V4H package, unpacked to real files at
**`/home/rich/Repositories/FS_Build_V4H_extracted`** -- deliberately *outside* the repo.
Nothing has been merged; this is derived data and is regenerable:
```bash
python3 MW4COMPARE/tools/extract-all.py \
/home/rich/Repositories/FS_Build_V4H/resource \
/home/rich/Repositories/FS_Build_V4H_extracted
```
857 packages -> **46,707 files, 1.24 GB**, about 3 minutes.
| directory | files | size | from |
|---|---:|---:|---|
| `core/` | 12,709 | 3.4 MB | `core.mw4` |
| `props/` | 10,779 | 178 MB | `props.mw4` |
| `textures/` | 10,419 | 358 MB | `textures.mw4` |
| `maps/<map>/` | 7,219 | 576 MB | 27 map packages |
| `Missions/<name>/` | 2,391 | 60 MB | 29 mission packages |
| `Variants/<name>/` | 3,188 | 4.8 MB | 797 saved loadouts |
| `Pilots/Tesla/options/` | 2 | 5 KB | |
| `_merged/` | 42,100 | *hardlinks* | flattened Content-shaped view |
Extraction is **per package**, because 842 entry paths are claimed by more than one
package and 70 of those hold genuinely different content (mostly a mission packing its own
copy of a global `props.mw4` asset -- 29 are Coliseum/Freezer crowd VO). A single flat tree
would silently lose data.
`_merged/` is a convenience view built with **hardlinks** (no extra disk) that mirrors
`Gameleap/mw4/Content`, so it diffs directly against our source tree. Conflicts resolve
`props > core > textures > maps > missions` and every one is listed in `_conflicts.tsv`.
Variants and Pilots are excluded from it -- their records have no directory part
(`{Mech}`, `<chassis>{Subsystem}`) so all 797 variants would collide.
Treat `_merged/` as read-only; hardlinks mean editing there edits the per-package copy.
Full details, including the duplicate-entry handling, are in the tree's own `_README.txt`.
### How close packed records are to real source
Verified against `mechs/jenner2c`:
* **Recovered as-is** -- `.contents`, `.damage`, `.data`, `.engine`, `.instance`,
`.subsystems`, `.torso`, `.animscript`, every `*.erf` geometry part, `*_cage.erf`,
`runninglights.erf`, `armaturedata/`, `armaturevideo/`. Decoded TGAs are valid
(`Targa image data - RGB 512 x 512 x 32`); `core/ablscripts/mwconst.abi` extracts
byte-identical to V4H's loose copy, which independently validates the decompressor.
* **Recoverable by renaming a qualified record** --
`X.data{solidobb}` -> `X_skeleton_SOLID.obb` and
`X.data{hierarchicalobb}` -> `X_skeleton.obb`. Both carry the `#BBO` magic and
byte-identical headers to our source `.obb` files.
* **Not directly present** -- `X.armature`. The packer splits it into per-joint
`X.contents[joint_*]{armature}` records.
Also worth knowing before any merge: **V4H's Jenner IIC is not a rename of ours.** Their
part prefix is `jec_` and chassis files are `jenner_2c.*`; ours are `j2c_` and
`jenner2c.*`. Zero files in common. They are separately authored assets.
### Pruning to differences only
Our own packages were extracted the same way to
**`/home/rich/Repositories/FS_Ours_extracted`** (102 packages, 51,382 files, 1.76 GB),
then everything V4H has that we already hold byte-for-byte was deleted:
```bash
python3 MW4COMPARE/tools/prune-identical.py /home/rich/Repositories/FS_Build_V4H_extracted --apply
python3 MW4COMPARE/tools/classify-survivors.py /home/rich/Repositories/FS_Build_V4H_extracted
```
**46,707 files / 1.24 GB -> 5,385 files / 66 MB.** 41,322 deleted (88.5%), 1,912 empty
directories removed. `_pruned.tsv` logs every deletion and which rule matched.
| package group | NEW | DIFFERS | |
|---|---:|---:|---|
| `Variants` | 3,188 | 0 | 797 loadouts x 4 records; we ship 1 |
| `core` | 746 | 440 | 6 new mechs + per-chassis changes |
| `props` | 155 | 419 | jenner2c animation + reworked MechLab |
| `textures` | 297 | 2 | new mechs' skins / footsteps |
| `Missions` | 1 | 114 | `.contents`, `.lights`, freezer crowd VO |
| `maps` | 0 | 21 | |
| `Pilots` | 0 | 2 | |
| **total** | **4,387** | **998** | |
#### Why our source tree alone is not a sufficient baseline
A `.mw4` is not a zip of `Content/`. Measured:
* **17,683 records (42%)** are qualified derivatives the packer *generates* and that have
no source file at all -- `foo.data{gamemodel}`, `foo.contents[joint_hip]{armature}`,
`bar.tga{hint}`.
* **~7,400 more** have a source path but the packer **rewrites the bytes**. Proven by
comparing *our own* `Content/` against *our own* `.mw4` output: 3,238 `.data`,
2,320 `.video`, 583 `.instance`, 361 `.contents`, 345 `.audio`, 235 `.damage`,
117 `.torso`, 103 `.subsystems`, 65 `.engine`, 58 `.lights`.
Matching against `Content/` alone would have "kept" roughly 24,000 files we already have.
So each candidate is tested in order: (1) same package + entry path in our extracted
packages, (2) same entry path in any of our packages, (3) unqualified name at the same path
in `Content/`. In practice 41,320 matched rule 1 and 2 matched rule 3.
> **Case warning.** V4H's packer lowercased every entry name; ours did not. The same record
> is `core/mechs/longbow/longbow.data{element}` in V4H and
> `core/Mechs/Longbow/longbow.data{Element}` in ours. Every comparison here is
> case-insensitive -- a plain `diff -r` between the two trees is meaningless.
#### Reading the 998 DIFFERS
By extension: 352 `data{gamemodel}`, 96 `torso{gamemodel}`, 93 `subsystems`,
114 `contents[joint_*]{armature}`, 55 `lights`, 54 `audio`, 38 `instance`, 37 `contents`,
29 `script`, 28 `tga`, 15 `wav`.
* **Probably packer noise** -- the `{gamemodel}` records encode cross-package references
(one side stores an inline path string where the other stores a numeric id), and the
`.lights` differences measured 3 bytes out of 33 KB. Verify before treating as content.
* **Real, needs a design decision** -- the 93 `.subsystems` (V4H's balance pass) and the
37 mission `.contents` (substantial layout edits).
* **Real, straightforward** -- the 29 `.script` files (their MechLab rework) and the
freezer crowd VO.
### Reshaped to the repo layout
Once pruned, the V4H tree has **zero cross-package collisions** -- 5,385 files map to 5,385
distinct destinations -- so it was flattened to the repo's shape:
```bash
python3 MW4COMPARE/tools/restructure.py /home/rich/Repositories/FS_Build_V4H_extracted --apply
```
```
FS_Build_V4H_extracted/
Content/ same shape and casing as Gameleap/mw4/Content
Resource/Variants/<name>/ 797 saved loadouts (records have no path)
Resource/Pilots/Tesla/options/
```
Path components are re-cased against the repo wherever a counterpart exists, so
`mechs/longbow/longbow.data{element}` became
`Content/Mechs/Longbow/longbow.data{Element}`. Components with no counterpart keep the
packer's spelling -- which is why the six new chassis sit at `Content/Mechs/champion`,
`dasher`, `griffin`, `jenner2c`, `marauder`, `thunderbolt` in lower case.
`_layout.tsv` records the originating package and entry name for every file, so the
per-package view can be reconstructed at any time.
**`FS_Ours_extracted` is deliberately left in per-package form.** Unpruned it still has 60
genuine collisions (mission-local copies of shared `Culturals/` props, e.g.
`arena_debris_large01.data` differing across Coliseum / Reduex / StormCanyon) plus 2,617
harmless duplicate paths. Flattening it would lose data, and it is regenerable in ~4 minutes
anyway. `restructure.py` refuses to run on a tree with collisions for exactly this reason.
### Source vs compiled -- only 502 of the 2,195 files are repackable
**A `.mw4` does not store the source tree.** Some records are the source file byte for byte;
others are what the packer produced from it. Which is which was established empirically by
hash-matching every record of *our own* packages against `Gameleap/mw4/Content`
(22,138 matched a source file, 28,728 did not):
| | |
|---|---|
| **repackable source** | `.tga` `.erf` `.mw4anim` `.bid` `.wav` `.bsp` `.material` `.abl` `.obb` `.ebf` `.bounds` `.animscript` `.fgd` `.tcf` `.mlr` `.d3f` `.gaf` `.script` `.h` `.abi` |
| **compiled** | `.data` `.instance` `.subsystems` `.damage` `.torso` `.engine` `.audio` `.video` `.lights` `.mw4`, and every `{hint}` `{handle}` `{gamemodel}` `{element}` `{footsteps}` `{zones}` `{nametable}` `[shadow]` `[joint_*]{armature}` `[joint_*]{sites}` record |
The decisive evidence: our source `Content/Mechs/Atlas/atlas.data` is **4,891 bytes of text**;
the record packed under that name is a **12-byte binary stub**. The real payload went into
the `{gamemodel}` / `{element}` records. So a mech's `.data`, `.damage`, `.subsystems`,
`.instance`, `.contents`, `.torso` and `.engine` **cannot be recovered from any package** --
they must be re-authored. `.armature` does not exist in a package either; the packer splits it
into per-joint `X.contents[joint_*]{armature}` records.
```bash
python3 MW4COMPARE/tools/split-source.py /home/rich/Repositories/FS_Build_V4H_extracted --apply
```
```
FS_Build_V4H_extracted/
Content/ 502 files repackable source, repo shape + casing - drop straight in
_compiled/ 1693 files packer output, reference only
Resource/ variants + pilot options (use the original .mw4 files)
```
Renames applied -- these genuinely are `.obb` files (`#BBO` magic, headers byte-identical to
ours):
| record | becomes |
|---|---|
| `champion.data{solidobb}` | `champion_skeleton_SOLID.obb` |
| `champion.data{hierarchicalobb}` | `champion_skeleton.obb` |
(`_SOLID.obb` / `.obb` outside `Content/Mechs`.) The real source spelling is declared inside
the `.data` text we don't have, so a convention was applied and logged in
`_source-vs-compiled.tsv` -- whatever `.data` gets authored later must reference these names.
What survives in `Content/`: 208 `.erf`, 154 `.mw4anim`, 73 `.tga`, 29 `.script`, 17 `.obb`,
15 `.wav`, 3 `.h`, 1 each `.animscript` `.abl` `.abi`.
The resulting `Content/Mechs/dasher/` now matches our own `Content/Mechs/Dasher_DNU/` naming
exactly (`das_hip.erf`, `dasher_cage.erf`, `dasher_skeleton.obb`, `dasher_skeleton_SOLID.obb`).
> For the six new chassis we therefore have **all geometry, animation and art**, but their
> definition files must be written by hand. Note we already hold complete source for two of
> them in our own repo: `Content/Mechs/Dasher_DNU` and `Content/Mechs/jenner2c`.
---
## 6. Reports
Written to `reports/` by `run-comparison.sh`. The two `manifest-*.tsv` files are ~5 MB
intermediates and are gitignored.
| file | contents |
|---|---|
| `packages-entrydiff.txt` | per-package entry-name diff, all 857 V4H packages vs our 102 |
| `core-decoded-diff.txt` | 433 real differences + 754 V4H-only entries in `core.mw4` |
| `props-decoded-diff.txt` | 419 real differences + 155 V4H-only in `props.mw4` |
| `textures-decoded-diff.txt` | 1 real difference + 299 V4H-only in `textures.mw4` |
| `maps-decoded-diff.txt` | per-map decoded diff (all <=1 real difference) |
| `missions-decoded-diff.txt` | per-mission decoded diff (`.contents` / `.lights` / freezer) |
| `hsh-diff.txt`, `hshoriginal-diff.txt` | HUD art trees |
| `content-diff.txt` | the 11 loose scripts + `mwconst.abi` |
| `resource-files-diff.txt` | package-file and loose-file level diff of `resource/` |
| `variants-inventory.txt` | 797 variants grouped by chassis with load status |
`reference/` holds verbatim copies of small, high-value V4H text files that are not in our
tree, so they survive independently of the `FS_Build_V4H` folder:
| file | source |
|---|---|
| `v4h-changelog.txt` | `firestorm build V4h.txt` -- their v4 -> v4h release notes |
| `Convert-hud.txt`, `Convert-radar.txt` | `hsh/hud/Convert.txt`, `hsh/radar/hud/Convert.txt` -- paper-doll authoring recipe, points at `coord.cpp` |
| `mechnames.script`, `buildmechnames.script` | the 71-entry mech name tables |
| `v4h-firestorm.h`, `v4h-firestorm64.h` | 20 console / 64 Dragon-MP callsigns |
| `v4h-options.h` | `USE_ALLOWED_MECHS`, `Battle_Ammo`, `USE_OBSERVER`, ... |
| `mwconst.abi.diff` | the `M_Champion..M_Griffin` / `LastMechID=70` delta |
| `options.ini.diff` | ours -> theirs |
Not copied (too large, read them from `FS_Build_V4H/content/shellscripts/mechbay/`):
`mechdescriptions.script` (74,883 B) and `old mechdescriptions.script` (74,438 B).
---
## 6b. Decompiling the compiled records
**Full write-up: [DECOMPILING.md](DECOMPILING.md).** Read that before touching any of this.
The compiled records are not a dead end. The engine source contains both the writer
(`MWTool::BuildResource` -> `MWObject::CreateSubsystemStream` -> `Tool::ConstructCreateMessage`
-> a per-class factory in one of 51 `mw4/Code/MW4/*_Tool.cpp`) and the reader, and the message
structs are plain PODs declared in our own headers. We also have a closed-loop test harness:
65 chassis where we hold both the source text and the compiled record.
| record type | status |
|---|---|
| `.armature` | **done and verified** -- 2,938 of 2,976 pages exact across 65 chassis, 1,849/1,849 child lists exact. All 38 residuals are data the packer itself discarded, so the game never sees them either. |
| `.subsystems` | **done and verified** -- 7,579 of 7,585 keys exact across 63 chassis. The 6 residuals are Behemoth/Behemoth II `GroupIndex`, changed in source by `e45a67a8` after the package was built. |
| `.data` `.damage` `.contents` `.instance` `.torso` `.engine` | same mechanism, not started |
```bash
python3 MW4COMPARE/tools/decompile/armature.py --verify # 65-chassis harness
python3 MW4COMPARE/tools/decompile/armature.py <recordDir> -o out.armature
```
`.armature` files have been generated for all six new chassis into
`FS_Build_V4H_extracted/Content/Mechs/*/`.
Two things found along the way that matter beyond this section:
* The extracted baseline reflects the **packages**, not `Content/` at HEAD. `core.mw4` and
`textures.mw4` have never been repacked since the initial mirror (`2b8ca921`); `props.mw4` is
current as of `8bfaf9b9`. Check this before concluding anything is missing. See
DECOMPILING.md section 8.
* `ResourceID` packs the package record id in its **high word**, which is how `Model=`
references resolve back to a path.
---
## 7. Merge assessment
Ranked by value/risk. **None of this has been done.**
### Low risk, high value -- take from V4H
1. **727 mech variants** that reference chassis we already have. Drop-in: they are
self-contained `.mw4` files in `Resource/Variants/`, keyed by chassis *name*, and our
chassis names match. Sanity-check a sample in the MechLab first -- their changelog warns
that a corrupt variant can corrupt a build.
2. **The 11 loose shell scripts** -- at minimum `mechdescriptions.script` (74 KB of mech
text we simply do not have) and the two mech-name tables. `firestorm.h`/`firestorm64.h`
callsign lists are a straight content lift.
3. **Freezer stadium crowd VO** (17 wavs) and the expanded `freezer_teamattrition.abl`.
4. **HUD/MFD/radar art for the six new mechs**, and the 24-bpp `Mechs/*.bmp` masters if we
want higher-quality portraits than our 8-bpp copies.
### Medium risk -- the six new mechs
The assets are all extractable from `core.mw4`/`textures.mw4`/`props.mw4`. The blockers are
on *our* side and are all code, not data:
* `render.cpp` `mechnames[]` -- append 6 entries (currently 65, `"dasher"` is commented out).
* `coord.cpp` -- MFD/radar paper-doll coordinate tables must gain 6 entries.
* `mwconst.abi` -- apply the `M_Champion..M_Griffin` / `LastMechID=70` delta.
* `ScriptStrings.dll` (`StringResource.rc`) -- add `DNL_CHAMPION`, `DNL_JENNER2C`,
`DNL_DASHER`, `DNL_MARAUDER`, `DNL_THUNDERBOLT`, `DNL_GRIFFIN`.
* MechLab scripts must learn about IDs 65-70 (this is what their +33 KB
`mechbay_main.script` is doing).
* `MW4Shell.cpp` / anything else with a hard-coded 65.
Because IDs 0-64 are identical, this is purely additive.
Extracting the mech *source* (rather than the packed records) is the open question -- the
records in `core.mw4` are the compiled forms, and we need `.erf`/`.obb`/`.data` sources to
rebuild `core.mw4` with our own packer. We already hold source for `dasher` and `jenner2c`.
### High risk -- do not merge blind
* **`.subsystems` for the 65 shared chassis.** All 65 differ. Taking theirs imports their
entire balance pass (3-ton ammo cap, revised weapon groups) wholesale; taking ours keeps
our balance. This is a *design* decision, not a merge decision.
* **Mission `.contents`.** Real, substantial edits on their side for every shared mission.
Needs a proper structural diff before any decision.
* **MechLab scripts.** Their MechLab is documented (by them) as crash-prone on save.
Do not import their `mechbay_*.script` wholesale.
### Not worth taking
`stats/zWorldStats` (dead 2001 telemetry), `RMTSHARE.EXE`, `uploadbuildv4h/` (777 MB
installer of the same content), `options.ini.rp411old`.
---
## 8. Open questions for the project owner
1. **Balance.** Do we adopt V4H's `.subsystems` (3-ton ammo cap, their weapon groups) for
the 65 shared mechs, keep ours, or merge selectively per mech--
2. **Mission layouts.** Their `.contents` differ for every shared mission. Is that
deliberate content work on their side (more drop zones? different props?) or drift we
should ignore--
3. **Mech source.** Is there a source tree for V4H anywhere, or is `core.mw4` the only copy
of the six new mechs? That decides whether we re-pack from source or have to write an
unpacker that reconstructs `.erf`/`.obb` from packed records.
4. **`bitdepth=32`.** V4H runs the console at 32 bpp; we run 16. Given the
`DWM8And16BitMitigation` dependency documented in the repo memory, is their build
actually rendering at 32 bpp, and should we try it--
5. **The 25 unregistered chassis on our side** (Canis, Gargoyle, razorback, reaver, ...) --
are these finished assets that were never wired up, or abandoned work? If usable, the
combined roster could be far larger than 71.
6. **Variant hygiene.** Their changelog warns that corrupt variants can corrupt the build.
Do we import all 727 loadable variants, or curate--
7. **`jenner_iic1`.** Three variants reference a chassis that exists in neither build.
Drop them, or re-point at `jenner_2c`--
---
*Generated 2026-08-08 against `FS_Build_V4H` (Oct/Nov 2021) and branch `5.1.0b-in-progress`.*
Reference in New Issue View Git Blame Copy Permalink