Holy shit. So the short news is we actually figured it out, but man the journey here was painful. I’ll give some of my own thoughts about the process here in a minute. But for the start, here’s Claude with the technical deep dive.

Take it away, buddy! (I’ll add a few notes along the way)

Claude with the technical deets

The Problem in One Sentence

Metal Gear Solid Integral (PSX) stores the byte-length of each codec call’s GCL script payload in a 2-byte big-endian field, capping any single call at 65,535 bytes. Translating the Integral staff calls (freq 140.07) from Japanese to English blows past that limit because each call contains 60–91 random trivia items in RND_OPTN blocks, and English text is simply longer.

What Needed to Change — and Why Both Sides Had to Move Together

The limit lives in two completely separate places that have to match:

The PSX executable (SLPM_862.47) — the GCL interpreter in radiomes.c reads the size field at runtime to know how big the call block is.
The Python recompiler (RadioDatRecompiler.py) — it writes those size fields when building a new RADIO.DAT from translated XML.

If you change one without the other, the game reads garbage. A 4-byte recompiler output fed to the original 2-byte-reading executable, or a patched executable given the old 2-byte files, both corrupt the call block pointer arithmetic.

The Binary Layout We Were Working With

Every codec call in RADIO.DAT has this header structure:

[freq 2B] [unk1 2B] [unk2 2B] [unk3 2B] [0x80] [size field] [GCL payload...] [font data...]
 bytes 0-1  2-3       4-5       6-7      byte8    bytes 9+
                                          ^pScript[0]

pScript is the pointer the game passes into menu_gcl_exec_block_800478B4. pScript[0] is the 0x80 marker (RDCODE_SCRIPT). The size field immediately follows, and the GCL payload follows that. After the payload comes the font/face tile data, which the codec task locates by computing fontAddrOffset = totalSize + 1.

Inside the GCL payload, every command uses the same structure:

[0xFF] [cmd_byte] [size field] [command payload...]

The same size field expansion had to happen at both levels: the outer call block, and every inner command.

The C Patch — `radiomes.c`

Before (original decompilation)

const int totalSize = ((pScript[1] << 8) | (pScript[2]));
unsigned char *ptr = pScript + 3;
// ...
size = (ptr[2] << 8) | ptr[3];
ptr += 4;  // skip 0xFF + cmd_byte + 2-byte size
// ...
ptr += size - 2;  // advance past payload (size includes its own 2 bytes)

The call returns totalSize + 1 + pScript — a pointer to the byte just past the call block.

After (patched)

const int totalSize = ((pScript[1] << 24) | (pScript[2] << 16) | (pScript[3] << 8) | pScript[4]);
unsigned char *ptr = pScript + 5;
// ...
size = (ptr[2] << 24) | (ptr[3] << 16) | (ptr[4] << 8) | ptr[5];
ptr += 6;  // skip 0xFF + cmd_byte + 4-byte size
// ...
ptr += size - sizeof(int);  // sizeof(int) == 4 on MIPS PSX

The sizeof(int) idiom is intentional — the stored size value includes the size field itself, so subtracting 4 gets you the payload length. The return value totalSize + 1 + pScript is unchanged; totalSize now reads 4 bytes, so it naturally handles values over 65535.

radio_moveToNext_80047880 (which skips over a call block without executing it, used in IF/switch branch-skipping) already used load_big_endian_int — a 4-byte macro — so it was automatically correct once the size field in the binary is 4 bytes.

The font address calculation in menu_radio_codec_task_proc_80047AA0 also reads radioDatIter[1..4] as a 4-byte value to compute fontAddrOffset, which means it too was already reading 4 bytes.

No other code paths in radiomes.c needed changes. The per-command dispatch table in exec_block is the only place that interprets the inner command sizes, and all command handlers receive a pScript pointer past the size field, so they’re size-field-agnostic.

The Recompiler Patch — `RadioDatRecompiler.py`

`createLength()`

This is the central function that all command builders call:

def createLength(payload_size: int) -> bytes:
    if USE_LONG:
        return struct.pack('>L', 4 + payload_size)   # 4-byte: includes own 4 bytes
    else:
        return struct.pack('>H', 2 + payload_size)   # 2-byte: includes own 2 bytes

The stored value is self-inclusive — size in the C code is the number of bytes from the start of the size field to the end of the payload. This is why the C patch subtracts sizeof(int) after reading size to get the payload length.

`USE_LONG` flag

Set by the --long CLI flag. When True, every call to createLength() throughout the recompiler emits 4-byte size fields. This covers:

The outer 0xFF cmd size fields for SUBTITLE, VOX_CUES, ANI_FACE, ADD_FREQ, MEM_SAVE, ASK_USER, IF_CHECK, ELSE, ELSE_IFS, RND_SWCH
The inner 0x80 size blocks inside VOX_CUES, IF_CHECK, ELSE, ELSE_IFS, RND_OPTN
The top-level call header 0x80 size field

Opaque blobs (`getContentBytes`)

FF06 (MUS_CUES), FF08 (SAVEGAME), and FF40 (EVAL_CMD) are stored as verbatim content hex blobs because they contain no translatable text. When USE_LONG=True, getContentBytes rewrites their 2-byte outer size field to 4-byte:

cmd_byte = blob[1:2]
old_sz = struct.unpack('>H', blob[2:4])[0]
payload_len = old_sz - 2
payload = blob[4:4 + payload_len]
return b'\xff' + cmd_byte + createLength(payload_len) + payload

Call header rebuild

When USE_LONG=True, the 11-byte opaque content blob for the call header is replaced with a freshly assembled header from named XML attributes:

callHeader = freq + unk1 + unk2 + unk3 + b'\x80' + createLength(len(elemContent))

This is necessary because the original 2-byte size bytes in the content blob would be wrong.

The Round-Trip Pipeline and Why We Needed It

Before any of this was meaningful, we needed a verified extract → XML → recompile pipeline that could reproduce the original binary byte-for-byte. Without that, we had no way to know whether a test failure was our change or a latent bug in the recompiler.

The test-roundtrip.sh script was the ground truth. It runs the extractor on the original binary, then the recompiler on the XML, then shasum -a 256 to compare. We didn’t move forward on the 4-byte work until both the jpn-d1 and integral-d1 baselines passed cleanly.

Getting there required fixing several bugs that had been masked by luck (the test never exercised certain call structures) or were simply incorrect without being detectable from the output hash alone.

The Bugs — The “Chaos” Part

Bug 1 — EVAL_CMD (FF40) length scanner (extractor)

The case b'\x40' branch in RadioDatTools.py used a manual scanner to find the end of an FF40 command because someone thought FF40 had a non-standard terminator. The scanner condition was:

if data[i] == b'\xff' or b'\x00':

Python operator precedence: b'\x00' is a truthy bytes object, so X or b'\x00' is always True regardless of X. The scanner always fired immediately, then roamed the entire file looking for the 14 31 00 terminator sequence, landing on far-off false matches. Short FF40 commands (under 16 bytes) got lengths in the hundreds or thousands of bytes. This blew past their containing element’s budget and corrupted all downstream elementStack accounting — every subsequent element in the call was mis-parsed.

Fix: FF40 uses a standard outer size field, same as every other FF command. Replaced the scanner with getLength(offset).

*J-Rush’s note: Originally i think i just found based on analyzing the USA/JPN version these were of static length, so i just used the raw hex to be re-inserted. After changing it we determined Integral had some different sized eval statements, so thats why this occurred here.

Overall fixing a bug like this makes the extraction and recompiler more robust overall.

Bug 2 — `getGraphicsData` blank tile handling (extractor)

Each call can define custom character tiles: 36-byte chunks stored consecutively after the GCL payload. The extractor scanned for tiles until it hit a null run longer than 36 bytes, treating that as sector padding.

Some calls have consecutive blank tiles — legitimate all-zero 36-byte tiles between real ones. The extractor stopped scanning at the first blank tile run, truncating the tile list.

Fix: when end_limit is known, peek ahead after a null run. If non-null data follows within the budget, the blank run is valid tile data, not padding. Continue scanning.

J-Rush: This is a misnomer, but we’ll talk about it later.

Bug 3 — Custom character index off-by-one (encoder)

translateJapaneseHex (the decoder) stores custom char tiles as a 1-based array — index 1 is the first tile. encodeJapaneseHex was computing the index as 0-based, so it wrote 96 00 for the first tile, 96 01 for the second, etc. The decoder would read those as “tile 0” and “tile 1” — one slot behind. Every call using custom chars displayed the wrong character.

Fix: +1 on the index in the encoder, matching the decoder’s 1-based convention.

J-Rush: I thought i fixed this! But i think the fix was in a different branch. Overall I have this project in maybe 4 places…. Here we have it once for the patch and another time for the integral staff fix working area. Then, I have it in my main translation project folder, and lastly there’s one more copy for the GUI branch. Overall this could be much cleaner, but we’ll get there.

Bug 4–10 — Stale size bytes throughout the recompiler

The original recompiler was written with the philosophy “pass through the original content blob for anything structural, re-encode only the translatable text.” That works if the text re-encoding never changes byte counts — but it doesn’t hold:

SUBTITLE re-encoding changes byte counts due to encoding normalisations (C0→80, C1→81, C2→82 prefix normalisation; #T{...}# furigana marker encoding differences).
IF_CHECK, ELSE_IFS contain SUBTITLE commands as children. If the child bytes change size, the IF outer and inner size fields — read from the stale content blob — are wrong.
RND_SWCH outer size and totalWeight field were stale. If RND_OPTN options were added or removed (which is exactly what we need for translation), totalWeight would be wrong.

Each command type required its own fix: extract the named fields into XML attributes, rebuild the binary from those fields + fresh child bytes, compute sizes from scratch with createLength(). This was pipeline changes P1 through P7, each with a round-trip hash update to verify.

The full list: VOX_CUES (P1), ANI_FACE (P2), ADD_FREQ (P3), MEM_SAVE (P4), ASK_USER (P5), IF_CHECK/ELSE/ELSE_IFS (P6), RND_SWCH/RND_OPTN (P7).

J-Rush’s Interlude:

Okay, I’m going to jump in and take over for a bit. Because I said I’d come back to the blank graphics tile thing, but Claude didn’t detail this little tidbit.

That ended up to be misleading, because silly me I forgot the Staff call frequency is not the only new frequency in Integral. There is also the “healing radio” originator, 140.66 which plays some notable tunes. Well, only 3, but it’s a cool rendition of “Theme of Solid Snake” from Metal Gear 2, but in the MGS sound font.

Not as amazing as J-War’s remix, but I give it a solid couple foot taps.

Still, so i was staring at some of the invalid graphics, and i realized they are indeed an actual call failing to be parsed….. turns out without that frequency we were ignoring that data. So a quick add to the frequency list, and of course it started identifying the characters.

Still, nothing would be as rough as trying to get this frankensteined binary to function. It comes back to an early oversight I really should’ve taken more seriously.

Emulator woes

So first off, doing dev work on a Mac can suck hard. I’ve moved my workflow over after investing in a Mac Studio, since it excels at LLM performance, but I run into tons of issues with various things:

Apple holding back versions of python, and outdated libraries. This can be fixed with brew and venv, but man it was a pain getting adjusted.
Most tooling is made for windows (and some linux folks out there). So we also had to make some adjustments to Foxdie_team’s run script to properly run the Macos binary of PCSX-Redux.
Further, the emulator crashes if I try to output the CPU instructions. I mean you cna see them but stopping it or copy pasting is a crash.

So with my heart in my throat I compile the game… tons of stages aren’t matching the scan against offsets, but no matter. We’re successfully generating the Dat files, so let’s go off to the races.

The dev version not only has overlays built in, it starts at a debug menu, which is ideal for quick tests. This way I can test the dock (stage d00a) to see the first codec call, 140.85. I’ve mentioned this was the one I started with, mostly because it’s simple. One voice clip, subtitles, and saving the frequency to codec memory. Easy.

Only…. it hung.

Okay, not totally unexpected. Claude went back and found various issues with our recompiler and with the binary adjustments. After everything looked confirmed fixed, I still couldnt figure out what was going on.

Debugging was a painful process, as I had to rely heavily on Claude to not only build the debugger script, but to know what the CPU was spitting out. As far as I can tell, we hit an error, looped awhile, and then threw an error. Only we got so far down the stack that the error we threw wasn’t even relevant to what was happening.

Near as we could tell, though, we weren’t even hitting the break points we set. So the issue was elsewhere, even though we didnt modify anything else.

I regret having Claude spin for awhile trying to piece together what was going on in the CPU instruction log. The motto is work smarter not harder, right? So. I came back yesterday with a couple new approaches.

Review source code further, see if there were any points at which we were looking for an offset we forgot to adjust to 4-bytes. Claude did a thorough analysis, but we were coming up empty.
Alright, maybe there was an issue with the overlays? Claude insisted no, but i tried the retail build just in case. That one actually failed to even start. So more work for the future…
Claude added some print statements to some of the areas where we thought we were hitting the disruptions to check the bytes being read, since stopping there was damn near impossible if we never reach the break points we were expecting.

Lastly I figured ok, let’s go back to those stage issues. At first i had mixed up the builds, so I had been running the japanese RADIO.DAT against our recompiler fixes, so of course it broke getting recompiled with the integral STAGE.DIR.

But thats part of why i wrote off the stage errors, so i figured time to come back. Claude assisted writing some debugging output. And while we expect some errors (stage s00aa is an unused beta dock stage, still included in some versions) it was just spitting out far too much.

Claude quickly ran the tooling to compare to the stage.dir offsets we pulled. I even added something i was thinking about, which was to classify the ones we find based on which stage we’re in. We can utilize the stage table of contents to map out which stage a call entry is from.

Lo and behold, nothing was matching, so I gave Claude a theory to run with. Integral’s RADIO.DAT is the only version where Radio calls are aligned to 0x800 bytes. So, it hit me… Maybe they patched Stage / GCX language to instead use the block offset.

Claude quickly adjusted its checks and bam… 100% matches!

Holy shit, could that really have been it? So we’ve added a few adjustments to the recompiler:

Instead of storing byte padding, we re-insert null padding until we hit a 0x800 block, regardless how many we had before.
We make the integral flag require the padding.
Lastly, the integral flag will use a block-based offset style when we replace offsets in STAGE.DIR.

And with that, long ints are not just possible, but working!

There remain two outstanding issues:

Will there be enough memory space for some of the egrigiously long calls? Initial reports were that there was a lot of memory space to spare but we shall see.
We’ll have to manually patch this into the Japanese version, since the binaries don’t match. That said I don’t think its a huge issue yet… there are 3 places to make the patch. And if we’re forced to use a jump instruction, I’m sure we can make it efficient.
Oh right, the retail version still won’t boot. Might be for the next bullet, but we’ll have to figure that out.
Lastly… if any overlays changed, they will also need to be injected into STAGE.DIR.

That said time will tell how much longer it takes for the rest. But we may be able to patch Integral’s staff calls and be done with that version soon!

Now back to Claude with an overview of some further next steps…

Current State

Both sides of the change are implemented:

radiomes.c in the mgs_reversing decompilation reads 4-byte size fields throughout menu_gcl_exec_block_800478B4 and radio_moveToNext_80047880.
RadioDatRecompiler.py writes 4-byte size fields when invoked with --long.
The recompiler rebuilds call headers from named XML attributes when USE_LONG=True, rather than passing through the stale 2-byte header blob.
The round-trip pipeline passes byte-for-byte on the original (2-byte) format for both jpn-d1 and integral-d1.

What hasn’t been tested yet:

End-to-end: compile the patched executable, run it in an emulator with a --long-compiled RADIO.DAT containing a call that exceeds 65535 bytes, and verify the call plays correctly.
The three encoding allowances flagged as needing in-game verification (C0→80 normalisation, #T{...}# furigana marker byte encoding, kanji vs. custom dict priority) haven’t been tested in a running game.

A couple last notes (J-Rush)

There are a couple other things I will need to review. I think text injection was wrong. I was seeing straight ASCII in the hex editor show \x00 hex escape characters where injected dialogue was. We’ll have to verify it was inserted properly.

That said, I want to continue working to adjust some tooling for inserting dialogue. We can fix the length of a call. But for subtitles themselves we’re limited to 4 lines and if the single Japanese line conveys too much, we need to insert more. Thats not as easy as it seems, given we get the timings from VOX. Which means, we need to edit both simultaneously.

I actually had Claude run forward with some progress for a GUI editor. I want to come back to that when it’s more polished, but it certainly accelerated my work on it. I was having a hard time finding motivation to keep working with the GUI; especially as it’s a lot of work for little payoff.

Still, I want to get it finished and let people loose on it to see what people think. Many people want to do the translation for their own means (I get a lot of asks for different localizations of the English version into other european languages). Having this set would make a lot of people not comfortable with the command line happy.

I’m still torn on if I’d monetize the gui. Leaning towards not more and more. But I also am excited by some of the opportunities Claude might provide me to get some apps written that might actually be of some use to other people.

We’ll see. In the meantime, there’s a lot ot look forward to! I am hoping maybe in the next month or two, I might be within reach of a public beta of Disc 1. It would likely only include story portions, (as the radio dialogue is VERY dense and will take most of my time) but it would accelerate my work if I had other people doing some beta testing for me.

But honestly the future for this project is looking very, very bright!