Incredibly Ambitious SMB 3 Hack Released After 12 Years

theseus's copyright infringement

These patch files are supposed to be diffs between the final version of the ROM after your modifications, against the original upstream ROM you started with. Ideally, they're only supposed to contain the code and assets you upserted into the game yourself.

And if you don't move anything around, then that's all they will contain.

But if you do start moving things around — sliding sections of code or data forward in memory and fixing up references to them so as to make more room for new stuff — then you run into the problem, at release time, that an .IPS file is literal binary bytewise diff. So, by moving a section forward by one byte, the .IPS file will now contain a complete copy of that section, in order to overwrite [SECT_START+1:SECT_START+N+1] with [SECT_START:SECT_START+N].

What is needed, to avoid this, is an alternative patch format, that doesn't just blindly say "overwrite [X:Y] with [binary]" but rather says something like (a binary encoding of) the following "data-unpacking command language" (here as pseudo-Elixir code):

    expected_output(size: 1000, sha256: <<0x012345...abcdef>>)
    require_external_data_source(id: :upstream, size: 1000, sha256: <<0x6789ab...beefee>>, description: "Super Mario World (U) v1.0 SFC ROM image")
    embed_data_source(id: :patch, size: 100, sha256: <<0x777777...cafefe>>)

    write_at(offset: 0, constant_u8(<<0x00>>, repeat: 100))
    write_at(offset: 100, embedded_data_slice(:patch, offset: 0, len: 100)) # some patch data
    write_at(offset: 200, external_data_slice(:upstream, offset: 0, len: 400)) # then some source data
    write_at(offset: 204, constant_u32(<<0xaabbccdd>>, repeat: 1)) # fixup one little bit of the copied source data

    embed_data(:patch, compression: :deflate, data: <<...>>)

...where the patcher tool would ask you for "Super Mario World (U) v1.0 SFC ROM image"; check that the file you provide matches the given SHA; allocate a 1000-byte output buffer; run through the write ops against the output buffer; validate that the output buffer now matches the given SHA; and then write out the output (or pass the pointer to the buffer back to the caller, if being used as a library.)

I'm not sure why no patchfile format with these semantics exists. It'd be relatively straightforward to write a library that computed such patch-op sequences from one or more infiles + one outfile, using (essentially) LZ77 to discover shifted partially-overwritten repeats, and gradient-descent optimization over alternative equivalent formulations to find minimal such descriptions.

---

Alternately, though, maybe we just shouldn't be making arbitrary low-level byte-delta patches to opaque ROM images in the first place!

IMHO we're at a point now with understanding the entire libraries of most of these old systems, that we could very well write a single program for each console that:

1. disassembles a source ROM into sections (individual assets, plus position-independent-code -ified assembly);

2. patches the text sections with git-commit-like textual diff hunks over the PIC ASM (where the tool could very well extract these from a git worktree!);

3. replaces assets wholesale with new per-asset binaries (that can be different sizes, or with different numbers in lists, or with individual assets in different formats — in which cases, the code that loads them will be automatically modified to compensate);

4. and then reassembles the ROM, computing all the fixups in the process.

Given the current state of whole-library game preservation (we've had GoodTools for a good 20 years now); and given the experience that game modders now have with building "perfect" build pipelines for game decompilation projects, that build the original ROM image... we should (IMHO) be able to set up these "high-level patchers" in such a way as to guarantee that each tool works on 100% of known titles, to losslessly reproduce the input ROM after a disasm+reasm pass.