This is a post on a bug that broke Fatal Rewind for the Sega Genesis, a port of the Amiga/Atari ST game The Killing Game Show.
I’m not sure why some Amiga-to-Genesis ports changed the game’s title. Leander is another one, which got ported to Genesis under the title Galahad. Marketing thing I guess.
The bug is related to emulation of VDP interrupts, so first, an overview of how those work.
Background: VDP Interrupts
The Genesis VDP (video display processor) exposes two interrupts to software running on the 68000 main CPU: VINT (vertical interrupt) and HINT (horizontal interrupt).
VINT fires once per frame at the beginning of the vertical blanking period (VBlank). This marks the end of the frame and is almost always what software uses for timing overall game speed, as well as any graphical updates that the game needs to make between frames (e.g. scrolling the backgrounds or changing sprite positions).
HINT fires once every N lines at the beginning of the horizontal blanking period (HBlank), with a configurable N. Software can also change N mid-frame to make HINT trigger on specific lines. This is useful for timing mid-frame raster effects without needing to poll the current line counter. Though some games poll the counter anyway because 68000 interrupt handling is rather slow, and that eats up valuable cycles if you’re doing an effect that requires making a change on every line.
VINT is raised as 68000 auto-vectored interrupt level 6 and HINT as level 4, so when the VDP is raising both simultaneously, VINT has priority over HINT.
Software can mask these interrupts on both the 68000 side and the VDP side. On the 68000 side by using the interrupt mask bits in the status register, and on the VDP side by setting/clearing interrupt enabled bits in VDP registers.
The VDP interrupt enabled bits are the part that’s relevant to this post.
Software can write to VDP registers by writing to the VDP’s 16-bit control port at $C00004 or $C00006. The control port is mapped to two consecutive word addresses so that games can use 32-bit longword write instructions (MOVE.L) to perform two control port writes in a single instruction, which is a little faster than two 16-bit word write instructions.
(The 68000 has 32-bit registers and 32-bit memory read/write instructions, but only a 16-bit data bus, so it implements 32-bit memory accesses by splitting them into two consecutive 16-bit accesses.)
The full details of how the control port works are a bit complex, particularly in how it interacts with the VDP data port and VDP DMA, but the only part that’s relevant to this post is how to use it to write to VDP registers: write a value where the highest two bits are 0b10, i.e. value & 0xC000 == 0x8000. Bits 8-12 specify the 5-bit register number and the lowest 8 bits contain the 8-bit register value:
The VDP has about two dozen registers, but only two are relevant to this post: registers #0 and #1, the “mode set” registers. Among other things, these registers contain the interrupt enabled flags: register #0 the HINT enabled flag (bit 4) and register #1 the VINT enabled flag (bit 5).
VDP registers #0 and #1
For example, if you wanted to enable HINTs, you could execute this instruction:
|
|
Bit 4 enables HINTs, and Sega’s documentation says bit 2 should always be 1. (Setting bit 2 to 0 enables an undocumented low-color mode that nothing uses.)
Similar for VINTs, you could execute this instruction:
|
|
Bit 6 enables display, bit 5 enables VINTs, and again Sega’s documentation says bit 2 should always be 1. (In this case, setting bit 2 to 0 puts the VDP in “Mode 4”, aka Sega Master System mode.)
Very importantly, the VDP has internal VINT and HINT pending flags that it sets independently of whether interrupts are enabled. It always sets the VINT pending flag at the beginning of VBlank, and it always sets the HINT pending flag on the line where its internal HINT counter reloads. The interrupt enabled flags only function as masks for these pending flags when deciding whether to assert the 68000 interrupt lines; they have no effect on whether the pending flags get set or cleared.
Once set, these pending flags can only be cleared by the 68000 handling and acknowledging the interrupt. There is no way for software to manually clear the interrupt pending flags, unlike on Sega Master System and Game Gear.
This means that if a game leaves VINTs disabled for more than a frame, when it next enables VINTs the 68000 will immediately receive and handle a VINT, assuming it’s not masking out level 6 interrupts on the 68000 side. Same thing happens if a game enables HINTs while the internal HINT pending flag is set, only with a level 4 interrupt instead of level 6.
First Attempt
Alright, let’s see what happens when we try to run this game.
It starts up and shows the EA logo animation, looks fine:
Unfortunately, immediately after this, it freezes - hangs forever at a black screen.
I have a few warning/error logs that are enabled by default because they indicate that something extremely unexpected happened, and, well:
address error triggered while handling illegal opcode exception (opcode=FFFF, address=FFFFFFFF)
Encountered 68000 address error; address=FFFFFFFF, op_type=Jump
address error triggered while handling address error; CPU is now frozen (address=FFFFFFFF)
It first executed an illegal opcode (!?), then it triggered an address error trying to jump to the exception vector (!?!?), and then it triggered another address error while trying to jump to the address error exception vector? Something has gone horribly wrong.
Brief Tangent: 68000 Address Errors
The 68000 does not support unaligned memory accesses. Performing a 16-bit or 32-bit access to an odd address triggers an address error exception.
Address errors are usually impossible for software to recover from due to some quirks regarding what PC value the 68000 writes into the exception stack frame. Even if software has an address error exception handler, it can’t reliably figure out where to jump back to after handling the exception. In an emulator, if a game triggers an address error, 99% of the time that indicates an emulator bug.
I have only ever seen one game trigger an address error that wasn’t caused by an emulator bug: Popful Mail on Sega CD. It triggers an address error if you run it with unformatted Sega CD backup RAM, which causes the game to crash into this error screen right after the title screen:
Given that no games depend on address errors to function correctly (assuming formatted SCD backup RAM), some emulators don’t emulate address errors at all due to the performance cost of checking the address on every 16-bit and 32-bit memory access. However I’ve found them quite handy for making games crash quickly and obviously when things go horribly wrong due to some emulation bug. Speaking of which…
What Happened?
Back to Fatal Rewind, execution logs leading up to the crash:
0224B0 move.w #0x8014, (0xFFC00004) ; VDP control port (register #0)
INT4
028AA0 tst.w (0xFFFFE048) ; working RAM
028AA4 beq 178 ; $028B56
028B56 move.w #0x9295, (0xFFC00004) ; VDP control port (register #18)
028B5E clr.w (0xFFFFB38A) ; working RAM
028B62 move.w #0x8AFF, (0xFFC00004) ; VDP control port (register #10)
028B6A move.w #0x8700, (0xFFC00004) ; VDP control port (register #7)
028B72 rte
0224B8 move.w #0x8164, (0xFFC00004) ; VDP control port (register #1)
INT6
028A94 move.l a1, -(sp)
028A96 move.l (0xFFFFE03E), a1 ; working RAM
028A9A jsr a1
000000 illegal ; 0xFFFF
line F
address error
address error
68000 execution logs
It first enables HINTs (register #0), which immediately triggers an HINT because the HINT pending flag is set. Its HINT handler writes to some VDP registers, but nothing that really seems relevant: it changes the window vertical position (register #18), the HINT interval (register #10), and the backdrop color (register #7).
After its HINT handler returns it then enables VINTs (register #1), which immediately triggers a VINT. Its VINT handler reads an address from RAM and jumps to that address, which seems reasonable, except it reads $000000 which is not so reasonable. $000000-$000003 is the location in ROM containing the 68000 initial stack pointer, so it’s definitely not a sensible address to jump to.
Clearly the game isn’t supposed to jump to $000000. Either the VINT handler is supposed to read a different value from RAM here, or the VINT handler is somehow not supposed to execute at all at this point in the code. All of this code is in cartridge ROM ($000000-$3FFFFF) so it’s not like the game could have corrupted its own code.
When the game crashes, the most recent write to $E03E in RAM is from this loop, which is just zeroing out a bunch of RAM:
|
|
Assuming that the game was supposed to read a non-zero value from RAM, this is the kind of emulation bug that is really difficult to debug. Some code that should have executed didn’t, but who knows when or where or why? Or maybe it was the opposite, that some code executed that shouldn’t have? Or maybe two pieces of code executed in the wrong order due to an inter-component timing issue? There are a huge number of possibilities for where execution could have diverged from actual hardware.
Thankfully, that assumption is false! The issue is not that the game expects its VINT handler to read a non-zero value - it’s that the game doesn’t expect its VINT handler to execute at all here.
This theory was fairly easy to test by making a change to VINT pending flag behavior: make it so that disabling VINTs clears the pending flag, and then never set the pending flag while VINTs are disabled. That change alone seems to fix this game:
However, this altered pending flag behavior is definitely not accurate to actual hardware, and there are other games that depend on the accurate behavior. Why doesn’t the game’s VINT handler execute here on hardware?
Interrupt Delay
Fatal Rewind actually depends on two different VDP interrupt quirks to work correctly. I’m going to start by briefly covering a different game that depends on only one of those quirks, Sesame Street: Counting Cafe.
Without emulating the quirk that it depends on, this game hangs forever at a black screen without ever displaying any graphics. It gets stuck in this loop:
|
|
It’s waiting for some value in RAM to change, which is pretty typical when a game wants to spinloop waiting for an interrupt to trigger - it will poll a RAM value that it expects its interrupt handler to modify. Problem is, the game has both VINT and HINT disabled here, so no interrupt is ever going to trigger while it’s in this loop.
Here’s more context leading up to where it gets stuck:
|
|
Its VINT handler ultimately disables VINTs again before it returns, so they’re no longer enabled by the time it gets into the loop.
The behavior that this depends on (along with some other details on interrupt behavior) is documented on the spritesmind forums by the author of BlastEm, which is how I figured out what’s wrong here: https://gendev.spritesmind.net/forum/viewtopic.php?t=2202
In summary, when a game enables VDP interrupts while the corresponding pending flag is set, there’s effectively a 1-instruction delay before the 68000 begins to handle the interrupt.
The VDP begins to assert the 68000’s IPL interrupt lines almost immediately, but the 68000 latches external interrupt state mid-instruction, before it performs the memory write that enables interrupts. It won’t see the interrupt lines change until it latches interrupt state again in the middle of the next instruction, and then it will handle the interrupt after that instruction completes.
There’s a noted exception for MOVE.L instructions where the 68000 latches interrupt state between the two word writes, but even for those the 68000 will latch interrupt state just slightly before the VDP asserts the interrupt lines due to the small latency between the VDP receiving the write and asserting the interrupt lines.
Given that, Sesame Street should actually handle VINT after the CLR.B instruction, not immediately after it enables VINTs:
|
|
Executing that CLR.B instruction before handling VINT causes the game’s VINT handler to go down a different code branch where it does not disable VINTs before it returns, so eventually a second VINT will trigger during the loop, and after that the game will move on and finish booting.
Sesame Street only depends on this delay for VINTs, but Fatal Rewind depends on it for both VINTs and HINTs.
After stumbling through a few possible implementations, I eventually emulated this by adding a 1-instruction delay on all changes to the VDP interrupt enabled flags. This is not exactly how actual hardware behaves, but it was much simpler to implement than polling interrupt state mid-instruction given the timing implications of doing that.
This fixed both Sesame Street and Fatal Rewind for me, though I think it’s worth mentioning that Fatal Rewind also depends on another behavior that’s maybe not obvious.
Interrupt Acknowledge Confusion
There’s another quirky VDP interrupt behavior that is documented in one of Sega’s official supplements, though the description here is a little confusing (badly translated maybe):
When the 68000 acknowledges a VDP interrupt, it doesn’t explicitly tell the VDP which interrupt to acknowledge - it only asserts a line (VPA) that signals the VDP to acknowledge an interrupt. The VDP will acknowledge the highest-level interrupt that it’s currently raising.
This creates a possible edge case where the VDP begins raising VINT (level 6) between the 68000 beginning to handle an HINT (level 4) and the 68000 sending the interrupt acknowledge signal. When this happens the VDP will acknowledge VINT, not HINT, and then the 68000 will handle the HINT a second time immediately after returning from its HINT handler.
If this isn’t emulated properly, Fatal Rewind’s execution trace will look like this, where VINT immediately interrupts the HINT handler and the game still crashes:
0224B0 move.w #0x8014, (0xFFC00004) ; VDP control port (register #0)
0224B8 move.w #0x8164, (0xFFC00004) ; VDP control port (register #1)
INT4
INT6
028A94 move.l a1, -(sp)
028A96 move.l (0xFFFFE03E), a1 ; working RAM
028A9A jsr a1
000000 illegal ; 0xFFFF
line F
address error
address error
Sega’s supplement describes the edge case in the context of a game executing a very long instruction (e.g. multiplication or division) right before an HINT on the final line of the frame, but it can also happen when a game enables HINTs then VINTs with two consecutive instructions, as Fatal Rewind does here.
The 68000 begins to handle HINT because that’s the state it latched in the middle of the second MOVE.W instruction, but the VDP is raising VINT by the time it receives the interrupt acknowledge, so the VDP acknowledges VINT instead of HINT.
The execution trace is supposed to look like this:
0224B0 move.w #0x8014, (0xFFC00004) ; VDP control port (register #0)
0224B8 move.w #0x8164, (0xFFC00004) ; VDP control port (register #1)
INT4
028AA0 tst.w (0xFFFFE048) ; working RAM
028AA4 beq 178 ; $028B56
028B56 move.w #0x9295, (0xFFC00004) ; VDP control port (register #18)
028B5E clr.w (0xFFFFB38A) ; working RAM
028B62 move.w #0x8AFF, (0xFFC00004) ; VDP control port (register #10)
028B6A move.w #0x8700, (0xFFC00004) ; VDP control port (register #7)
028B72 rte
INT4
028AA0 tst.w (0xFFFFE048) ; working RAM
028AA4 beq 178 ; $028B56
028B56 move.w #0x9295, (0xFFC00004) ; VDP control port (register #18)
028B5E clr.w (0xFFFFB38A) ; working RAM
028B62 move.w #0x8AFF, (0xFFC00004) ; VDP control port (register #10)
028B6A move.w #0x8700, (0xFFC00004) ; VDP control port (register #7)
028B72 rte
0224C0 jmp 0x02B79C
02B79C jsr 0x022A12
...
No INT6, at least not that the 68000 sees. Instead it handles two INT4s.
The game then goes on to set $FFE03E to 0x028B74 before its VINT handler executes for the first time, so it jumps to an actual code address rather than $000000.
As an aside, this only accidentally worked with my implementation of delaying VDP interrupt enabled changes by 1 instruction. I split 68000 interrupt handling into two “instructions” for timing purposes, with the second “instruction” performing the interrupt acknowledge, so the VINT enabled delay had already passed by the time the 68000 acknowledged the interrupt.
Summary
Fatal Rewind depends on two non-obvious VDP interrupt behaviors due to enabling HINTs then VINTs with two consecutive instructions while both interrupts are pending:
- Since the 68000 latches external interrupt state mid-instruction, when a game changes VDP interrupts from disabled to enabled while an interrupt is pending, from software’s perspective there’s a 1-instruction delay before the 68000 handles the interrupt (Sesame Street: Counting Cafe also depends on this)
- When the 68000 sends an interrupt acknowledge signal to the VDP, the VDP acknowledges the highest-level interrupt that it’s currently raising, which is not necessarily the same interrupt that the 68000 is handling
Emulating a 1-instruction delay on changes to the interrupt enabled flags is enough to fix this in an emulator that doesn’t have completely accurate sub-instruction timings, as long as VDP interrupt acknowledges ignore this delay and use the most recently written flag values.