Debugging fickle crashes/bugs

Discussion in 'Discussion & Q&A' started by Pacca, Nov 2, 2018.

  1. Pacca

    Pacca Level 1 inflatable otter thing Member

    Jul 5, 2014
    Twinleaf Town
    The issue has been fixed now. As it turns out, the game assumes that the title cards will be active and will write to their object space, even if there is no object there! So removing the title cards causes junk data to load there, which makes the Game over/time over text malfunction! The old post is spoilered below in case your curious:

    I've run into a funny kind of issue that I've never really had to deal with before. Most issues are really easy to track down, particularly if they trigger a crash with helpful info. Most bugs happen consistently, and are linked to an obvious problem once you figure it out. I've recently been running into much, much worse problems to debug, however, and I have no idea where to even begin with them...

    For example, there's a crash that occurs when the player dies sometimes. It is extremely rare, but I've seen it happen several times myself and bug testers have reported it, so it's not a non-issue. It instantly negates any progress the player has made up to that point by stopping the game entirely and forcing a reset, so I can't exactly just ignore it either.

    Normally, this is the point where I'd start proper debugging. Determining what I can from the crash message itself, adding crash handler stuff to the most relevant looking code, narrowing down where the problem occurs, etc. etc.. But this only works on a problem that can be consistently triggered. The issue I'm stuck with is seemingly entirely random and extremely rare, so I can't even resort to git to track down versions that don't have the problem, as any of them might have it regardless of how much I playtest them.

    And the few things I can glean from the observations made thus far aren't particularly helpful, although they do lead me down a certain path.
    • The way you die does not seem to matter; Time Overs, death pits, Oil Oceans Oil, spikes, they all seem to be subject to it.
    • The game jumps partway through a random objects mappings and runs into illegal instructions there (obviously), so where the crash occurs isn't even remotely helpful, and all the stack tells me is that an object caused the issue (it goes back to the dynamic jump in "RunObjectLoop"). Normally, I'd use the value in a0 to pin down the exact object that caused it, but the value in a0 is always $FFFFB082, which isn't a valid object pointer (and the closest valid object pointer, $B080, is reserved for the title cards and will always be 0 when the crash occurs, so no object will run there).
    • It happens on multiple stages in various places, so I doubt that any single stage object could be responsible.
    • It does occur regardless of which character is being played, so second player tails isn't responsible for the crash, and it's likely that it's caused by a mess up that got copied between characters, either from vanilla code or code intentionally added to them by me. It seems most likely that the player objects are at fault, since very few other objects are able to run in the after death state. Although it's hard to tell whether the crash happens before or after said state occurs.

    It seems I've finely found a bug that's truly beaten me, I can't even think of a theoretical convoluted plan that could possibly nail it down...
    Last edited: Nov 2, 2018
  2. Clownacy

    Clownacy UP - ON - CPU Staff

    Aug 15, 2014
    Is d7 clobbered too? That would tell you the object's RAM slot index.

    Failing that, you could push the object's code address to somewhere in RAM right before you jump to it, and check that when the game crashes. It wouldn't take much RAM/CPU, so you'd be safe leaving it enabled for a while.

    But yeah, from my experiences with unreproduceable bugs, you're at their mercy. No way around it. The best you can do is just be prepared.
    Last edited: Nov 2, 2018
    Pacca likes this.