Advanced Error Handler and Debugger

Discussion in 'Utilities' started by vladikcomper, Apr 5, 2016.

  1. Pacca

    Pacca Having an online identity crisis since 2019 Member

    Joined:
    Jul 5, 2014
    Messages:
    1,168
    Location:
    Limbo
    I stumbled across a bug that makes the error message quite difficult to read in very specific circumstances:
     
    AURORA☆FIELDS likes this.
  2. vladikcomper

    vladikcomper Well-Known Member Member

    Joined:
    Dec 2, 2009
    Messages:
    404
    Location:
    Russia
    Pacguy, wow, that's some amazing find. It's actually the first time since the release (that I'm aware of, at least) someone managed to effectively crash the crash handler~

    Comments in your video are mostly correct. This is indeed error handler trying to decode corrupted label name, which causes it to hang. All symbol data is encoded and compressed to save space and due to the nature of my compression, the decompressor is really sensitive to random/incorrect data. However, I have all sorts of safe checks to avoid feeding it with wrong entries (v1 also implemented a rough counter just to avoid endless loops during decompression, but I had to drop it in v2 because I was out of space and registers). But there are intentionally added edge cases of course, to make finding symbols smarter of course.

    * * *

    Please try the hotfix of the error handler blob (from the attached archive), to see if it fixes this issue.
    If it does, congratulations, you've discovered quite a rare edge case!
    In fact, nearly impossible set of circumstances had to be met: 1) trying to resolve label (symbol) for a very specific offset in a specific 64-kb section past the ROM area, 2) so an incorrect block in the symbols data was loaded, 3) which had to be corrupted in a specific way to be able to point to a specific random data, 4) which had to have a certain sequence of bytes to manage to put symbol decoder in an endless loop. Just wow, you should've been really unlucky to pull that off.

    I believe it's your case, seeing the offset of exception is 872000D4, which error handler decodes to 2000D4, because M68K on MD has 24-bit address bus and upper 8 bits of address do not matter. Offset 2000D4 is just slightly after 2 MB of ROM space. In case your ROM is 2 MB, the offset points just slightly after the last block of your ROM. This triggers an edge case to help resolve offsets that may refer to the last label in the ROM.

    Say, you have the following at the end of your ROM:

    Code:
    block 1F, full offset: $1FFFFF: EndOfROM:
    block 1F, full offset: $1FFFFF:   dc.b   'This is the end of my ROM'
    block 20, full offset: $200018:   dc.b   '(c) 2018, me'
    block 20, full offset: $200024:   dc.b   0
    
    The "EndOfROM" is the last label ConvSym will record, but there's some data past it that we can point to.
    Being smart, error handler will decode offset $200018 into EndOfROM+$19 ($1FFFFF+$19 = $200018).

    This is actually a tricky edge case. There are technically no labels declared in block 20 (offsets $20xxxx), it's non-existent.
    Normally, the error handler could've stopped and return '<undefined>' for this offset. But if it detects the offset to be located after the last defined block, it'll try to resolve it with the last label of the last block. There was a small oversight/bug in this logic, that is hopefully gone with the hotfix.

    Please let me know if it works. If it does, I'll update Error handler with this fix.
     

    Attached Files:

    Niko and AURORA☆FIELDS like this.
  3. Pacca

    Pacca Having an online identity crisis since 2019 Member

    Joined:
    Jul 5, 2014
    Messages:
    1,168
    Location:
    Limbo
    The hotfix actually seems to have no effect at all! The behavior still occurs, exactly as it did before. I'm happy to supply more info if necessary.
     
  4. vladikcomper

    vladikcomper Well-Known Member Member

    Joined:
    Dec 2, 2009
    Messages:
    404
    Location:
    Russia
    Now this gets even more interesting. I was quite sure it was an oversight for the edge case I described.

    Yes, more details would be nice. I'd like to see what really causes this crash to occur. If you can, please PM me your ROM and your .LST-file that is used to generate symbols data, that's all I need to to debug the issue.
     
  5. Pacca

    Pacca Having an online identity crisis since 2019 Member

    Joined:
    Jul 5, 2014
    Messages:
    1,168
    Location:
    Limbo
    As it turns out, the issue was a user error type situation. Although I didn't notice it, the rom actually exceeded 4mb after the symbol data was appended, so the error handler ended up interpreting whatever was past the 4mb range as symbol data, since the data was truncated.
     
    vladikcomper likes this.
  6. Clownacy

    Clownacy Well-Known Member Retired Staff

    Joined:
    Aug 15, 2014
    Messages:
    930
    I've never used this before, but it looks like there are a few problems with the installation guide, at least for Sonic 2.

    The first problem is that ConvSym is only ran if AS didn't print any warnings. It's an edgecase, but I ran into the problem after deliberately inserting a 'move.w (1).w,d0' into my ROM to trigger the debugger.

    The second problem is that, after ConvSym is ran, the ROM's header isn't amended to account for the new ROM size, causing emulators with an auto-checksum-fix to break the checksum check.

    Both of these problems can be fixed by just editing build.bat:

    First, undo the installation guide's edit by changing this:
    Code:
    IF EXIST s2built.bin (
       convsym s2.lst s2built.bin -input as_lst -a
       exit /b
    )
    ...back to this:
    Code:
    IF EXIST s2built.bin exit /b
    Then insert a new call to ConvSym at the proper place by inserting this:
    Code:
    IF EXIST s2built.bin convsym s2.lst s2built.bin -input as_lst -a
    ...before this:
    Code:
    IF EXIST s2built.bin "win32/fixheader" s2built.bin
     
    FireRat and Pacca like this.
  7. Alriightyman

    Alriightyman I'm back! Member

    Joined:
    Oct 3, 2007
    Messages:
    156
    Location:
    USA
    I know this is an old thread but I just wanted to say this is by far the most interesting and helpful error handling screen period. Sooooo much better than referencing the address with the list file. I haven't tried the custom screens yet, but I most certainly will soon. And the choice to raise an error where you want, is brilliant! Fantastic work on this Vlad!
     
  8. AURORA☆FIELDS

    AURORA☆FIELDS so uh yes Retired Staff

    Joined:
    Oct 7, 2011
    Messages:
    759
    Hey, I am here to bring you a small update! Or rather, a large one because this one will double the size of the handler. However, it will allow you to decode instructions! That's right, the error debugger can now show the instruction (or the one after it - thanks 68k) which caused the exception. In addition, you can also disassemble your own instructions or even disassemble multiple instructions sequentially! Before I tell you what to do, here are a few considerations:
    • This will increase the size of the error handler package, and although not terribly large, it is much larger than the original
    • The decoder uses RAM from $FF0000 to $FF0080 for processing.
    • The decoder also uses more stack, meaning that any areas above the stack space may also get corrupted, so be careful when debugging RAM.
    • The decoder is not guaranteed to correctly guess what each instruction should look like, it can only do what it assumes is intended
    • The decoder does not show the labels in immediate operations.
    However, it also has some of these nice features:
    • The decoder stores the next instruction address in $FF0000, which allows you to sequentially decode instructions
    • The decoder can also be used as a standalone piece of code, which will just output text in ASCII with console subsystem commands (for example for switching palette lines or character sets)
    • Will not decode instructions that do not exist in 68k (looking at you, IDA "Pro").
    • Will not attempt to decode instructions from addresses that do not exist (this checks the header to see how large ROM is)
    • You can use the asm formatter to disassemble from the address provided.
    • Comes with the source code!
    First, you need to follow the installation instructions for v2.0, then choose either AS or ASM68K versions from the downloads below, and replace the files. After that, by default all standard errors will also show the disassembled instructions. Here are a few screenshots:

    [​IMG] [​IMG] [​IMG]

    Oh! And let me know if you find any issues~
     

    Attached Files:

    Niko, TheInvisibleSun, Kilo and 8 others like this.
  9. MarkeyJester

    MarkeyJester ♡ ! Member

    Joined:
    Jun 27, 2009
    Messages:
    2,873
    This is quite a delightful update here! I am glad someone is continuing from where vladikcomper left off~

    This might be a bit of a fiddle, but a potential work around to avoid 68k RAM from being accessed might be; to use VDP memory instead, it would be a fiddle to constantly set the VDP to read and write mode over and over, but it would spare 68k RAM from being tampered with. VSRAM 0028 to 003F might be a potential free space which could be abused for this, since it's not display required, though, I am not sure if this space exists on hardware internally or if memory shuts off at 0027. I would assume 0028 to 003F would exist (for obvious reasons), though I haven't tested on hardware.

    I'm guessing it might be a bit much effort to waste on setting up just to save on RAM, probably require massive rewrites everywhere, but I thought it would be an option...

    ...SRAM might be another option.

    EDIT: Thinking in doubles...
     
    ProjectFM and vladikcomper like this.
  10. AURORA☆FIELDS

    AURORA☆FIELDS so uh yes Retired Staff

    Joined:
    Oct 7, 2011
    Messages:
    759
    Unfortunately setting this up would be a nightmare. This code relies a lot in single-byte writes and uses a sort of argument stack to store temporary variables. Trying to convert this all to VDP writes and reads is definitely possible, but would be a nightmare to program for and incredibly slow. This already is not horribly fast, but that would make this even worse. To me, this is a good enough tradeoff, especially since originally this code was not at all written with Vladik's debugger in mind.
     
    Inferno, Kilo, MarkeyJester and 2 others like this.
  11. vladikcomper

    vladikcomper Well-Known Member Member

    Joined:
    Dec 2, 2009
    Messages:
    404
    Location:
    Russia
    That's one fantastic debugger extension, @AURORA☆FIELDS !
    I happened to witness its development process a little, and I'm genuinely impressed you were able to put everything together in a matter of a few days. Even though the disassembler's core was written before it, it still had to be properly integrated with the debugger, and you did a perfect job implementing it deeply into the console subsystem, so it can be used in any console programs and not only in the exception screen alone.

    Frankly, I was planning a similar thing myself, but eventually, I couldn't find enough time and motivation to get into it, as it seemed like enormous amount of work and testing to get it right. So I'm really glad you came and implemented something that was on top of my planned features list for years.

    As for a few implementation trade-offs that were pointed out, while I admit the solution is not fully "clean" and memory-safe, I believe, any issues that come from it won't affect anyone in 99% of cases, as long as the game doesn't recover from the crash state (which it cannot, in the current implementation).

    * * *
    Also, this may be a good opportunity to remind everyone that the Advanced Error Handler has been open-sourced for years now~
    If you're interested in learning its internals, or you have an urge to modify it for your own needs, check its repo on the GitHub: https://github.com/vladikcomper/md-modules (it even has yet unreleased SEGA-32X branch!)

    It's also worth noticing, the development wasn't halted after the initial version 2.0 release. I've still been working on it over the past years, and it's slowly moving towards the next release (with largely improved ConvSym and a few performance optimizations), which may see the light of day in not-so-distant future.
     
    MarkeyJester, ProjectFM and Kilo like this.
  12. AURORA☆FIELDS

    AURORA☆FIELDS so uh yes Retired Staff

    Joined:
    Oct 7, 2011
    Messages:
    759
    It might actually be worth releasing the original source code for the decoder and the tools I used to make it. These use a different output format but more or less work with a similar method anyway. Anyone is free to use this as well for any purpose too. There are 2 versions of the decoder here: fast and slow. Slow is much smaller (in fact, a fair bit smaller than the one in the error handler) while fast is way faster. Slow also requires more RAM use, because it relies on a software stack approach for command processing, while fast has no such restrictions. However, fast requires way more instructions to do the same thing, which makes it larger essentially.
     

    Attached Files:

  13. Dragonightmare

    Dragonightmare Still alive Member

    Joined:
    Jan 31, 2021
    Messages:
    39
    Location:
    A place in italy
    Wow, amazing job @vladikcomper. I think this will fit better than those annoying error lines seen in sonic 1 and sonic 2. :D