MD Debugger and Error Handler (v.2.6 update)

I stumbled across a bug that makes the error message quite difficult to read in very specific circumstances:

 

Pacguy, wow, that's some amazing find. It's actually the first time since the release (that I'm aware of, at least) someone managed to effectively crash the crash handler~

Comments in your video are mostly correct. This is indeed error handler trying to decode corrupted label name, which causes it to hang. All symbol data is encoded and compressed to save space and due to the nature of my compression, the decompressor is really sensitive to random/incorrect data. However, I have all sorts of safe checks to avoid feeding it with wrong entries (v1 also implemented a rough counter just to avoid endless loops during decompression, but I had to drop it in v2 because I was out of space and registers). But there are intentionally added edge cases of course, to make finding symbols smarter of course.

* * *

Please try the hotfix of the error handler blob (from the attached archive), to see if it fixes this issue.
If it does, congratulations, you've discovered quite a rare edge case!
In fact, nearly impossible set of circumstances had to be met: 1) trying to resolve label (symbol) for a very specific offset in a specific 64-kb section past the ROM area, 2) so an incorrect block in the symbols data was loaded, 3) which had to be corrupted in a specific way to be able to point to a specific random data, 4) which had to have a certain sequence of bytes to manage to put symbol decoder in an endless loop. Just wow, you should've been really unlucky to pull that off.

I believe it's your case, seeing the offset of exception is 872000D4, which error handler decodes to 2000D4, because M68K on MD has 24-bit address bus and upper 8 bits of address do not matter. Offset 2000D4 is just slightly after 2 MB of ROM space. In case your ROM is 2 MB, the offset points just slightly after the last block of your ROM. This triggers an edge case to help resolve offsets that may refer to the last label in the ROM.

Say, you have the following at the end of your ROM:

Code:

block 1F, full offset: $1FFFFF: EndOfROM:
block 1F, full offset: $1FFFFF:   dc.b   'This is the end of my ROM'
block 20, full offset: $200018:   dc.b   '(c) 2018, me'
block 20, full offset: $200024:   dc.b   0

The "EndOfROM" is the last label ConvSym will record, but there's some data past it that we can point to.
Being smart, error handler will decode offset $200018 into EndOfROM+$19 ($1FFFFF+$19 = $200018).

This is actually a tricky edge case. There are technically no labels declared in block 20 (offsets $20xxxx), it's non-existent.
Normally, the error handler could've stopped and return '<undefined>' for this offset. But if it detects the offset to be located after the last defined block, it'll try to resolve it with the last label of the last block. There was a small oversight/bug in this logic, that is hopefully gone with the hotfix.

Please let me know if it works. If it does, I'll update Error handler with this fix.

 

The hotfix actually seems to have no effect at all! The behavior still occurs, exactly as it did before. I'm happy to supply more info if necessary.

 

Now this gets even more interesting. I was quite sure it was an oversight for the edge case I described.

Yes, more details would be nice. I'd like to see what really causes this crash to occur. If you can, please PM me your ROM and your .LST-file that is used to generate symbols data, that's all I need to to debug the issue.

 

As it turns out, the issue was a user error type situation. Although I didn't notice it, the rom actually exceeded 4mb after the symbol data was appended, so the error handler ended up interpreting whatever was past the 4mb range as symbol data, since the data was truncated.

 

I've never used this before, but it looks like there are a few problems with the installation guide, at least for Sonic 2.

The first problem is that ConvSym is only ran if AS didn't print any warnings. It's an edgecase, but I ran into the problem after deliberately inserting a 'move.w (1).w,d0' into my ROM to trigger the debugger.

The second problem is that, after ConvSym is ran, the ROM's header isn't amended to account for the new ROM size, causing emulators with an auto-checksum-fix to break the checksum check.

Both of these problems can be fixed by just editing build.bat:

First, undo the installation guide's edit by changing this:

Code:

IF EXIST s2built.bin (
   convsym s2.lst s2built.bin -input as_lst -a
   exit /b
)

...back to this:

Code:

IF EXIST s2built.bin exit /b

Then insert a new call to ConvSym at the proper place by inserting this:

Code:

IF EXIST s2built.bin convsym s2.lst s2built.bin -input as_lst -a

...before this:

Code:

IF EXIST s2built.bin "win32/fixheader" s2built.bin

 

I know this is an old thread but I just wanted to say this is by far the most interesting and helpful error handling screen period. Sooooo much better than referencing the address with the list file. I haven't tried the custom screens yet, but I most certainly will soon. And the choice to raise an error where you want, is brilliant! Fantastic work on this Vlad!

 

Hey, I am here to bring you a small update! Or rather, a large one because this one will double the size of the handler. However, it will allow you to decode instructions! That's right, the error debugger can now show the instruction (or the one after it - thanks 68k) which caused the exception. In addition, you can also disassemble your own instructions or even disassemble multiple instructions sequentially! Before I tell you what to do, here are a few considerations:

• This will increase the size of the error handler package, and although not terribly large, it is much larger than the original
• The decoder uses RAM from $FF0000 to $FF0080 for processing.
• The decoder also uses more stack, meaning that any areas above the stack space may also get corrupted, so be careful when debugging RAM.
• The decoder is not guaranteed to correctly guess what each instruction should look like, it can only do what it assumes is intended
• The decoder does not show the labels in immediate operations.

However, it also has some of these nice features:

• The decoder stores the next instruction address in $FF0000, which allows you to sequentially decode instructions
• The decoder can also be used as a standalone piece of code, which will just output text in ASCII with console subsystem commands (for example for switching palette lines or character sets)
• Will not decode instructions that do not exist in 68k (looking at you, IDA "Pro").
• Will not attempt to decode instructions from addresses that do not exist (this checks the header to see how large ROM is)
• You can use the asm formatter to disassemble from the address provided.
• Comes with the source code!

First, you need to follow the installation instructions for v2.0, then choose either AS or ASM68K versions from the downloads below, and replace the files. After that, by default all standard errors will also show the disassembled instructions. Here are a few screenshots:



Oh! And let me know if you find any issues~

 

This is quite a delightful update here! I am glad someone is continuing from where vladikcomper left off~

This might be a bit of a fiddle, but a potential work around to avoid 68k RAM from being accessed might be; to use VDP memory instead, it would be a fiddle to constantly set the VDP to read and write mode over and over, but it would spare 68k RAM from being tampered with. VSRAM 0028 to 003F might be a potential free space which could be abused for this, since it's not display required, though, I am not sure if this space exists on hardware internally or if memory shuts off at 0027. I would assume 0028 to 003F would exist (for obvious reasons), though I haven't tested on hardware.

I'm guessing it might be a bit much effort to waste on setting up just to save on RAM, probably require massive rewrites everywhere, but I thought it would be an option...

...SRAM might be another option.

EDIT: Thinking in doubles...

 

Unfortunately setting this up would be a nightmare. This code relies a lot in single-byte writes and uses a sort of argument stack to store temporary variables. Trying to convert this all to VDP writes and reads is definitely possible, but would be a nightmare to program for and incredibly slow. This already is not horribly fast, but that would make this even worse. To me, this is a good enough tradeoff, especially since originally this code was not at all written with Vladik's debugger in mind.

 

That's one fantastic debugger extension, @AURORA☆FIELDS !
I happened to witness its development process a little, and I'm genuinely impressed you were able to put everything together in a matter of a few days. Even though the disassembler's core was written before it, it still had to be properly integrated with the debugger, and you did a perfect job implementing it deeply into the console subsystem, so it can be used in any console programs and not only in the exception screen alone.

Frankly, I was planning a similar thing myself, but eventually, I couldn't find enough time and motivation to get into it, as it seemed like enormous amount of work and testing to get it right. So I'm really glad you came and implemented something that was on top of my planned features list for years.

As for a few implementation trade-offs that were pointed out, while I admit the solution is not fully "clean" and memory-safe, I believe, any issues that come from it won't affect anyone in 99% of cases, as long as the game doesn't recover from the crash state (which it cannot, in the current implementation).

* * *
Also, this may be a good opportunity to remind everyone that the Advanced Error Handler has been open-sourced for years now~
If you're interested in learning its internals, or you have an urge to modify it for your own needs, check its repo on the GitHub: https://github.com/vladikcomper/md-modules (it even has yet unreleased SEGA-32X branch!)

It's also worth noticing, the development wasn't halted after the initial version 2.0 release. I've still been working on it over the past years, and it's slowly moving towards the next release (with largely improved ConvSym and a few performance optimizations), which may see the light of day in not-so-distant future.

 

It might actually be worth releasing the original source code for the decoder and the tools I used to make it. These use a different output format but more or less work with a similar method anyway. Anyone is free to use this as well for any purpose too. There are 2 versions of the decoder here: fast and slow. Slow is much smaller (in fact, a fair bit smaller than the one in the error handler) while fast is way faster. Slow also requires more RAM use, because it relies on a software stack approach for command processing, while fast has no such restrictions. However, fast requires way more instructions to do the same thing, which makes it larger essentially.

 

Wow, amazing job @vladikcomper. I think this will fit better than those annoying error lines seen in sonic 1 and sonic 2.

 

There is an issue with the AS version of the Sonic 1 Hivebrain 2005 disassembly.

If the Error Handler is installed, the rolling sound effect will be corrupted.
Depending on how much and what data is in ROM, you can expect different results:
- The sound effect will be truncated;
- The sound effect will produce static noise of length that depends on what data is in ROM.

The latter is what happens with a blank copy of Sonic 1 with just the Error Handler installed.

Whether or not symbol data is present in the ROM does not change the result in any way whatsoever.

 

the fact that i'm hyped over an exception handler is..... something else. this is some really nice work vladikcomper, well done!

(also apologies for bump)

 

Since this thread got bumped recently, I might as well post this (this is a slight-copypaste from the GitHub issue page):

As of the switch to the Lua build system for the three Sonic disassemblies (Sonic 1, Sonic 2, Sonic 3K) on GitHub between July and August of 2022, the old way of installing ConvSym (the symbol extraction tool) does not apply to these revised disassemblies anymore. However, I have found that it is not that hard for me to implement it into my own hacks with some trial-and-error and got it to work. Here I have compiled a short guide on how to use the symbol extraction of the error handler in newer Lua-based Sonic disassemblies. I am going to be using Sonic 2 here as an example, since I mostly use the error handler there. I am also using the most recent release of the error handler at the time (version 2.0).

Using symbol extraction in Lua-based Sonic disassemblies (Sonic 2)

The steps of installing the error handler in Sonic 2 remains exactly the same as outlined in the guides, so there should be no difference. Refer to the original guide on porting the error handler into Sonic 2. However, during the ConvSym installation, after the step of turning off the ROM padding, move the ConvSym tool into the build_tools/Windows-x86 directory, and under build.lua do the following:

Under "-- Obtain the paths to the native build tools for the current platform." insert "ConvSym" after the build tools. Here's an example:

Code:

-- Obtain the paths to the native build tools for the current platform.
local tools, platform_directory = common.find_tools("build tool bundle", "build_tools/source_code/", repository, "fixpointer", "saxman", "ConvSym") <-- ConvSym is added after the usual build tools

Then after that, under the code for fixing the ROM header (after "common.fix_header("s2built.bin")" but before "-- A successful build; we can quit now."), insert the following:

Code:

if common.file_exists("s2built.bin") then
    os.execute(tools.ConvSym .. " s2.lst s2built.bin -input as_lst -a")
    print("Symbols extracted from s2.lst, ready for use with the Error Handler.") <-- I wrote this message to indicate that ConvSym works
end

This is the code based on vladikcomper's original guide, modified to work with Lua-based Sonic disassemblies.

Once all that was done, build the ROM, and do some crash-testing. The easiest way to do this is to use a sample debugger as outlined in the guide, or go to an invalid level slot (such as Level ID 11, which is invalid in Sonic 2).

That's it! You've now got working symbol extraction for the error handler in a Lua-based Sonic 2 disassembly! Note that this is my way of doing it, so you may find a better way with how you want to do it this way. Also this method only works in the Lua-based Sonic 2 disassembly, as I could not get it to work satisfactorily on the Lua-based Sonic 1 and Sonic 3K disassemblies. If anyone can build from my work, that would be appreciated.

EDIT (06/13/2023): Removed "os.exit(false)" from the ConvSym code, as Clownacy mentioned it is only used for when an error occurs in assembling the ROM. I had it there just to test and verify if the ConvSym code works as intended. Also other minor typographical changes and fixes.
EDIT (06/14/2023): Slight wording changes.

 

Nice work, though that `os.exit(false)` in the last block of code shouldn't be there: it's meant for signalling an error.

 

Thanks for the reply. The reason why "os.exit(false)" was there is because I wanted to “properly” see the message after it had finished to indicate that the ConvSym tool works as intended, otherwise it would only be shown for a split second before closing and I wouldn’t have any confirmation that it works or not (unless I use the Command Prompt).

I’ll remove it from the guide.

EDIT: Removed.

 

Since the thread is bumped anyways, let's give a small announcement early on!

The Advanced Error Handler and Debugger 2.5 Update is almost here!

Well, the coding has just been fully finished today, all features are finalized. I now have to go through a much more tedious process of updating documentation and testing disassembly integrations. Since S1/S2 Github disasms switched to Lua build system in particular, brand new installation guides are underway.

Given my free time (or lack thereof, to be precise), it may take weeks to get everything in order for a proper release (hence this announcement is early). But if you're used to building from source codes, just clone the MD-Modules repository, build and copy-paste ready-to-use utilities and debugger bundles for your assembler (ASM68K or AS). There are instructions that should have you covered~

Upcoming features

While more than 5 years have passed since version 2.0 release in 2018, the 2.5 update isn't anything big or ground-breaking, but rather an incremental update with a good amount of stability and quality-of-life improvements and just a few new features.

Error Handler and Debugger:

• Assertions! Super-charge your debugging technique with self-testing code!
  • Assertions is one of the most powerful tools for debugging supported by many high-level programming languages. The Advanced Error Handler and Debugger 2.5 implements them in a form of "assert" pseudo-instruction, which is only compiled in DEBUG builds. This means zero run-time cost for your final (RELEASE) builds to implement self-testing code.
  • Code:

      assert.w  a1, eq, #v_player  ; a1 points to "v_player" RAM (Sonic object slot)
      assert.b  (a0), eq, #$01     ; make sure current object's ID is $01 (Sonic)
      assert.l  obMap(a0), ne      ; make sure "obMap(a0)" is set (not zero)

• Slightly improved error info display. The header of exception screen should now be even more readable: raw offsets are now displayed next to symbols and one instance where the same offset and symbol are shown as separate fields is now merged into one line to avoid ambiguity ("Location" and "Module" in the old version).
  • 

• ASM68K debugger: support projects with "case-sensitive" switch. If you ASM68K project uses "case-sensitive" mode for whatever reason ("/o c+" flag), it's now properly supported by the debugger header.

• A lot of optimizations, stability fixes and QoL improvements. Smarter stack analyzer, support for non-sign-extendable stack pointers and more. A number of non-critical bugs was found and fixed over the course of development. Some code was also optimized and reduced.

• Slightly smaller and easier to install Error Handler blob. Error Handler blob is still within ~3 KB boundary (quite small for the number of features it provides!). Since version 2.5, installation also requires less files, as blob ("ErrorHandler.bin") is now inlined in the assembly file instead.

Symbol generation (ConvSym):

• Stable AS support! Extracting symbols for AS-based disassembles was a bit unstable before due to an experimental parser. It since was completely re-written and is now fully accurate to the symbol table that AS itself generates. This means a much better experience for AS users.

• Full support for native ASM68K symbol files. You can take full advantage of native ASM68K symbol files now, with a fast and reliable parser. Previously, some features weren't fully supported (mainly local labels) so a sophisticated ASM68K listing file parser was recommended as a workaround.
  • You need "/o v+" compile flag to make ASM68K output local labels to the symbol file.

• Tons of new features for advanced users. ConvSym's development continued since 2018 and the new update will ship ConvSym v.2.9.1 instead of v.2.0! It's a huge gap meaning a ton of new features. Just to scratch the surface:
  • Improve symbol data compression by force-converting your symbols to upper or lowecase;
  • Extract symbols from .txt/.log files via "log" parser;
  • Support for multiple labels on the same offset;
  • Support for symbols in RAM section (symbols must be properly implemented in your project);
  • Advanced offset transformations: mask, upper/bottom boundary, add/subtract base address (makes SEGA 32X symbol generation doable, but must be implemented by your project);
  • And much more! See ConvSym's README for more information.

• Fully cross-platform: MacOS, Linux and Windows support! ConvSym utility now has builds for all the major platforms, it's no longer Windows-exclusive.

(And just a small reminder that all of these features are already available, if you're able to build from source code and don't need extra guides =)