Advanced Error Handler and Debugger

Discussion in 'Utilities' started by vladikcomper, Apr 5, 2016.

  1. vladikcomper

    vladikcomper Well-Known Member Member

    Joined:
    Dec 2, 2009
    Messages:
    382
    Location:
    Russia
    14/01/2018 -- Brand new Version 2.0 is out!
    See release post: http://sonicresearch.org/community/...-for-your-hacks-and-homebrew.4442/#post-78456

    ----

    I believe, many of you know how important debugging and testing is in any development process. Not only it affects the final production's quality, completeness and stability, but, most importantly, your own work's efficiency, which may be represented as correlation of the result and effort you put into it.
    If one spends a lot of their time just on trying to get something to work as they wish, yet one bug spawns after another, they clearly must be doing it wrong: be it using inappropriate strategy or lack of appropriate tools, or even both.
    Now, when it comes to Sonic hacking, homebrew development or anything that involves modifying or creating assembly code, having proper tools to help debug your code is essential. A single misspelled register, forgotten branch or uncoordinated call may crash everything right away, and the bug is usually rather hard and time-consuming to spot if you can't control your program's flow and don't know any circumstances behind program crashes.

    This leads us to a very important conclusion: the most essential thing you should care about first when working with assembly code is error handling. If anything crashes, you should instantly know what, where and why. This way, you'll find a problematic part of your code in a matter of seconds.

    Unfortunately, neither of classic Sonic games implements an error handler to meet this simple demand.
    Moreover, Sonic 1 is the only game among the other to have one. But it isn't too informative with its one-line error messages, which, by the way, are simply hacked into the current game mode's foreground and easy to break occasionally as they depend on screen's scrolling and palette (hence you'll never be able to read them if screen has faded or you may not see parts of it because of awkward scrolling).
    The only custom error handler to be released was flamewing's, which was hardcoded only to bus and address errors back in the day, but eventually got extended to handle all standard errors. And I think it's really good and flamewing has done a really impressive job, so make sure to check it out as well!
    To be absolutely honest, I never used that error handler, I had my own since 2011, which certainly not to be compared with the amazing work flamewing did on his. It just occurred I had a little different views on overall error handler appearance, behavior and mobility.

    I used my old error handler in several projects I was involved in, as well as my own ones, including Sonic 3 in 1 and Sonic Winter Adventures. However, as soon as homebrew game, Sonic Warped, has grown really big I felt like I needed a little more given the same formula to help speed up code debugging even more.
    Initially designed for my own homebrew, I presented it to a few people and they seem to really love the idea, some of them even asked if I can adapt it to their hacks. So I went a little further and expanded it to be completely flexible, independable and easy to use.
    Since a few people found it really helpful, I thought it would be a good idea to release it for everyone's use, so it may help out a lot more people.

    So, today I present to you... The Advanced Error Handler that will work in every hack and homebrew.

    Overview

    [​IMG]

    • Displays the actual subroutine names in the crash report!
      Yes, you heard me right. You won't just see some raw offsets that you have to search for, but the actual subroutine names and locations inside them. This way, you may know where to look at in the matter of seconds, you don't even need to waste you time searching through the listing file to debug many kinds of errors!
      This is done by storing symbol information in the end of your ROM with a tiny little tool to make this possible. This information is compressed and encrypted in my custom format, this way it will take as less space as possible, while no one will be able to read your labels and their offsets easily. More on that below.

    • Analyses the stack to identify the caller of the problematic routine.
      In many cases the crash address alone will tell you little to nothing about its origin. The subroutine calls hierarchy, on the contrary, may tell you everything.
      For instance, one of the most common error, Illegal instruction is usually caused by jumping to wrong offsets as the result of calculating wrong addresses, be it a variable pushed out of the boundaries or just a register getting garbaged. The crash is triggered at that wrong offset, as the processor starts fetching random bytes instead of proper instructions. The location of it, however, will tell us nothing, it's usually completely random.
      So, in this particular case it's more important to know, which subroutine called this wrong offset in the first place.
      My error handler will analyze stack to find out, what called the problematic routine and display its name. Some analysis required because of the nature of the stack: it's not only used to store the return addresses, various routines may save variables and registers on stack as well. My simple analyzer will check every word of the stack to see if it can form a proper ROM offset, it also checks not to go out of the stack boundaries. This works in most of the cases.

    • Displays all main registers and the stack state before the crash.
      My error handler will also check the stack's upper boundary to highlight the data that's actually on the stack. The boundary checking relies on the correct top of the stack address in the vector table. There are games that don't store it correctly though, changing stack pointer to a different address later in the startup code (one game to do so is Sonic 3 & Knuckles). Which is why my error handler will display as many words as possible in its stack dump. This can also be helpful to watch the RAM addresses after the stack.
      Another little notice, regarding the stack: When the processor enters error state, an exception to handle this error is triggered. There are several groups of exceptions, even the interrupts fall in one of these, however, I won't describe them all here as it's beyond the scope of this little paragraph. Two simple things you should know: 1) When exception occurs, processor always saves system register and the return address on the stack; 2) Address and Bus Errors are the only ones to store some more additional information on the stack.
      My error handler won't display any information stored by the exception itself in the stack dump. For the most part, it's already represented in the other fields. For instance, the return address goes to the "Location" field, and in case of Address Error, an extra "Address" field appears, representing the odd address accessed.
      The only field that I don't actually display either way is SR, because I never really needed it in my practice. Same goes for instruction data fields for address error.

    • Displays current vertical and horizontal interrupt handlers, if they are dynamic.
      This applies mostly to Sonic 3 & Knuckles and other games that may do this trick. If your ROM doesn't have them, the additional fields simply won't appear.
      You can change VInt and HInt vectors to point to RAM instead of particular routines in the ROM. This way, it is possible to modify VInt and HInt handlers as the game wants. Many games simply write JMP (xxx).l opcode at those RAM addresses to redirect to whatever interrupt routine they want at the specific time.
      My error handler is able to detect this particular opcode and retrieve the jump offset. This may be useful for debugging interrupt-related errors, in case your game uses many changeable interrupt handlers (like mine).

    • Works anywhere, works anyhow.
      Be it a hack or a specific homebrew, written from scratch, the error handler will work the same for both. It also adapts to what you have and displays what it can.
      Don't want to include symbol information? Simply don't follow the guidelines to get it generated. The Error handler will understand if it's missing and will display offsets instead of symbols.
      Don't have changeable interrupts handlers? The error screen will notice that and won't try to display them.

    • Easy to install, independable, reliable.
      You don't have to include any source code and compile it, hoping your assembler not to throw a bunch of errors or unresolved conflicts.
      Just download an already compiled light-weight binary module and include it in the end of your ROM. Then simply modify your vectors table to refer to that module and that's it! If you want it to make it display labels instead of offsets, you'll only have to add a few extra lines in your build script, but it will work either way. A quick installation guide can be found below.
      It doesn't depend on a particular assembler or a game, the module just works on its own.
      As the error handler does some complex operations like stack analysis and offsets decoding using encrypted data, I put a special attention on making the code reliable and highly error-resistant. It won't break if the symbol information is missing, it won't break if compressed symbol data is corrupted, it won't break even if the most of environment is corrupted.

    Technical specs

    If you are interested in technical details behind it for some good reason (like being a Tech Member), or you just like to read those "Interesting facts" kind of lists, enjoy the spoiler!

    • The compiled error handler module takes only $7E4 bytes of ROM space and it doesn't depend on any external routines. It includes font graphics in 1bpp format, which reduces its size by 4 times, as good as some commonly used compression formats like Nemesis or Kosinski would do. Using the latter would either make module dependable or make me include the decompressors inside (which would be really pointless due to their size, as I loose all the space saved by the actual compressed data).

    • The error handler is doing its initialization really throughoutly, as if everything was completely broken or wrong: it will ensure there aren't any pending writes to VDP, wait if DMA is going and VDP cannot be actually accessed, it will reset all important VDP registers to make sure they are set properly. This protects from a wide range of conditions which would otherwise prevent the error screen from displaying properly.

    • The Error Handler doesn't use any variables, doesn't require any memory for them. Believe or not, processor's registers are more than enough to store data in-between calculations. The screen even uses the stack to save register only once. No memory requirements means none of your game's variables will be accidentally touched.

    • Symbol information is collected based on assembler's output when compilation is finished, it is then appended to the end of the ROM, where the Error Handler will try to find it.

    • Symbol information is stored in my custom format, symbol names themselves are compressed. The compression method adapts to the actual data compressed, so characters will get different representations in different symbol lists.

    • Indexed search is used to find offsets and corresponding symbols quickly. My symbol information format is designed in a way that offset indexing not only helps the search performance, but also reduces space required to store them.

    • Indexing can address up to 128 KB of data, which means a compressed symbol file can be really large, even exceed 128 KB a little.

    Installation

    As mentioned earlier, installation is really easy, it literally takes just a couple of steps.
    If you want to try it your in your own project, be it a hack or something more, first of all, download and extract this in your source code's root folder:
    https://www.dropbox.com/s/75mqu09l28jwer6/Error Handler.7z?dl=0
    I'll provide an example for Sonic 1 disassembly here, but installation isn't much different for the others.

    1. Adding the Error Handler module to your ROM
    • Add the following code at the very end of your main source file: http://pastebin.com/J8it8T0w
      In case of Sonic 1, your should open sonic1.asm file and find "EndOfROM:" label. Place that code just before "EndOfROM".
      Including the error handler module at the end is very important if you want it to load symbol information, which will be appended to the end of the ROM by a special tool later. Otherwise, the error handler won't "see" any symbol information; it will work fine, but will display raw offsets instead of the actual labels.
      NOTICE: If you're using the AS macro assembler, replace "incbin" with "binclude"
      NOTICE 2: Also, make sure your build script doesn't do any padding, which may add a gap between the module included and the actual end of the ROM.


    • Make sure to delete old error handlers if there are any.
      In case of Sonic 1, open sonic1.asm and find "BusError:" label. Delete everything before the "loc_43A:" label.

    • Make sure to modify the exceptions vectors in the vector table to accommodate to the labels in the code you've included in step 1.
      In case of Sonic 1, nothing is to modify, as the new code already uses the correct labels from the vectors table.
      For other projects/disassemblies that may use different label names and handler setup in the vector table, here's how it should look like for the reference: http://pastebin.com/FQH5xTsw (ignore Stack, EntryPoint, HBlank and VBlank labels, they may use your own names, the others should refer to labels used by error handler module)

    That's it! You now have the advanced error handler in your ROM.

    2. Getting symbol generation to work


    The ConvSym.exe tool is designed to convert symbols file generated by assembler to my custom compressed format that the Error Handler module is able to read and retrieve information from.
    Unfortunately, it only supports ASM68K assembler so far, all projects to use the AS macro assembler or other incompatible assemblers will have to do without this step. As mentioned earlier, the error handler is able to work without any symbol information included.
    The following example targets Sonic 1 disassembly.

    • Open build.bat. You should see something like this:
      Code:
      @echo off
      asm68k /o op+ /o os+ /o ow+ /o oz+ /o oaq+ /o osq+ /o omq+ /p /o ae- sonic1.asm, s1built.bin
      pause

    • Modify it as follows and save:
      Code:
      @echo off
      asm68k /o op+ /o os+ /o ow+ /o oz+ /o oaq+ /o osq+ /o omq+ /p /o ae- sonic1.asm, s1built.bin, sonic1.sym, sonic1.lst
      convsym sonic1.sym sonic1.symcmp
      copy /B s1built.bin+sonic1.symcmp s1built.bin /Y
      del sonic1.symcmp
      pause

    Again, that's it! Now try running the build script. You should see additional message, saying "Encoding symbol table for run-time debugger...". Ensure it doesn't display any errors.

    3. Testing with intentional crash

    If you want to see it in the works, simply provoke a crash intentionally! The easiest way is to add "illegal" opcode in a routine that will run either way when your game starts.
    Here's an example for Sonic 1. Just open sonic1.asm, find "Obj01:" label and add "illegal" right after it (make sure there is at least one space or tab before this command). Compile your ROM, load any level and you should see this: https://www.dropbox.com/s/flo5rr3g51zeqh3/Debugger_in_S1.PNG?dl=0

    If you do, then congratulations, you did everything right! Happy errors and debuggin~
     
    Last edited: Jan 15, 2018
  2. Novedicus

    Novedicus Well-Known Member Member

    Joined:
    Aug 26, 2013
    Messages:
    799
    I love you for this. Thank you! Just implemented this and I'm already liking it.
     
    LuigiXHero likes this.
  3. FireRat

    FireRat "The grand imitator..." Member

    Joined:
    Oct 31, 2009
    Messages:
    526
    I really want to give this baby a try, but Dropbox comes up with the Error 500. Is there any available mirror? thank you!
     
  4. vladikcomper

    vladikcomper Well-Known Member Member

    Joined:
    Dec 2, 2009
    Messages:
    382
    Location:
    Russia
    That's weird, Dropbox usually works for everyone. Error 500 means internal server error. Perhaps, the server was encountering problems the time you tried to download it. I'm pretty sure it was a temporary failure, otherwise, no one would be able to download it or see the pictures in my post, which are also hosted on Dropbox.
    I'm afraid I don't know any reliable file sharing sites where I can set up the mirror without having to register and log in. So I just attached the file to my post, hope that works well for you.

    On a small note for everyone, I've slightly edited the first post:
    1) It appeared there was a paragraph missing in the "Overview" section after I copy-pasted it from Sonic Retro. It's under the very first spoiler in the post, describing why knowing the caller can be important for Illegal Instruction and Line Emulator errors.
    2) I've also added a small notice in the "Installation" section for those using the AS macro-assembler. You'll have to replace "incbin" with "binclude" when including the binary module. But I think that what most of you guys here already knows.
     

    Attached Files:

    Last edited: Apr 5, 2016
    FохConED and FireRat like this.
  5. Natsumi

    Natsumi Markey's Member

    Joined:
    Oct 7, 2011
    Messages:
    621
    Location:
    Otter's lap
    I saw a bug where the error handler does not seem to account for 24-bit addresses when looking for the caller. I had the error handler not show the caller correctly when the actual caller address was $FF0724DC. I also did not use the symbol table because I added the debugger for finding out why a bug happens.
     
  6. vladikcomper

    vladikcomper Well-Known Member Member

    Joined:
    Dec 2, 2009
    Messages:
    382
    Location:
    Russia
    Natsumi, it's not a bug per se, it's Error Handler being too careful when analyzing stack.
    As you clearly know, there is no certain way to trace back which routine called which; you have no real clue about previous processor states, your one and only hint is... stack of course.
    Then again, stack is used for everything: not only it stores return addresses upon executing jsr/bsr opcodes, it also saves registers, variables; basically all the stuff your program may want to temporarily save...

    Let's say, you have to tell return addresses from data in the stack (what Error Handler has to do); in the process, you find FF 07 24 DC sequence. How do you tell it's a return address? Is it longword FF0724DC or words FF07 and 24DC stored separately? It could be all kinds of data, like XY-position of something saved on stack via MOVEM, registers dump. It doesn't look like a longword that forms a proper offset, which is why Error Handler skips it.

    I clearly see your point in counting this as a bug, but those were absolutely necessary restrictions in order to reduce cases where data in stack is falsely treated as offsets.
    Storing data in upper byte of return address (like you did) is a really rare case.
     
  7. Natsumi

    Natsumi Markey's Member

    Joined:
    Oct 7, 2011
    Messages:
    621
    Location:
    Otter's lap
    Now, I do know very well why you would feel its necessary to observe it as full 32-bit offset to reduce faulty data, and to an extent you are right. However, you have a trick up your sleeves. You know the address of the Error Handler. Assuming no bank switching, you already know the end address of the ROM. You can pretty safely assume, that anything coming from even addresses and below that point, will be often a valid address in ROM. Will it fail sometimes? Yeah, of course. But, you could also show a longer caller stack, which would help weed out cases where you interpreted data as an address.

    Not to say I am doubting you on this matter or that you wouldn't understand these things, but I am about to release my sound driver which RELIES on this trick very often. It can seriously confuse someone who may not realize the caller is missing some of the time. In fact, this caught me out while I was debugging some very persistent issue, to then think it might even be breaking the stack (which has happened before, too). I know you might not think this is a major issue, but at 3am, trying to debug something really stupid and finicky, these are the kind of things that just make your life that much worse. Of course, it's up to you if you care enough to fix it, but I would urge you to.
     
  8. vladikcomper

    vladikcomper Well-Known Member Member

    Joined:
    Dec 2, 2009
    Messages:
    382
    Location:
    Russia
    Heads up everyone, as here comes a really huge update, or should I say, an evolution of error handler.
    Insane amounts of time and effort were actually put into this update; the entire thing was rewritten from the ground up at least twice to make the impossible possible and to implement it in the best way; I still can't believe I actually made it~
    Please, meet the ...

    Advanced Error Handler and Debugger 2.0
    * Now with AS support and much much more...

    So, it's been almost 1.5 years since version 1.0 was originally released.
    To my surprise, quite a few people started using my debugger in their projects, some of them even told me in person it really helped them track down code issues faster and spend less time in a listing file. Still, people were asking me about some new features, to add more customization and extensibility. I myself was aiming to develop a much powerful and sophisticated debugging system to meet my new demands.
    So I took all the feedback and requests into consideration and completely reworked debugger's internals to make some insane new features possible and yet preserve a bit of back-compatibility and the same appearances.

    [​IMG] [​IMG] [​IMG]

    While the key feature in version 1.0 was decoding raw offsets into actual labels/symbols in the source code, version 2.0 takes the existing concept and expands it on a whole new level with its huge additions:

    [​IMG]

    So, the key concept behind version 2.0 and the focus of further additions is...

    Create your own debuggers and error handlers with ease!
    No real assembly programming required. Forget about digging through emulator's memory viewers all over again to find that damn RAM value. Forget about coding your own diagnostic or debug screens, worrying if the code prints it right.
    Log all the data you want, present it as you like.
    This becomes possible with debugger's new high-level features, available natively for ASM68K and AS with a set of new macros -- no additional tools or parsers required!

    [​IMG]

    Speaking of new features, there are so many of them, I'd rather just describe highlights:
    • The debugger is now built around an extensible modular architecture
      • Its core includes a powerful console subsystem which may change the way you debug..
      • The console subsystem among with a set of macros implements formatted strings, like in high-level languages (think of printf() or std::cout from C++), but even more powerful in a sense:
        • Supports printing any register or memory value using any addressing mode supported by the M68k;
        • Supports decoding raw offsets into symbols (labels) from the source code,
        • Supports C-strings (null-terminated ASCII strings),
        • Supports colour formatting, newlines, as well as automatic newlines (with configurable length of the line).
    • Write your own console programs and custom error handlers!
      • As demonstrated above, you may quickly write your own subroutines that feed data to the console and execute some code.
      • You can assign this programs to specific errors instead of the standard error handler (which is internally the same type of console program!)
      • You may generate your own error messages at any point of code execution using RaiseError macro.
        • It's always useful to include error handling logic in your code to identify failure states
        • Generating custom error messages helps to describe conditions of the error and provide necessary information for debugging
        • Formatted strings are fully supported by RaiseError macro, so you can easily log specific RAM addresses and registers touched in your code
    • Meet the new and improved symbol storage format: DEB v.2.0 (or simply, DEB2)
      • Supports extra large ROMs (>4 MB)
      • Has no limits on storage capacity
        • While DEB1 only store and address up to 128 kb of compressed symbol data (slightly more in extreme cases), DEB2 can address up to 4 GB of data, which means no real limit in terms of MD
        • For instance, S3K symbol base is so huge, it takes ~330 kb in a compressed storage. It wasn't even possible to store and address so much data using DEB1 format, but DEB2 can handle this and much more.
      • Despite many additions, maintains compression rations comparable to much simpler DEB1 format
        • DEB2 bases are usually only 5--8% larger than their DEB1 counterparts
      • Has an improved storage format with much faster symbol searching
        • The new format not only provides fast offset indexing by blocks (already found in DEB1), but also improves drastically on search performance by implementing binary searching.
        • For the sake of comparison in an extremely bad scenario, looking up certain symbols may take up to 5,000 accesses for DEB1 (if blocks are large enough), but it takes at most 12 accesses in DEB2 (because binary search has logarithmic complexity).
    • Meet the updated symbol extraction utility (ConvSym.exe) with tons of new features
      • The utility is used to collect symbols and their offsets from assembler's output and store them in DEB2 format, accessible by Error Handler.
      • It usually runs by build.bat script upon assembly completion and connects symbol data to an already compiled ROM.
      • Now supports both ASM68K and AS assemblers
        • Symbol data can be extracted from assembler-generated symbol file (ASM68K format) or listing (both AS and ASM68K formats)
      • Support for local labels! (optional, can only be extracted from LST-files)
      • Now has plenty of useful command line options:
        • You can filter unnecessary symbols by offsets range or/and using regular expressions
        • You can append an existing ROM instead of generating a separate file
        • You can log all symbols into text files
        • And much more~
    • And finally, it's same as always: lightweight, easy to install and 100% back-compatible
      • Believe or not, the entire debugger fits into 3 KB blob!
      • Despite being so feature-packed, version 1.0 is as easy to install and maintain as the first one.
      • Back-compatibility literally means you can just copy version 2.0 files over version 1.0 installation and it'll work right away. But in order to have all new additions available, you need to add new macros and modify build scripts, so I recommend following the installation guides.
    Downloads

    Pick the one that suits your assembler!

    Installation guides

    Sonic 1 Disassembly (GitHub / Hivebrain)
    Step 1. Download and unpack
    Download ASM68K version from Downloads section above (unless you're using AS-branch of the said disassembly, which you know if you are...)
    Copy all files into disassembly's root directory.

    Step 2. Install debugger macros
    At the very beginning of sonic1.asm (sonic.asm in GitHub version) add include directive for debugger's main definitions:
    Code:
        include   "Debugger.asm"
    This will allow you to use all the macros and commands shipped with the Error handler (RaiseError, Console.Write, Console.SetXY etc).

    At this point, make sure the new definitions don't conflict with the existing labels in your disassembly! The easiest way do so is to try to compile your ROM. Does it build successfully?
    During assembly, you'll most likely get the following error:
    Code:
    D:\SEGA\DEV\PROJECTS\ERROR HANDLER\SONIC 1 (GITHUB)\SONIC.ASM(44) : Error : Label 'console' multiply defined
    console: dc.b 'SEGA MEGA DRIVE '
    This is because debugger definitions use Console as a namespace for new set of console-related commands (Console.Write, Console.Run etc), which conflicts with now ambiguously named label in the ROM header.
    As an easy fix, you can just remove the label itself or rename it. But do not remove the DC.B part or the entire line, this will shift the remaining data in the ROM header!

    Fix the issue as you like and make sure the ROM build successfully before going to the next step...

    Step 3. Include Error handler blob.
    At the very bottom of sonic1.asm (sonic.asm in GitHub version), just before "EndOfRom:", add the following snippet:
    Code:
    ; ==============================================================
    ; --------------------------------------------------------------
    ; Debugging modules
    ; --------------------------------------------------------------
    
       include   "ErrorHandler.asm"
    
    ErrorHandler.bin is debugger blob itself, while ErrorHandler.asm specifies error handler options (that you can alter if you want) and also declares entry points for error vectors and their standard error messages (again, alter if you like, they also support formatting).

    However, to avoid more name conflicts, the old error handlers must to be removed.
    Find "BusError:" label, which is the first exception handler for Bus Error vector.
    Remove all the code from "BusError:" through the end of "ErrorWaitForC:" (look for "; End of function ErrorWaitForC")

    Make sure ROM builds successfully. You now have error handler properly installed!

    Step 4. Configure symbol extraction
    This is what ConvSym.exe comes for!
    Now, for ASM68K version, there are two configurations for you to choose from, both are equally easy to set. The first one is the fastest, but lacks some features; the second is slower (though modern machines would barely notice...) but has more to offer. Which one do you choose?

    Option A: Extraction from ASM68K-generated symbol file (fastest, no local labels support, all labels are lowercase)
    Open build.bat in a text editor and replace or modify its contents with the following:
    Hivebrain 2005 version
    Code:
    @echo off
    asm68k /o op+ /o os+ /o ow+ /o oz+ /o oaq+ /o osq+ /o omq+ /p /o ae- sonic1.asm, s1built.bin, s1built.sym, sonic1.lst
    convsym s1built.sym s1built.bin -input asm68k_sym -a
    Notice that no ROM padding utilities should be used, as symbols storage should append directly after the error handler blob!

    Github version

    Code:
    @echo off
    
    IF EXIST s1built.bin move /Y s1built.bin s1built.prev.bin >NUL
    asm68k /k /p /o ae- sonic.asm, s1built.bin >errors.txt, sonic.sym, sonic.lst
    
    convsym sonic.sym s1built.bin -input asm68k_sym -a
    fixheadr.exe s1built.bin
    

    Option B: Extraction from listing file (slower, local labels support, all labels have original case)
    Open build.bat in a text editor and replace or modify its contents with the following:
    Hivebrain 2005 version
    Code:
    @echo off
    asm68k /o op+ /o os+ /o ow+ /o oz+ /o oaq+ /o osq+ /o omq+ /p /o ae- sonic1.asm, s1built.bin, s1built.sym, sonic1.lst
    convsym sonic1.lst s1built.bin -input asm68k_lst -inopt "/localSign=@ /localJoin=. /ignoreMacroDefs+ /ignoreMacroExp- /addMacrosAsOpcodes+" -a
    fixheadr.exe s1built.bin
    Notice that no ROM padding utilities should be used, as symbols storage should append directly after the error handler blob!

    Github version

    Code:
    @echo off
    
    IF EXIST s1built.bin move /Y s1built.bin s1built.prev.bin >NUL
    asm68k /k /p /o ae- sonic.asm, s1built.bin >errors.txt, sonic.sym, sonic.lst
    convsym sonic.lst s1built.bin -input asm68k_lst -inopt "/localSign=@ /localJoin=. /ignoreMacroDefs+ /ignoreMacroExp- /addMacrosAsOpcodes+" -a
    fixheadr.exe s1built.bin

    NOTICE: Run ConvSym.exe in command line without arguments to see full options list.

    Congratulations, you've done it!
    Now, if you'd like to test and explore, read the next two sections for references and examples~

    Sonic 2 Disassembly (Github)
    Step 1. Download and unpack
    Download AS version from Downloads section above. Copy all files into disassembly's root directory.

    Step 2. Install debugger macros
    Near the beginning of s2.asm find the following directives:
    Code:
       CPU 68000
       include "s2.macrosetup.asm"
    Add include directive for debugger's main definitions below, like that:
    Code:
       CPU 68000
       include "s2.macrosetup.asm"
        include "Debugger.asm"           ; ++
    This will allow you to use all the macros and commands shipped with the Error handler (RaiseError, Console.Write, Console.SetXY etc).

    Step 3. Include Error handler blob


    Near the very end of s2.asm, just before "; end of 'ROM'", insert the following code snippet:
    Code:
    ; ==============================================================
    ; --------------------------------------------------------------
    ; Debugging modules
    ; --------------------------------------------------------------
    
       include   "ErrorHandler.asm"
    
    ErrorHandler.bin is debugger blob itself, ErrorHandler.asm specifies error handler options (that you can alter if you want) and also declares handlers for standard error exceptions and their respective messages (which you are free to alter as well, they also support formatting).

    Now, you should add new error handlers defined in ErrorHandler.asm to the vectors table in the beginning of the ROM.
    Since Sonic 2 Final doesn't even has code to display error messages, all error exceptions in the vector table refer to the "ErrorTrap" subroutine, which simply freezes processor in an infinite loop.
    Near the beginning of s2.asm, swap the vectors table seen between ";Vectors:" and "; byte_100:" lines to the following:
    Code:
    ;Vectors:
       dc.l System_Stack   ; Initial stack pointer value
       dc.l EntryPoint       ; Start of program
       dc.l BusError       ; Bus error
       dc.l AddressError   ; Address error (4)
       dc.l IllegalInstr   ; Illegal instruction
       dc.l ZeroDivide       ; Division by zero
       dc.l ChkInstr       ; CHK exception
       dc.l TrapvInstr       ; TRAPV exception (8)
       dc.l PrivilegeViol   ; Privilege violation
       dc.l Trace           ; TRACE exception
       dc.l Line1010Emu   ; Line-A emulator
       dc.l Line1111Emu   ; Line-F emulator (12)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved) (16)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved) (20)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved) (24)
       dc.l ErrorExcept   ; Spurious exception
       dc.l ErrorExcept   ; IRQ level 1
       dc.l ErrorExcept   ; IRQ level 2
       dc.l ErrorExcept   ; IRQ level 3 (28)
       dc.l H_Int           ; IRQ level 4 (horizontal retrace interrupt)
       dc.l ErrorExcept   ; IRQ level 5
       dc.l V_Int           ; IRQ level 6 (vertical retrace interrupt)
       dc.l ErrorExcept   ; IRQ level 7 (32)
       dc.l ErrorExcept   ; TRAP #00 exception
       dc.l ErrorExcept   ; TRAP #01 exception
       dc.l ErrorExcept   ; TRAP #02 exception
       dc.l ErrorExcept   ; TRAP #03 exception (36)
       dc.l ErrorExcept   ; TRAP #04 exception
       dc.l ErrorExcept   ; TRAP #05 exception
       dc.l ErrorExcept   ; TRAP #06 exception
       dc.l ErrorExcept   ; TRAP #07 exception (40)
       dc.l ErrorExcept   ; TRAP #08 exception
       dc.l ErrorExcept   ; TRAP #09 exception
       dc.l ErrorExcept   ; TRAP #10 exception
       dc.l ErrorExcept   ; TRAP #11 exception (44)
       dc.l ErrorExcept   ; TRAP #12 exception
       dc.l ErrorExcept   ; TRAP #13 exception
       dc.l ErrorExcept   ; TRAP #14 exception
       dc.l ErrorExcept   ; TRAP #15 exception (48)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved) (52)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved) (56)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved) (60)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved)
       dc.l ErrorExcept   ; Unused (reserved) (64)
    ; byte_100:
    Make sure ROM builds successfully.
    You now have error handler properly installed!

    Step 4. Configure symbol extraction

    It's what ConvSym.exe comes for!
    This utility collects data about all symbols used in your source code and their corresponding offsets and stores them in a compressed base in a special format that Error Handler understands.
    The symbol data is appended to the end of the ROM, so Error Handler can access it. In case of Sonic 2, which uses AS macro assembler, data can be only extracted from a listing file generated by the assembler.

    In order for symbol data to be accessed correctly, you should first disable ROM padding.
    Open s2.asm and at the begging, find "padToPowerOfTwo" option. Change it as follows:
    Code:
    padToPowerOfTwo = 0
    ;   | If 1, pads the end of the ROM to the next power of two bytes (for real hardware)
    Now open build.bat and find the following line:
    Code:
    IF EXIST s2built.bin exit /b
    Replace it with this code:
    Code:
    IF EXIST s2built.bin (
       convsym s2.lst s2built.bin -input as_lst -a
       exit /b
    )
    NOTICE: Run ConvSym.exe in command line without arguments to see full options list.

    Congratulations, you've done it!
    Now, if you'd like to test and explore, read the next two sections for references and examples~

    Sonic 3 & Knuckles (Github disassembly)
    Step 1. Download and unpack
    Download AS version from Downloads section above. Copy all files into disassembly's root directory.

    Step 2. Install debugger macros
    Near the beginning of sonic3k.asm find the following directives:
    Code:
           cpu 68000
           include "sonic3k.macrosetup.asm"   ; include a few basic macros        
           include "sonic3k.constants.asm"       ; include a few basic macros
           include "sonic3k.macros.asm"       ; include some simplifying macros and functions
    Add include directive for debugger's main definitions below, like that:
    Code:
           cpu 68000
           include "sonic3k.macrosetup.asm"   ; include a few basic macros        
           include "sonic3k.constants.asm"       ; include a few basic macros
           include "sonic3k.macros.asm"       ; include some simplifying macros and functions
           include "Debugger.asm"               ; include debugger macros
    This will allow you to use all the macros and commands shipped with the Error handler (RaiseError, Console.Write, Console.SetXY etc).

    Step 3. Include Error handler blob
    Near the very end of sonic3k.asm, find "EndOfROM:" label, replace everything below it with the following:
    Code:
    EndOfROM:
       if Sonic3_Complete=0 
           include "ErrorHandler.asm"
           include "Lockon S3/LockOn Pointers.asm"       ; lock-on compatibility is broken, this was left only to keep the labels declared
       else
           include "Lockon S3/LockOn Data.asm"
           include "ErrorHandler.asm"
       endif
      
       END
    Please note that appending Error handler blob to the end of Sonic & Knuckles ROM will definitely break the lock-on compatibility, unless you manage to fit both blob and generated symbol data (if included) in 2 MB. The Sonic 3 Complete version shouldn't be affected though.

    ErrorHandler.bin is debugger blob itself, ErrorHandler.asm specifies error handler options (that you can alter if you want) and also declares handlers for standard error exceptions and their respective messages (which you are free to alter as well, they also support formatting).

    Now, you should add new error handlers defined in ErrorHandler.asm to the vectors table in the beginning of the ROM.
    Since Sonic & Knuckles doesn't even has code to display error messages, all error exceptions in the vector table refer to the "ErrorTrap" subroutine, which simply freezes processor in an infinite loop.
    Near the beginning of sonic3k.asm, swap the entire vectors table found at "Vectors:" label with the following:
    Code:
    Vectors:   dc.l   $00000000,   EntryPoint,   BusError, AddressError   ; 0
           dc.l   IllegalInstr, ZeroDivide, ChkInstr, TrapvInstr   ; 4
           dc.l   PrivilegeViol, Trace, Line1010Emu,   Line1111Emu   ; 8
           dc.l   ErrorExcept, ErrorExcept, ErrorExcept, ErrorExcept   ; 12
           dc.l   ErrorExcept, ErrorExcept, ErrorExcept, ErrorExcept   ; 16
           dc.l   ErrorExcept, ErrorExcept, ErrorExcept, ErrorExcept   ; 20
           dc.l   ErrorExcept, ErrorTrap,   ErrorTrap,   ErrorTrap   ; 24
           dc.l   H_int_jump,   ErrorTrap,   V_int_jump,   ErrorTrap   ; 28
           dc.l   ErrorTrap,   ErrorTrap,   ErrorTrap,   ErrorTrap   ; 32
           dc.l   ErrorTrap,   ErrorTrap,   ErrorTrap,   ErrorTrap   ; 36
           dc.l   ErrorTrap,   ErrorTrap,   ErrorTrap,   ErrorTrap   ; 40
           dc.l   ErrorTrap,   ErrorTrap,   ErrorTrap,   ErrorTrap   ; 44
           dc.l   ErrorTrap,   ErrorTrap,   ErrorTrap,   ErrorTrap   ; 48
           dc.l   ErrorTrap,   ErrorTrap,   ErrorTrap,   ErrorTrap   ; 52
           dc.l   ErrorTrap,   ErrorTrap,   ErrorTrap,   ErrorTrap   ; 56
           dc.l   ErrorTrap,   ErrorTrap,   ErrorTrap,   ErrorTrap   ; 60
    Now let's try to build Sonic & Knuckles ROM and see what happens. Run "Build Scripts\buildSK.bat" and check out the console output.

    Normally, you may found out assembly results in hundreds of errors within the original code, that we haven't even touched. Unfortunately, the AS Macroassembler, which is used in Sonic & Knuckles disassembly is unstable and has many issues, especially when complex macros are involved. There's not much we can do to fix assembler itself, but the are always some workarounds...

    Step 4. Fixing disassembly issues
    1) The first (and main) thing you'll notice in error logs is that named temporary labels are totally broken.
    For intance, you'll get the following error inside the init code:
    Code:
    > > >sonic3k.asm(303): error: symbol undefined
    > > > matchingChar688
    > > >        beq.s   $$matchingChar
    While you can clearly see the label below the said line in sonic3k.asm:
    Code:
           beq.s   $$matchingChar
           moveq   #1,d3
    
    $$matchingChar:
    This is due to a bug of the AS Macroassembler, when the internal label counter used to resolve named temporary labels isn't restored properly after the previous pass.

    The easiest workaround is to get rid of these type of labels entirely in favor of so-called composed local symbols, which start with the dot (".").
    Thus, "$$matchingChar" should become ".matchingChar" et cetera. The main difference is that composed symbols doesn't use numbering, but instead refer to the nearest global symbol preceding them. Hence, if the ".matchingChar" temporary symbol is found below the "Test_SystemString" global symbol, it implicitly resolves to "Test_SystemString.matchingChar".

    Replace every occurance of "$$" with "." in sonic3k.asm. If you also use $$-prefixed labels somewhere else in your custom .asm-files, fix them as well.

    Try assemblying the Sonic & Knuckles ROM.

    2) Additionally, if you receive "Compressed sound driver might not fit" error upon generating the final ROM, change the "Size_of_Snd_driver(2)_guess" constant in the beginning of sonic3k.asm to whatever error description suggests.

    Step 5. Configure symbol extraction
    It's what ConvSym.exe comes for!
    This utility collects data about all symbols used in your source code and their corresponding offsets and stores them in a compressed base in a special format that Error Handler understands.
    The symbol data is appended to the end of the ROM, so Error Handler can access it. In case of Sonic 2, which uses AS macro assembler, data can be only extracted from a listing file generated by the assembler.

    In order for symbol data to be accessed correctly, you should make sure ROM padding is disabled.

    Find the "strip_padding" constant at the beginning of sonic3k.asm, change it as follows:

    strip_padding = 1
    ; If 1, strips all unnecessary padding

    Finally, open "Build Scripts\buildSK.bat" and find this line:
    Code:
    IF EXIST skbuilt.bin goto LABLEXIT
    Replace it with:
    Code:
    IF EXIST skbuilt.bin (
       ConvSym sonic3k.lst skbuilt.bin -input as_lst -a
       goto LABLEXIT
    )
    Do the same for "Build Scripts\buildS3Complete.bat", if needed. In case of S3 Complete, you should target sonic3k.bin instead of skbuilt.bin.

    NOTICE: Run ConvSym.exe in command line without arguments to see full options list.

    Congratulations, you've done it!


    Quick debugger commands (macros) reference


    Debugger macros
    Code:
            RaiseError text[, handler]
    Displays error screen with the specified message.
    • text -- a formatted string representing error message, for example: "Object at address %<.w a0 hex> crashed"; displays in error screen header.
    • handler (optional) -- label of the console program (subroutine) used to print error screen body; if omitted, standard error handler is used.

    Code:
            Console.Run handler
    Switches game into console mode, runs the console program specified.
    • handler -- label of the console program (subroutine) to be run in console mode.
    RaiseError and Console.Run may call the same console programs, as they basically initiate the same mode. In case of RaiseError, an error header is automatically printed, while Console.Run just starts with an empty screen.
    Console programs are basically the same subroutines and must end with "rts" opcode. The only difference from ordinary routines is that they are executed by console subsystem, hence additional commands can be used in this mode to address console output:

    Code:
            Console.Write text
    Code:
            Console.WriteLine text
    Can be used only in console programs, called by RaiseError or Console.Run.
    • text -- a formatted string to write in console
    .WriteLine version automatically puts a newline after string is written.
    .Write version doesn't add newline, so the next write will append to the same line. You can use %<endl> flag though to put newlines manually, for example: "Ready...%<endl>Set...%<endl>Go!"

    Code:
            Console.SetXY  x, y
    Sets write position to the specified position of screen (in tiles).
    • x -- operand, represents x position in tiles
    • y -- operand, represents y position in tiles
    Screen size is 40 x 28 tiles.
    Leftmost position: Console.SetXY #0, #0

    Since x, y are operands, all M68K addressing modes may be used to pass these parameters, for instance:
    Console.SetXY d0, PositionData+1(pc,d1)


    Code:
            Console.BreakLine
    Adds a newline.
    Alternative: Console.Write "%<endl>"

    Formatted strings reference
    Formatted strings may include flags or formatted values, which are encapsulated in "%<...>" tokens, for example:
    Code:
            Console.Write "d0 equals to %<.b d0 hex|signed>, and... %<endl>d1 is %<.w d1>"
    List of flags:
    • %<endl> -- end of line flag, adds a newline;
    • %<cr> -- carriage return, jump to the beginning of the same line;
    • %<pal0> -- use palette line #0;
      %<pal1> -- use palette line #1;
      %<pal2> -- use palette line #2;
      %<pal3> -- use palette line #3;
    • %<setw,X> -- set line width: number of characters before automatic newline
      • by default, X=40 in console-only mode, X=38 on error screens
      • WARNING! In AS version, you have to write "%<setw>%<X>" due to macros limitations
    • %<setx,X> -- set X-position of the next character on the line;
      • WARNING! In AS version, you have to write "%<setx>%<X>" due to macros limitations
    Flags can be merged into a single token, for example: instead of "%<endl>%<setx,2>%<pal0>" you can just write "%<endl,setx,2,pal0>".
    WARNING! In AS version, merging flags is not supported

    Formatted values tokens:

    %<type operand[ format]>
    • type -- must be .b, .w or .l
    • operand -- source register or memory address, supports the same addressing modes as MOVE command
    • format (optional) -- specifies value formatter and it's arguments, if needed. Default format is hex.
      • hex -- display as hexadecimal number
      • dec -- display as decimal number
      • bin -- display as binary number
        • hex|signed, dec|signed, bin|signed -- treat value as signed number (additionally displays + or - before the number depending on its sign)
      • sym -- treat value as offset, decode into symbol+displacement
      • str -- treat value as offset, display C-string pointered by it
    Examples:
    Code:
        move.w   #$F211, d0
       Console.WriteLine "%<.b d0>"            ; prints "11"
       Console.WriteLine "%<.b d0 dec>"       ; prints "17"
       Console.WriteLine "%<.w d0>"            ; prints "F211"
       Console.WriteLine "%<.w d0 hex|signed>"   ; prints "-0DEF"
       Console.WriteLine "%<.b d0 hex|signed>"   ; prints "+11"
       rts
    
    Code:
       lea       SomeData, a0
       moveq   #1, d0
     
       Console.WriteLine "a0 = %<.l a0 sym>"       ; prints "a0 = somedata"
       Console.WriteLine "%<.b SomeData(pc,d0)>"   ; prints "19"
       addq.w   #1, d0
       Console.WriteLine "%<.b SomeData(pc,d0)>"   ; prints "B3"
    
       Console.WriteLine "%<.l #SomeString str>"   ; prints "Apples!"
    
       rts
     
    SomeData:
       dc.b   $AE, $19, $B3, $10
     
    SomeString:
       dc.b   "Apples!", 0
    
    WARNING! Trying to display the value of register SP (also known as A7) or address using it (e.g. %<.l -4(sp)>) leads to unexpected results. This is because formatted strings arguments are stored on stack at run-time, so stack pointer's value is different by the time it's requested.
    Do not try to print SP directly, the results are unreliable.

    WARNING! AS version has limitations and only supports register direct, register indirect and absolute addressing modes. Absolute mode is only supported when passing symbols, not raw addresses, e.g.:
    Console.Write "%<.w $FFFFEE00>" doesn't work in AS, but Console.Write "%<.w Camera_X_Pos>" does.
    ASM68K version supports all standard addressing modes that M68K provides.

    NOTICE: Please also refer to Debugger.asm for throughout technical description of constants used by macros.


    Testing with a sample debugger

    Sonic 1 (Github and Hivebrain disassemblies)
    Now, if you want to see the debugger in action, let's play with some sample code, shall we?

    Create SampleLevelDebugger.asm file with the following sample code:
    Code:
    SampleLevelDebugger:
       Console.WriteLine "Camera (FG): %<.w $FFFFF700>-%<.w $FFFFF704>"
       Console.WriteLine "Camera (BG): %<.w $FFFFF708>-%<.w $FFFFF70C>"
       Console.BreakLine
     
       Console.WriteLine "%<pal1>Objects IDs in slots:%<pal0>"
       Console.Write "%<setw,39>"       ; format slots table nicely ...
    
       lea       $FFFFD000, a0
       move.w   #$2000/$40-1, d0
     
       @DisplayObjSlot:
           Console.Write "%<.b (a0)> "
           lea       $40(a0), a0
           dbf       d0, @DisplayObjSlot
    
       rts
    
    Now, include it in sonic1.asm (sonic.asm in GitHub version):
    Code:
        include   "SampleLevelDebugger.asm"
    I recommend including your console programs near the actual debugger, right above
    Code:
        include   "ErrorHandler.asm"
    but not below it! (or error handler expects symbol data to be appended below later...)

    Almost done! Now, let's actually call the debugger, for instance, when player presses A on the level.
    Find "Obj01_Normal:" ("Sonic_Normal:" in GitHub version) and right below the label, place the following code as shown:
    Code:
    Obj01_Normal:
       btst   #6, $FFFFF602
       beq.s   @skip
       RaiseError "Intentional crash test:%<endl>Level ID = %<.w $FFFFFE10>%<endl>Frame = %<.w $FFFFFE04>", SampleLevelDebugger
    @skip:
    
    That's it. Compile ROM, go to any level, press A and see the results!

    [​IMG]

    Sonic 2 (Github Disassembly)
    Now, if you want to see the debugger in action, let's play with some sample code, shall we?

    Create SampleLevelDebugger.asm file with the following sample code:
    Code:
    SampleLevelDebugger:
       Console.WriteLine "Camera (FG): %<.w Camera_X_pos>-%<.w Camera_Y_pos>"
       Console.WriteLine "Camera (BG): %<.w Camera_BG_X_pos>-%<.w Camera_BG_Y_pos>"
       Console.BreakLine
    
       Console.WriteLine "%<pal1>Objects IDs in slots:%<pal0>"
       Console.Write "%<setw>%<39>"   ; format slots table nicely ...
    
       lea      $FFFFB000, a0
       move.w   #$2600/next_object-1, d0
    
    -  Console.Write "%<.b (a0)> "
       lea       next_object(a0), a0
       dbf       d0, -
    
       rts
    
    
    Now, include it in s2.asm:
    Code:
        include   "SampleLevelDebugger.asm"
    I recommend including your console programs near the actual debugger, right above:
    Code:
        include   "ErrorHandler.asm"
    but not below it! (or error handler expects symbol data to be appended below later...)

    Almost done! Now, let's actually call the debugger, for instance, when player presses A on the level.
    Find "Obj01_Normal:" and right below the label, place the following code as shown:
    Code:
    Obj01_Normal:
        btst   #6, $FFFFF602
       beq.s   +
       RaiseError "Intentional crash test:%<endl>Level ID = %<.w Current_ZoneAndAct>%<endl>Frame = %<.w Timer_frames>", SampleLevelDebugger
    +
    
    That's it. Compile ROM, go to any level, press A and see the results!

    [​IMG]
     
    Last edited: Apr 22, 2018
  9. Novedicus

    Novedicus Well-Known Member Member

    Joined:
    Aug 26, 2013
    Messages:
    799
    Now this...

    this is fucking awesome.
     
    vladikcomper likes this.
  10. MarkeyJester

    MarkeyJester Blue hair? What a freak! Member

    Joined:
    Jun 27, 2009
    Messages:
    2,622
    Well done vlad!

    I am surprised you have had it complete so soon, it felt like only yesterday you were in London showing me your printf functionality. It looks as solid here as it did back then~

    The responses are slow here at the moment, but I do hope people will slowly appreciate the effort you've put into this, and the amount of control you have just handed to everyone on a plate, let's hope they comprehend and respect it enough to put it to its optimal use.

    Congratulations on your release.
     
    vladikcomper and Natsumi like this.
  11. Natsumi

    Natsumi Markey's Member

    Joined:
    Oct 7, 2011
    Messages:
    621
    Location:
    Otter's lap
    This is pretty cool, but there seems to be an issue; I can not fit the code for the RaiseError macro into my code, so I want to jump out to a subroutine to do this call elsewhere. But as it turns out, this means that the location/module is incorrectly set to this routine. Any way to add a function so I can jsr into this routine, and the source routine gets displayed instead (and make sure the caller is also correct)? Also, calling a subroutine from the console handler breaks the location also.
     
    vladikcomper likes this.
  12. DigitalDuck

    DigitalDuck Active Member Member

    Joined:
    Apr 27, 2017
    Messages:
    38
    Location:
    Lincs, UK
    This is definitely something I'd be able to make use of, and it looks amazing!
     
    vladikcomper likes this.
  13. LazloPsylus

    LazloPsylus A Certain Scientific Railgun The Railgun

    Joined:
    Nov 25, 2009
    Messages:
    Location:
    Academy City
    Congratulations are indeed in order, vlad. One of the best general tools for debugging now has even more extensibility? Yes, please.
     
    vladikcomper likes this.
  14. vladikcomper

    vladikcomper Well-Known Member Member

    Joined:
    Dec 2, 2009
    Messages:
    382
    Location:
    Russia
    Aww, thank you so much for your kind words guys :3
    I'm so happy you appreciate all the hardwork that went into this.

    Also, I've got a little update (see below)

    I feel the same, but... oh my, time goes so fast! I can't believe it's been almost 5 months already. I hope to visit London again, maybe one day, in the future~
    By the way, console core went through one last overhaul since the last time you saw it. In fact, I revised it so much, I can't even remember. If it wasn't for weeks of planning, testing and rewrites, it wouldn't have been even remotely as powerful, let alone stable. I decided to take all the time I needed to get the core as fast, clean and stable as possible.

    If I get you right, you need to trick error handler into thinking RaiseError originated from a different location?
    This can be easily done, actually, since RaiseError macro generates code that explictly pushes exception location to the stack:

    Code:
    RaiseError &
       macro   string, console_program, opts
    
       pea       *(pc)
       move.w   sr, -(sp)
       <...>
    
    As you see, it's current location by default, but you can modify or extend macros to swap this location to anything you want.

    But in your case, I suggest you to add option to disable "pea *(pc)" in RaiseError completely, so the last return address pushed to stack will be treated as exception location instead. This means, if you JSR to a code that does RaiseError, error handler will think the exception happened at that JSR, but not at the actual location of RaiseError.
    This should also get the caller you want instead of caller to the code containing RaiseError command.

    * * *

    UPDATE:
    • Added Installation guide for Sonic 2 Disassembly (AS version)
    • Added Sample Debugger for Sonic 2
    • Hotfix for AS version of debugger (please re-download if you use this version)
    See version 2.0 release post for all these updates.

    --
    It appears that some builds of AS v.1.42 have a bug in one of its built-in functions, strstr() specifically.
    The strstr() function is meant to return position of a substring in a given string (which is really frequently used by debugger macros that parse formatted strings), but in some builds for some weird reason it fails to find substring if it's located at the very end of the string.
    For example, strstr("abc","c") returns -1 as if "c" is not found in the string, while it's clearly on position 2. However, strstr("abc","b") will work correctly, since "b" is not at the end of the string.
    The bug is present in v.1.42 Bld 88 found in the current version of Sonic 2 disassembly, but it doesn't happen in v.1.42 Bld 55 that I used for testing earlier.
    Due to this unexpected behavior, formatted strings failed to compile correctly under certain circumstances, unexpected errors were thrown during assembly.
    The recent update fixes this issue and works on both bugged and stable AS versions.
     
    Last edited: Jan 15, 2018
    Pacguy, FохConED, Novedicus and 3 others like this.
  15. Spanner

    Spanner The Tool Administrator

    Joined:
    Aug 9, 2007
    Messages:
    2,253
    In terms of colour formatting, can that include the background colour? A blue screen of death would be nice.
     
    NoneofyourBusiness and Pacguy like this.
  16. vladikcomper

    vladikcomper Well-Known Member Member

    Joined:
    Dec 2, 2009
    Messages:
    382
    Location:
    Russia
    Alright, another update everyone!

    I actually planned to do this on the very first day, but decided to take extra time for some clean-up, organizing and documenting...


    The Advanced Error Handler and Debugger goes full open-source!
    Source code is now available at Github repository: https://github.com/vladikcomper/md-modules
    This includes all the custom utilities and test suites used throughout the development.
    The repository has the following structure:
    • /utils -- the collection of cross-platform utilities used for/by debugger, written in C++
      • /utils/convsym -- source code for ConvSym, the symbol extraction utility
      • /utils/cbundle -- source code for CBundle, custom pre-processor used to build AS and ASM68K bundles from the shared "cross-assembler" source files
      • /utils/core -- base C++ classes used by utilities
    • /modules -- the collection of debugging modules, written in M68K assembly
      • /modules/errorhandler -- source code for Error Handler blob
        • /modules/errorhandler/bundles -- bundles source code that define debugger macros and integrate pre-compiled Error Handler blob into your projects.
      • /modules/core -- source code for debugger's core, used by Error Handler, which includes Console subsystem
    Utilities can be compiled with GCC (build scripts included) under Windows, Linux or MacOS.
    Unfortunately, I hardly know Linux, I don't own Mac either, so I never made proper builds for these platforms.
    However, Linux versions were built and partly tested (under 64-bit Ubuntu), so I believe it should work perfectly on all platforms. Linux binaries are included in the source code if someone needs them (but again, I'm too inexperienced in Linux, sorry; and I currently don't have Linux machine, so I cannot support this platform yet)

    * * *

    But wait, there's more~

    Tutorial: Adding your own header on the error screen

    aka Inserting strings in error messages

    This was #1 request for a while, so I guess I should cover this.
    Trust me, there's really nothing extraordinary here, nothing that you couldn't pick up by yourself by carefully reading strings reference and definitions files.
    Open ErrorHandler.asm file and scroll down a little until you see this bit:
    Code:
    BusError:
       __ErrorMessage "BUS ERROR", _eh_default|_eh_address_error
    
    AddressError:
       __ErrorMessage "ADDRESS ERROR", _eh_default|_eh_address_error
    
    IllegalInstr:
       __ErrorMessage "ILLEGAL INSTRUCTION", _eh_default
       <...>
    
    These are standard error vectors and __ErrorMessage is just a special macro that calls error screen, almost identical to RaiseError, except it can only use standard error handler and it works inside error exception already formed by the processor (RaiseError generates one itself).

    As you can easily guess, you can edit or extend these error messages as you wish:
    Code:
       __ErrorMessage "Game crashed. Error exception raised: ILLEGAL INSTRUCTION", _eh_default
    And don't worry about the text length, everything will fit as console has automatic newlines:
    [​IMG]
    More over, since __ErrorMessage is just another variation of RaiseError, formatted strings are supported here as well:
    Code:
        __ErrorMessage "%<pal0>And then the game crashed...%<endl>The following error exception raised: %<pal1>ILLEGAL INSTRUCTION", _eh_default
    [​IMG]
    We could just stop now and copy-paste the same "header text" into all standard error messages...
    But isn't that redundant? It certainly is. Not to say, it's harder to alter the text when there are so many copies of it...

    It's time to take advantage of debugger's high-level features!
    More specifically, support of processing C-strings (null terminated strings).

    Alter standard error messages code as shown below:
    Code:
    Str_ErrorHeader:
       dc.b   "~This is common header string~", 0
       even
          
    ; ---------------------------------------------------------------
    BusError:
       __ErrorMessage "%<.l #Str_ErrorHeader str>BUS ERROR", _eh_default|_eh_address_error
    
    AddressError:
       __ErrorMessage "%<.l #Str_ErrorHeader str>ADDRESS ERROR", _eh_default|_eh_address_error
    
    IllegalInstr:
       __ErrorMessage "%<.l #Str_ErrorHeader str>ILLEGAL INSTRUCTION", _eh_default
    
       <...>
    That's right, you can define a C-string and insert it anywhere you want (in debugger's formatted strings).
    Now we have a proper common header string, stored separately from the error messages:

    [​IMG]
    Well, looks like it still lacks something? Proper formatting, that is.

    Now, here's another hidden gem: C-strings can use formatting flags as well!
    The formatting approach is different here though: no pre-parsed tokens (%<...>) and only flags are supported (you cannot insert and format register and memory values).
    Try this one:
    Code:
    Str_ErrorHeader:
       dc.b   pal0,"This hack is experiencing technical", endl
       dc.b   "difficulties, please stand by~", endl, endl
       dc.b   pal3,"Please send the following information", endl
       dc.b   "to ",pal2,"megahacker777@mail.kool",pal3,":",pal0,endl,endl
       dc.b   0
       even
    
    [​IMG]

    IMPORTANT!
    Never forget to put null-terminator in C-strings that you type with DC.B directive:
    Code:
            dc.b   "This is null-terminated string -->", 0
            even
    A good practice is also to put "even" after DC.B that stores the string, to ensure the following data gets aligned properly.
    Don't confuse C-strings with formatted strings you pass to debugger's commands (macros) as arguments, C-strings are just plain byte arrays!

    WARNING AS-USERS!
    AS versions of debuggers macros has limited support for formatted strings (see "Quick formatted strings reference" in the release post).
    Specifically, inserting data using immediate addressing mode (%<.. #ImmediateValue ..>) is not supported, so the tutorial may appear broken.
    To bypass this limitation you have to use absolute addressing and load string pointer instead:
    Code:
    Str_ErrorHeader:
       dc.b   "~This is common header string~", 0
       even
    
    PtrStr_ErrorHeader:
       dc.l  Str_ErrorHeader
          
    ; ---------------------------------------------------------------
    BusError:
       __ErrorMessage "%<.l PtrStr_ErrorHeader str>BUS ERROR", _eh_default|_eh_address_error
    
    AddressError:
       __ErrorMessage "%<.l PtrStr_ErrorHeader str>ADDRESS ERROR", _eh_default|_eh_address_error
    
    IllegalInstr:
       __ErrorMessage "%<.l PtrStr_ErrorHeader str>ILLEGAL INSTRUCTION", _eh_default
    
       <...>

    * * *

    Well, yes and no.
    If you mean set background color of each character, this is supported by the console core, but not implemented in error handlers.
    There are even undocumented flags for it (setpat and setoff) which you can find in Debugger.asm definitions. They have no desired effect in error handler, because it initializes console with a single copy of font, while additional copies of the font with different colour indecies must be produced in order for the trick to work (otherwise, you're stuck with only 4 color choices determined by the current palette line alone).

    But since I've just released the source code for Error Handler blob, you can easily modify and recompile it to support background colors!
    In ErrorHandler.asm (in the source code, not in the bundle!) alter "ErrorHandler_ConsoleConfig:" as follows:
    Code:
    ; ===============================================================
    ; ---------------------------------------------------------------
    ; Console loading programme for Error Handler
    ; ---------------------------------------------------------------
    
    ErrorHandler_ConsoleConfig:
    
       ; ---------------------------------------------------------------
       ; Font decompression programme
       ; ---------------------------------------------------------------
    
       dcvram   VRAM_Font                   ; font offset in VRAM
       dc.w   $1111, $1112, $1121, $1122   ; decompression table for 1bpp nibbles
       dc.w   $1211, $1212, $1221, $1222   ; ''
       dc.w   $2111, $2112, $2121, $2122   ; ''
       dc.w   $2211, $2212, $2221, $2222   ; ''
                                         
       ; Produce additional font for background colour #1
       dcvram   (VRAM_Font+$2000)           ; font offset in VRAM
       dc.w   $3333, $3332, $3323, $3322   ; decompression table for 1bpp nibbles
       dc.w   $3233, $3232, $3223, $3222   ; ''
       dc.w   $2333, $2332, $2323, $2322   ; ''
       dc.w   $2233, $2232, $2223, $2222   ; ''
    
       dc.w   -1                           ; end marker
    
       ; ---------------------------------------------------------------
       ; Console RAM initial config
       ; ---------------------------------------------------------------
    
       dcvram   VRAM_PlaneA                   ; screen start address / plane nametable pointer
       dc.w   40                           ; number of characters per line
       dc.w   40                           ; number of charasters on the first line (meant to be the same as the above)
       dc.w   0                           ; base font pattern (tile id for ASCII $00 + palette flags)
       dc.w   $80                           ; size of screen row (in bytes)
    
       dc.w   $2000/$20-1                   ; size of screen (in tiles - 1)
    
       ; ---------------------------------------------------------------
       ; CRAM data
       ; ---------------------------------------------------------------
    
       dc.w   $0000, $0EEE, $0600, -6*2                   ; line 0: white text
       dc.w   $0000, $00CE, $0600, -6*2                   ; line 1: yellow text
       dc.w   $0000, $0EEA, $0600, -6*2                   ; line 2: lighter blue text
       dc.w   $0000, $0E86, $0600, -6*2                   ; line 3: darker blue text
    

    That's it! Compile and you can now set your text an additional background color, which is dark blue ($0600):

    Code:
    MyErrorHandler:
       Console.WriteLine 'Hello, this is a dummy debugger~'
     
       Console.WriteLine '%<pal0>d7 is %<pal2>%<.w d7>%<pal0> in hex'
       Console.WriteLine '- which is %<pal2>%<.w d7 dec>%<pal0> in dec'
       Console.WriteLine '- which is %<pal2>%<.w d7 bin>%<pal0> in binary'
     
       Console.WriteLine 'Look, %<setpat,1>bac%<pal2>kgro%<pal3>und %<pal1>colors%<setpat,0>!'
    
       rts
    [​IMG]
     
    Last edited: Jan 21, 2018
    Calvin, Crash, StephenUK and 9 others like this.
  17. Natsumi

    Natsumi Markey's Member

    Joined:
    Oct 7, 2011
    Messages:
    621
    Location:
    Otter's lap
    Could you add an ability for the debugger to return execution into user code? I would specifically like to poll the controller, to be able to move the error screen around (with skipping certain writelines), so I can display much bigger error report than there is space.
     
    Calvin likes this.
  18. Tanner

    Tanner I play video games and do computer stuff! Member

    Joined:
    Dec 23, 2016
    Messages:
    28
    Could you create a tutorial for installing this into the Sonic 3 & Knuckles GitHub Disassembly?
     
  19. vladikcomper

    vladikcomper Well-Known Member Member

    Joined:
    Dec 2, 2009
    Messages:
    382
    Location:
    Russia
    This is pretty easily done by modifying the macros. You don't even have to modify or recompile the blob (ErrorHandler.bin).

    For Console.Run command:
    In Debugger.asm, inside definition of Console macro, find the following code (line 129 in ASM68K version):
    Code:
        elseif strcmp("\0","run")
           jsr       ErrorHandler.__extern__console_only
           jsr       \1
           bra.s   *
    
    The first call to "__extern__console_only" initializes the console environment.
    The \1 in the following "jsr \1", as you know, substitutes the first argument that you pass to Console.Run, which is the location of your console program.
    The last "bra.s *" opcode is executed once your console program is finished (returned), and it obviously just lock-ups execution.

    Well, you get the idea~
    Just replace "bra.s *" with whatever handler you like to be able to manipulate console after your console program finishes.

    For RaiseError command:
    In Debugger.asm, inside definition of RaiseError macro, find the following code (line 91 in ASM68K version):
    Code:
        if strlen("\console_program")           ; if console program offset is specified ...
           dc.b   \opts+_eh_enter_console|(((*&1)^1)*_eh_align_offset)   ; add flag "_eh_align_offset" if the next byte is at odd offset ...
           even                                                           ; ... to tell Error handler to skip this byte, so it'll jump to ...
           jmp       \console_program                                       ; ... an aligned "jmp" instruction that calls console program itself
    
    The strategy is pretty similar here,
    Replace "jmp \console_program" with "jsr \console_program", add "jmp YourHandler" next to it.

    Speaking of scrolling handlers, in ErrorHandler.asm, you may find this code:
    Code:
        if ref(ErrorHandler.__extern_scrollconsole)
    ErrorHandler.__extern__scrollconsole:
    
       endc
    
    This is an empty routine, intended to handle console scrolling. I simply didn't have enough time to implement it in the first version 2.0 release, but it is planned for the future updates.

    Console subsystem itself also has a little setup to help implement scrolling easier.
    First of all, the draw area is 512 x 512 pixels large (64 x 64 tiles), which means you can draw a lot of stuff offscreen (mainly vertically) and then scroll down to see it. You can keep up to 64 lines of text for scrolling (good for logging stuff), without need to redraw/erase them to fit new lines.
    Second of all, the text drawing functions, .Write and .WriteLine, have logic to warp drawing positions inside this 64 x 64 tile drawing area, so you won't have to worry about warping yourself. The horizontal warping is controlled by <setw,X> flag (where X defaults to 40 tiles in "clean" console mode, which is exactly the width of the screen), and the vertical warping happens automatically once you fill all 64 lines to allow for continuous logging.

    Yes, S3K installation guide is planned. Unfortunately, I've been too busy recently to finish this and a few other guides planned.
    But since you're asking, I may hurry up with this one and get it done during the next week.
     
  20. vladikcomper

    vladikcomper Well-Known Member Member

    Joined:
    Dec 2, 2009
    Messages:
    382
    Location:
    Russia
    Small update everyone,
    As promised, I've added an installation guide for Sonic 3 & Knuckles. Check out version 2.0 release post above for the new guide.

    It actually took some time because of another portion of AS bugs that I encountered, and the overall disassembly structure that I didn't know very well.
    Sonic & Knuckles is by far the most "unstable" AS-based disassembly I had to deal with due to compressed sound driver and macro shenanigans. For some reason, AS likes to fail a lot on this one, has anything been even slightly altered or expanded.
    I've already heard from several people about AS randomly "forgetting" defined labels, failing to calculate branch size correctly or even throwing dozens of addressing mode for no logical reason. Some of these issues I've already faced myself when working on the AS branch of the debugger, then on the Sonic 2 installation guide. Interestingly, in Sonic & Knuckles, chances you trigger these hardly avoidable assembler bugs are amplified for some reason, especially when sophisticated debugger environment is installed. Bear this in mind~