clownmdemu - The Greatest Mega Drive Emulator Ever (Someday)

You're right, I forgot that some Mega CD rips lack a CUE file. Thanks for the reminder - that's yet another format that I should add support for.

 

v0.7.1
Try it in your web browser: clownmdemu.clownacy.com
Download: https://github.com/Clownacy/clownmdemu-frontend/releases/tag/v0.7.1

Being that it is currently a weekend, I have some spare time to release another update. Mainly, this update adds support for more Mega CD disc rip formats, but there is another improvement that I wanted to highlight as well:

Fix Window Icon Blurriness on High-DPI Displays
A user shared this screenshot of the emulator in a GitHub Issue:



This desktop is running at an extremely high resolution, which is made evident by the high-resolution font rendering. However, when looking at the very top-left of the screenshot, something odd can be seen...



Right next to the crisp, high-resolution font is an awfully blurry, low-resolution icon. I found this to be very strange, as the icon which I made for the emulator supported resolutions up to 1024x1024, and yet here the window is clearly using the 16x16 version of the icon.

I eventually found that setting the icon of the window is the responsibility of the SDL2 library, and that it was using a peculiar method of extracting "default" icons from the running executable: it would obtain the full path to the executable, use the ExtractIconEx WinAPI function to extract specific large and small icons from it, and then assign those to the window. I found this odd because it seemed unnecessary to obtain the path to the current executable when SDL already has a handle to the executable. Additionally, WinAPI provides functions to obtain icons from the current executable by loading them as resources, which, to my understanding, is the standard method which the majority of people use.

Icons can exist on their own or in a group. It appears that ExtractIconEx returns lone icons rather than icon groups, and that the icons which it returns are not the ideal resolution for high-DPI displays. Windows relies on icon groups in order to support high-DPI displays, selecting the icon with the appropriate resolution from the icon group to be shown. Because ExtractIconEx was not returning an icon group, Windows was unable to do this, resulting in the 16x16 icon being used at all times.

SDL2 has a 'hints' system, allowing some degree of configuration. With it, I can force the window to use a particular icon or icon group, instead of relying on SDL2 to choose one automatically. Indeed, doing this fixes the issue, causing an icon with a decent resolution to be shown instead:



But this was not good enough for me: why should I, and every other user of SDL2, have to set a hint to make SDL2 use the correct icon resolution? SDL2 should get it correct by default! So, I took it upon myself to fix SDL2. According to this StackOverflow answer, in situations where Windows itself has to assume an executable's icon (such as when viewing it in File Explorer), it uses the first icon group within it. Using WinAPI's EnumResourceNames function, I made SDL2 do just this. As I had hoped, this fixed the issue completely, ensuring that the best icon resolution is always used!

With my work done, I submitted it as a Pull Request to SDL2's GitHub repository, and it was promptly merged! When SDL2 v2.30.3 or v2.32.0 is released, expect to see this bugfix included as a part of it! This improvement also benefits the upcoming SDL3, which boasts improved high-DPI support as one of its main features!

Support Many More Mega CD Disc Rip Formats
v0.7 only supported rips in the 'BIN+CUE' format. This is a basic format where the BIN file contains a dump of the disc's sectors, and the CUE file contains a transcription of the disc's Table of Contents. While this format does well to preserve data accurately, it uses a lot of space due to two things: 13% of the CD-ROM data being useless junk data, and the CD-DA audio being uncompressed PCM.

To address this, other formats exist, such as 'ISO+OGG+CUE'. In this format, the CD-ROM data is moved to its own ISO file with the junk data removed, and the CD-DA audio is also split to its own files and encoded in the Ogg Vorbis format. This greatly reduces the size of the rip: for example, the Japanese version of Sonic CD is reduced from 556MiB to 126MiB.

The downside to such a format is complexity: the rip is no longer a straightforward dump of raw, uncompressed sectors, with each one being exactly 2352 bytes; now, CD-ROM data is in sectors that are 2048 bytes in size, while CD-DA audio is encoded and lacks the notion of sectors altogether. The encoding in particular is a massive complexity, as formats such as Ogg Vorbis are extremely complicated and not very feasible to create a new decoder from scratch for, necessitating the use of pre-existing software libraries instead.

It is with such libraries that I have extended my emulator to support additional formats! In particular, stb_vorbis, dr_flac, dr_mp3, and dr_wav are used to support Ogg Vorbis, FLAC, MP3, and WAV audio! This should encompass the majority of formats commonly used by Mega CD rips. What is special about the libraries which I have chosen is that they minimise the concerns about portability and licensing, due to them being written in ANSI C and released under extremely-permissive licenses.

Another format that is now supported is the 'CUE-less' rip: in this format, there are only ISO and audio files, and no CUE file. With no CUE file linking them all together, these files are instead linked through their naming: the ISO file is named 'XXXX.iso', and the audio tracks are named 'XXXX 02.ogg', 'XXXX 03.ogg', etc. If the user makes the emulator load the ISO, then the emulator will automatically search for the audio files and load them too.

The 'CUE-less' rip in particular is the best format for homebrew developers, as they do not have to maintain a big fragile CUE file, nor do they have to go through the rigmarole of inserting junk data into their ISO file, decoding their audio to raw PCM, and combining it all into a gigantic BIN file.

With these formats supported, archivists, homebrew developers, and typical end-users alike are all catered to!

There is one format that still needs to be added, however: CHD. This is a format from MAME, which encodes the audio in FLAC, compresses the ISO in one of several compression schemes, and combines it all into a single file. This format has found popularity relatively recently, being added to many emulators under the RetroArch umbrella. It walks the line between targeting end-users and archivists, as it preserves the original data while still saving space through the use of compression. Unfortunately, the library which is used to decode CHD files is woefully undocumented, preventing me from integrating it into my emulator. I hope to eventually figure out how to use this library so that every common rip format is supported.

Closing
With this, the biggest shortcoming of the previous update has been addressed. The emulator should now be well-rounded enough to last people until the next update. I wonder what I will work on next?

 

(Cross-post from my blog).

v0.8
Try it in your web browser: clownmdemu.clownacy.com
Download: https://github.com/Clownacy/clownmdemu-frontend/releases/tag/v0.8.0.1

v0.8 is an update that was burdened by attempting a major change to the codebase. This change was meant to greatly improve the emulator's frontend, allowing for the elimination of numerous hacks within its codebase while making new features possible. Unfortunately, this did not go to plan, and all of the work on it cannot be released today. This change was the migration of the frontend from version 2 of the SDL library to version 3.

This would have been perfect for adding native multi-window support, but deficiencies in the pre-release SDL3 meant that the frontend would also be rendered extremely unstable and buggy. Instead, today's release is a refinement of the emulator based on its prior software stack. While it would be great to announce and show off native multi-window, that feature will have to wait until SDL3 matures. With that out of the way, here are the features that avoided disappointment.

Dark Title Bar on Windows 11


It is about time! Thanks to a tip from Ewan, the frontend now features a little WinAPI hack to set the title bar to the same colour as the Dear ImGui menu bar. This will only work on Windows 11 due to using a recent API.

Now Windows 11 users no longer have to put up with the ebony-and-ivory look that the emulator previously had.​

Linux AppImage Build
Linux executables tend not to be anywhere near as universal as Windows executables: their heavy reliance on system libraries and lack of standalone installers make it very difficult to share precompiled software in a form that will work without any hassle on a variety of Linux distributions. Because of this, users either have to compile software from source code, or wait for their distro to provide it as a package. Unfortunately, clownmdemu is not provided by any distro, meaning that compiling from source is the only option. However, this is not a simple process, as it requires recursively checking-out the Git repo and then bootstrapping the build system with CMake - things which beginner programmers are apparently unfamiliar with. All of this combined makes it quite difficult for a Linux user to obtain a useable copy of my emulator.

This is where AppImage comes in - AppImage is a standard for Linux software which bundles all files (executables, libraries, and data) into a single executable, making it very convenient to share, download, and run. Additionally, software is encouraged to be compiled on the oldest still-supported version of Ubuntu LTS, allowing it to run on a huge range of Linux distributions without risk of glibc incompatibilities. With AppImage, I can produce a Linux build that is easy to download and run while also being compatible with the vast majority of current Linux distributions!

Supporting AppImage was not particularly difficult: it merely requires that the emulator's build system be capable of installing the software in a way that adheres to some freedesktop.org standards - particularly providing an icon and basic metadata. Likewise, compiling with an old version of Ubuntu was not very difficult, with it only requiring that I mark some CMake scripts as being compatible with older versions of CMake, and also needing to switch from C++20's format library to its backward-compatible middleware version.

With that done, I am now able to create a clownmdemu AppImage! It is provided alongside the Windows executable in the GitHub releases.

But why stop there? AppImage and its two competing standards - Flatpak and Snap - all provide catalogues for users to conveniently browse and download compatible software. It is one thing to make my software easy to run, but it would be even better to also make it easy to find! So, I have added clownmdemu to appimage.github.io!

Fix for Sprite Data Not Being Cached on Prior Scan-Lines
The ROM-hack 'Gametap Pro Gamer 6' had a couple of splash screens which were not rendering correctly in this emulator:

Hey, that's me!​

The cause of this bug was surprisingly difficult to uncover, taking me several days. Eventually, I found that this issue was not the product of a bug, but rather a deliberate design choice: back when I had implemented sprite scan-line caching, I made it so that it would not cache the sprite data of scan-lines which had already been rendered. In hindsight, this makes absolutely no sense. I think this was a misguided optimisation, but, after all these years, I no longer remember why I implemented it in the first place. Regardless, by removing this anti-feature, this bug is now fixed:

I really am the best!​

Approximation of the YM2612's 'Ladder Effect'
A feature which purists often mention in relation to the Mega Drive is a curious bug that is present in early models of the console: the 'low-volume distortion' effect which is present in the YM2612 sound synthesiser, but not the YM3438 that is used in later models.

The exact details of how this bug works do not yet appear to be understood, however the general idea of it is that an offset is applied to samples on one half of the sound wave. Exactly how large this offset is is unclear, but I have seen at least one mention of it being approximately 4 decrements of the YM2612's internal 9-bit mixer value.

This bug causes quiet sounds to be louder than normal. Many songs and sounds were designed with this behaviour in mind, causing the absence of this bug to make them sound incorrect. For instance, here is a song from Earthworm Jim 2:

Without the low-volume distortion, the grating whining in the background is absent. According to Tommy Tallarico, this whining is an intended part of the song, rendering its absence a flaw.

Another major example of this is the After Burner II soundtrack, with three songs bearing distinct differences:

YM2612:

YM3438:

• Red Out has a descending note at the start which is inaudible without the low-volume distortion.
• Super Stripe has an electric piano-sounding instrument playing some backing notes which are much quieter without the low-volume distortion.
• After Burner has a lead melody whose notes decay much faster without the low-volume distortion.

These differences were egregious enough that they simply needed to be addressed, and so I have introduced a rough approximation of this bug into clownmdemu. Given its rough nature, and the fact that this bug does not exist (at least to the same extent) in later Mega Drives, an option to disable this is has also been added.

Have it your way!​

Support for 'KDEBUG' Debug Messages
It is not often that standards manifest within Mega Drive emulation: there is no standard save state format, no standard layout for modern gamepads, and not even an agreed standard for how to convert Mega Drive colours to modern sRGB RGB888. KDEBUG, however, is an exception:

KDEBUG is an extension to the Mega Drive's API that provides a way for software to log text for the user to view. While useless for players, this is great for developers, in the same way that C's 'printf' function is: it can be used to make the game report things to the developer that are difficult to determine otherwise, such as how many of a pool's objects are allocated, or how many entries are currently populating a linked-list.

This API originated in Gens KMod, which is a modified version of the historical Mega Drive emulator Gens. KMod introduced a vast number of debugging utilities to Gens, with this API being one of them. Many years later, this API was adopted by another emulator, BlastEm. This API would go on to be used by Vladikcomper's debugging framework (which is widely used by Sonic the Hedgehog ROM-hacks) as well as SGDK (which is widely used by Mega Drive homebrew). In all, KDEBUG has become quite ubiquitous.

Indeed, it has become so ubiquitous that I have received multiple requests for KDEBUG support be added to my emulator too.

The API itself is pretty simple: it uses one of the VDP's unused registers as a character stream output, allowing strings to be logged by feeding each character into the register one after the other. After the last character is sent, a null character (literally the byte 0x00) is sent, which the emulator responds to by presenting the complete string to the user. On a real Mega Drive, writing data to this register does nothing whatsoever, making it a harmless no-op. This allows software which uses the KDEBUG API to remain compatible with platforms which do not support it.

As with other developer notifications, KDEBUG messages are recorded to the emulator's Log window. By enabling the 'Log to Console' option, the messages will be sent to 'stderr', allowing them to be viewed from the terminal just like a native PC program.

An example of Vladikcomper's debugger logging which chunk data is being loaded.​

Accurate Motorola 68000 Instruction Durations
CPUs work by processing instructions. These instructions can do things like add two numbers together, perform a multiplication, and so on. Different instruction take a different amount of time to execute... and this is what my emulator has been failing to do for its entire existence.

The Z80 CPU emulator has had proper instruction durations for a long time now, but that was due to the necessity of it: games rely on the Z80's timing to be correct so that audio samples do not play at the wrong speed. However, for the Mega Drive's main CPU, the 68000, this necessity did not exist: many games worked perfectly with every instruction taking 10 cycles, and so my emulator did just that.

This remained the case until I overhauled how Z80 interrupts worked, which exposed an issue: in Sonic 2, with each 68000 instruction taking 10 cycles, the Z80 interrupt would consistently occur right in the middle of a long Z80 bus request operation, causing the interrupt to be lost, which meant that the music would play at an uneven, slow speed.

Rather than mitigate the consequences of this hack by adding another hack, I opted to eliminate the hack by finally implementing proper 68000 instruction durations. This was done by creating a test program which ran the 68000 CPU interpreter in isolation and compared the results to a sprawling series of tests whose data was derived from an extremely accurate microcode-level emulator. This way, I could compare my emulator's behaviour to that of the accurate emulator. This is something that I had already done previously back in late 2021, but I only verified instruction behaviour rather than instruction durations. With this new program, I could verify both!

All instructions now have accurate durations! With this, this music in Sonic 2 no longer plays at a wonky speed, and games should lag (or not lag) more consistently with how they do on a real Mega Drive.

True Multi-Window
Ever since debug menus were first added, they looked like this:



The menus mostly worked, but they had one glaring limitation:



They were all fake! It was not possible to see a 'fake' window outside of the 'real' window! Yes, you could maximise the emulator, but that meant obscuring all of the other programs that you were using, like your web browser!

The reason that the windows were this way was because they were made with Dear ImGui, which is a library for quickly and easily creating menus. This library works across many different operating systems, rendering its windows inside the program's main window by design, so that it works on video game consoles, which lack a proper windowing system like what a PC has. Unfortunately, this illustrates how Dear ImGui is better-suited for embedding in a video game than for developing a sprawling desktop application.

Despite this, I continued using Dear ImGui because I hoped that its developers would eventually address this shortcoming and allow for the creation of proper windows on platforms that support it. Indeed, such a feature has already been in development for quite some time!



There was just one problem: it did not work on Linux. Not only was the X11 support extremely buggy, but it was fundamentally incompatible with Wayland, meaning that it would never support the protocol without first undergoing major modifications. This was a deal-breaker.

For years, I hoped that something would change, but nothing ever did. Even now, Dear ImGui's 'multi-viewport' support is completely unusable on modern Linux.

Finally, I had waited long enough, and decided that, if Dear ImGui was not going to support multiple windows for me, then I was going to do it for myself. This is the result:

It's finally here!​

At last, all menus are now proper windows! This was achieved by bypassing Dear ImGui and manually creating windows using the underlying API, SDL2. Unlike Dear ImGui, SDL2 supports creating windows on all major operating systems, including Wayland-powered Linux.

Long ago, I had refactored the frontend so that a window was represented by a class, with which it was easy to create multiple windows. By inheriting from this class, I could create a class which represented a window that used Dear ImGui widgets, and then, by inheriting from that, I could create a class that represented a menu window. In this hierarchy, the main window is a 'window with Dear ImGui', while all other windows are 'menu windows'. Each menu can then inherit from the menu-window class and extend it with a unique interface. By structuring the code this way, all menus share a great deal of code, making it very easy to create new ones. Additionally, by centralising so much code into common classes, I made it easy to apply fundamental modifications to all menus at once! This would be very useful for solving one particular problem...

Unfortunately, not all platforms support using more than a single window. One platform in particular is one that my emulator only recently gained support for - Emscripten. By making my emulator use native windows, I had broken compatibility with platforms that cannot use them! To address this, I made it so that a menu can be either a real window or a fake window. Due to the aforementioned centralisation of the core menu code, this was very easy to do! As a result, the Emscripten version of the emulator will use Dear ImGui windows, while other platforms will use normal windows!

Allowing the menus to be used outside of the main window has been a feature that many users have requested over the years, so I am very happy to finally see this fulfilled!

There is just one problem: these menus are jank. On Windows, Linux, and presumably every other platform, these windows are not considered 'subwindows' of the main window. This means that each one gets a slot on the operating system's task bar, and that un-minimising one window will not cause the rest of them to be un-minimised along with it. These sound like small problems, but they really bring the whole experience down.

This is annoying.​

SDL2, despite flaunting its ability to create multiple windows as one of its main improvements over SDL1, has no API to overcome this. There is one function for designating a 'modal' window, but it does not solve this particular problem. Not to mention, that function is only implemented on Linux; it does absolutely nothing on Windows and macOS, and was only introduced because the Linux port of the Unreal Engine's editor needed it. It is things like this which call SDL2's role as a platform-abstraction library into serious question.

Because of this, Dear ImGui windows remain the default for now, until something can be done to make native windows less clunky. For those feeling adventurous, native windows can be enabled by editing the frontend's configuration file, which can be found at '%APPDATA%/clownacy/clownmdemu-frontend/configuration.ini' on Windows and '~/.local/share/clownacy/clownmdemu-frontend/configuration.ini' on Linux. Just change the 'dear-imgui-windows' setting from 'on' to 'off', and you are good to go!

SDL2's successor, SDL3, provides a function for making a window the parent of other windows, which would solve all of these problems. One would think that the obvious thing to do is migrate this project to SDL3 so that it can make use of this feature, and that is exactly what I did! Unfortunately, SDL3 is still a work-in-progress, being far too unstable for regular use. While the day will come that the switch to SDL3 can be made, and the improved native windows can finally be released for everyone to use, that day is not coming any time soon.

Preliminary Migration to SDL3
Background
From the very beginning, this emulator has used the SDL2 library for handling the various differences between operating systems, allowing the emulator to work on any platform with minimal effort. There is one problem with it, however: it is quite old.

SDL2 as we know it began as SDL 1.3 in the late 2000s. Since its inception, it has maintained a rock-stable programming interface (both API and ABI), which allowed it to continue working with all software that had been built upon it, even after many years of releases, features, and heavy refactoring. This interface also had the unfortunate downside of cementing various poor design decisions into SDL2, forcing developers to deal with them for well over a decade.

At long last, SDL's developers have opted to shed these years' worth of flaws by developing SDL3, which boasts a refreshed API. Gone are quirks like functions returning 0 to indicate success while others would do it to indicate failure, the filtering of textures being specified by clunkily altering global state, and high-DPI support on Windows being enabled completely differently to how it is for other platforms.

In addition to a cleaned-up interface, SDL3 boasts new features like a 3D rendering API, file dialogs, and a replacement for the conventional 'main' function that makes software immediately compatible with Emscripten.

None of this is particularly beneficial to my emulator, since it does not need 3D rendering, it already uses its own custom file dialog logic, and it provides special-case code for Emscripten to work around the incompatibility with 'main' functions. However, by discarding this custom code and offloading the work to SDL3, not only would the codebase be simplified and lightened by several hundred lines of code, but improvements made to SDL3's implementation of these features would directly benefit the emulator, whereas it would remain completely unaffected if it continued using its own implementations.

What ended up pushing me over the edge, however, was the ability for a window to declare itself as the child of another window. By doing this, the child window is not allocated an icon on the task bar, and it is presented to the user whenever its parent is. This may seem minute, but it goes a long way towards making a multi-window program not feel like a mess to use.

Wanting to make the emulator's multi-window support the best that it can be, I decided to migrate my emulator from SDL2 to SDL3.

Migration
The process of migrating to SDL3 was very complex. Fortunately, the developers of SDL provide extensive documentation for how to port software to the new library, along with a few scripts to automate certain tasks such as renaming functions.

One of these scripts gave me a great deal of trouble, that being the Coccinelle script. Coccinelle is a program meant for applying patches to source code in a syntax-aware manner, but, on Windows and Arch Linux, it merely crashes when attempting to run the script. It lacks a Flatpak and neither MSYS2 nor Arch Linux provide a package for it, requiring that it be compiled from source, which makes it unclear whether the fault is with the program or merely how it was compiled. The only way that I got the script to run was by installing a recent version of Ubuntu in a virtual machine and installing and running its Coccinelle package from there, which is far more effort than it is worth. I do wish SDL's developers had chosen a more reliable tool.

After applying the Coccinelle script, I was able to run the accompanying Python scripts without any problems. From there, I consulted the migration documentation to address every compiler error that I encountered until the emulator successfully compiled. Unfortunately, a few bugs had sneaked in along the way, due to some API changes not producing compiler errors despite the issues they caused, such as the 'SDL_Event' struct's 'cdevice' ('controller device') member being renamed 'gbutton' ('gamepad device'), and then a new member being added that was also called 'cdevice' (short for 'camera device' instead). This problem could have been avoided by using a more verbose naming scheme.

With the emulator now running atop SDL3, I could replace its custom file dialog and 'main' function logic with usage of SDL3's new APIs. I also considered doing the same to replace the audio mixer, but I am uncertain about whether SDL3's mixer features dynamic rate control like mine does, so I have not made the switch.

Result
As of now, the SDL3 migration is complete. The codebase is benefitted by this in multiple ways: being free of many lines of code that were made redundant by SDL3, having clearer code due to targetting a better API, and being able to make use of features that did not exist in SDL2. With this, the emulator is prepared for the future!

...Unfortunately, it is not prepared for the present: SDL3 is still pre-release software, meaning that it is lacking in polish and stability. A glance at SDL's issue tracker will show many flaws in its current implementation. The worst problem occurs when closing a window. On Windows 11, this causes the desktop to behave erratically, as if the compositor or GPU driver had crashed, while on Linux and Windows XP, the emulator itself crashes. Strangely, this only seems to occur with the software and Vulkan renderers; OpenGL and DirectX appear to work just fine.

Because of all of this, the SDL3 migration was moved to its own dedicated branch. The main version of the emulator will remain with SDL2 until things improve. Those who are interested in trying the SDL3 port can find its source code here.

Closing
I think this is the first time in the project's history that I have implemented features which were either completely reverted or locked behind a hidden setting. Both native window support and the SDL3 migration held this update back for months, so it is a tremendous shame to not see all of that effort pay-off. This has me wondering if it would be worth abandoning SDL in favour of another library, such as GLFW. Unfortunately, SDL is so deep-rooted into the codebase that removing it would involve rewriting a great amount of code. While it is possible, it would not be easy. Still, refactoring is one of my favourite programming tasks, so maybe I will take the plunge. Only time will tell.