Twizzler Compression

Discussion in 'Utilities' started by MarkeyJester, Oct 19, 2016.

  1. MarkeyJester

    MarkeyJester ♡ ! Member

    Joined:
    Jun 27, 2009
    Messages:
    2,867
    A while ago, I was toying with the idea of making a PC game, and needed a compression format. I didn't want to use open compression formats like PNG for graphics, since that makes the art liable for hacking. So I needed a compression format that compressed as good as (if not better than) PNG, but was also unknown to the majority of the public.

    I've since given up the project, but did succeed in writing a fairly decent compression algorithm, that in most cases, did compress better than PNG (though, this is difficult to measure since PNG comes with an uncompressed bloated header...). But I figured it'd be really cool to have a 68k decompressor so that the compression could be used in hacks/homebrews.

    --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---

    Twizzler

    [​IMG]

    Twizzler is an LZSS compression format, with a sliding window of $20000 bytes, fitted with two huffman trees (one for retrace (offset), one for copy (length)).

    Twizzler
    Twizzler Moduled
    68k Source

    Provided above are two compressor/decompressor programs and a 68k decompression algorithm source.

    To use the compressors/decompressors, simply pass the file you want to compress/decompress, as the first argument (for the n00bs of you out there, just drag and drop the file(s) onto the program's icon, windows will do the rest...), it compresses/decompresses depending on the extention name. If the extension is ".twiz" or ".twim", the programs will decompress them to a ".bin" file, if the extension is NOT ".twiz" or ".twim", it'll compress them.

    So why are there two compressors? One is "Moduled", the other is not:
    • "Twizzler" (or ".twiz") is for data that is decompressed to 68k RAM
    • "Twizzler Moduled" (or ".twim") is for data that is decompressed to VDP VRAM (for example, art data)
    In simple terms, if you want to compress art/graphic data, use "TwizMod.exe", otherwise, use "Twizzler.exe". A moduled version is necessary because the sliding window is $20000 bytes, that's so large, it doesn't even fit in 68k RAM which is in total $10000 bytes. And so, it has been reduced to $1000 bytes.

    The 68k source file has two algorithms, one is normal, the other is for "Moduled", here is example code on using the normal decompression routine:
    Code:
    		lea	(TwizzlerData).l,a0			; load compressed data address
    		lea	($00FF0000).l,a1			; load 68k RAM location we want to decompress to
    		jsr	TwizDec					; decompress and dump
    And here is example code on using the "Moduled" decompression routine:
    Code:
    		lea	(TwizzlerArt).l,a0			; load compressed art data address
    		move.w	#$2000,d0				; set VRAM address to decompress to (2000)
    		jsr	TwimDec					; decompress and dump to VRAM
    --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---

    Is it worth using?

    There's honestly no reason for me to tell you whether it's worth it for your needs, that's down to you to decide, give it a trial and decide for yourself.

    One thing I can say, is that it compresses smaller than Kosinski (even when using the best kosinski compressor known to date), in all known tests bar a few. I wouldn't however say it's faster than Kosinski, that would likely be far from the truth, I have however, optimised the 68k code to be as fast as possible, without going way too overboard, but maybe someone else can make improvements on speed/size to their liking. This format is meant for size rather than speed, so keep that in mind.

    The compressors however, are EXTREMELY fast, I spent months on them learning some major optimisation techniques for C, what you might find is the program will simply "flicker" on the screen, so don't blink!! I've considered writing a KENS series of programs actually, all of them seems to be "somewhat" slow, not that it's a problem though considering the files we're dealing with here are small and insignificant, but umm... I donno, let me know your thoughts on that...

    That about wraps it up!

    Happy hacking folks~

    --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---

    A very special thanks to Natsumi and MainMemory for some heavy tests, and bug reporting.

    Some of the bugs, I would never have noticed until months later~
     
    Last edited: Oct 23, 2016