I've been kicking this idea around for a while, so here I've finally gotten around to some sort of implementation. The fx-imglib toolset is intended to consist of a few PC-side utilities for image compression and embedding in G3A files, and a device-side (Prizm) library for decompressing images.

Rationale, features
It makes me sad when people embed graphics and resources in source code, as everybody's been doing with Prizm programs thus far. With a few simple conversion programs (such as the convert565 tool in the mkg3a source tree) and a little bit of objcopy trickery, it's easy to embed resources in your binaries directly. I intend to make the necessary additions to the Prizm SDK package's makefile to do this.

Rolling your own compression and decompression routines is a pretty big time-waster. Preliminary tests on Kerm's Obliterate title screen with LZF compression got it down to about 85 kilobytes as RGB8 (34% of its original size, about half the size it is as RGB565, and only ~10 kB larger than the PNG). Further improvements are expected as I implement pre-filtering and perhaps tweak the LZF bitstream to work better with RGB565 data rather than plain bytestreams.

What's implemented
I have a working LZF compressor (that needs optimization) and untested decompressor. I prototyped resource-embedding with objcopy a while ago but didn't test it- shouldn't be hard to make it work. Filtering is the next thing on my list to implement- will have to see how well it actually works.
Bump.

Improved the compressor for greatly-improved speed at the cost of additional memory usage, but that also makes it compress worse. Should be able to recover the original compression ratio with a bit more memory and CPU time, which should be worth it (the current code gives me 100k of data out, compared to 85k yesterday).
This is stupendous, something that I've been wanting for a long time. Would you mind documenting it on the wiki one of these days? Smile I'm also happy to hear that you continue to optimize it via IRC.
Improving my hash table compressor more. Tracking the changes:

Code:
> python compressor.py obliterate.png
# lzf.py changeset f583a7323072
85544 bytes out in 244.6 seconds
# changeset 16be1c05f024
100700 bytes out in 15.07 seconds
# changeset e8c5df4b4f58
91411 bytes out in 3.82 seconds

The latest changeset should be able to match the compression ratio of the original incredi-slow version, so I'm probably missing something. Improvements continue, anyway.
More than twice as fast as the original compressor, but it's still pretty slow. I seem to have achieved perfect compression within the limits of LZF, however:
Code:
> python compressor.py obliterate.png
84759 bytes out in 96.76 seconds

I'll need to consider ways to improve performance more, since 90 seconds is still too slow. I suspect improving the hashing scheme will help, and pruning the backref candidates on the fly will afford a large speed improvement (it was considering nearly 25000 candidates when encountering a run of 0, which involves a search of around 7 million operations in the worst case).
Out of curiosity, how much does decompression speed vary relative to the packed size? I'm very impressed with the results you've obtained on the compression side, of course, and I don't think the current speed is too much of a problem considering it's basically a one-time operation.
Decompression speed is dependent mainly on output size. Encountering a backreference involves copying from a point somewhere behind the leading edge of the input stream (up to 32k back), but it all ends up as a memcpy operation in the end.
  
Register to Join the Conversation
Have your own thoughts to add to this or any other topic? Want to ask a question, offer a suggestion, share your own programs and projects, upload a file to the file archives, get help with calculator and computer programming, or simply chat with like-minded coders and tech and calculator enthusiasts via the site-wide AJAX SAX widget? Registration for a free Cemetech account only takes a minute.

» Go to Registration page
Page 1 of 1
» All times are UTC - 5 Hours
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

 

Advertisement