Okay so I have a correction to make about the previous message and some progress about the slowdown at the start of E1M2. First, the following claim is incorrect:


This is because, being used to libprof in its original setting (ie. with my custom kernel), I quickly forgot that I only ported the part about counting timer and not the one that determines the clock speed. The profiler assumes Pϕ=29.49 MHz so results are scaled if the frequency is different. Note that values are still proportional to real time, so comparisons between sections of the code and WADs are completely fine.

I realized this after testing under overclock with Ptune3. Ptune3 increases Pϕ so the delays were inflated, and ended up totaling 1700 ms even though the profiling completed in about 1100 ms during the test (18 FPS). Ptune2, in addition to pushing the performance further, does it with a lower Pϕ setting, deflating the numbers. This means I can end up with comically different numbers for the same performance in E1M1 (20 FPS with Ptune3, 21 FPS with Ptune2), which prompted confusion on my end. (So the black-magic-level numbers are not real time, even though I welcome the added boost.)

Now in your original report you alternate between 18 FPS (1111 ms profiling time) and 26 FPS (769 ms profiling time), which represents a 342 ms difference over 40 ticks. I've reproduced your setup by configuring Ptune2 as you indicated, and the Pϕ value in that particular situation is 13.36 MHz. Therefore, we can un-scale your measures to correct the libprof bias:


As you can see, there is really no room for other activity. I don't profile all the AIs and physics so a 10% difference is nothing unexpected. Now the difference between the two GR values is 288 ms which is about the 18 FPS vs 26 FPS difference of 342 ms. The extra time in DA and LL is likely spent during GR so we can't simply count it too, but I think by this point the difference is small enough that all doubt is cleared and we can be sure that the extra loading is the source of the FPS drop.

Now what could be loaded during rendering? Geometry is loaded entirely before the level starts, which leaves only textures (I checked the code to make sure there was nothing else).

The primary way in which textures are loaded during rendering, as far as my understanding goes, is through the R_GetColumn() function which simply returns a column of a texture. This function loads the texture into memory (if not already loaded) with purge level PU_CACHE, meaning that the loaded texture is considered temporary data and Z_Malloc is allowed to reuse the space immediately if it needs it. Therefore, it is possible that through random and/or complex interactions on the heap, textures get evicted to answer other allocation requests and thus need to be reloaded. It would be hard to make statistics about this, though...

Also, because you trusted unaligned lumps during the test, it is clear that these textures being loaded are fragmented, making the bug dependent on the structure of the WAD file in storage memory (thus difficult to reproduce).

I'm tempted to leave the problem as is now, because I'm fairly sure there's no easy way to address it. I'll spend some time looking at split textures instead, if by a stroke of luck and genius they can be implemented this bug would disappear at the same time, likely with performance improvements too. It's kind of a high-reward low-probability-of-success thing, but who knows. ^^

I've looked briefly into split textures. I hoped that textures with 2 fragments in the WAD file could be referenced without loading by changing the in-memory format of textures to allow breaking them into two pieces. Unfortunately, the texture format for walls (which is best described as "column-major sparse") is not used for display directly and instead wall textures are generated at startup on the heap using the lump contents. So there's little to do that could reduce heap consumption. I haven't looked at flats because I believe the benefit would be too small.

I will now be focusing on the last differences between the original DOS Doom and CGDoom, as well as trying to obtain compatibility for Doom II.


I still have happy news to report, because I was able to reprogram the melting effect without using any additional heap! \o/

There are three memory regions used in the melting effect, and all three of them are screen-sized (320x200 = 64 kB). One is used as temporary memory to transpose rows and columns, which makes the melting faster. The other two are used to hold the pixels of the before-frame and after-frame which mix up to form the effect.

I got rid of the temporary buffer by not transposing, since the effect is already fast enough for us. I moved the extra full-screen images to PRAM, which was previously almost empty (17/160 kB) and is now almost full (145/160 kB). With a little bit of reprogramming the effect can be made to work in this special memory region.

Technically only one buffer is needed, but I'm not going to try that optimization yet.

I had to fight with a particular bug for a couple of hours. The increased pressure on the PRAM heap (which uses a custom, extremely minimal allocator of mine) revealed that the Zalloc function (Malloc + initialize to zero) didn't work properly. I had carefully used a 32-bit pointer to initialize the PRAM but failed to make it volatile, so GCC was once again turning the whole thing into a memset (which did not use a 32-bit pointer as needed). You might remember that this is not the first time I've had this problem... xD


I will look at both of these next. Also, there's the game save/load which I can attempt to support.

I've restored dynamic palettes, which is the basis for these effects (and costs a couple kB of heap due to loading the extra palettes). This handles picking up items, taking damage, being under the berserk effect, and wearing a radiation suit.

This allowed me to remove the hardcoded, default palette. (It was already compacted from 24-bit to 16-bit RGB, I assume all done to save some memory. We can afford the extra bytes right now. ^^)

I added back the title screen. Unfortunately the demos are disabled; the code for that had been removed previously (not by me), and I believe the demo data may not be present in modified WADs. (They don't show up for these WADs in GZDoom at least.)

I wasn't able to "blur" the title screen when the menu is open, so the readability is suboptimal. I've left a "skip title screen" option before WAD selection to avoid a large number. Finally, I've pondered over "End Game" going back to the title screen but decided against it in order to keep controls fluid (otherwise leaving the application requires two "End Game" in a row). I saw that GZDoom does it the same way so I figure it's acceptable.

The final items on my TODO list, barring any new bugs, are an attempt at Save/Load (not too easy because the fx-CG 50 filesystem is rather hard to deal with) and Doom II support (mainly other Z_Malloc failures). ^^

Perfect, that you could implement so much! Everything is working great!


Well, that's bad. But I know that the demos weren't removed because I can find 4 demo lumps in the Ultimate Doom WAD when opening it in SLADE.


DOS Doom actually never blurs the screen because it only shows the title screen graphic and then plays a few demos. GZDoom doesn't do that and therefore has probably that yellowish blur. I think if you still manage to implement the demos, readability shouldn't be a problem because most Doom levels are quite dark. Maybe you can get the removed source code from the original Doom Repo ( https://github.com/id-Software/DOOM/blob/77735c3ff0772609e9c8d29e3ce2ab42ff54d20b/linuxdoom-1.10/g_game.c#L1582)?
If not I think you could darken the background a bit when the menu is open to help with the readablility.

I tested also a bit Critor's WADs that only have a single episode of the Ultimate Doom:
- System ERROR on the second image when pressing Read this!
- you can't select any weapon after the rocket launcher (these are Plasma Gun and the BFG9000, probably because the game handles the WAD as a shareware WAD and therefore only uses the weapons which are actually used in the shareware WAD)
- level names are incorrect (it uses them always from Episode 1)

And here is some other stuff I found:
- there are sometimes Z_Malloc errors on the intermission screen
- bad performance in a few areas (for example E4M2 and E4M3, around 15-17 FPS with OC)
Maybe you can find some more performance improvements to get these areas to 20 or even 25 FPS. I think a good goal for the entire game could be 30FPS or even better 35FPS which is the maximum framerate of DOS Doom.

I can, in fact a lot of code I've committed recently is "restored" from there. I had had a brief glance and believed a lot of code was needed, but it's actually fairly straightforward so I've enable demos again.

Things get difficult from here. Shareware Doom has demos of version 108, Ultimate Doom has demos of version 109 and the source port is in version 110. I've found informal references of the demo format, based on which the demo formats are compatible, and forced playback. The decoding is fine, but the levels play differently. Either enemy positions or enemy AIs changed in incompatible ways.

As a result, all demos die quickly and then fail horribly from there. Single-episode Ultimate Doom is particularly golden as the maps don't even match (they're renamed, see below).

I've extended the "Skip title screen" option to select between no title screen, title screen with no demos, and title screen with demos to let you test it.


I've checked, but I don't think I can. This is because there is one single global palette that applies to both the background and the menu itself. I can change the palette to be yellow (using one of the palettes shown when applying the bonus effect), but it makes everything yellow. There is no way to apply different palettes to different objects on-screen, and also no way to not use the palette.


Alright so the single-episode WADs are unofficial and have some structural flaws. In particular, the engine always expects to start at E1M1 and then determines game version based on what other levels are present. Thus the levels in doomu2.wad for example, are all named E1M1 to E1M9, and there is not E2M1. This is why it's identified as shareware, why you can't select weapons 6 and 7, and why level names are incorrect.

I've worked around this in a pretty hacky way, by using the WAD file name doomu[1-4].wad to indicate the starting episode. I don't know of another method based on the file contents alone, if you have any ideas I'd be happy to hear them.

Based on this information, the Read This! bug should now be fixed because the game mode is identified properly (and it influences which lump is loaded). I was only able to reproduce this bug today and a limited number of times, so confirmation would be welcome.

Weapon selection is also fixed.

The number and names of episodes in the New Game menu have been adjusted by adding some custom work-around code.

Level names have also been adjusted with more custom work-around code. This accounts for both the world map and intermission screens.


I know you've mentioned it quite a few times, but I still haven't been able to reproduce it. Do you have any sort of consistent setup for this? What WAD, level, etc?

I will soon add much more detail in Z_Malloc errors to try and analyze them more quickly.


I'd hate to disappoint you, but I don't think we're going anywhere near this level. Look at shareware Hangar for instance. With the basic clock settings you start at about 12 FPS, with two thirds of the time spent in the renderer, one third spent display frames to the screen, and absolutely nothing else.

The display interface time (DI) time is the cheekiest of all. The more frames you display, the more time you spend communicating with the display, and you eventually converge towards the max FPS that the display bandwidth allows. So far, it's about 43.5 FPS. We can't go any higher than that even if we optimize away the entirety of the renderer.

Then, the renderer. I'm not going to touch it, both because of the complexity and the limited time I have left to work on this project. It's still the number 1 time consumer and it alone limits us. Currently, the limit imposed by the rendered is about 20 FPS (on the first screen of Hangar).

Fixing memory usage and lump loading to avoid extremely bad framerate is something I can do, but these limits still prevent any sort of extremely high framerate. Now this isn't too bad; because of the frameskip 35 FPS ends up being 17.5 FPS for us to keep the game speed consistent. This might be reachable in some locations, but to be honest with you I'm almost out of optimization ideas.

On a loosely-related note, I previously tried the F4 reset of Ptune2, which drove Bϕ too high and caused screen glitches and later a crash. Some screen commands seem to have failed. I didn't notice any damage fortunately, although my brightness level now resets way too often.

Yeah, the demos seem to be working but it's a real mess. All of them die super quickly. The interesting thing is that the demo level never changes. It always starts in E1M5 even if the first demo starts in a different level. Also after the demo dies it still stays on E1M5 even though it's now playing a demo which was meant for another level. The same thing happens also in the single episode WADs 2-4 only that it loads here the 5th level of the episode.
Another thing I noticed is that in the Ultimate Doom WAD the start of the first demo looks actually fine to me but then it fails at the first enemys. At what difficulty do these demos play normally? Maybe it's just the wrong difficulty that destroys the demo?


Sadly I don't have a consistent setup for this. It sometimes happens and sometimes not.
I got it working once with the Ultimate Doom WAD when leaving E3M5. It gave me Z_Malloc failure and froze the calculator. For some reason I couldn't continue the game like I could before when I got a Z_Malloc error on the intermission screen. I tried it two times to get it working again but no sucess... I tried also some of the single episode WADs because I got Z_Malloc errors before with them. But also here I couldn't get them again. If I find a consistent one I'll let you know!


That works but if you change the filename nothing will work anymore. I think the best and easiest thing would be to modify the single episode WADs and put a string that tells the game it's a single episode WAD with episode 1, 2, 3 or 4. With that it's pretty much the same like with the filename but you won't have any problems renaming the file to whatever you want.

I think theres also another way but it also requires modifying the WAD but maybe also with with some benefits like smaller file size. You know that every episode has it's own intermission graphic:
- E1: WIMAP0
- E2: WIMAP1
- E3: WIMAP2
- E4: INTERPIC

So why we don't remove all unneeded intermission pictures from the WAD and only leave the one we actually need for the episode? With that we would also remove ~200KB of unnessary data and know exactly which episode we have in the WAD.
There's only one problem with this. The shareware WAD has also only one intermission picture (WIMAP0). So we would need to differenciate the Doom Ultimate E1 WAD and the shareware WAD.
One way would be by looking at the number of demos in the WAD. Both the shareware WAD from Planet Casio and my one have 3 demos but the Ultimate Doom E1 WAD has 4 demos! Or we could just compare the file size. Shareware WADs should never be bigger than 4.1MB and the Ultimate Doom E1 WAD is larger than 7MB in size.


I was already sorta expecting this because I saw before how the palette change affected the whole screen.


Yes, it seems to be fixed now.

In the various WADs I have DEMO1 is always in E1M5, so that's not a bug. After a respawn the demo continues because there are still inputs to be processed. It will only leave the level once the demo is complete... and the code for that was missing, so it would stay dead on E1M5 forever. This is now fixed. In the version you currently have demos for other levels were never played in the first place, now (with the next G3A) you will see them if you wait a minute or two for the first one to finish. The levels load normally.


It thought so too. The difficulty is encoded in the demo lump, and I previously checked both "adjacent" difficulty levels in case the decoding was incorrect. But actually all difficulty levels fail similarly, and the closest one is the setting I left in the G3A. Sometimes it fails by a really thin positioning margin. This is why I suspect that enemies are different, either their initial position or the AI.

Also GZDoom don't play these demos, I suspect this is the reason.


I'd like to support TI-Planet's WADs at least decently so I'll probably leave the filename heuristic on to help with that (there are few reasons the rename anyway), and add some WAD detection on top of that. I'll go with the number of demos unless I try and remove them, thanks!


Yes it's a nice idea! I've wanted to look at the WADs for a little while now. In addition to these intermission screens I'm suspecting other graphics can be cleaned up. SLADE can remove some textures from doomu2.wad, saving ~600 kB. There are also these intermissions screens. I will try to have a slightly deeper look into that, since saving space is really useful.

So I've looked at reducing single-episode WADs. There are actually quite a bit of lumps that can be removed, mainly: unused flats, unused textures and patches, unused sky textures, unused intermission screens and finale screens.

I haven't tested the reduced WADs all the way through (you know my ability to get around these levels...) but I reduced them in SLADE and played around a bit. If these changes are successful then it will save:

doom1.wad: saves 2.3 MB (now 5.2 MB)
doom2.wad: saves 1.0 MB (now 6.6 MB)
doom3.wad: saves 1.1 MB (now 6.3 MB)
doom4.wad: saves 2.0 MB (now 5.8 MB)

For now, you can download the WADs here: https://linx.breizh.pm/ptbxzkvm.zip (this is from a Planète Casio member's server - expires 1 month after this post), and test them with this post's G3A update.

I've added single-episode WAD detection based on either the file name (intended for unmodified TI-Planet WADs) or number of demos combined with intermission images, as you suggested (intended for the WADs I just modified). I also selected the sky texture properly for each episode.


So I had this happen to me just now. I have just added detailed Z_Malloc error messages, in the sense that the free heap and largest heap block are shown on the Z_Malloc error screen. You might also notice that I removed the old error screen and made a new one which supports printf-like messages and wraps them at the screen width so the results are much more pleasant. Anyway I had removed a bunch of memory to test that extra Z_Malloc error information, and got one such error during an intermission.

Basically in my case the only zone available (I had removed the others) was too fragmented. I realize that intermissions use 60 kB images so to load that we need 60 kB of heap (unless I can split them, which may work, unlike textures). I always assumed that the level was unloaded before the intermission, resulting in a lot of heap being available, but actually I can't find evidence of this in the code. Thus it makes sense for these errors to occur at intermissions.

A quick question about the new G3A:
Can you reenable all the heaps because there is only one heap available (the one with 448KB). You probably turned all the others off to get my intermission errors but now I'm getting in every WAD very fast Z_Malloc errors. I mean it's great that I can get a look at the new detailed error message but I can't test like that.

Darn it. I spent a lot of time making sure to not commit that temporary restriction to the code, but then I forgot to rebuild one last time before committing the new G3A!! (ノ*°▽°*)

I force-pushed over the previous commit after rebuilding, it should be fine now.

Wow! I expected there would be still something else to remove but this is a massive difference in file size compared to before! This will definitly come in handy so I don't have to delete as much stuff off my calculator to get the WAD on it.

I tested all WADs and found two levels that gave me errors:
E2M7: R_TextureNumForName: Compute1Compute1- not found
E3M3: R_TextureNumForName: Compute1- not found / R_TextureNumForName: Planet1 not found

Both level do load after you click through all of the error messages but there will be some corrupted/missing textures (stripes of random colors). Every other level worked without any problems.

All the demo levels do load now but it's pretty much the same like with E1M5. Everything works as expected until the demo gets to the enemies and it completly falls apart. Is there any way to fix the Enemy AI or enemy positions to get them working with the demos properly?

Also here is something I noticed on the main menu WAD selector: it only shows the name for the Ultimate Doom E[1,2,3,4] WAD when the file isn't renamed.

Well well well, I have some news, and they're excellent.

Remember how the two bottlenecks are rendering and sending the image to display? I took a deep dive into the second one to test some advanced ideas. Now this post is going to look very silly, because the current method ended up being superior to these ideas, so I'm not going to talk about any of it.

But in the process of understanding why the current method was faster, I stumbled upon one major mistake in the current code: VRAM access is uncached. There are reasons for that (the OS uses the DMA thus cache-coherency is needed and that's the simplest option), and historically every single time I've tested caching made no difference on VRAM (which puzzled me for a long time). Well, no longer. Caching here makes a really big difference, and it just happens that the current display method works well with it. I only had to flip a bit to obtain way more than I hoped with all the fancy approaches.

This change multiplied the speed of the Display Interface step by 3.4x (459ms → 135ms in profiler), which results in FPS improvements all across the board. The first screen of shareware Hangar is now 16 FPS without overclock and about 25 FPS with Ptune3's F5 preset. The first screen of E1M2, with which we had trouble not too long ago, is also 15 FPS - 23 FPS in the same situation.

I do note, however, that the variations in smoothness in areas with a lot of geometry are more noticeable now.

I've updated the G3A for this change, since you've been asking for performance improvements and I'm happy that I got something consistent to show you


Thanks, these textures (Compute1 and Planet1, the repeat is an artifact of the error message) were detected as unused by SLADE. I will keep them in the next "version" of these single-episode WADs.


I don't believe so. If enemy positions are the problem, we would need to replace the levels in the WAD with older versions. If enemy AIs are the problem, we would need to replace the AIs with a previous version of the code, which is likely not as good and may not be available at all. Currently I don't see a way to fix them.


This is normal, the menu only looks at the file names. It can't look into the WADs as this is only possible once Doom's engine is started, which only occurs later.

Alright, you know what, let's not stop the good news there. There's something I've been keeping in store for a while now, and I think it's time to play with it.

You see, the fx-CG 10/20 had a 2 MB memory chip. However the fx-CG 50 has an 8 MB one. This is a huge difference because the OS doesn't usually change a lot, so a big chunk of the extra 6 MB might be free!

Using extra chip memory like this has to be done carefully because any new OS version might decide to exploit the added space. For instance on the black-and-white series of CASIO calculators, there has been 512 kB chips with only 256 kB used for years, but the recent release of the G-III series changed that by using the second half, so some programs had unpleasant surprises along with the calculators running them (a couple of crashes mainly, nothing too bad fortunately).

Still, as I'm testing with OS 3.40/3.50, I see a consistent 3 MB of zeroes followed by about 3 MB of completely random data. I don't know what it's doing here or what it's for, but I can use the zeros, so I've added an option for that.

The new menu option, "Use memory beyond 2MB", allows CGDoom to look for a large free area after the 2 MB position in RAM (since the first 2 MB are still used as they were on the fx-CG 20). It's available on OS 3.50 and earlier, and usually results in a ~2900 kB heap area, for a 4.3x increase in available heap!

If there's any way that Z_Malloc errors are going away for good, then that's the one!

I am hoping that wherever this option is supported we can play as many of the WADs as possible. I haven't tested all of these, but the scope expands so drastically with this change that it's probably worth looking at them all.

Wow! You did it. You just blew my mind two times in a row! (btw this is your 100th post and it's one of the best) I guess ComputerNerd was right about that the CG50 has more RAM! I'll write a bit more but first I got again something to say about the new G3A update:
there isn't a "Use memory beyond 2MB" option on the main screen. I'm currently on OS 3.60 so I hope that this isn't a problem.

Aah, I see! No worries, I'll install OS 3.60 and check whether the memory is still unused as it is on OS 3.50. It's very likely to be the case, but I enable the option on a whitelist basis to make sure we don't get unpleasant surprises later down the road.

Everything's fine on OS 3.60, so I pushed a new G3A.

Also you're right, this was my 100th message. I didn't even realize!

Wow! This simple bit flip gives a lot more performance!
Stuff that really gets affected with this change is simple geometry!
All of that is tested with an not very fragmented flash and Ptune2 213MHz overclock:

Doom II WAD:
looking at walls: over 80FPS!!!
Overall performance: 30-40FPS

The Ultimate Doom WAD:
Start E1M1: 40FPS!
looking at walls: over 85FPS!!!
Overall performance: 25-50FPS

The problem with the insane FPS sometimes is that the game runs super fast and you can't really control it anymore. We really need to rate-limit the game now so regardless what framerate it's running on the game speed is always 100%.
Also do we sometimes really need that many frames? I think a framerate cap at 35 FPS would be great so the batteries don't get eaten away too fast.

In more demanding areas (like E4M2) the framerate doesn't change too much but overall it's a very nice performance improvement everywhere. Sometimes even too good.
Something else to note that changing the screensize now really affects the framerate and can make slower areas really fast.

The only performance bottleneck would be now the renderer. Stuff that really gets the calculator in trouble are transparent textures and stairs.

Now let's talk about the huge heap upgrade! With around 3MB more it's safe to say that Doom II will never ever run out of memory again. I also tested all the levels and no problems here.

Another WAD I tested all the way through is the Heretic WAD and it got to the end. All levels that had textures missing (LAVAFL1, LAVAFL2, LAVAFL3, WATRWAL1) did freeze the calculator probably when trying to render them (Level 5, 6, 11, 12, 13).
Level 16 restarts the calculator or gives System ERROR.

Taking a look at the heap consumption it uses at maximum ~600KB in level 32 from our 3MB heap.

Thanks! Things are starting to look really good now.

I forgot to mention earlier that I reprogrammed the display of full-screen images for the title screen, Read This! screen, and intermissions. Now the images are rendered directly from ROM, without loading to RAM, even if they are fragmented. These images are each ~66 kB in size and could cause Z_Malloc errors before, as you have seen, especially at the end of levels. This is no longer possible now (even without memory beyond the 2 MB line, which I added only at this late stage to ensure it's not a mandatory prerequisite for a pleasant experience).

I've also started to look at the Save/Load system. So far I've enable the save menus, and it's possible to choose one of the 6 save files, type an alphabetic description of choice, and save. The file seems to be generated properly, however writing a file changes the structure of the storage memory, invalidating the file mapping and every access to the WAD in ROM!! Instead of rebuilding all of it I think I'll use the safest way and quit the program after saving.

I'll give you another update later today with the rate limiting and hopefully some Save/Load goodness!

Good news again, the Save/Load feature is now working! Obviously it's not cross-WAD compatible even though the files are shown in the same way with every WAD, but apart from that I think it's quite true to the original.

I've also rate-limited the game to 35 tics per second, based on your 35 FPS figure, which seemed to be reasonably close to GZDoom. This means 17.5 FPS with the default frameskip, which is rather smooth and also pleasantly easy to achieve even without overclock. The movement is not very fast but there's still no run button yet, so I think it will be fine. 35 FPS might be slightly too fast based on my impression of the firing rate of the shotgun, but it's a very good starting point.

Because this rate-limiting uses libprof to measure the time elapsed between tics (the RTC would not be precise enough), I decided to implement the proper logic to obtain the speed of Pϕ, so now the profiler and rate-limiter will always behave faithfully even under overclock - finally!

I usually don't play with overclock so I never really reach 35 full FPS, but I agree with you that it's probably time to add no-frameskip to the VARS setting. I'm also considering a frameskip setting that automatically detects laggy areas and skips frames as needed to maintain the game speed. I'll let you know when I experiment with that.

Also, unfortunately the rate limiter does not save up the batteries for various reasons (I can't put the CPU to sleep as I don't have interrupt control to wake it up), so the best saving option is to use less overclock. I'm thinking maybe it can be done automatically, similar to the frameskip, to maximize consistency in the gameplay.


This is awesome! So now we can officially support Ultimate Doom and Doom II. I'll have a look at Heretic to find the crashes, and progressively make my way to the other games as well.