Ok so I was screwing around with my TI-84+CE a little, and I know its impossible to boot a custom OS on the CE, but what if it was possible to package that OS as a program, then from that program, unload TI-OS and take over the calculator. This would not replace anything in ROM or the bootloader, this would simply replace TI-OS with my own OS in ram.

The code for TI-OS is never* loaded into RAM, it's run directly from flash memory. You could certainly write a program that uses all of the memory that's normally reserved for the OS and then just never exit from that. But you wouldn't be able to run programs designed for TI-OS, since those call into the OS at a fixed location in flash, nor would you be able to use interrupts.

*aside for a few routines that get loaded into RAM temporarily while performing operations on flash memory, but those barely count.