A long-overdue project that I had was to get some way to have libraries on the Prizm. Either the use of a shell or having the individual addins dynamically link in libraries.
I have been doing a lot of research into the MMU, how it is set up, the TLB pages (still a WIP, although in theory I can override it on my own), how g3a-based processes are setup, and ELF-formatted binaries. If anyone has any info on any of these topics, I encourage you to share info.
I have some documentation on wikiprizm on my user pages, however I am currently trying to understand how elf binaries store their runtime linking info and how they should be handled. I am also reworking libfxcg to allow for compiling .so's.
On a side-topic, there is the question of if a shell or standalone launcher should be used, or if the g3a should contain loader code and the elf attached. An external launcher would require either tapping into the OS's TLB page storage or adding in a custom exception handler to intersect TLB misses. This depends on how linking happens, I am still looking into this.
Any insight would be greatly helpful.
First of all here is the ELF specification
http://www.skyfree.org/linux/references/ELF_Format.pdf and here is a nice short summary article on ELF that outlines how to load an ELF
http://wiki.osdev.org/ELF
I would advocate for a loader primary due to the simple fact that I think the g3a format is bloated significantly primarily due to storing two copies of the icon uncompressed adding 28kb per application. I do not support your position that the elf file should be bundled as part of a g3a. There are two main reasons for this:
1. Inherits g3a bloat for each application taking an extra 28kb per application.
2. What if you release a bugfix to your loader code and the author abandons their software and it is closed source? The application will be stuck with the out of date loading code. By having a loader you are free to improve the loader without worrying about the author not updating the code when you do. If it is open source the concern is reduced but not eliminated. I know I am sitting on a few open source prizm applications that I improved but never got to releasing.
Also I see no problem with modifying the OS's TLB page. Just change what is unused or put it back and there will be no issue.
I too have thought about shared libraries primarily to reduce any code duplication but have not much to bring on the table due to me being busy with other stuff. I will try to get some work done on this myself.
OSDev seems to either lack details of runtime linking or just puts TODOs on those sections
I'll have a go at the spec to try and find the structures.
If a shell is made that can launch any elf, sure. (Note that if the ELF is appended to the end, you can still extract the ELF and rebuild the g3a. ) Otherwise it may be beneficial to bundle everything into a standalone g3a for ease of transfering.
For now, just running an ELF would be awesome. Although I may be slightly delayed as I appear to have completely broken my dev tools from updates in the past, I cannot build libfxcg anymore as I get ICEs thrown
Yes you could look at the spec but I forgot to mention open a terminal and type man elf or see
http://man7.org/linux/man-pages/man5/elf.5.html Admittedly I have little prior experience with ELF. I just used google.
Yes why did I not think of extracting the ELF file. But still I do not like an extra 28K+ per application. Maybe the developer could make both options available.
Anyway you got me interested in this. I will try to write a bin loader then maybe an ELF loader. Just don't count on anything.
Also about breaking your dev tools due to addions. What are you refereeing to? I am using gcc 4.9.1 and binutils 2.2.4 and it works great.
I recommend a github repo for this.
I upgraded fedora 16 to 20, libmpfr had a version change which required a rebuild of gcc (I had 4.6.03 and 4.7.0, both when they were released), I am working on building gcc-4.9.0 right now.
<edit>
Github repo has been made, but nothing is there yet. That specific repo will contain the code for loading an ELF and running it. I am messing around with the MMU right now, will report back what I find. The most I have done is read the magic bytes of the ELF directly from the flash chip.
General rundown of what must happen from irc:
Quote:
[21:21:58] <EinHelfer> Here's the gist of it, since there is no documentation of the OS's handling of TLB pages, you have to override the OS's exception handlers
[21:22:20] <EinHelfer> TLB exceptions go to one specific handler, the other 2 you can jump into the OS
[21:22:59] <EinHelfer> This exception vector must be in the ROM or RAM and must be carefully handled to revert the VBR
[21:23:32] <EinHelfer> From there, you must do the usual register copying, check the exception type, and jump to the OS if it is the wrong one
[21:24:22] <EinHelfer> Given that you have the TLB exceptions, you should be safe to use the MMU Entry registers with LDTLB to insert a new entry from your own page table
[21:25:02] <EinHelfer> If you have yet another TLB miss in your code, you have to jump into the OS's handler
Here's what my plan is, add a custom exception handler to override certain TLB misses. This will not allow a page to suddenly get removed and have code crash. The actual linking part will be added later, but for now:Code: ! eh.S
! Goal of this is to handle TLB misses and not explode on quits
EH_Init:
! Cache the offsets into the OS's exception vector
! Save the old VBR
! Init the TLB table
! Do the VBR swap
EH_Deinit:
! Swap the old VBR back
EH_AddPage: (args: int source, int dest, char flags, returns 1 on success, 0 on failure)
! Considering pages can be 4KB, you can have a LOT of pages. Maybe a self-balancing binary tree?
! But you can always improve, bare essentials first:
! Get the position of the end of the table
! If at the end, return false
! Insert a cache of entry registers for quick access
! Bump up the end pointer
! Return true
EH_DelPage: (args: int source, returns 1 on success, 0 on failure)
! Scan the table for a matching source (slow, needs logic ops on the table)
! If not found, return false
! If the end ptr points to the current entry, jump to EH_DelPage_Success
! Copy the entry at the end point over the current entry
EH_DelPage_Success:
! Decrease the end ptr
! Return true
EH100: (VBR+H'100, ignore)
! Load old VBR, add 0x100, jump
EH400: (VBR+H'400)
! Save registers to stack as-per OS's requirement
! The OS does a bunch of logic, I don't really care. Check if EXPEVT against 0x040, 0x060.
! If they don't match, jump in past the register copying in the OS's handler
! Since they do, just need to insert into the UTLB. (The ITLB, ignore it, the MMU has a hardware exception handler, it can look it up on its own)
! Search for the address in the cache (It is just an address, you'd have to look at the flags for the page table to determine the range the entry covers)
! If found it, load up the entry registers, flushing writes when needed. Else, clean up and jump to the OS, maybe it knows what it is doing.
! Use LDTLB and do a dance to get the MMU to update itself.
! Return back to the code
EH600: (VBR+H'600, ignore)
! Load old VBR, add 0x600, jump
! Variables, can be put in-between these entry points
EH_OLD_VBR:
.long 0
EH_UTLB_PAGE_TABLE:
! Fill for a good amount of entries, can get quite large.
Working on the exception handler now, aiming to make a dummy. If nothing fails, I am good to go.
That's a very intersting work ! I don't have the level to help, or discuss of very technical things here I think, but I follow that very closely !
I have successfully hijacked the exception handler and can reroute exceptions. I am experimenting around a bit to make this "safe" by finding out what affects the VBR and how I can clean up after myself. I will try to get the code pushed to github asap.
I have found that the custom exception handler still works in the main menu, however once another process is launcher, the stack is wiped and the EH is corrupted and reboots the calc.
I was able to watch a non-error exception (probably a mmu tlb miss) get handled cleanly (I keep a counter of all exceptions) and was able to crash the process using asm("/* IT'S A */ TRAPA #1!!1!"); to generate an "INTERRUPT" error. The VBR, however, does not get reset. This is not a problem, however, as I can do a bit of processing to temporarily reset the VBR back to the original position. Upon returning (if it was a handled exception), I can restore the EH and keep moving. The only problem is that a quit handler MUST be added to uninstall the EH.
<edit>
The code is up on github, I am formalizing the code in there as it is really rough. The crtEH.S and eh.x include my C++-specific modifications and can get cleaned up quite a bit. Working on temporary unloading on exception.
Note that the only way we know of setting a quit handler, using SetQuitHandler, means the quit handler doesn't always get called. So far, I have found out that this is the case when the add-in is running as a eActivity strip; while most add-ins don't work as eActivity strips, finding a way for the shared library system to work even when running as a strip would be nice. But then again, I suspect that the contents of the MMU and the like will be different when running as a strip, and much of the things you've done so far would need to be reworked or at least tested again.
To keep priorities in check, I will not focus on eActivity strips just yet.
Does the quit handler fail on other times? Switching to another addin is my current problem, and I was hoping that would solve it.
I have the code made to keep the handler safe by reverting the interrupt when it leaves the OS, but restores it by faking the saved PC. I'm working on a trapa handler to see how to properly handle interrupts.
When add-ins are running from the main menu, I don't think the handler ever fails to get called, those willing to test but not willing to compile an add-in, can use the latest version of my Eigenmath port, after ensuring the "save session" option is enabled.
Don't forget to test the behavior of your hacks on power-off.
Trap handling support added, I was able to set a trap handler, issue a trap, then have the handler called with the trap code.
For a quick idea how this was done from the API:
Code: void handle_trap(int code) {
printf("You've activated my '%d' trap card\n", code);
}
void main() {
EH_Install();
EH_SetTrapHandler(handle_trap);
// ...
asm("trapa #21");
// ...
// Required when leaving the addin to the main menu.
EH_Uninstall();
}
Does anyone know what addresses in virtual memory are used by the OS? I see a lot of scattered regions mapped, but have no idea if they are actually used or if this is from some bug or something. (I find 100's of MB mapped to.... somewhere, I don't think the OS really uses all mapped memory).
I am looking for a few MB available that would work across multiple OSes. I don't want to use something that will cause errors or data loss/damage.
<edit>
I have successfully mapped 0x00100000 to 0x88000000, 4KB, r/w all, cached, copy-back, not shared.
<edit>
I have a mmap and munmap implementation similar to Linux's, except that it uses a char* filepath instead of a int fd. I also have a mremap to update all mmaps in the event that bfile writes are used.
I don't know if powering off writes to the filesystem and will fragment it.
This is at the point where it is stable enough to use. Due to the ongoing contest and my intent on using it for an entry, all commits will be pushed after the contest is over.
Yeah ! That's great news (both the fact your work is on a good way, and the fact we're going (I hope) to have an entry for the Prizm on this contest) !
But personally, as I already say, I don't think I'd be able to help... :s
I just can tell you to be careful with the different addresses you use, and check their validity on multiple device (if it's possible for you).
I say that because I remember Kristaba working on his alternative OS (
FiXos) for the fx-9680's devices, and using specifics addresses to load the OS on RAM for example, or later to use / make the filesystem, and some addresses were differents on some devices, so this wasn't working with them, he had to fix that.
As the Prizm's are perheaps more "unified" (probably not the good word ^^) (I want to say that they the CPU are the same on all devices or similarities like that (to illustrate, some fx-9680 are working with an SH3 CPU, other with an SH4).
That's just an advice or a warning at some point, but, that's very interesting work !
AHelper wrote:
I don't know if powering off writes to the filesystem and will fragment it.
I think it doesn't, but since the power off routine allows for getting to many things (including the diagnostic menu and the calculator reset dialog which undoubtedly allows for changing the filesystem), I think it's best to assume that powering off will write to the filesystem, yes.
There seems to be a lot of bogus mapping done by the OS if it doesn't know what should go there. That said, the OS does not have a real TLB table stored, it dynamically generates mappings, pulling from various sources such as addin code pages and other random addresses mapping to invalid addresses. If there was a table, then it would at least have a cache of the RAM pages and would cache those values. If there is a table, then it only stores some entries.
EH_AddPage currently doesn't invalid pages that are already mapped to an address it wants to use, but from what I gather, on a clean start, you'd have access to lots of addresses. 0xC0000000-0xDFFFFFFF seems to get random mappings (Neither the RAM, ROM, or peripherals), so those could be used. I am using for now 0x00100000 upwards. ofc there is data mapped in a bit before 0x00300000, but I don't have a clear map of what is used. Finding out requires crashing the addin, so :/
For the poweroff, I'll have to put wrappers around all syscalls that can power off/write to the flash FS.
I put up all of my commits to github. If someone wants to continue working on this project, that would be wonderful. I am curious if this code works on the emulator.
If anyone wants to know anything about how the EH code works, ask away while I still remember
It is not 100% certain of any side-effects that come from the EH code, but from my development, it works enough that I can use mmap without fuss from the OS.
AHelper wrote:
Note: This project is no longer supported after bricking my hardware on another project
I am very sorry to hear that. I think a good holiday gift for the community would be more understanding of the bootloader and maybe if we are lucky a means of recovery. I will take note of that and get on it once we get closer to said holiday. If I may ask does your prizm show signs of being able to power up as in valid code is being executed by the CPU? This would make recovery easier assuming said code provides either a means to run arbitrary code and/or modify flash memory. I thank you for your hard work on this. I am confident that we will see benefit from this.
I have a broken prizm coming to look into recover of them soon as well programmernerd. I plan to start a thread for it come weeks end. Keep an eye out for it I most certainly can use your help!
You should contact geekboy as he is planning on communicating with the flash chip with a microprocessor as a mean of reading and writing the chip.
I will describe what happened from what I remember: I started my map code, I was able to get events processed. At least 1 timer interval ran and returned fine. More events were processed and was able to render everything (high priority event). After that, something uses the invalid state of the engine that caused an exception that forced a reboot. From there, the LCD powers off, and 1-2 seconds pass before the LCD turns on white with the loading icon in the top corner. So far, so good, I have this happen in the past, but the OS continues booting like normal.
After a few seconds of the cursor displayed, not sure if it rotated, it want to a "Updating OS..." in blue text with a image background (One of the ones in the OS). Of course, this is on a Linux machine, not a windows one that I have the updater on.
Removing the USB cable from the calc, there isn't much I can do with it. The AC/on key does the normal power of screen (With username shown). Hitting AC/on again just turns on the LCD in a low brightness (all white, maybe the cursor), then off in about 0.5 seconds. Reset button pressed, and... black. Emergency OS update key combo yields nothings for me.
It is interesting how the OS was able to get to that Updating OS screen. If an exception happened (probably), then the CPU jumps to the start of the bootloader. Yet, it ran the OS. Could this have been from running an invalid opcode in the bootloader, causing an exception (VBR might still be kept) that the OS handled, detected this was from the bootcode, and tried to rescue it?
Also, does the OS update program have the bootcode in it? Last I read, it can't fix a wiped bootloader and a protocol 7 program would be needed to save it.