Iambian, sorry to respond to such a long and well-composed set of documentation with a short reply, but looking good. Smile I'm glad that you decided to use the 8-byte DCS AP header for the sake of cross-compatibility. A small note, in your Illusiat example, the ZNG0/ZNG0/ZNGO is a typo, right?
KermMartian wrote:
Iambian, sorry to respond to such a long and well-composed set of documentation with a short reply, but looking good. Smile I'm glad that you decided to use the 8-byte DCS AP header for the sake of cross-compatibility. A small note, in your Illusiat example, the ZNG0/ZNG0/ZNGO is a typo, right?

Got that right. They're all supposed to be zeroes. Really difficult to tell from what I was using to edit the file. Ought to figure out what font distinguishes between 'O' and '0' the best and still remain fixed-width.

On another note, I spent the better part of yesterday working on the Z80 side of things. The program is a bit larger than I would've liked. I estimate prgmATHENA coming up to about 2KB, though it's 1794 bytes at the moment. So far, it does the following (it's not much) :

1. Scan the VAT for eligible metadata files for each project.
2. Allows you to select which one you want to "install"
3. Reads the metadata file to show you the statistics in the following order: Number of data files that are expected, the number of files that will be extracted, the number of bytes of FlashROM the installation will use, the number of bytes of RAM the install will take up, the filetype of the large data files, the 14 character description, and whether or not all the files the installer expects actually exist on the calculator.

The stuff I have on my immediate todo list are as follows, and comes straight from the source file:
Quote:
;Todo: EDIT THE FOLLOWING IN PUCRUNCH DECOMPRESSOR
;All input bytes to respect FlashROM storage
;All output by attaching the handler call
;All output to write to the circular buffer, and LZ routines to reference that
;
;Todo: Enable a test link to the ProcessFile call. HL points to name in meta.


I'm not too aware of how the decompressor works, so I'm just guessing and testing here. Hoping that the edits that I make will make it work, but now I'm afraid that it's gonna make the decompressor ... slow. It might not hurt it too much, but performance issues are starting to enter my mind. Probably shouldn't give it too much thought until I get it all working.

EDIT: I shouldn't say that I'm not aware of how the decompressor works, but I can easily recognize what's going into the routine and what's coming out. With some modification, I'm sure I can get it to work. I understand it well enough to do just that. I'm just hoping that the edits actually work, since I'm modifying how the decompressor outputs its data (which requires an awareness of how the LZ77 portion works. Something I'm not *too* aware of, but I can make my guesses)
Double post.

Athena has a load of test stuff in it, so I'm not sure about the final size. At the moment, it's at 2068 bytes. I've finished making the edit to the decompressor, but it also means that the overall compression on the PC side is going to be weaker, since I had to restrict the search range of the LZ77 part to 255 bytes.

I've written enough so that Athena can now test an entire install. It won't create any files (haven't written the code) but it will "decompress" them into nowhere. The bytes of interest are kept, but the rest of the data other than its checksum is discarded. That checksum is compared against the one in those bytes of interest.

The step after this is to actually stuff the data into files and do stuff with them, like archiving. Or check for previous installations or other files that has names that match files in the install.
That's no double-post, that's a bump. Smile I hope that our earlier discussions about the structure of .8xg group files was a help, and that you're making progress towards constructing groups that TI-Connect won't complain about.
I haven't worked on the group thing at all, though I did get the Z80 Athena to work quite well. It can now test and install.

Question: Should I have the thing automatically test an archive prior to doing anything with it? I mean, it can take a while for larger projects (13 seconds for the 134KB game Illusiat 13), but I feel like it should happen. I mean, I wouldn't want to try to do anything with an archive to find out that it was corrupted or something.

Later on, I'll get to doing things like adding .8xg support for the PC Athena, and get the Uninstall and Repair options for the Z80 Athena. After that, I may start working in some sort of DCS compatibility. And maybe optimize the pucrunch decompressor for Athena (since editing it left it less than optimum.)

I have a post on Omnimaga that shows a screenshot of Athena testing and installing Kevin's Illusiat 13, if you wanted to see what it looks like. Maybe comment on its genericness (is that even a word?)
How does the decompressor work? Does it copy stuff directly to Flash or to RAM first?
souvik1997 wrote:
How does the decompressor work? Does it copy stuff directly to Flash or to RAM first?


Not sure entirely, but it involves RLE, LZ77, and inverted Elias-Gamma codes. If you really want to know how the thing works, you'll give this really long document a good read. I did and I feel as though I understand enough to comfortably make edits to preexisting code. Most of these edits weren't hard, since they were redirecting its input and output. The only iffy part was figuring out the LZ77 part and why you couldn't just decompress without making sure that ALL the decompressed material stayed put until it was finished. Solved that by restricting the LZ search range at compress-time and wrote code that deals with a 256 byte circular buffer, where all bytes would be temporarily written (in addition to its designated output), just to keep the LZ part of the algorithm happy.

To answer another question, this writes to RAM first, then uses the appropriate romcall to move to FlashROM. This is okay since Athena packages all files that needs to be moved into Flash first. That way, there won't be anything in RAM to possibly mess with it except the stuff that needs to be in RAM to begin with. And they'll be extracted long after the archiving was done.

I'm not sure if this thing is gonna support directly writing to Flash. That's a pain I don't want to have to deal with. I mean, if I did want to deal with it, I'd know exactly how I'd do it to make it faster than how the TI-OS is doing it. Just don't feel like adding to code that shouldn't be this large to begin with.
Very impressively done, Iambian, although I'm disappointed that Omnimaga gets a screenshot and we don't. Smile To repair that frightful omission, here's the screenshot to which you referred:

Athena is mostly done. Other than not having a repair option (which could easily be simulated by uninstalling, then reinstalling), the Z80 portion is pretty much done. Works great. Got Illusiat 13 and Reign of Legends 3 installed using it. The only problem I had was the FlashROM check, which seemed to flakey for me to use. If anyone has any suggestions, I'm open to it.

The PC Athena util now recognizes group files. It wasn't that painful to add in support to it due to the way it was coded. Hooray for decent (though not altogether good) coding practices. It successfully saw through how Reign of Legends 3 was packaged. It was all in group files.

The following is a screenshot showing the testing (but not installing - that takes like 5-10 minutes) and then viewing the memory manager to show how much memory the archives themselves take up. Overall, it seems to be a good thing. (If you're wondering, I deleted the install files for Illusiat 13 to make it clear that what you're viewing is just the Reign of Legends 3 files)



If you're paying close attention to the start of the test, those are pic files that are being represented. Packing them with the PC Athena caused my computer to beep Smile
Haha, a superb job, Iambian! Can you tell me about the FlashrOM check, what it is supposed to do, and it what way it is being flaky?
KermMartian wrote:
Haha, a superb job, Iambian! Can you tell me about the FlashrOM check, what it is supposed to do, and it what way it is being flaky?

This is the offending piece of code. I commented it out because it doesn't seem to work very well.

Code:
;_ChkfreeArc   EQU 5014h  ;Checking FlashROM is too flakey at the moment...
; bcall(_ChkfreeArc)
; ld hl,$839F  ;4 bytes, written big-endian (by byte)
; ld a,(hl)
; inc hl
; or a
; jr nz,GetMetadataInf2S
; ld de,athfsize+2
; ld b,3
;GetMetadataInf2:
; ld a,(de)  ;Needed
; cp (hl)    ;Needed-Have
; dec de
; inc hl
; jr c,GetMetadataInf2S
; djnz GetMetadataInf2
GetMetadataInf2E:
 PRINTF(Aths.OutOfMem)
 call KeyWait
 jp AthenaInstaller
GetMetadataInf2S:


On another note, I've uploaded what I had of Athena to both Omnimaga and ticalc.org . I did Omnimaga because I don't really know how to attach files here other than use some sort of 3rd party file hosting service. I intend on updating these later when I fix some problems.

Right now, a problem I'm facing is the fact that files won't send to a real calc because of some invalid file error. I'm guessing I'm not doing the checksumming right. This is the following code I have for it:

Code:
int packto8xp(char *datastream,int datasize,unsigned char ftype,char *fname) {
    int a=0;
    int b=0;
    long int c=0;
    char tempfname[8+1+3+1];  //8.3 filename standard
    char chr = ' ';
    for (a=0;a<8;a++) {
        if (!(chr=fname[a])) break;
        tempfname[a]=chr;
    }
    char tempfnametype[]=".8xp\0";
    if (ftype==0x15) {
        tempfnametype[3]='v';
    }
    for (b=0;b<5;a++,b++) {
        tempfname[a]=tempfnametype[b];
    }
    FILE *outputfile;
    unsigned char temphead[74];
    outputfile=fopen(tempfname,"wb");
    if (outputfile==NULL) {
        printf("Cannot write out .8xp/.8xv file!\n");
        return 1;
    }
    for (a=0;a<74;temphead[a]=0,a++); //initializing the variable space
    for (a=0;a<53;temphead[a]=tiheader[a],a++);
    temphead[53]=(unsigned char) ((datasize+17+2)&0x00FF);
    temphead[54]=(unsigned char) (((datasize+17+2)&0xFF00)>>8);
    temphead[55]=0x0D;  //Checksumming starts at 55
    temphead[56]=0x00;
    temphead[57]=(unsigned char) ((datasize+2)&0x00FF);
    temphead[58]=(unsigned char) (((datasize+2)&0xFF00)>>8);
    temphead[59]=ftype;
    for (a=0,b=60;a<8;temphead[b]=0,a++,b++);
    for (a=0,b=60;a<8;a++,b++) {
        if (fname[a]==0) break;
        temphead[b]=fname[a];
    }
    temphead[68]=0;
    temphead[69]=0x80;
    temphead[70]=(unsigned char) ((datasize+2)&0x00FF);
    temphead[71]=(unsigned char) (((datasize+2)&0xFF00)>>8);
    temphead[72]=(unsigned char) ((datasize)&0x00FF);
    temphead[73]=(unsigned char) (((datasize)&0xFF00)>>8);
    for(a=55;a<74;c=c+temphead[a],a++);
    for(a=0;a<datasize;c=c+datastream[a],a++);
    if (74!=fwrite(temphead,1,74,outputfile)) return 1;
    if (datasize!=fwrite(datastream,1,datasize,outputfile)) return 1;
    temphead[0]=(unsigned char) ((c)&0x00FF);
    temphead[1]=(unsigned char) (((c)&0xFF00)>>8);
    if (2!=fwrite(temphead,1,2,outputfile)) return 1;
    fclose(outputfile);
    return 0;
}


EDIT: Solved the C code problem thing. When I was checksumming, I forgot to cast the input as unsigned char. Those changes aren't reflected above, but I guess you ought to know. For some reason, declaring the variables as unsigned char wasn't enough.
FWIW, I use this:


Code:
   bcall(_ChkFreeArc)
   ld hl,0839Fh
   ld d,(hl)
   inc hl
   ld e,(hl)
   inc hl
   ld a,(hl)   ;don't care about low byte @$83A2 though
Now that I've got most of the project squared away, I think I might want to go a step further with Athena, given that I don't start working on Escheron as soon as I can.

The idea would've been splitting the installer up into one very small ASM program (like 100-200 bytes) and the rest of it as a big library that can be put into saferam locations as the need arises. Doing it this way will keep much of RAM clear for the actual install, since the current Athena weighs in at 2600 or so bytes. Not as lightweight as I had hoped. With that, I may be able to achieve DCS compatibility.

So, that's the decision. E:SoR, or this.

EDIT: Here's a link to the current build of Athena. The on on ticalc needs to be updated to this: http://www.mediafire.com/?ll3nf7wf8c5335d
Iambian wrote:
Now that I've got most of the project squared away, I think I might want to go a step further with Athena, given that I don't start working on Escheron as soon as I can.
*want to?

Quote:
The idea would've been splitting the installer up into one very small ASM program (like 100-200 bytes) and the rest of it as a big library that can be put into saferam locations as the need arises. Doing it this way will keep much of RAM clear for the actual install, since the current Athena weighs in at 2600 or so bytes. Not as lightweight as I had hoped. With that, I may be able to achieve DCS compatibility.
I think that would be cool, but what part of the current setup is incompatible with Doors CS?

Quote:
So, that's the decision. E:SoR, or this.

EDIT: Here's a link to the current build of Athena. The on on ticalc needs to be updated to this: http://www.mediafire.com/?ll3nf7wf8c5335d
I think this is an awesome project, but if you're concerned that you're neglecting E:SoR too much, you should definitely work on what you're more interested in.
I did the upload thing and now Athena is on Cemetech, with added Cemetech-only goodies. Actually, the only "goodie" I put in was source for both the packager and installer. Hope you guys can get something out of it. Or tell me that my code sucks and provide something better. And I sure hope you guys know how to compile this stuff for yourself. Speaking of stuff, it's located here.

At some point, I'll update this installer with something that'll actually work with DCS. To start, I'd like a pointer to where I can find out what saferam areas are available when running in DCS.
That's great! Here's the download link for Athena:

http://www.cemetech.net/programs/index.php?mode=file&path=/win/AthenaPI.zip

As far as memory areas go, you get the same five SafeRAM chunks as with Ion, unless you use the GUI system or CALCnet. The following describe what memory areas you lose from using the GUI systems or CALCnet:

http://dcs.cemetech.net/index.php?title=GUI_Memory_Areas
http://dcs.cemetech.net/index.php?title=Interfacing_CALCnet2
It's been just a bit over two years, and now I'm revisiting this old project. What I'm proposing now is less of an update and more of an overhaul with the project.

The Athena Packager is getting a face-lift. Instead of a black console box and some batch file, it'll be an actual window. To add files to the project, you just drag and drop into the area that will accept those files. The window currently looks like this:


(Nevermind the bottom line. I forgot to remove it when I realized I could just use the console to print debug info)

Hovering over some of the text and boxes will reveal further information about what you can do with it, and more importantly, what you should not or cannot do.

The Athena Installer is also getting an overhaul. I realize that some BASIC projects are so large that they will want every last scrap of RAM the calc can offer, and Athena's current memory footprint may make that impossible. I'm going to rework it so absolutely nothing of Athena remains in User RAM when the install process happens. To accomplish this, I'm going to be ... uh. Juggling memory. No other way to describe that.

And now here's hoping that in the next couple of weeks, I get this thing released.

Note: The packager is being coded in Java. The reason for that is long and convoluted and doesn't need to be discussed here. Also, this isn't cross-platform, since I don't feel like figuring out how to get pucrunch to work cross-platform. If anyone can solve this problem, I'd like to hear it.
That looks great, Iambian; likely well-worth the no doubt very hard effort you put into it. Memory juggling on the calculator-side sounds challenging but necessary here. Regarding pucrunch, why not just re-implement it in Java?
Decided not to re-implement puCrunch in Java. Really didn't feel like trying to understand the source well enough to do that.

I've finally got the Java app to spit out files that resemble the original Athena package files, the test case being Reign of Legends 3. The Athena installer will read the package but for some reason, refuse to install them to calc (it's skipping over the contents as if it was a package test).

I'll have to run more tests "tomorrow". I don't remember coding any sort of fail-safe in case the archive is damaged...

EDIT: Turns out that the packager failed to properly store file types into its header. Due to the logic it was using, every file was registering as a real-type variable that should go in RAM. Got that fixed and got to playing RoL3 for a little bit.

All that's left is overhauling the installer file and allow self-extracting archives.
Great, it sounds as if this project is nearing some sort of releasable form! Plus I'm always happy when more people understand the nitty-gritty behind the file format and other details of the calculator beyond the challenges of programming the device itself. What do you mean by overhauling the installer file?
  
Register to Join the Conversation
Have your own thoughts to add to this or any other topic? Want to ask a question, offer a suggestion, share your own programs and projects, upload a file to the file archives, get help with calculator and computer programming, or simply chat with like-minded coders and tech and calculator enthusiasts via the site-wide AJAX SAX widget? Registration for a free Cemetech account only takes a minute.

» Go to Registration page
Page 2 of 3
» All times are UTC - 5 Hours
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

 

Advertisement