Oh, silly me, yeah, it seems the final product is fuctional

OK, very good, I needed to make sure because one of the final issues was in the calculation of the checksums. With 2.x, I was dealing with characters; with 3.x, I was dealing with integers. Pain in the neck strict typing. :/

A few updates.

First, I've settled on a name for the project (at least temporarily, anyway)- it will be called "z89 IDE". I am, however, considering naming the on-calc component "Solar89" as a reference to Lunar IDE. I'm bad at coming up with names.

Second, the Python end of things is working perfectly. I tested it with a variable generated from an older version of the tokenizer and it produced a *.8xp file, which I ran successfully in WabbitEmu! Admittedly, all it did was clear the screen, but that's progress.

I'm now resuming work on the tokenizer. I just need to finish getting it to read tokens in from a text file, then actually write that text file, and then I can release a beta.

Sounds great! Solar89 is a fun name; I like how you came up with it. Good luck with your tokenizer.

Update: the tokenizer is working now. And it works fine with the Python converter as well, and these screenshots prove it- the text file and the tokenizer in TiEmu, and then the program in WabbitEmu.

I now need to finish writing the token file (which as you can probably tell doesn't have many tokens in it yet...) and implement error checking to stop the program from running an infinite loop when it finds a character that's not a valid token.

Very impressive! Tell me about the tokenizer: what kind of speed does it achieve? I bet even my PITA brute-force algorithm would get 10-20 tokens per second on a 68k calc.

It's very fast for really small files (all I've tried thus far).

A major speed drain is going to be parsing the token text file, which is why I'm considering writing a seperate program to "compile" the token text file into another file containing only the array of tokenInfo structures that can then be read directly by the tokenizer.

Since the token file is small at the moment, this speed drain isn't yet noticeable.

Very nice. What algorithm are you using for the token file? Iterative substring searching, or a tree-based structure of some sort?

Iterative substring search- I'm not comfortable enough in C programming to try implementing any more complex algorithms. So that's why I'm worried about speed here.

Makes sense. If you wanted to show me the code, I'd be happy to help you with a tree-based method, but if you think it's not worth it, I'm sure what you're doing is fine.

Update: All one-byte tokens are now supported.

I've been working on an error-handling system, but it has some bugs at the moment (sometimes it works, sometimes it doesn't)... however, here's how it will work: if the converter reduces a line to a single character, and that character is not a token, an error message appears in a dialog box. Then, the program then writes a question-mark token (not one of the unused tokens, the actual AFh question mark) in place of that character.

This is necessary because otherwise the program will enter an infinite loop, trying to break down the one-character line further.



Maybe- we'll see as the token file(s) grow in size. At the moment, with only the one-byte tokens, my text file is 3,233 bytes in size. I haven't used that file in TiEmu yet with the current build of Solar89, though, so I don't know how laggy it would be.

Yeah, I came up with an estimate of about 5-6KB for a full one- and two-token file including string and token value information, hence why I decided not to go with that method.

And multi-line programs, and multi-token lines both work now, as you can see from the Hello World program I've tokenized and loaded into WabbitEmu.

I'm still not sure if error checking is working properly- I'll need to look at that.

The entire process of initializing the one-byte tokens, then tokenizing Hello World, takes five seconds. For small programs that's okay but once I add the two-byte tokens it might become problematic, so I might that tree-based algorithm after all. But I'm going to focus on fixing error-checking and making sure two-byte tokens actually work first...

TC01, impressive! I think it would take Tokenizer DE two passes of two seconds each on a TI-83+ to do that, so that's a decent speed.

I have uploaded v0.1, an alpha and testing version, since everything seems to be working (even though there are only one-byte tokens supported at the moment).

If you use it on a real calculator, I highly recommend doing a backup and then installing a crash protection program (either PreOS or KerNO) if you don't use one already, simply because it's an alpha, and even though it seems to work, there's always the chance of a bug that I missed.

I have provided a demo that you can try tokenizing (the same program in the screenshot above). There's documentation on what exactly to do with all of the programs included (in the various readme files), and once you get everything setup I recommend trying the demo (since I know it should work).

I'd like people with a Voyage 200 or TI-92 Plus to try it on those calculators, to see if it works- I have versions provided for them but I obviously can't test them. You can also try testing other calculator models- the only one that definitely won't work is the 85 since BinPac8x makes strings, not program files for it. (And the 86 probably uses different token codes than the 82, 83, and 84+).

Download here

ho, ho, that was quick, cant wait to try it

Wait, I thought 85 strings were directly comparable to 85 programs, now? Doesn't it tokenize on-the-fly or something?

Oh... they share the same format, it seems. (I obviously don't own a TI-85).

And I would guess they're converted to tokens when leaving the program editor like on the 89?

I guess all I'd need to do is change BinPac8x to produce programs instead of strings to get it to work (with a 85 token file, of course). I don't suppose you know how to do that too?

Two more updates.

First, two-byte tokens now work! I've only tested (and added) the matrix tokens, but they tokenize fine. I did have a bug involving any token with a 00 byte, but with TravisE's help I fixed it. Now I can start adding the other two-byte tokens.

Second, I've added a new feature (of a sort): the ability to run this program by passing it three arguments instead of from the dialog box. This might not sound too useful, but I did it because it would allow you to call this program from a 68k BASIC program, allowing you to quickly retokenize a program that you are editing without needing to fill in the dialog again (something that has annoyed me in my testing). It also doesn't break the dialog- if you do not have three arguments, the dialog box appears instead.

Also, I can see that a few people have downloaded the alpha I uploaded... if you have, has it worked? Any bugs to report?

Awesome, glad to hear that two-byte tokens work, and I can imagine that a token with $00 in it might cause problems. Having the API exposed sounds like a good idea as well, as you said, for possible BASIC integration.