I noticed that there isn't any online IDE for 68k Basic, so I decided to create one myself. I've nearly finished adding the regex rules for the language itself (using the Simple Mode addon), which means I have to turn to writing a tokenizer.

The question is, what do I need to know to write a tokenizer for 68k calculators? I would appreciate any help. Thanks

I feel like the easiest way (although it would take some considerable amount of elbow grease) would be to send programs containing all tokens over to your computer, then noting down what each token is represented as, and using a small search and replace script to remove the header and convert each token. I have worked on this a little with ti-83/84 basic, but got bored after 100-150 tokens. Also, I had a little trouble with some tokens which didn't convert well like these:

Code:

Those two tokens become strange when tokenized by a computer.

EDIT: Here is my vbs script to convert a 8xp file to a tokenized txt file, the file browsing function is borrowed from the internets

Code:

However, you need to remember that because of the way tokens work on the calc, if you want to complete this script and make it work for all tokens, you will need to check for 2-byte tokens first, as the 2nd byte of those tokens would get converted if you just added all 1 byte tokens and ran the script.

I've revisited this, turns out the above script is bad because it searches and replaces the entire string each time, which means for example if the token for sin( got converted, and then it searches for "n" and replaces that with the token associated with n, then the output will be something like "si xor (", which is bad. I instead go through each token one at a time, convert it and add it to a new string. I added a handfull of 1 byte tokens, and also all the letters. The 2-byte tokens will need to be broken up into chunks depending on what the indicator byte is. I merged 2 indicator bytes to put all the lowercase letters in one line, but if the 2nd bytes of those two indicator bytes overlap, then they will need to be seperated. Anyways, here is the revised code.

Code:

If I remember correctly, the way programs are tokenized on the 68K calculators is quite different from the Z80 models. It's a much more complex system; I believe that expressions are even translated into an RPN-like format internally. It's not merely a one-to-one correspondence between token and text.

I don't know of a complete, definitive token list, so you may have to get really dirty and dive deep into the bowels of the GCC4TI documentation, especially estack.h, to get some kind of idea of how it works. The Command Post Plus flash app may be handy to mess around on the calc and check/edit the raw hex codes of variables, including tokenized programs, to see what they look like.

Another important thing to know is that the 68K models have an untokenized program format, too, which is essentially the source code in plain text using the calculator's character encoding. Programs in RAM that have been newly created on the calculator, or edited since last run, will be untokenized instead of tokenized. You'll want your IDE to be aware of these as well. This can get ugly, as I've heard that the untokenized form is not localization safe. The TI language packs for the calculators translate all the function names into different languages, which really makes things fun when users of different languages try to transfer programs that weren't tokenized.…

Do you mean these? I might start here, then go up to understand how estack.h works.

The state of the art, computer-side tokenizer for the TI-68k series remains the horrible Visual Basic code from Tokens89, whose most recent version is in Peter Engels' TI-Edit ( http://pengels.bplaced.net/index.php/tiedit ). Tokens89 was created by Kevin Kofler, before he both turned free software extremist and strongly despised TI-Basic and TI-Basic programmers, i.e. a looong time ago - give or take 15 years. He was already a jerk.

Excale had started converting Tokens89 to PHP and JS, I'll ping him.

I found out that opening an 89P file in notepad gives me this (not that the program isn't mine, I just used it for testing purposes):

Code:


As you can see, most of the code is readable. Would this mean (in theory) that I could just compare the notepad file and the actual file, and take what tokens are missing/converted improperly and use that to create the tokenizer (replacing the .txt extension with the proper extension)?

When the program is stored in untokenized form, yes, much of it is readable.
What you describe looks like reinventing a known wheel, which can be a valuable learning exercise, but probably isn't the fastest way to achieve the aim of creating a TI-68k BASIC IDE

No, when you want to convert the file back to an actual program, you will need to convert the tokens back to their jumbled state. Remember, the only reason you were converting it to begin with was to make it human readable.
But yes, you will be able to create the tokenizer by comparing the actual code and the notepad version, assuming there is one of each token that doesn't convert correctly in the program you are using.

The mostly-readable version is what it looks like when the program is currently untokenized on the calculator. Other than some binary headers and stuff, it's pretty much all text, with a special character encoding for the non-ASCII characters.

You could certainly make the IDE read these text versions, convert it to Unicode (or whatever) for editing, and then back to calculator format for sending. However, this would only work if the program is untokenized. This happens when the program is opened in the program editor on the calculator and the program is not archived. But as soon as an attempt is made to run the program, the calculator converts it back into tokenized format, and it will not look like text anymore if you send it to the computer. If the user sends the program to you IDE in that state, it will not be readable unless your IDE can interpret the tokens (or call some library or code that can do so).

Someone familiar with the 68K calc localization would have to confirm this, but I've also heard that the untokenized text will only work if the calculator is set for the same language as the untokenized text, because supposedly the non-English language packs translate the names of all the built-in functions and commands. This could be an inconvenience for users working in non-English languages. I don't think the tokenized versions have this problem. Also, sending untokenized programs to the calculator causes a delay the first time the program is run while the program is retokenized (the larger the program, the longer the delay), but this is a relatively minor issue. (The worst case is archiving the program before running/tokenizing it first; then the delay happens every time it is run instead of just the first.)

There are localization issues with both the untokenized form and the tokenized form:
* the former won't (always) tokenize properly in a different language;
* in some cases (*), the latter will work properly only if the calculator's current language is the one the programmer depends on. The only way to make several functions which return localized strings, such as the one which returns the type of a variable, work properly on all languages is to... you guessed it, compare said string against every possible expected value inside the set of language localizations provided for the TI-68k series, IIRC ~13 of them. One TI-68k developer attempted to cope with this brokenness, but he likely remained alone in this endeavour...

I don't think I want to do that on my own . If I ever find someone to help me with localization, I'll do that, but my main focus is on completing the IDE first in English.


I think this is the path I'll most likely go down. I would probably do this by scanning each line for tokens that do not convert properly, then find the index of the token in a predefined string array thingy (this would contain all of the 'readable' tokens that do not convert properly) and use the index to match up with another string array thingy that contains all the jumbled up text that notepad spits out and swap the tokens. Then this would be sent to the calculator.

If anybody made sense of that (I hope so!), could this be possible?

Yep, that's exactly what the code I posted a while back does.