Login [Register]
Don't have an account? Register now to chat, post, use our tools, and much more.
I noticed that there isn't any online IDE for 68k Basic, so I decided to create one myself. I've nearly finished adding the regex rules for the language itself (using the Simple Mode addon), which means I have to turn to writing a tokenizer.

The question is, what do I need to know to write a tokenizer for 68k calculators? I would appreciate any help. Thanks Smile
Battlesquid wrote:
I noticed that there isn't any online IDE for 68k Basic, so I decided to create one myself. I've nearly finished adding the regex rules for the language itself (using the Simple Mode addon), which means I have to turn to writing a tokenizer.

The question is, what do I need to know to write a tokenizer for 68k calculators? I would appreciate any help. Thanks Smile

I feel like the easiest way (although it would take some considerable amount of elbow grease) would be to send programs containing all tokens over to your computer, then noting down what each token is represented as, and using a small search and replace script to remove the header and convert each token. I have worked on this a little with ti-83/84 basic, but got bored after 100-150 tokens. Also, I had a little trouble with some tokens which didn't convert well like these:

Code:
   log(      ΐ
   ^^2      ΐ

Those two tokens become strange when tokenized by a computer.

EDIT: Here is my vbs script to convert a 8xp file to a tokenized txt file, the file browsing function is borrowed from the internets

Code:
Function BrowseForFile()
BrowseForFile = CreateObject("WScript.Shell").Exec("mshta.exe ""about:<input type=file id=f>" &"<script>resizeTo(0,0);f.click();new ActiveXObject('Scripting.FileSystemObject')"&".GetStandardStream(1).WriteLine(f.value);close();</script>""").StdOut.ReadLine()
End Function
filePath = BrowseForFile()
If filePath = "" Then
MsgBox "Operation canceled",vbcritical
else
arr=Array("sin(","cos(","tan(","srt(","^","/","*","-","+",vbcrlf,".","-","ln(")
arr2=Array("Β","Δ","Ζ","Ό","π","ƒ","‚","q","p","?",":","°","Ύ")
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFile = objFSO.OpenTextFile(filePath, 1)
strText = objFile.ReadAll
objFile.Close
begin=InStr(strText, "")+2
strText=Mid(strText,begin,len(strText)-1-begin)
For i = 0 To 11
strText = Replace(strText, arr2(i), arr(i))
Next
Set objFile = objFSO.OpenTextFile(filePath, 2)
objFile.WriteLine strNewText
objFile.Close
objFSO.MoveFile filePath, Replace(filePath, "8xp", "txt")
End If

However, you need to remember that because of the way tokens work on the calc, if you want to complete this script and make it work for all tokens, you will need to check for 2-byte tokens first, as the 2nd byte of those tokens would get converted if you just added all 1 byte tokens and ran the script.

I've revisited this, turns out the above script is bad because it searches and replaces the entire string each time, which means for example if the token for sin( got converted, and then it searches for "n" and replaces that with the token associated with n, then the output will be something like "si xor (", which is bad. I instead go through each token one at a time, convert it and add it to a new string. I added a handfull of 1 byte tokens, and also all the letters. The 2-byte tokens will need to be broken up into chunks depending on what the indicator byte is. I merged 2 indicator bytes to put all the lowercase letters in one line, but if the 2nd bytes of those two indicator bytes overlap, then they will need to be seperated. Anyways, here is the revised code.

Code:
Function BrowseForFile()
BrowseForFile = CreateObject("WScript.Shell").Exec("mshta.exe ""about:<input type=file id=f>" &"<script>resizeTo(0,0);f.click();new ActiveXObject('Scripting.FileSystemObject')"&".GetStandardStream(1).WriteLine(f.value);close();</script>""").StdOut.ReadLine()
End Function
filePath = BrowseForFile()
If filePath = "" Then
MsgBox "Operation canceled",vbcritical
else
arr=Array("sin(","cos(","tan(","sqrt(","^","/","*","-","+",vbcrlf,".","-","ln(","^^-1","(pi)","e^(","[i]",":"," ","(theta)","?","]","[","{","}","=","!=",">",">=","<","<="," and "," or "," xor ","not(","abs(","round(","iPart(","fPart(","int(","min(","max(")
arr2=Array("Β","Δ","Ζ","Ό","π","ƒ","‚","q","p","?",":","°","Ύ"," ","¬","Ώ",",",">",")","[","―","","","","   ","j","o","l","n","k","m","@","<","=","Έ","²","","Ή","Ί","±","","")
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFile = objFSO.OpenTextFile(filePath, 1)
strText = objFile.ReadAll
objFile.Close
fileNamep=InStrRev(filePath,"\",-1)
fileName=Mid(filePath, filenamep+1,Len(filePath)-filenamep-4)
begin=InStr(strText,filename)+14
strText=Mid(strText,begin,len(strText)-1-begin)
Dim strNewText
For i = 0 To Len(strText)-1
FindThisString = Left(strText,1)
If FindThisString="»" or FindThisString="b" Then
FindThisString=Left(strText,2)
strNewText=strNewText&Mid("abcdefghijklmnopqrstuvwxyz",InStr("΅Ά·ΈΉΊΌ½ΎΏΐΑΒΓΔΕΖΗΘΙΚ",Right(FindThisString,1)),1)
strText=Right(strText,Len(strText)-2)
i=i+1
else
For j = 0 To 41
    If arr2(j)=FindThisString Then
strNewText=strNewText&arr(j)
    End If
Next
If InStr("ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789",FindThisString) then
strNewText=strNewText&FindThisString
End If
strText=Right(strText,Len(strText)-1)
End If
Next
Set objFile = objFSO.OpenTextFile(filePath, 2)
objFile.WriteLine strNewText
objFile.Close
objFSO.MoveFile filePath, Replace(filePath, "8xp", "txt")
End If
If I remember correctly, the way programs are tokenized on the 68K calculators is quite different from the Z80 models. It's a much more complex system; I believe that expressions are even translated into an RPN-like format internally. It's not merely a one-to-one correspondence between token and text.

I don't know of a complete, definitive token list, so you may have to get really dirty and dive deep into the bowels of the GCC4TI documentation, especially estack.h, to get some kind of idea of how it works. The Command Post Plus flash app may be handy to mess around on the calc and check/edit the raw hex codes of variables, including tokenized programs, to see what they look like.

Another important thing to know is that the 68K models have an untokenized program format, too, which is essentially the source code in plain text using the calculator's character encoding. Programs in RAM that have been newly created on the calculator, or edited since last run, will be untokenized instead of tokenized. You'll want your IDE to be aware of these as well. This can get ugly, as I've heard that the untokenized form is not localization safe. The TI language packs for the calculators translate all the function names into different languages, which really makes things fun when users of different languages try to transfer programs that weren't tokenized.…
Do you mean these? I might start here, then go up to understand how estack.h works.
The state of the art, computer-side tokenizer for the TI-68k series remains the horrible Visual Basic code from Tokens89, whose most recent version is in Peter Engels' TI-Edit ( http://pengels.bplaced.net/index.php/tiedit ). Tokens89 was created by Kevin Kofler, before he both turned free software extremist and strongly despised TI-Basic and TI-Basic programmers, i.e. a looong time ago - give or take 15 years. He was already a jerk.

Excale had started converting Tokens89 to PHP and JS, I'll ping him.
I found out that opening an 89P file in notepad gives me this (not that the program isn't mine, I just used it for testing purposes):

Code:

**TI92P* main bo                                          R   bomberma  π  ₯Z    –()
Prgm
setFold(bomberma)
Local a,b,c,c01,c02,c03,d,d01,d02,d03,e,e1,e2,f,f1,f2,m1,m2,m3,n1,o,on1,on2,on3,on4,r,x,ˆ,ˆ2
RplcPic b1
For ˆ,1,225
getKey()ˆ2:If ˆ20:Goto st
EndFor
ClrDraw
PxlText "Presents",30,55
For ˆ,1,225
getKey()ˆ2:If ˆ20:Goto st
EndFor
For ˆ,30,36
getKey()ˆ2:If ˆ20:Goto st
PxlHorz ˆ
EndFor
For ˆ,30,36
getKey()ˆ2:If ˆ20:Goto st
PxlHorz ˆ,­1
EndFor
Lbl st
RplcPic b2
Toolbar
Title "  Play  ",p
Title "  Options  ",o
Title "  About  ",about
Title "  Exit  ",e
EndTBar
Lbl p
RplcPic b3
For ˆ,0,o11-1
Lbl bl
rand(7)a
a-1a:a*8a:a+3a
rand(23)b
b*6b:b+4b
If pxlTest(a+3,b+3)=true or a=3 and b=10 or a=3 and b=16 or a=11 and b=10:Goto bl
RclPic b4,a,b
EndFor
Lbl rup
rand(5)e1
e1-1e1:e1*8e1:e1+3e1
rand(11)f1
f1*6f1:f1+4f1
If pxlTest(e1+3,f1+3)=true:Goto rup
RclPic b6,e1,f1
Lbl rdo
rand(5)e2
e2-1e2:e2*8e2:e2+3e2
rand(11)f2
f2*6f2:f2+4f2
If pxlTest(e2+3,f2+3)=true:Goto rdo
RclPic b7,e2,f2
Lbl m1
rand(7)c01:c01-1c01:c01*8c01:c01+3c01
rand(23)d01:d01*6d01:d01+4d01
If pxlTest(c01+3,d01+3)=true or c01=3 and d01=10 or c01=3 and d01=16 or c01=11 and d01=10:Goto m1
Lbl m2
rand(7)c02:c02-1c02:c02*8c02:c02+3c02
rand(23)d02:d02*6d02:d02+4d02
If pxlTest(c02+3,d02+3)=true or c02=3 and d02=10 or c02=3 and d02=16 or c02=11 and d02=10:Goto m2
Lbl m3
rand(7)c03:c03-1c03:c03*8c03:c03+3c03
rand(23)d03:d03*6d03:d03+4d03
If pxlTest(c03+3,d03+3)=true or c03=3 and d03=10 or c03=3 and d03=16 or c03=11 and d03=10:Goto m3
3a:10b:0c:0d
0e:9f
o12n1:1r
1m1:1m2:1m3:3x
RclPic b3
PxlText "#X:   :   :  # :",68,10
RclPic b5,68,46:RclPic b5,68,76:RclPic b5,68,106
Lbl main
PxlText string(x),68,28
PxlText "  ",68,118
PxlText string(n1),68,118
Loop
RclPic b6,e1,f1
RclPic b7,e2,f2
If a=e1 and b=f1:r+1r
If a=e2 and b=f2:r-1r
If r>5:5r:If r<1:1r
PxlText string(r),68,58
getKey()ˆ
If e=1 and ˆ=kbo or fœ0
Goto bo
If e=0 and ˆ=kbo Then
1e:ac:bd:n1-1n1
EndIf
If e=1 Then:f-1f
RclPic b5,c,d:EndIf
PxlText string(f),68,88
If ˆ0:PxlText " ",a,b
If ˆ=kup and pxlTest(a-2,b+3)=false:a-8a
If ˆ=kdo and pxlTest(a+8,b+3)=false:a+8a
If ˆ=kle and pxlTest(a+3,b-1)=false:b-6b
If ˆ=kri and pxlTest(a+3,b+7)=false:b+6b
RclPic b8,a,b
If m1=1 Then
rand(4)ˆ2
PxlText " ",c01,d01
If ˆ2=1 and pxlTest(c01-2,d01+3)=false:c01-8c01
If ˆ2=2 and pxlTest(c01+8,d01+3)=false:c01+8c01
If ˆ2=3 and pxlTest(c01+3,d01-1)=false:d01-6d01
If ˆ2=4 and pxlTest(c01+3,d01+7)=false:d01+6d01
PxlText "X",c01,d01
If a=c01 and b=d01:Goto w2:EndIf
If m2=1 Then
rand(4)ˆ2
PxlText " ",c02,d02
If ˆ2=1 and pxlTest(c02-2,d02+3)=false:c02-8c02
If ˆ2=2 and pxlTest(c02+8,d02+3)=false:c02+8c02
If ˆ2=3 and pxlTest(c02+3,d02-1)=false:d02-6d02
If ˆ2=4 and pxlTest(c02+3,d02+7)=false:d02+6d02
PxlText "X",c02,d02
If a=c02 and b=d02:Goto w2:EndIf
If m3=1 Then
rand(4)ˆ2
PxlText " ",c03,d03
If ˆ2=1 and pxlTest(c03-2,d03+3)=false:c03-8c03
If ˆ2=2 and pxlTest(c03+8,d03+3)=false:c03+8c03
If ˆ2=3 and pxlTest(c03+3,d03-1)=false:d03-6d03
If ˆ2=4 and pxlTest(c03+3,d03+7)=false:d03+6d03
PxlText "X",c03,d03
If a=c03 and b=d03:Goto w2:EndIf
EndLoop
Lbl bo
0on1:0on2:0on3:0on4
PxlText " ",c,d
If pxlTest(c-3,d+3)=false Then:PxlText " ",c-8,d
Else:1on1:EndIf
If pxlTest(c+9,d+3)=false Then:PxlText " ",c+8,d
Else:1on2:EndIf
If pxlTest(c+3,d-2)=false Then:PxlText " ",c,d-6
Else:1on3:EndIf
If pxlTest(c+3,d+8)=false Then:PxlText " ",c,d+6
Else:1on4:EndIf
If rž2 and c-10ž4 and on1=0 Then
If pxlTest(c-11,d+3)=false Then
PxlText " ",c-16,d
Else:1on1:EndIf:EndIf
If rž2 and c+16œ59 and on2=0 Then
If pxlTest(c+17,d+3)=false Then
PxlText " ",c+16,d
Else:1on2:EndIf:EndIf
If rž2 and d-7ž7 and on3=0 Then
If pxlTest(c+3,d-8)=false Then
PxlText " ",c,d-12
Else:1on3:EndIf:EndIf
If rž2 and d+13œ148 and on4=0 Then
If pxlTest(c+3,d+14)=false Then
PxlText " ",c,d+12
Else:1on4:EndIf:EndIf
If rž3 and c-18ž4 and on1=0 Then
If pxlTest(c-19,d+3)=false Then
PxlText " ",c-24,d
Else:1on1:EndIf:EndIf
If rž3 and c+24œ59 and on2=0 Then
If pxlTest(c+25,d+3)=false Then
PxlText " ",c+24,d
Else:1on2:EndIf:EndIf
If rž3 and d-13ž7 and on3=0 Then
If pxlTest(c+3,d-14)=false Then
PxlText " ",c,d-18
Else:1on3:EndIf:EndIf
If rž3 and d+19œ148 and on4=0 Then
If pxlTest(c+3,d+20)=false Then
PxlText " ",c,d+18
Else:1on4:EndIf:EndIf
If rž4 and c-24ž4 and on1=0 Then
If pxlTest(c-25,d+3)=false Then
PxlText " ",c-32,d
Else:1on1:EndIf:EndIf
If rž4 and c+32œ59 and on2=0 Then
If pxlTest(c+33,d+3)=false Then
PxlText " ",c+32,d
Else:1on2:EndIf:EndIf
If rž4 and d-19ž7 and on3=0 Then
If pxlTest(c+3,d-20)=false Then
PxlText " ",c,d-24
Else:1on3:EndIf:EndIf
If rž4 and d+25œ148 and on4=0 Then
If pxlTest(c+3,d+26)=false Then
PxlText " ",c,d+24
Else:1on4:EndIf:EndIf
If r=5 and c-34ž4 and on1=0 Then
If pxlTest(c-35,d+3)=false Then
PxlText " ",c-40,d
Else:1on1:EndIf:EndIf
If r=5 and c+40œ59 and on2=0 Then
If pxlTest(c+41,d+3)=false Then
PxlText " ",c+40,d
Else:1on2:EndIf:EndIf
If r=5 and d-25ž7 and on3=0 Then
If pxlTest(c+3,d-26)=false Then
PxlText " ",c,d-30
Else:1on3:EndIf:EndIf
If r=5 and d+31œ148 and on4=0 Then
If pxlTest(c+3,d+32)=false Then
PxlText " ",c,d+30
Else:1on4:EndIf:EndIf
If pxlTest(a+3,b+3)=false:Goto w2
If pxlTest(c01+3,d01+3)=false and m1=1 Then:0m1:x-1x:EndIf
If pxlTest(c02+3,d02+3)=false and m2=1 Then:0m2:x-1x:EndIf
If pxlTest(c03+3,d03+3)=false and m3=1 Then:0m3:x-1x:EndIf
If m1=0 and m2=0 and m3=0:Goto w1
If n1=0:Goto w2
0e:9f
RclPic b3
Goto main
Lbl w1
Pause
Dialog
Title "BomberMaster!"
Text "You Win!"
EndDlog
Goto st
Lbl w2
Pause
Dialog
Title "BomberMaster!"
Text "You Lose!"
EndDlog
Goto st
Lbl o
0o
PopUp {"General Options","Select Keys"},o
If o=0:Goto st
If o=1:Goto op1
If o=2:Goto op2
Lbl op1
Unarchiv o11,o12
If o11=10:1o11
If o11=20:2o11
If o11=30:3o11
If o11=40:4o11
If o11=50:5o11
If o11=60:6o11
If o11=70:7o11
If o11=80:8o11
If o11=90:9o11
If o12=5:1o12
If o12=10:2o12
If o12=15:3o12
If o12=20:4o12
Dialog
Title "General Options:"
DropDown "# Blocks",{"10","20","30","40","50","60","70","80","90"},o11
DropDown "# Bombs",{"5","10","15","20"},o12
EndDlog
If o11=1:10o11
If o11=2:20o11
If o11=3:30o11
If o11=4:40o11
If o11=5:50o11
If o11=6:60o11
If o11=7:70o11
If o11=8:80o11
If o11=9:90o11
If o12=1:5o12
If o12=2:10o12
If o12=3:15o12
If o12=4:20o12
Archive o11,o12
Goto st
Lbl op2
ClrDraw
Unarchiv kup,kdo,kle,kri,kbo
PxlText "",8,6
Loop
getKey()ˆ:If ˆ0:Exit
EndLoop
ˆkup
PxlText "",8,6
Loop
getKey()ˆ:If ˆ0:Exit
EndLoop
ˆkdo
PxlText "",8,6
Loop
getKey()ˆ:If ˆ0:Exit
EndLoop
ˆkle
PxlText "",8,6
Loop
getKey()ˆ:If ˆ0:Exit
EndLoop
ˆkri
RplcPic b5,8,6
Loop
getKey()ˆ:If ˆ0:Exit
EndLoop
ˆkbo
Archive kup,kdo,kle,kri,kbo
Goto st
Lbl about
Dialog
Title "BomberMaster!"
Text "Joshua Entertainent"
Text "By Joshua Masato Chernin"
EndDlog
Goto st
Lbl e
ClrDraw:DispHome
setFold(main)
EndPrgm δε άS§


As you can see, most of the code is readable. Would this mean (in theory) that I could just compare the notepad file and the actual file, and take what tokens are missing/converted improperly and use that to create the tokenizer (replacing the .txt extension with the proper extension)?
When the program is stored in untokenized form, yes, much of it is readable.
What you describe looks like reinventing a known wheel, which can be a valuable learning exercise, but probably isn't the fastest way to achieve the aim of creating a TI-68k BASIC IDE Smile
No, when you want to convert the file back to an actual program, you will need to convert the tokens back to their jumbled state. Remember, the only reason you were converting it to begin with was to make it human readable.
But yes, you will be able to create the tokenizer by comparing the actual code and the notepad version, assuming there is one of each token that doesn't convert correctly in the program you are using.
The mostly-readable version is what it looks like when the program is currently untokenized on the calculator. Other than some binary headers and stuff, it's pretty much all text, with a special character encoding for the non-ASCII characters.

You could certainly make the IDE read these text versions, convert it to Unicode (or whatever) for editing, and then back to calculator format for sending. However, this would only work if the program is untokenized. This happens when the program is opened in the program editor on the calculator and the program is not archived. But as soon as an attempt is made to run the program, the calculator converts it back into tokenized format, and it will not look like text anymore if you send it to the computer. If the user sends the program to you IDE in that state, it will not be readable unless your IDE can interpret the tokens (or call some library or code that can do so).

Someone familiar with the 68K calc localization would have to confirm this, but I've also heard that the untokenized text will only work if the calculator is set for the same language as the untokenized text, because supposedly the non-English language packs translate the names of all the built-in functions and commands. This could be an inconvenience for users working in non-English languages. I don't think the tokenized versions have this problem. Also, sending untokenized programs to the calculator causes a delay the first time the program is run while the program is retokenized (the larger the program, the longer the delay), but this is a relatively minor issue. (The worst case is archiving the program before running/tokenizing it first; then the delay happens every time it is run instead of just the first.)
Quote:
I've also heard that the untokenized text will only work if the calculator is set for the same language as the untokenized text

There are localization issues with both the untokenized form and the tokenized form:
* the former won't (always) tokenize properly in a different language;
* in some cases (*), the latter will work properly only if the calculator's current language is the one the programmer depends on. The only way to make several functions which return localized strings, such as the one which returns the type of a variable, work properly on all languages is to... you guessed it, compare said string against every possible expected value inside the set of language localizations provided for the TI-68k series, IIRC ~13 of them. One TI-68k developer attempted to cope with this brokenness, but he likely remained alone in this endeavour...
Lionel Debroux wrote:
Quote:
I've also heard that the untokenized text will only work if the calculator is set for the same language as the untokenized text

There are localization issues with both the untokenized form and the tokenized form:
* the former won't (always) tokenize properly in a different language;
* in some cases (*), the latter will work properly only if the calculator's current language is the one the programmer depends on. The only way to make several functions which return localized strings, such as the one which returns the type of a variable, work properly on all languages is to... you guessed it, compare said string against every possible expected value inside the set of language localizations provided for the TI-68k series, IIRC ~13 of them. One TI-68k developer attempted to cope with this brokenness, but he likely remained alone in this endeavour...


I don't think I want to do that on my own Shock. If I ever find someone to help me with localization, I'll do that, but my main focus is on completing the IDE first in English.

Travis wrote:
If the user sends the program to you IDE in that state, it will not be readable unless your IDE can interpret the tokens (or call some library or code that can do so).

I think this is the path I'll most likely go down. I would probably do this by scanning each line for tokens that do not convert properly, then find the index of the token in a predefined string array thingy (this would contain all of the 'readable' tokens that do not convert properly) and use the index to match up with another string array thingy that contains all the jumbled up text that notepad spits out and swap the tokens. Then this would be sent to the calculator.

If anybody made sense of that (I hope so!), could this be possible?
Battlesquid wrote:
Lionel Debroux wrote:
Quote:
I've also heard that the untokenized text will only work if the calculator is set for the same language as the untokenized text

There are localization issues with both the untokenized form and the tokenized form:
* the former won't (always) tokenize properly in a different language;
* in some cases (*), the latter will work properly only if the calculator's current language is the one the programmer depends on. The only way to make several functions which return localized strings, such as the one which returns the type of a variable, work properly on all languages is to... you guessed it, compare said string against every possible expected value inside the set of language localizations provided for the TI-68k series, IIRC ~13 of them. One TI-68k developer attempted to cope with this brokenness, but he likely remained alone in this endeavour...


I don't think I want to do that on my own Shock. If I ever find someone to help me with localization, I'll do that, but my main focus is on completing the IDE first in English.

Travis wrote:
If the user sends the program to you IDE in that state, it will not be readable unless your IDE can interpret the tokens (or call some library or code that can do so).

I think this is the path I'll most likely go down. I would probably do this by scanning each line for tokens that do not convert properly, then find the index of the token in a predefined string array thingy (this would contain all of the 'readable' tokens that do not convert properly) and use the index to match up with another string array thingy that contains all the jumbled up text that notepad spits out and swap the tokens. Then this would be sent to the calculator.

If anybody made sense of that (I hope so!), could this be possible?

Yep, that's exactly what the code I posted a while back does.
  
Register to Join the Conversation
Have your own thoughts to add to this or any other topic? Want to ask a question, offer a suggestion, share your own programs and projects, upload a file to the file archives, get help with calculator and computer programming, or simply chat with like-minded coders and tech and calculator enthusiasts via the site-wide AJAX SAX widget? Registration for a free Cemetech account only takes a minute.

» Go to Registration page
Page 1 of 1
» All times are GMT - 5 Hours
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

 

Advertisement