In answer to all the users that have requested a custom oncalc TI-BASIC editor, Tokenizer DE takes a first step in that direction. Just as SourceCoder (http://sc.cemetech.net) can convert text files into .8xp programs online, Tokenizer can take text documents typed in Document DE 6 or Document DE 7 and convert them into fully-tokenized BASIC programs editable with the TI-OS's TI-BASIC editor. Weighing in at slightly over 1KB, this small but efficient program takes advantage of the power of the DCS GUI system, and requires Doors CS 6.8 beta or higher. It uses a naive reverse substring-match brute force algorithm that takes very little memory (roughly 300 bytes for the core tokenization algorithm) at the expense of going at a moderate rather than blazing speed. Future improvements may integrate this with a Document DE-style editor; feel free to give it a try and post any comments or suggestions in the associated topic. This release also includes the source code to Tokenizer DE for those wishing to learn more about using the Doors CS ASM libraries.

Download
Tokenizer DE v1.0

Haha, now this is great, a DCS basic editor, as bwang said, no more problems with more than 7 lines
qazz42 wrote:
Haha, now this is great, a DCS basic editor, as bwang said, no more problems with more than 7 lines
Indeed, and it fits wider lines than the TI-OS editor. Thanks qazz. Smile
Will I have to have lowercase enabled to use it?
qazz42 wrote:
Will I have to have lowercase enabled to use it?
You type out the programs themselves in Document DE 7 or any other text editor that saves AP-formatted text documents oncalc, which means you'll be using the Doors CS text input routines that keep track of their own alpha flags separately from the TI-OS. Long story short, no, you do not.
Ah ok, I think I shall have alot of fun with this >Smile





Hehe I was reading it like this

"Balh blah blah blah blah no"
qazz42 wrote:
Ah ok, I think I shall have alot of fun with this >Smile

Hehe I was reading it like this
"Balh blah blah blah blah no"
Hehe, fair enough. I do have vague plans to eventually integrate Tokenizer with an editor and perhaps a detokenizer (which would be much much easier than the tokenizer) which would allow a more complete on-calc editing experience.
I'm curious- what other algorithms could be used that would be faster? My 68k C program uses the same one (or at least something extremely similar).
TC01 wrote:
I'm curious- what other algorithms could be used that would be faster? My 68k C program uses the same one (or at least something extremely similar).
As elfprince13 mentioned in my other topic on the subject, some kind of tree-based algorithm would be the best, where each branch in the tree represents a longer and longer substring, and full strings terminate at a leaf with some flag value (that it's a leaf) as well as the one- or two-byte value for that string's token. It would be extremely fast compared to what I'm doing, but the memory overhead would be probably close to 8-10KB.
Woah, holy tokenizer crash batman!


Ok, it seems that on my nspire TI-84+SE keypad, tokenizeer crashes,

The OS of it is 2.54 MP




Note, this only happens on my nspire, not my TI-84+SE
What specifically did you do to make it crash? Like maybe list steps.
1. Make a DocDE7 file (with ti-basic in it)

2. save it

3. open tokenizer

4. press tokenize

5. Click the file from number 1

6. Ram Reset
qazz42 wrote:
1. Make a DocDE7 file (with ti-basic in it)
2. save it
3. open tokenizer
4. press tokenize
5. Click the file from number 1
6. Ram Reset
And this happens only on the Nspire; with the exact same steps, your 84+SE doesn't crash? I know almost precisely what is causing this, then, and it's something I had hoped wouldn't show up on the Nspire.
I dont know, let me try it on my nspire emu! Ill give you the error


also, it is fine with the regular calc, the ARM really emulates the TI-84 like s**t
qazz42 wrote:
I dont know, let me try it on my nspire emu! Ill give you the error


also, it is fine with the regular calc, the ARM really emulates the TI-84 like s**t
That would be great, much appreciated. I think it's because I'm abusing the af' register and due to having too many things on my mind at once, omitted the necessary step of disabling interrupts around the section where I'm abusing af'.
ok, after clicking the file I get the following



Code:
Boot Loader Stage 2 (1.4.1571)
Build: 2008/2/19, 23:19:10
Copyright (c) 2006, 2007, 2008 Texas Instruments Incorporated
Using production keys

Clocks:  CPU = 90MHz   AHB = 45MHz   APB = 22MHz
Checking for NAND: NAND Flash ID: ST Micro NAND256R3A


Initializing graphics subsystem.
Boot option: Normal


Initializing filesystem.
Datalight Reliance v2.10.1150
Copyright (c) 2003-2006 Datalight, Inc.
Datalight FlashFX Pro v3.00 Build 1358
Nucleus Edition for ARM9
Copyright (c) 1993-2006 Datalight, Inc.
Patents: US#5860082, US#6260156.
Filesystem ready.
Purging temporary files...

Loading Operating System...

100%

BOOT2: loading complete (705 ticks), launching image.
Wrote to translated code at a400989c. Deleting translations.


Beginning system initialization.
Preparing file system. This takes a while...
   POSIX layer initialized.
   POSIX devices initialized.
Datalight Reliance v2.10.1150
Copyright (c) 2003-2006 Datalight, Inc.
Datalight FlashFX Pro v3.00 Build 1358
Nucleus Edition for ARM9
Copyright (c) 1993-2006 Datalight, Inc.
Patents: US#5860082, US#6260156.

   POSIX file system initialized.
File system ready.
* P3 mode battery door detection
System build date: Mar 24 2010, 19:05:02
Available memory: 17331720 bytes
Purging temporary files...
Launching system...



EDIT: Ohhh, this happens even in a stable OS, 1.6 O_o something is screwey with tokenizer!
That's weird, I don't see anything interesting there except for this:
Code:
* P3 mode battery door detection
Mind if I built a quick test build to throw at you?
Hmmm? Ok
qazz42 wrote:
Hmmm? Ok
I just emailed you a version that wraps the TokenizeSub routine in the same ld a,i \ push af \ di, then pop af \ ret po \ ei \ ret stuff that I use to conditionally disable and re-enable interrupts around iFastCopy in Doors CS.
ok, ill test it
  
Register to Join the Conversation
Have your own thoughts to add to this or any other topic? Want to ask a question, offer a suggestion, share your own programs and projects, upload a file to the file archives, get help with calculator and computer programming, or simply chat with like-minded coders and tech and calculator enthusiasts via the site-wide AJAX SAX widget? Registration for a free Cemetech account only takes a minute.

» Go to Registration page
Page 1 of 2
» All times are UTC - 5 Hours
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

 

Advertisement