"Tokens" BASIC IDE Project (Pre-2013)

tifreak8x · Guru Joined: 16 Mar 2005 Posts: 9064 Location: Sol System

Looking awesome merth! Very Happy

Updated my computer and TIFW archives Smile

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

Alright, there was an issue with backslashes, still, and I have made some more improvements with how they're handled. Not sure how "\r" will get handled if it's actually in a token.xml file, but for now that's not a huge deal since the calcs don't use it. Here's a screen shot:

The backslashes seem to be behaving.
Download at the normal spot: http://merthsoft.com/Tokens.zip
_________________
Shaun

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

I know this is soon for a double post, but I thought it important to let everyone know of a bug I found: If you press ctrl+s after having opened an .8xp or .8xv file, it will overwrite the file with a text-version of the program. On top of that, saving makes it have double new lines. I'm working on making this work right, but I'm at the airport right now and my battery is about to die. So, just be careful saving things Smile

EDIT:
Also, a user was seeing issues with it loading in extra tokens at the end. This has now been fixed.
_________________
Shaun

tifreak8x · Guru Joined: 16 Mar 2005 Posts: 9064 Location: Sol System

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

Hmm, I thought I did. The other user hasn't reported if it's working or not, so I guess I'm not positive. I'll make sure to upload one when I get home. Sorry!
_________________
Shaun

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

Alright, I've updated a new version that should have this and the saving issue fixed. If it's still not working, please let me know.
_________________
Shaun

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

Thanks to some help from Deep Though, I've fixed (once again) how saving is handled. Note that any programs you had previously compiled with tokens may now have those extra bytes at the end, but cleaning and rebuilding should fix it. Download at the normal place:
http://merthsoft.com/Tokens.zip
_________________
Shaun

tifreak8x · Guru Joined: 16 Mar 2005 Posts: 9064 Location: Sol System

~shaunnag

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

I had some bugs that I fixed:

Deep Thought · Posted: 29 Jul 2012 03:49:55 pm Post subject:

Heh, just a few days ago I was stumped wishing one of the many TI-BASIC editors out there would have a count-tokens-in-selection feature. I really like the updates here—the features TokenIDE has aren't really standard code editor features, but they're perfect for TI-BASIC.

As for the last bug you mentioned in your post, I'm not sure if this is related (it might even be fixed by now and I hadn't noticed), but I noticed a few months ago that if you import a program containing three consecutive periods (\.\.\.), TokenIDE converted them into the ellipsis character (\xBB\xDB) because it failed to add the backslashes between the dots that would separate them into three characters instead of one. (It caused some problems with formatting of homescreen graphics, but again I haven't checked since then and I don't know if it's been fixed or not.)
_________________

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

It's not related, and it's not a bug, per se, but it is something I've been meaning to fix. As it stands now, there's really no way for it to know if you mean "\.\.\." or "...". I plan on adding the functionality to determine it--it shouldn't be terribly difficult. Thanks for reminding me that it needs to be added Smile

_________________
Shaun

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

Due to some rather incessant nagging on tifreak's part (thank you, Daniel Smile

), I have added the functionality to count blocks, such as While/End etc. Here's an example:

You can specify any number of groups. It's under Tools > Block Counts.
Limitations:
- It currently doesn't handle "Else" gracefully. It just treats it as another token. Ideally it would unindent the "else" line and indent the following line:

- The indentation is automatic, so if you're doing this with Axe and have included your own indentation, it'll extra indent it.
- The AxeTokens.xml file doesn't have any groups defined, so check out Tokens.xml and see how it's doing it if you want to add them to it. I don't know if Axe does it the same way or what.
- Eventually this will be integrated into the main editor so that you don't have to go to a tool to use it. For now, though, it's separate.

Technical details:
There are two new tags that you'll use in the XML:
- indentGroup - String - Which group this token belongs in.
- indentGroupTerminator - Boolean - If it's a terminator.

For example, here's what the Tokens.xml looks like:

tifreak8x · Guru Joined: 16 Mar 2005 Posts: 9064 Location: Sol System

Looks fantastic. I hope it doesn't try to indent when code is all on a single line, because then that might cause some issues (Like ZITU)

Can't wait to play with it!
_________________
Projects: AOD Series: 70% | FFME: 80% | Pokemon: 18% | RPGSK: 60% | Star Trek: 70% | Star Trek 83+: 40% | TI-City: 5%

Look here for Assembly Resources.

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

It won't try to indent in-line, but if you're unbalanced and start a new line it may indent, if that makes sense. Here's what it would look like:

Which is a good way to see if it's not balanced.

Also, this doesn't ignore strings, which is, I'm pretty sure, how you want it to behave. What I mean is that it will count tokens within strings, so the actual string "While 1:End" will be balanced, but if it's part of text you're displaying ("While the world came crumbling down" or whatever), it will be unbalanced. The only good way around that for you, since you do generated code, would be to escape the While ("\While").
_________________
Shaun

KermMartian · Posted: 31 Jul 2012 11:25:59 pm Post subject:

But that eliminates the five-byte optimization of actually using the token, a trick I use in my programs all over the place! Sad

There's no way to do easy syntactic detection of which ones are in strings and which are not?
_________________

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

There is (and you can, of course, remove the escaping once you're sure you're balanced), but he generates code using strings, so that would defeat the whole point.
_________________
Shaun

KermMartian · Posted: 31 Jul 2012 11:29:16 pm Post subject:

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

Generally I wait to see people's workflows before introducing complexity and configurability. If there's enough of a demand for it, I'll add it, but I imagine people will generally just escape it where need-be.
_________________
Shaun

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

I've made another update. This has two things that are important to tifreak:
1. You can now toggle the indentation in the Block Counts tool. If you toggle it with ZITU note that it'll need a second to update (maybe I should add the Live Update checkbox, too)
2. The Live Update checkbox was broken in the last release. It's fixed now.

As always:
http://merthsoft.com/Tokens.zip
_________________
Shaun

merthsoft · File Archiver Joined: 09 May 2010 Posts: 2735

Just for fun, here's a little call graph of the program. The size is "Time in Method", so it shows me what's taking to most amount of time (which is drawing the tokens):

This program is neat because it does it all in real time and shows you he calls and stuff. I want to put it through a full profiler to hopefully optimize it some more.
_________________
Shaun