Thanks a lot for that post.
I have a small problem with the "thisout" variables since I don't understand what they're their for.
I'm trying to make a PHP to Python conversion and it's going pretty easy, though.
However, something made from scratch could work better than this, but I need to try this out too
ScoutDavid wrote:
I have a small problem with the "thisout" variables since I don't understand what they're their for.
The shortest version is that the point is to deal with the fact that there are both two-byte and one-byte tokens. The pair of "thisout" variables store SourceCoder's current best guess for what token the maximal substring at the beginning of the remainder of the input text will be encoded to.
"The shortest version is that the point is to deal with the fact that there are both two-byte and one-byte tokens. The pair of "thisout" variables store SourceCoder's current best guess for what token the maximal substring at the beginning of the remainder of the input text will be encoded to." :O
It's a guess, what if it's wrong, or it is not that much of a guess?
By guess I mean that it might first find that the best guess for "Xmax*3" is "X", then as it continues to iterate through all the available tokens, it notices the Xmax token, which becomes its new best guess.
:O
Like in:
Disp "Disp"
It may think that the Disp inside the tokens is one token only, but then it is not?
ScoutDavid wrote:
:O
Like in:
Disp "Disp"
It may think that the Disp inside the tokens is one token only, but then it is not?
No, it always picks the longest (maximal) substring that matches. The second Disp will never be a one-byte token, rather four tokens, because it's missing the space that comes as the fifth character of the Disp token.
Yes, of course:
Code:
"Disp "
differs from
Code:
"Disp"
SourceCoder is great, how long did it take you to do a first stable version?
ScoutDavid wrote:
SourceCoder is great, how long did it take you to do a first stable version?
Thanks, I appreciate it. If you delve into the Cemetech News Archive, you'll see that May 2005 was the first release of SourceCoder 1.0 in public beta form, capable only of turning .8xp files into plaintext source code. It reached a more stable form in October, and by late November, I was starting to work on the re-tokenizer.
What is the variable 'tr' in your script?
Thanks
ScoutDavid wrote:
What is the variable 'tr' in your script?
Thanks
That's my giant array of token to value mappings.
KermMartian wrote:
ScoutDavid wrote:
What is the variable 'tr' in your script?
Thanks
That's my giant array of token to value mappings.
Hum... value mappings... no ideal of what you saying soz...
Tokens ("Disp ") to value ($DE) mappings...
KermMartian wrote:
Tokens ("Disp ") to value ($DE) mappings...
So, it's like a database, an array-database?
ScoutDavid wrote:
KermMartian wrote:
Tokens ("Disp ") to value ($DE) mappings...
So, it's like a database, an array-database?
KermMartian wrote:
ScoutDavid wrote:
KermMartian wrote:
Tokens ("Disp ") to value ($DE) mappings...
So, it's like a database, an array-database?
Exactly, does it use MySQL?
Code:
def token2bin(inp):
i = 0
maxtokenlenght = 16
while (i<len(inp)):
out = ''
out1 = -1
out2 = -1
for a in range (maxtokenlenght, 0, -1):
k = inp[i:i+a]
try:
if tr[k] == [] or out == '':
pass
else:
out = k
j = tr[k]
if(j<999):
out1 = j
out2 = -1
else:
out1 = int (j / 1000)
out2 = j - 1000 * out1
except:
pass
if (out1 == -1):
pass
else:
i += len(out)
if (out2 == -1):
o *= str(out1)
else:
o *= str(out1) * str(out2)
return o
print token2bin("DISP")
What I have so far from PHP to Python conversion 
ScoutDavid wrote:
KermMartian wrote:
ScoutDavid wrote:
KermMartian wrote:
Tokens ("Disp ") to value ($DE) mappings...
So, it's like a database, an array-database?
Exactly, does it use MySQL?
KermMartian wrote:
Errr, no, it's an array. Are you familiar with the use of arrays?
Or rather, a dict. PHP "arrays" are what most languages call dictionaries or maps.
Kllrnohj wrote:
KermMartian wrote:
Errr, no, it's an array. Are you familiar with the use of arrays?
Or rather, a dict. PHP "arrays" are what most languages call dictionaries or maps.
Yes, indeed I am aware of arrays, duh!
Okay, you use a dictionary, I'll probably use if conditionals or an array
I found a new way to search tokens, I mean, two ways, gotta test both!
ScoutDavid wrote:
Yes, indeed I am aware of arrays, duh!
No duh necessary, everyone learns at different speeds, so if you didn't know about them it wouldn't necessarily indicate a deficiency in your programming education. 
Quote:
Okay, you use a dictionary, I'll probably use if conditionals or an array
I found a new way to search tokens, I mean, two ways, gotta test both!
Register to Join the Conversation
Have your own thoughts to add to this or any other topic? Want to ask a question, offer a suggestion, share your own programs and projects, upload a file to the file archives, get help with calculator and computer programming, or simply chat with like-minded coders and tech and calculator enthusiasts via the site-wide AJAX SAX widget? Registration for a free Cemetech account only takes a minute.
»
Go to Registration page
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
