lirtosiast wrote:
M. I. Wright, that's a rounding error in the 10^(log()) calculation (it's off by, I think, 10^-12), as OldMathTeacher said earlier. Using Ans10^(-int(log(Ans instead eliminates the error entirely.

You're right, I had foreseen that this sort of problem may occur. It's not really the TI-OS as such, but the calculation itself which appears to give 0.2379 [displayed result] but is actually a tiny bit out. The calculator works to 14 s.f. and displays to 10 s.f., so in most contexts those tiny errors are unimportant and undisplayed, whereas here they are all-important, influencing how many digits there are.
Your workaround is nice; manipulating the figure only by an integer power of 10 should, as you say, keep exactly to the decimal digits you want it to.
  
Register to Join the Conversation
Have your own thoughts to add to this or any other topic? Want to ask a question, offer a suggestion, share your own programs and projects, upload a file to the file archives, get help with calculator and computer programming, or simply chat with like-minded coders and tech and calculator enthusiasts via the site-wide AJAX SAX widget? Registration for a free Cemetech account only takes a minute.

» Go to Registration page
Page 2 of 2
» All times are UTC - 5 Hours
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

 

Advertisement