A few days ago I watched a video about benchmarking every iOS. This gave me an idea, benchmark all CE OSes! I came up with a few small programs to benchmark, I used this program to time everything.

Here are my results. Green=fastest time, red= slowest time.

ABENCH (runs every benchmark and stores the results into a list)

Code:

SetUpEditor L₆
9→dim(L₆
prgmT01RAND
Ans→L₆(1
prgmT02GRAPH
Ans→L₆(2
prgmT03GRAPH
Ans→L₆(3
prgmT04SUM
Ans→L₆(4
prgmT05LIST
Ans→L₆(5
prgmT06LIST
Ans→L₆(6
prgmT07LIST
Ans→L₆(7
L₆


T01RAND (random integer test)

Code:
0→B
For(A,1,5
Asm(prgmTIMER
rand(100
Asm(prgmTIMER
Ans+B→B
End
B/5

T02GRAPH (Horizontal line test)

Code:
0→W
For(A,1,5
Asm(prgmTIMER
ClrDraw
For(B,Ymax,Ymin,­Y
Horizontal B
End
Asm(prgmTIMER
Ans+W→W
End
W/5

T03GRAPH (vertical line test)

Code:
0→W
For(A,1,5
Asm(prgmTIMER
ClrDraw
For(B,Xmax,Xmin,­X
Vertical B
End
Asm(prgmTIMER
Ans+W→W
End
W/5

T04SUM (summation test)

Code:
0→W
For(A,1,5
Asm(prgmTIMER
Σ(X²,X,0,1000
Asm(prgmTIMER
Ans+W→W
End
W/5

T05LIST (creating list test)

Code:
0→W
For(A,1,5
DelVar L₁
Asm(prgmTIMER
999→dim(L₁
Asm(prgmTIMER
Ans+W→W
End
W/5

T06LIST (sorting list of only 0s)

Code:
0→W
For(A,1,5
DelVar L₁
100→dim(L₁
Asm(prgmTIMER
SortA(L₁
Asm(prgmTIMER
Ans+W→W
End
W/5

T07LIST (sorting list of only 7s)

Code:
0→W
For(A,1,5
DelVar L₁
100→dim(L₁
Fill(7,L₁
Asm(prgmTIMER
SortA(L₁
Asm(prgmTIMER
Ans+W→W
End
W/5

RAYTRSPR ( this is a 3D modeling test using raycasting) (I didn't make this program, this was made by TheScienceElf, you can view his video here.)

Code:
Asm(prgmTIMER
ClrDraw:1→G
{12,24,23,22,21,20}→L₁

For(Y,0,162,2G)
(81-Y)/164→B
For(X,0,262,2G)
(X-131)/164→A

A²+B²+1→E:­2B-12→F
34.75→H:F²-4EH→D

­1/B→T

AT→U:T→V
remainder(abs(int(U)+int(V)),2)→C

12→L:12→M
12-(T>0)(C+1)→C

If D≥0:Then
(­F-√(D))/(2E)→T
AT→I:BT-1→J:6-T→K
J/1.5→M

If M>0:Then
int(10M)→M
L₁(1+int(M/2))→L
L₁(1+int(M/2+.5))→M
Else:12→M
End

{I,J,K}/1.5→L₃
{A,B,1}→L₂
2sum(L₃L₂)→S

SL₃-L₂→L₄
L₄(1)→P:L₄(2)→Q:L₄(3)→R

­(J+2)/Q→T

If T>0:Then
PT+I→U:RT+6-K→V
11-remainder(abs(int(U)+int(V)),2)→C
Else:12→C:End

Else:If T>0:Then
1→E:­4→F
U²+V²-12V+37.75→H
F²-4EH→D

12(D≥0)+C(D<0)→L
L→M
End:End

Pxl-On(Y,X,C)
Pxl-On(Y+1,X,L)
Pxl-On(Y,X+1,M)
Pxl-On(Y+1,X+1,C)
End:End
Asm(prgmTIMER
Ans


If you spot any errors in my testing, or want me to benchmark some other aspect of the CE, please post below!
I made a raycaster once. You could use mine (look up ti-basic raycaster in the Cemetech search thingie)
Thus begin the conspiracies…
It is all barely a difference to me, I don't notice.
what on earth happened to the list sorting on 5.0.0?
first, did you re-test that one to confirm that the value was correct?
If so, why is there such a discrepancy? As far as I know, there was no mention of the sorting algorithm being improved. Did you try SortD() to see if similar results appear? What about a list that has different values rather than 0s?
I think the fact that the sortA() command seems 7x faster is worth looking into Confused
mr womp womp wrote:
what on earth happened to the list sorting on 5.0.0?
first, did you re-test that one to confirm that the value was correct?
If so, why is there such a discrepancy? As far as I know, there was no mention of the sorting algorithm being improved. Did you try SortD() to see if similar results appear? What about a list that has different values rather than 0s?
I think the fact that the sortA() command seems 7x faster is worth looking into Confused

This was a startling result, but I ran the thing 5 times and got the same result.
I have no idea why it was so slow.
I did not try SortD( I'll try that later since my calculator recently recovered from 3 OS fails.
I'll also try values other than 0.

Good points! Smile
Hey, coming at you with a 2 year necro-follow-up!

Since there's been several OS's released since my last benchmark, I've decided to re-do all of them. Hopefully the table is a little bit easier to understand than last time since Excel color coded all of them. I decided to check some of womp's questions about sortA( handling non-zero numbers, that became T07LIST. Unfortunately I forgot to make a test for sortD( Sad

Anyways, enjoy the graph: green is the best time and red is the worst time!

Each test was performed from the RAM, on the same calculator, in the same order, and after a full mem clear. The calculator was plugged in and fully charged the entire time. Each test, with the exception of prgmRAYTRSPR, was completed 5 times then the times were averaged into the table above. I only ran prgmRAYTRSPR once because it takes over 30 minutes per run and ain't nobody got time to do that 5 times over! I have yet to retest OS 5.1.0, 5.1.5, and 5.2.1 for new prgmRAYTRSPR times.

Overall it seems if you want the best BASIC performance, then OS 5.0.0 or OS 5.3.1 is the way to go. However, if you just care about the speed of specific functions, then you can't go wrong with OS 5.1.x or 5.2.1.

I'd like to dig into the numbers a bit more though.
    Fortunately, my numbers for my new test line up pretty well with the 2 year old test where OS 5.0.0 is the champ for random numbers, graphing horizontal and vertical lines, and overall BASIC performance. Then OS 5.3.0 (relatively) crushed the times for the summation and list tests but fell behind a bit in BASIC performance.

    There was something different though, comparing the numbers to my old test, my CE somehow got faster! Perhaps reverse-wear is happening to it. Razz

    As you can see by the graph, the only reason the newer OSes had good total scores was because the raytracing program was such a huge factor in this test. However, if you look at each function separately, it's clear they are generally getting slower the newer the OS.

    TI changed something for the better between OS 5.0.0 and OS 5.1.0 that made list sorting almost 9x faster on newer OSes! Even for numbers that weren't 0s.

    TI also changed something for the worse between OS 5.2.1 and OS 5.3.0 because every OS newer than 5.2.1 is typically 3 seconds slower at displaying horizontal and vertical lines.


So there you go, personally I think OS 5.3.0 has the best performance to match its features (Asm84CEprgm token, yay!). If you'd like to try out the programs yourself, I'll be updating the first post with all the programs I ran.
The reason I asked about sorting numbers that weren't all 0s is because some algorithms will take more or less time depending on how sorted the initial list is. I know the calculator used a version of selection sort, but that might have changed. Selection sort has quadratic time complexity (O(n²)), while Quicksort takes at best quasilinear time (O(n log(n))) and at worst quadratic time (ie. the pivot was chosen to be a unique minimum or maximum)

I would assume they would stay away from some fancy stuff like Timsort, but maybe because lists will often contain few elements, they might have gone with insertion or something else if memory usage is a concern, since insertion sort has the same space complexity as selection sort. For best-case scenarios, selection sort is really one of the worst, so it would make sense that if they changed the algorithm, we would expect a drastic improvement for best-case scenarios like the one you tested.

It wasn't so much that the test case was all 0s, but rather that the list was already sorted, which isn't a good way to measure a sorting algorithm. It is hard to figure out what would be a good test case though, since different algorithms and implementations benefit from different inputs. I would suggest trying a sorted list (like 0-99) and a random list of the same length, but using the same random list for all tests. Generally, a sorted list (sorted in the right way of course) will be the best case for most common sorting algorithms, so its a pretty good metric.
OK, a heck of a lot has changed since last I did this test, so I decided to go back and do all these tests over again on my Pre-A TI-84 Plus CE. I made a few changes so the graph won't line up completely with last test's results.

Specifically I changed:
  • I added a few more OSes to test. 5.2.0, 5.2.2, 5.4.1, 5.5.5, and 5.6.0
  • T06LIST to SortA( a randomly generated list with the seed 1->rand.
  • T07LIST to SortD( a randomly generated list with the seed 1->rand.
  • I added T08ASM which tests how long it took for the timer program to launch. This measures the speed overhead it took for the program to start and return.
  • I removed the ray trace benchmark for now because it just takes too long to test.
  • I added an extra row that shows how much faster a certain OS was over the baseline (the slowest OS).

You can download the whole benchmark group here.

Enough talk, let's see the numbers!
Green is the best time and Red is the worst time

Each test was performed from the RAM, on the same calculator, in the same order, and after a full mem clear. The calculator was plugged in and fully charged the entire time. Each test was completed 5 times, then the times were averaged into the table above. arTIfiCE and AsmHOOK were only installed on OS versions that required them; all jail breaking utilities were stored in the Archive.

OK, we see a lot of similarities to last years results. OS 5.0.0 continues to be the slowest at sorting but it's generally pretty darn fast at everything else.

As you get to the newer OSes which added removed features, it's clear that the calculator is getting slower and slower at doing every function. Only by about 1% from OS 5.3.0, but if we see another drop in performance like OS 5.3.0 brought for the graphing tests, there could be significant speed issues for those with a pre-M TI-84 Plus CE.

One result I was surprised to see was that OS 5.5+ was negligibly slower at launching ASM programs than OSes that supported ASM. I figured TI would have done more to block ASM programs from launching.

So those are the results for this year! I'm probably going to stop re-testing every single OS every time a new OS gets released because this took 4 hours to organize, test, collect data, and post. I'll just be testing the newest OS and tacking it onto the end of my Excel sheet. I hope to improve my Excel sheet next time because I know my current method isn't super great at providing an accurate depiction of the OSes speed, it's just a rough measurement for now. Let me know what you thought of the results! Anything you guys want me to test next time? Are there any results that surprised you?
I received some criticism on my last table but thanks to the help of Kerm and LogicalJoe, I've created a new table that's actually useful and doesn't exaggerate the color of a negligible difference! Not only that, but I'm sharing my results so you can mess with the data however you'd like! Go take a look at it here: https://1drv.ms/x/s!Al_ZukYl7XyyyzQRzhPZcp0ZZCD4?e=6fXiGA!

This table compares the speed of each OS to OS 5.0.0 which I used as the baseline:


By the request of Adriweb, I also made a table that compares the speed of a new OS over the previous OS:


My previous conclusion needs some adjusting. Although new OS versions are still generally getting slower, most of the time the difference per generation is negligible and you'll never notice it. The only exception to this is OS 5.1.0 significantly speeding up sorting lists by about 4 times! Along with OS 5.3.0 slowing down graphing by almost 14%.

So that's the new table format! Thanks again to those who helped me figure out Excel and requested data!
  
Register to Join the Conversation
Have your own thoughts to add to this or any other topic? Want to ask a question, offer a suggestion, share your own programs and projects, upload a file to the file archives, get help with calculator and computer programming, or simply chat with like-minded coders and tech and calculator enthusiasts via the site-wide AJAX SAX widget? Registration for a free Cemetech account only takes a minute.

» Go to Registration page
Page 1 of 1
» All times are UTC - 5 Hours
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

 

Advertisement