UPDATE

File download now works and there's a nice UI that displays download statuses of files


The status states are:
waiting: the file is still being installed, or not started yet
done: the file is installed, CRC has been matched, and library entry updated
error: the file download completed, but the CRC didn't match, so the downloaded file was discarded
skipped: the current CRC matched the server's CRC so the file was skipped.

I'll probably turn the file not found into a status we list as well, rather than a pop up error.
This is pretty awesome!

I'm curious why you error out after the download. Can you save some bandwidth and time and match the CRC before downloading? I have no idea how fast the calculator is so maybe it's moot?

On the topic, I feel like error is pretty broad. I believe a MISMATCH error would be better, it's the same number of characters as WAITING and then ERROR can be reserved for a proper error.
Alex wrote:
Can you save some bandwidth and time and match the CRC before downloading? I have no idea how fast the calculator is so maybe it's moot?

On the topic, I feel like error is pretty broad. I believe a MISMATCH error would be better, it's the same number of characters as WAITING and then ERROR can be reserved for a proper error.


Well let me clarify what's actually happening. There are two CRC matches that occur (also this morning I made a few edits that removed one leg of the download process).

1) Calculator tells server what service you would like to download.
2) Server opens the service.conf file and figures out what files the service requires. That list is sent to the calculator.
3) The calculator retrieves the library entry for each file, filling in a CRC of 0 if the file doesn't exist or is non in the library. The library entry contains a name, type, an a CRC.
4) The calculator begins looping, for each file, a FILE_DL_START packet containing the name, type, and CRC of the file on the calculator.
5) The server fetches and opens in 'rb' mode the appropriate .bin file, and calculates a CRC for it. If the CRC matches the CRC supplied by the calculator, the server responds with FILE_DL_SKIP. It's a 1-byte packet.
6) Else, the server responds with a series of FILE_DL_DATA... max-1024 byte packets containing the file data. then terminating with a FILE_DL_END, which contains the name, type, and CRC.
7) If the file was not skipped, the calculator matches the CRC of the new file with the CRC sent by the server. If the two match... this means that the download occurred without data loss. If they do not, it means that some packet loss occurred and the new file is invalid.
Maybe I'm misinterpreting what you wrote here then where it sounds like Step 5 happens AFTER the file is downloaded. So it sounds like that's not the case then?

Quote:
error: the file download completed, but the CRC didn't match, so the downloaded file was discarded
There are two CRC matches.

[Client] ==> [Server]
(CRC for current file compared with server's CRC for same file; used to verify version) *

[Server] ==> [Client]
(CRC for server copy compared with newly downloaded device copy;
Along with file size, used to verify errorless transmission) **

*, **
If someone is of the belief that a different mechanism for versioning/error checking would be more effective/efficient, please recommend it!

The use of CRC for these purposes is not to be confused with the file integrity mechanisms I plan to implement server-side. I plan to use a SHA-1 hashfile with very strict permissions to verify the integrity of the hosted download. That is the intended security mechanism. The CRC is not. Smile
CRC's are basically pointless here- they do nothing to actually pinpoint and fix errors. What you really want is an error-correcting code, like Reed-solomon, or even nothing at all because whatever connection you have from the server to the bridge almost certainly has error correction and USB is good enough unless you're sending insane amounts of data and also wiggling the connector while also drying your hair (and pointing the hairdryer at the USB cable) and also maliciously inserting bit errors.
Fair enough. So if I eliminate the CRC for error checking and perhaps just match number of bytes transmitted, what mechanism should I be using to check version numbers (the CE libraries I know have their own version numbers embedded, but I'm talking more about other programs people may submit that I would rather not demand users modify to provide metadata)?

[EDIT] How would one add a version number to a program? Is there a makefile option you can specify or something?


Also, just to share a development I made last night. Rather than reinventing the wheel and implementing my own hash check for user-submitted files, I merely opened /etc/rkhunter.conf, and added the prgm and deps directory to my USER_FILEPROPS_FILES_DIRS directives. This allows the system file-integrity checker to also track that directory, monitoring an SHA-1, as well as other types of metadata, for the files. I've also written a cron-job, designed to run before the system checks, that looks for a file in the logs directory of the VAPOR service with a list of all user-files that have been modified by service owners that day, and calls `rkhunter --propudp $file` on them before actually checking file properties for the system.
I see no issue with using the SHA for checking if a file is an update. Just not a CRC :p
MateoConLechuga wrote:
I see no issue with using the SHA for checking if a file is an update. Just not a CRC :p

Interestingly enough, rkhunter doesn't maintain an SHA-1 for monitoring file properties, it actually maintains an SHA-256, and can go as high as an SHA-512 since it has the binaries to do so, but it defaults to SHA-256.
For checking versions, I wound up, at least temporarily, using the 'file last modified' timestamp. After a bit of testing, it appears that regardless of the method of upload (via the php file uploader, or scp) the file modified date reflects the build date of the program or appvar, even after it is converted with convbin. So I think I'll use that for versioning, unless it happens to break in some use cases, in which case I'll switch to an SHA-1 for that.
I also no longer use the CRC to validate the file, I simply verify the bytes written matches the expected file sizes communicated by the server.

The server itself is coded in a way that packets have explicit lengths, and a packet of a certain type with an invalid length, as well as a packet of an undefined type, logs "invalid packet sent by {ip}" to the server logfile. This does not trigger a disconnect, but there is a fail2ban jail watching the logfile, set to 5 hits in 10 minutes.
Ok, I am in need of some assistance from whoever can provide it.
I plan to give VAPOR the ability to update and reload itself automatically upon launch. I slightly modified a version of the updater ASM code beckadamtheinventor contributed for project TI-Trek.

First, I'll explain what happens, then the code:
The file is updated properly. The file is relaunched properly, the screen temporarily goes white and then back to the main menu.
The program also works properly in the reloaded version... with all downloads and menu navigation working totally fine.
However, as soon as I [Clear] out of the program, it crashes, yielding 3 write to flash from unprivileged errors. Here's a screenshot:


And lastly here is the code itself that I am using:
https://raw.githubusercontent.com/acagliano/vapor-client/main/src/asm/launch_self.asm

If anyone who knows assembly better than me can identify possible causes for this, please lmk. I'm next to useless at asm.
Development seems to be coming along very nicely Razz . But I installing python to grab programs from a server sound like going to cemetech but with more steps. I think it would be best if you added a browser application down the future (to get rid of the extra step paradox).

Good Job! Razz
Alvajoy123 wrote:
[i]But installing python to grab programs from a server sound like going to cemetech but with more steps. I think it would be best if you added a browser application down the future (to get rid of the extra step paradox).

I tried to implement WebSerial + Web Sockets to connect to the server, but it failed. Would anyone have any other suggestions for that?

Also bear in mind that commandblockguy is likely going to be implementing an update to InternetCE that would eliminate the bridge and just use a standard internet device.
ACagliano wrote:
Alvajoy123 wrote:
[i]But installing python to grab programs from a server sound like going to cemetech but with more steps. I think it would be best if you added a browser application down the future (to get rid of the extra step paradox).

I tried to implement WebSerial + Web Sockets to connect to the server, but it failed. Would anyone have any other suggestions for that?

Also bear in mind that commandblockguy is likely going to be implementing an update to InternetCE that would eliminate the bridge and just use a standard internet device.

I think updating InternetCE will still lead to a (extra step paradox). As the calculator owner would still need to purchase a different external cord, leading back to (extra step paradox). It would be best to implement what is already in the box (the standard cord).

"Sorry for double posting"
Alvajoy123 wrote:
I think updating InternetCE will still lead to a (extra step paradox). As the calculator owner would still need to purchase a different external cord, leading back to (extra step paradox). It would be best to implement what is already in the box (the standard cord).
"Sorry for double posting"

That's incorrect as far as I know. The cable that comes with the device would work perfectly fine, as it's a data-capable USB cable. This may simply require the user to run a packaged command-line script to in a way bind the serial port to the network port, but it shouldn't require any extra dependencies... but command would be the authority on that, not me.
My favored project to implement would be Nanotube... which commandblockguy was writing that would let the calc use a Wifi adapter, but even that would require a few extra pieces of hardware.
Whether you are going to be downloading from cemetech, then launching up TiLp or TI-Connect, connecting with your cable, and transferring, or using VAPOR, you're still going to be connected to your computer with a standard usbA-mini data cable.
I will very soon be releasing a public RC of VAPOR's client, with online service detection, file hash retrieval and updating, automatic self-updating of VAPOR (literally you need only connect, and if there's a newer version, VAPOR will be updated and "relaunched" in place, and all you will see is a simple flicker on your screen.

Fair warning, the first time you run VAPOR it will appear to be frozen for a while... not to worry, it is merely taking an SHA-1 hash of itself. Hashes are used by VAPOR's online network for versioning... but once this initial hash is taken, you will not need to wait again for so long to run the program (unless you lose the metadata file stored on your calculator).

Also, for now, downloads are slow. This is because VAPOR calls hashlib_sha1update() with every file data frame it receives, and the code is relatively unoptimized, given that it was done in C (but not to worry, beck has offered to do a full assembly re-write, making it smaller and faster).

While not every planned feature is implemented yet in VAPOR, updates and feature additions can be pushed out via autoupdate without a problem.
Update

This is less an update and more of a "this project isn't dead" post. lol. The "release" is more or less ready but there are two small things holding back the release:

1) As some of you may be aware, beckadamtheinventor offered to re-write the HASHLIB code in pure assembly (I had written it in C and then modified it). As of right now, the checksums work, but the SHA1_final overruns the digest buffer, resulting in an incorrect hash and corrupted program memory. This bug will need fixing before VAPOR can be released.

2) There is an obscure bug within VAPOR itself where file transfers sort of just...stall. With no error condition, crash, or anything. The program appears to just stop downloading, pressing any key returns you to normal functionality, and usually re-attempting the download succeeds. I suspect that this may be a stray dropped frame somewhere due to backlog on the device hashing the incoming buffer and to this end I plan to re-work the file download, such that the server doesn't send the next file packet segment until it receives a FILE_DATA_NEXT packet from the device.
Update

This time it's actually an update. Imagine that.
Bug detailed in #2 above appears to be resolved.
#1 is still in progress, but for now disabled in VAPOR.

A release will be coming very soon (hopefully...once I get over my sneaking suspicion it's not as stable as I think it is and do a bit more testing), but there are a few currently toggle-able, but un-implemented, settings. Here is a teaser of the settings UI, showing some of the (known) features and some of the newer, planned ones.

UPDATE

I am now formally, informally releasing VAPOR v1.1 beta.
Currently implemented features as of right now:

<> Toggling values in the Settings tab and saving them
<> Viewing the Library of all files VAPOR has downloaded
<> Using the services tab to view all services (servers) VAPOR is hosting.
<> Using the services tab to "launch" a specific service, which as of right now, merely downloads its dependencies.
<> Automatically updating the main program, the toolchain libraries, and any 3rd party libraries on launch.
<> Automatically restart when VAPOR itself is updated (I will expand this to be when VAPOR or a library is updated).

Other notes about this release:

<> VAPOR uses HASHLIB but doesn't require it. It is compatible with V2 of HASHLIB which is not in Cemetech's archives yet. V2 of HASHLIB alters some function names and finalizes their order, but is not fully optimized yet. Download it from here: https://github.com/acagliano/hashlib-ce. If HASHLIB is not present, VAPOR disables certain options in the settings menu.

<> If HASHLIB could not be loaded, VAPOR will not verify file downloads. It will assume a file download completed without error and save the server's sha-1 into the library without checking against its own. The SHA-1 will be compared against the server's SHA-1 in the future to determine if the file needs to be updated.

<> I may yet support the ability for people to host "packages" of software that do not associate with any service, but that can still be downloaded, similar to a package manager. If people think this would be a good feature, let me know.

<> VAPOR no longer regards the libraries (toolchain or otherwise) as part of individual packages. If you do not check Toolchain Libs and/or 3rd Party Libs on the settings tab, VAPOR will not download them, and running programs that require them will error (if you don't already have them). That being said, telling VAPOR to download them is as simple as checking Toolchain Libs and/or 3rd Party Libs and then restarting VAPOR. Toolchain libs are considered anything that is currently distributed with the toolchain and anything currently in beta but expected to be added. 3rd party is anything that is not part of the toolchain--like HASHLIB and perhaps beck's expected large integer library.

VAPOR 1.1b can be downloaded at: https://vapor.cagstech.com/pkgs.php?t=soft
It's the 3rd file on the list.
Documentation can be found here: https://vapor.cagstech.com/downloads/docs.php . This includes a list of requirements.
VAPOR, just like TI-Trek, needs to be bridged to the server at present. The latest version of the titrek bridge by commandblockguy will do the job: http://titrek.us/components/server/downloads/bridge/1.2/trek-bridge.zip
Feel free to beta test and report back on any bugs.
Nice!
I will definitely try it!
UPDATE

Vapor is no longer primarily a game and game server hosting platform. It is being rebranded as a more single function tool, and split into two independent, but related, projects.

The VAPOR client will be scrapped and replaced with a library, called Vapor Package Manager (VPM). The library will provide commands such as:


Code:

vpm_install(<package_name>)
vpm_uninstall(<package_name>)
vpm_reloadprogram(void)

The `vpm_install` command will query the mirror server for the package name you have indicated. Upon locating it, the server will compile a list of that program's dependencies and stream them all to your calculator. If the mirror server has a package with a non-local URI, then the URI provided will be curl'ed or otherwise downloaded to provide the requested file. Should a requisite file not be found, the server will serve what it can find and then return an error. Should the package itself not be defined on the mirror server, the server will also return an error. Every file updated with this command will cause the library to update a database file with a SHA-256, size, and type for the file, used for versioning.

`vpm_uninstall` will perform the opposite of this... it will uninstall the program by name and type (not touching dependencies). Uninstall does not require a connection to the mirror server, install does.

`vpm_reloadprogram` can be called and it will reload the currently-running program by copying its own code to ram, jumping to that address, copying in the program's new data to pgrmstart. This will be more or less the same mechanism present in the VAPOR client currently.

Despite this re-brand, the Vapor network, now called Vapor Proxy Service (VPS) will remain up for people (myself included) to host calculator games and other utility-type servers, such as VAPOR or the BOS equivalent beck is planning to make (BPM - BOS package manager). I also at some point intend to give the Vapor proxy the ability to act as an SSL authentication service by request. Since it is currently not planned to implement RSADecrypt() in hashlib, and verifying servers' with RSA requires the decryption of the host's signature with the certificate authority's public key, a workaround I came up with is allowing a client to set up an AES connection to a service I host, and send that service the host's certificate and let the service respond with valid or invalid. I realize this is not a totally secure setup, and it is vulnerable to MITM between you and my validation server, but without RSADecrypt on the client, it's the best I can come up with at the time.
  
Register to Join the Conversation
Have your own thoughts to add to this or any other topic? Want to ask a question, offer a suggestion, share your own programs and projects, upload a file to the file archives, get help with calculator and computer programming, or simply chat with like-minded coders and tech and calculator enthusiasts via the site-wide AJAX SAX widget? Registration for a free Cemetech account only takes a minute.

» Go to Registration page
Page 2 of 3
» All times are UTC - 5 Hours
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

 

Advertisement