I have a TI89 (non Titanium) with the original TI Graph Link (Gray cable) connected to a 25-9 pin serial adapter, with a Serial to USB adapter. I'm running Windows 7 64-bit as well. TILP II seems to recognize the TI89 and can send the ROMDUMP file over to the calculator, but the ROMDUMP file doesn't transfer anything and causes it to timeout.

I tried it first with just TILP, and when I got no joy, I ran TI Connect which detected the calculator just fine.

Any ideas?

Most users are equipped with BlackLink / "$4 cable", or ParallelLink / "$5 cable", in which case serial to USB adapters are recipes for disaster (because hardly any such adapters implements direct access to the wires, which is required for TI's protocol).
However, the GrayLink is more adapter-friendly. And if TILP II (TILP I is thoroughly outdated) recognized the calculator, and even sent the ROMDUMP file to the calculator, that means your setup is alright (unlike that of the vast majority of users trying to use serial to USB adapters).

Is the failure to perform a ROM dump consistent ? Given that communication with the calculators, I'd expect the ROM dumping procedure to eventually work (maybe after increasing the timeout ?).
Erase the ROMDUMP file on the calculator side if you want.

Yes, the failure is very consistent. It shows 0/2048 kB on my TI-89 screen, then it times out. I tried increasing the timeout to 50 - but never increased the time out more than that amount.

Hmm.
ROM dumping switches to its own higher-level protocol (rather than using TI's, which does not contain such commands in the first place). However, I'm not aware this is supposed to trip the GrayLink, as we're not using a different MAC (unlike, say, CALCnet, which cannot go through e.g. a SilverLink for that reason).

Are there any settings for the Serial to USB adapter that I should try? I turned down the baud rate and UART buffers to see if it'd do anything, but nothing worked.