Ah, well now for some reason the serial uploads are starting, but losing sync shortly thereafter. I guess that's progress. 
Really odd though. When I upload a sketch via ISP that uses the serial monitor, everything looks fine. So I think I can rule out wiring and clocking issues.