Preparing for a move to a new server, I needed to offload about somewhat over a gigabyte of newsfeeds that my website collects, and that I had been saving on the server. I tarred them and zipped them into about a dozen smaller files of about 150Mb each. All seemed well. I downloaded them onto my Windows pc. The website was moved (http://schema-root.org). My plan was to move them back to the new server, strip them from their rss formats and load the news items into a database. However, in my newbieness I managed to transfer the gzipped files in ascii mode (both directions!). So they won't unzip now, either on my pc, or on the server.
Using:
> gunzip < d200512.tar.gz | tar xvf -
I was able to extract a few percent of the files from the first archive I tried, maybe a hundred of five thousand or so.
My question is: Would it be possible to get rid of the linefeed-carriage returns that were inserted into the zip file by being ftp'ed in ascii mode, back to what they were before I screwed them up? In my innocence, I am imagining that every existing linefeed byte in the original zip file (bytes that happened to be linefeeds) had a carrage return byte added after it during the ftp transfer in ascii mode. And so I am wondering whether there might be some utility somewhere that would strip them back out, and if there were such a utility, whether it would be likely to produce a zip file that could be unzipped.
You do have programs that convert files from MS-DOS/Windows format to Unix. Check dos2unix on Solaris and dos2ux on HP. However, those are meant to work on ascii files transferred from Dos/Win env to Unix - I dont think that they will be of any help to you with zipped files.
You can of course try using the utilities - best of luck!
Thank you blowtorch for the suggestion. I tried dos2unix on the .gz files, and it did produce smaller files. But they were still unacceptable to gunzip. I guess it would only take one "legitimate" occurance of the byte pair 0D0A in the .gz file to make the approach I am taking not work, because dos2unix would replace it with 0A (I think). I would want that to happen most of the time, but not if the original 0D0A were in the original good .gz file.
I appreciate your help, though. You provided information that I had already wanted to try, but I didn't know the name dos2unix.
Since I can extract a few of my original files from the corrupted .gz files, I will keep them around. At this point I think I need to learn more about how .gz files are organized, and about how gunzip works.
(My files contain several million newsfeed blurbs related to over 8,000 topics. They represent a form of current events history, so I really want them back. But, at the same time, there is no real rush. )
[This forum posting is two years late, but I'm adding it because this is a frequently asked question and somebody out there may search for it again and find some hope.]
It is very difficult to fix gzip files corrupted by FTP ASCII transfer. The problem is that (on average) 1/256 of the bytes have 3/8 of their bits flipped, and there is no way to distinguish whether one of those CR/LF bytes was supposed to be the way it is, or got that way by the ASCII transfer - so it can't be inverted in any simple way.
So-called zip recovery programs only fix CRC/checksum errors (to avoid error messages), they don't actually fix the data. That's only useful if the file is truncated or has a bad block late in the compressed data. It doesn't work for which about 1/256 of their bytes are corrupted, throughout the file.
So for 99% of the cases, give up, find the original if you can, and re-transfer in binary mode.
Despite what I said above, there is a computationally expensive way to search for the necessary repair. However, it requires custom coding of a search heuristic, and lots of computation. Therefore it's not a turn-key process. It's only feasible if you have no other backup and the data is so critical that you're willing to invest in some custom coding. I documented some details on what it took to recover a pile of 20MB gzip files containing 250MB million-line web server logs, at my web site.