I've been trying to figure this out since last night, and I'm just stumped. The last time I did any shell scripting was 8 years ago on a Unix box, and it was never my strong suit. I'm on a Mac running Leopard now. Here's my dilemma - hopefully someone can point me in the right direction.
I'm trying to process product datafeeds - plain text, thousands of lines, pipe delimited. My shell script prepares a directory, then downloads via curl a zip file. It unzips the file. It greps out lines containing a keyword and redirects into a new file. I use a perl command to find/replace my ID number in the file. Everything I've described works perfectly - but the next step is the problem.
I need to add the field names to the top of the file so I can put the file into WebMerge and process it. The field names are on a single line in a separate text file called FirstLine.txt. I use this:
cat FirstLine.txt kids.txt > readyToProcess.txt
to combine the files. What I expect is that if FirstLine.txt contains foo and kids.txt contains bar, then readyToProcess.txt should contain foo(new line)bar. What I'm getting instead is that the file contains foobar.
Is there a situation where the cat command wouldn't put the contents of the first file on a new line?
Thanks for any help you can offer - I'm rather stuck until I figure this out!
Ah-ha. Okay, I think I'm onto something... I spent some time researching the line terminator thing (have been since last night actually) but everything I came up with kept telling me that my line terminator was Unix-style. Just now though, I punched file FirstLine.txt into Terminal and it came back with FirstLine.txt: ASCII text, with no line terminators.
Great. So... how do I insert a line terminator? I found many, many resources that talk about the tr command and changing \r to \n but nothing in the way of adding a line terminator where none exists.
Open the file in your preferred text editor, hit <End>, press <Enter>, save, close.
Or, on the command line, enter printf "<lt>" >> FirstLine.txt, where <lt> is one of the following:
I was hoping you wouldn't say that See, I figured there must be some trick or something, because when I simply add a new line like that (by opening in TextWrangler and hitting Enter at the end of the line), my output gets... weird.
This is a short example of what it was doing when I first posted.
FirstLine.txt:
kids.txt:
cat FirstLine.txt kids.txt > readyToProcess.txt
After adding the new line to FirstLine.txt, I try cat'ing again, but this time, the resulting readyToProcess.txt file looks like this:
So, it added the first line great - but why on earth would it throw an extra line break between all the rest of the lines in the file?
Also - I ran printf "\n" >> FirstLine.txt and the result was the same as when I open the file in TextWrangler and add the new line manually. Oddly, when I cleared that new line and ran printf "\r" >> FirstLine.txt the output is almost correct. I get this:
There's an extra space at the beginning of every line starting with line 3 which I've noted with because vBulletin doesn't allow a space at the beginning of a line in a quote tag
That's strange. You could try to edit it with VIm, adding an empty line and deleting it again, as it usually adds the appropriate EOL when saving a file. Also, if you enter ':set list', you'll see any special characters (eg. $ for EOL, ^I for tab, ...)
Give this a try.. And yes, it's kind've a kludge...
cat foo|echo >>foobar
cat bar|echo >>foobar
Echo should add a line terminator at the end of each file....
But what I'm suspecting is that these files are kinda munged in the first place. Since they're delimited, you could open them up in a oocalc and import them. Then reopend the oocalc file and export it back out to | delimited.
Then give your cat foo bar >>foobar a try...
Also, cat -A should show you all non-printing characters which (in theory) should tell all.
New development...
I found this line of code and figured that if cat wasn't going to cooperate, I'll just put my first line right into the script:
(echo '0a'; echo 'first line of text'; echo '.'; echo 'wq') | ed -s readyToProcess.txt
The resulting file was double-spaced, like I described earlier:
I created a new text file called test.txt that looked like this:
and ran:
(echo '0a'; echo 'first line of text'; echo '.'; echo 'wq') | ed -s test.txt
and it worked. So Scott, I think you're right, I think there is an issue of some sort with the datafeed file. I ran file readyToProcess.txt and it came back with readyToProcess.txt: ASCII English text, with very long lines, with CRLF line terminators
See, I figured there must be some trick or something, because when I simply add a new line like that (by opening in TextWrangler and hitting Enter at the end of the line), my output gets... weird.