You don't need to use a stand alone utility to trash the carriage-return characters when using an awk to parse your data. Just add an extra gsub() and that should work:
awk '
BEGIN { n = 0; k = 1; }
FILENAME != last {
fn = FILENAME;
gsub( ".txt", "", fn );
printf( "%s%s%s", k ? "" : ",", n ? "\n" : "", fn );
last = FILENAME;
n = 1; # signal need for newline before fn (prevent lead newline)
k = 0; # when set, signals comma
}
/ntp server/ || /ntp peer/ {
gsub( "\r", "", $0 ); # ditch the charriage return
printf( ",%s", $0 );
k = 1;
}
END {
printf( "%s\n", k ? "" : "," );
} ' *.txt
I suspected it might be something like a bloody carriage-return (ctl-m) but assumed that these files were generated on a UNIX box; most UNIX applications don't add the unnecessary character.
As for xargs, you might try something simple to see what is happening.
ls *txt | xargs echo "files:"
You should see one "files:" string followed by the list of files. If that seems to work, try it again with the awk programme. If you see something different, then have a peek at the man page.
Yes, that works like a champ. This too completes in one second or less, so my lack of xargs isn't hurting me too much. But I will look into it and see if it's something I can get implemented on my Mac. If so, I'll take the above code and the rest of the script you earlier posted to arrive at a final solution.
I can't thank you both enough. If it's any comfort, I've been remotely helping someone to set up a network lab using the Dynamips hypervisor at the same time you've been helping me! I doubt I'd ever be of much help to anyone around here due to my entry-level knowledge of scripting. But I do try to give back in other ways. Thanks again.
Cheers!
---------- Post updated at 01:26 PM ---------- Previous update was at 01:19 PM ----------
Strange. I do indeed see the expected output when I do the above. However, when I try to invoke that in ksh, I still just get the hostname of the very first device and then silence on the line. The full code at this point:
#!/bin/ksh
ls *.txt | xargs awk '
BEGIN { n = 0; k = 1; }
FILENAME != last {
fn = FILENAME;
gsub( ".txt", "", fn );
printf( "%s%s%s", k ? "" : ",", n ? "\n" : "", fn );
last = FILENAME;
n = 1; # signal need for newline before fn (prevent lead newline)
k = 0; # when set, signals comma
}
/ntp server/ || /ntp peer/ {
gsub( "\r", "", $0 ); # ditch the charriage return
printf( ",%s", $0 );
k = 1;
}
END {
printf( "%s\n", k ? "" : "," );
} ' > out5.txt
Am I screwing something up in the script that you can see at this point?
Glad it worked for you... always a pleasure to help.
I cut/pasted your code and it works fine for me, so from my point of view there's nothing that you've done wrong. For now your solution will work. Be aware that if your number of files grows large, you might get a message along the lines of "argument list too large." At that point you'd need to come back to the xargs issue.
You can try this simpler awk and see what the results are. Should print each filename on a new line:
ls *txt | xargs awk ' FILENAME != last { print FILENAME; last = FILENAME; }'
Your above abbreviated code led me to a discovery. It printed the very first hostname and then threw me an error on the very next file, which is something akin to "batchfile.txt." This is obviously not output from my TCL/Expect script, but rather is a big long list of TCL commands and Expect interactive stuff that I paste into my TCL shell to create all of the .txt files I'm performing post-analysis on with these new scripts you have helped me to create. At the end of every one of those TCL commands is something like "router1.txt," which is precisely what creates the .txt files with an embedded hostname in the file. Not sure how, but that was clearly a problem for xargs. I converted all non-router/switch text files to a .xtx extension and then ran the script and now I have something more along the lines I was hoping for. HOWEVER...
My output file right now is "out5.txt." Towards the very end of this file, I actually see an entry of "out5" as if it were a hostname, and then a very VERY long comma-separated line of 'ntp server' and 'ntp peer' strings. So somehow the script is actually catching up to itself before it completes. Right? I suppose I can just exclude output file from being evaluated as input, no?
---------- Post updated at 02:27 PM ---------- Previous update was at 02:00 PM ----------
Just dawned on me that I had stopped using .csv in my output file so that I wouldn't have to do "Open With" and could just open it directly in my text editor. I will, however, go back to .csv for production stuff and then this simply wouldn't be an issue! :o
Sounds like you've figured out the work around, but it would be interesting to see what was triggering it. Is batchfile.txt something that you can post?
I would deny you no request at this moment in time! Of course there is no "router1" or "router2" or any of that - everything has been sanitized along the way. In keeping with that spirit, here is a single line from "batchfile" - this repeats as required by IP address for the given scenario...
So I doubt that any of this requires explanation for you but just for the record, I have a long list of commands that get sent to the routers/switches via "commandlist.txt." Within the TCL script, the three AAA entries above get sent via Expect. The output file part is obvious. In this specific case, I only had one host output .txt file that alphanumerically preceded "batchfile" (and obviously that wasn't "router1"!) And also that one device lacked an 'ntp server/peer' command in it's config, hence this scenario where I was seeing the very first hostname in my out.txt/csv file and then nothing else!!
---------- Post updated at 07:43 PM ---------- Previous update was at 07:36 PM ----------
Oh, and you rightly asked me for the specific error. It looked like this:
svermill's-MacBook-Pro:folder$ ksh "NTP Script.ksh"
awk: can't open file batchfile
input record number 16, file batchfile
source line number 1
Hope that helps to understand what the issue was - I can see it in a very pixilated way but it's likely crystal-clear to yourself and many other code ninjas here...
