Anyone like a challenge?

I have searched through google, and this forum to try and find the answer, but alas, nothing quite hits the whole answer.

I am trying to read the last line (or lines) of some log files. I do this often.

The files are named sequentially, using the date as part of the file name, and appending the iteration so like this:

$ls -ltr |tail -6
h20150208.log.001
h20150209.log.001
h20150210.log.001
h20150211.log.001
h20150211.log.002
h20150211.log.003

And I want generally to tail -1 each of the files to see the end result

There may be 33 files - or may be 3000 files - different each time - I don't want to have to manually specify the filenames - but "*log.0??" works for a "filter". Aged files are compressed (.Z) or gzipped (.gz), and obviously I don't want to read these files.

However.
All of the methods I have tried , using find - don't sort the files into the correct order - I need it to be in chronological order, and I also need to see the filename - preferably on the same line as the resultant output - which is effectively succeeded or failure.
All the attempts I have made using ls -ltr error:

tail -1 `ls -ltr *.log.0??`
ls -ltr *.log.0??|xargs tail -f

Just to make this a little more challenging - this will be across multiple different O/Ss, including HP-UX, SunOS, RHEL and more.

I would prefer a solution which doesn't need to create a script file - but any amount of typing (or pasting) at the command prompt is acceptable.

The result I'm looking to achieve is rather like:

h20150208.log.001 Output from last line of file
h20150209.log.001 Output from last line of file
h20150210.log.001 Output from last line of file
h20150211.log.001 Output from last line of file
h20150211.log.002 Output from last line of file
h20150211.log.003 Output from last line of file

So far, the closest I have to what I need is:

for f in $(find . -name '*.log.0??'); do printf $f && cat $f | tail -1; done

But this fails because it isn't sorted in the correct way, and it concatenates the filename and timestamp (with no gap) like so:

./h20150131.log.00123:23:06  completed successfully
./h20150201.log.00123:01:47  completed successfully
./h20150208.log.00104:47:18  completed successfully
./h20150205.log.00102:55:49  completed successfully
./h20150126.log.00123:52:02  completed successfully
./h20150207.log.00105:50:30 JOBFAILURE: COMPLETED WITH ERRORS RC=201

If anyone has any fabulous ideas or can point me in the right direction, I'd be most grateful...

Background:
For anyone who's interested, I'm a DBA currently working in 2nd line support - looking after about 8000 database instances over 3500 hosts. This particular query is to look at backup log files, and try & determine where it all went wrong....
------
Regards,

Ian

Try:

find . -name '*.log.0??' |
sort -t. -k1.2,1n -k2 |
while read f
do
  printf "%s " "$f"
  tail -1 "$f"
done

Note: Using *.log.0?? would limit the result to the first 99 iterations for any particular day. Perhaps a better alternative would be to use: *.log.[0-9][0-9][0-9]

Damn - I've been trying to do that for days!

If you have the time - I'd be most grateful for an explanation -I get the first line!
Also, can we exclude the timestamp?
And the "./" if possible?

------
Regards,

Ian

Given that the filenames contain the date and a sequence number within that date, I don't see the need for sorting by time in ls , and unless there are files in multiple directories, there is no need for find . Doesn't this do what you need?:

for lf in $(ls *.log.0??|tail -n 6);do printf '%s ' "$lf";tail -n 1 "$lf";done

Hi Don,

Thanks - but that doesn't work - the output is "unsorted", still contains the timestamp which I don't need.

h20150204.log.001 05:10:30 completed successfully
h20150207.log.001 05:50:30 JOBFAILURE: COMPLETED WITH ERRORS RC=201
h20150210.log.001 00:57:23 JOBFAILURE: COMPLETED WITH ERRORS RC=201
h20150211.log.003 00:35:39 OTHER MESSAGE
h20150127.log.001 02:32:00 completed successfully

-----
Regards,

Ian

---------- Post updated at 01:07 AM ---------- Previous update was at 12:59 AM ----------

Yes, I get that, thanks. Generally, there won't be more than 99 files - but I know how to deal with it if there are - the syntax I used, because some of the files are compressed, so I couldn't use *log.*

Hi,

find . -name '*.log.[0-9][0-9][0-9]' |    # get a the names of all the hosts
sort -t. -k1.2,1n -k2 |                   # sort this using a "." as a field separator; sort
                                          # numerically with the 2nd character to the last
                                          # character of the first field and for the second
                                          # key use a regular sort from filed 2 onwards.
while read f
do
  printf "%s " "$f"
  tail -1 "$f"
done

To get rid of the ./ and the timestamp, aa quick fix would be:

...
do
   printf "%s" "${f##*/}"
   tail -1 "$f" | awk '{$1=x}1'
done

Or a bit more efficiently if the file name do not contain spaces, perhaps

...
do
   printf "%s " "${f##*/}"
   tail -1 "$f" 
done | awk '{$2=x}1' 

Perfect, Scrutinizer, thank you so much

-----
Regards,

Ian

---------- Post updated at 01:17 AM ---------- Previous update was at 01:13 AM ----------

Bizarrely, my previous post is off to be moderated!

Anyway - both methods are perfect - thanks so much.

---
Regards,

Ian

That is strange. By default, the output from ls is sorted by filename and (as long as the character(s) in your log file names before the date stamp is(are) constant) the output should be in sorted order. And, tail won't rearrange the output order, and the for loop won't rearrange the output order. Are there hidden characters in your filenames? Did you pass options to ls that I didn't use? Is there a filename ending with .log. followed by three digits that has a minus sign as its first character?

I didn't see your additional requirement to strip out the timestamp until after I submitted my response. You already have suggestions on how to deal with that (as long as the last lines of your log files don't contain adjacent sequences of multiple spaces and tabs that you don't mind being converted to single spaces).

Don,

Don't want to take too much of your time - given that I have a result - however, to answer your points:

I copied & executed your code exactly from your reply - and pasted the output. As far as I know, there are no hidden characters, no minus signs and nothing added.

Interestingly, if I execute the bracketed ls statement, I get this:

>ls *.log.0??|tail -n 6
h20150126.log.001   h20150129.log.001   h20150201.log.001   h20150204.log.001   h20150207.log.001   h20150210.log.001   h20150211.log.003
h20150127.log.001   h20150130.log.001   h20150202.log.001   h20150205.log.001   h20150208.log.001   h20150211.log.001   h20150212.log.001
h20150128.log.001   h20150131.log.001   h20150203.log.001   h20150206.log.001   h20150209.log.001   h20150211.log.002   h20150212.log.002

this lists them top to bottom, left to right
###
looking at the output from the whole command, it is presenting the files in left to right top to bottom from the above output - if you know what I mean - ie the top row of the output above is represented as the start of the output below....

 > for lf in $(ls *.log.0??|tail -n 6);do printf '%s ' "$lf";tail -n 1 "$lf";done
h20150126.log.001 23:52:02 completed successfully
h20150129.log.001 20:58:34 completed successfully
h20150201.log.001 23:01:47 completed successfully
h20150204.log.001 05:10:30 completed successfully
h20150207.log.001 05:50:30 JOBFAILURE: COMPLETED WITH ERRORS RC=201
h20150210.log.001 00:57:23 JOBFAILURE: COMPLETED WITH ERRORS RC=201

Don't know why that would be. I tried adding "-l" to the bracketed command - to list one file per line, but that failed spectacularly!

----
Regards,

Ian

What OS are you using?

When ls output is directed into a pipe (or, for that matter, anything that is not directed to a terminal), the standards require it to produce one filename per output line; nothing at all like what you showed in your last post!

On the machine from which the output has been taken:

 > uname -srv
HP-UX B.11.23 U

I'm not making it up - the results are absolutely copied & pasted from the live server I happen to be working on at the moment - I'll try it elsewhere when I have the opportunity.
---
Regards,

Ian

---------- Post updated at 02:48 AM ---------- Previous update was at 02:42 AM ----------

Here's another:

<hostname removed>> ls *.log.0??|tail -n 6
usage: tail [+/-[n][lbc][f]] [file]
       tail [+/-[n][l][r|f]] [file]
<hostname removed> > ls *.log.0??|tail -6
h20150127.log.001   h20150131.log.001   h20150204.log.001   h20150208.log.001
h20150128.log.001   h20150201.log.001   h20150205.log.001   h20150209.log.001
h20150129.log.001   h20150202.log.001   h20150206.log.001   h20150210.log.001
h20150130.log.001   h20150203.log.001   h20150207.log.001   h20150211.log.001
<hostname removed> > uname -srv
SunOS 5.9 Generic_118558-28
<hostname removed> >

---------- Post updated at 02:50 AM ---------- Previous update was at 02:48 AM ----------

Same thing:

<hostname removed> > for lf in $(ls *.log.0??|tail -6 );do printf '%s ' "$lf";tail -1 "$lf";done
h20150127.log.001 04:26:27 completed successfully
h20150131.log.001 04:33:33 completed successfully
h20150204.log.001 04:31:47 completed successfully
h20150208.log.001 07:46:14 completed successfully

The output you are showing would be what one would expect if you had an alias installed for ls that was defined by:

alias ls='ls -C'

What do the commands:

type ls;alias ls

print for you?

Here is the output (almost identical) from the 2 boxes:

> type ls;alias ls
ls is an alias for ls -FC
ls=ls -FC

and

 > type ls;alias ls
ls is an alias for ls -FC
ls='ls -FC'

I guess that's standard throughout the "estate" then.
----
Regards

Ian

So could you try using \ls instead of ls ?

(or put Don's suggestion into a script and run it like that instead of from the command line)

Bingo!

Where's the "gold star" smiley when you need one?

----
Regards,

Ian

If your sysadmin has set things up like this for all users on your system, in my not so humble opinion, he or she should be fired. If you did this to yourself in one of your shell initialization files, you need to be very aware that lots of commands that work for everyone else will fail due to this unexpected behavior.

If you issue the command:

unalias ls

and then try the command I suggested earlier (with some adjustments for the age of a Solaris Release 9 system and the removal of the timestamp):

for lf in $(ls *.log.0??|tail -6);do printf '%s %s\n' "$lf" "$(tail -1 "$lf")" | awk '{$2=x}1';done

That should work on the HP/UX system. Change the awk to /usr/xpg4/bin/awk or nawk on the Solaris 9 system.

The tail -number is no longer listed in the standards, but is still accepted as a synonym for the today's standard's tail -nnumber preferred form on HP/UX and more recent Solaris releases.

All of this assumes that you're using ksh or bash on your Solaris system. The pure Bourne shell installed in /bin/sh on Solaris systems won't understand the $(command) form of command substitution.

@Don shouldn't tail -6 be taken out? I think it was only used in post #1 to produce the listing, no?

I picked up the tail -6 from BitterBits' post #1. If the intent is to process all files, neither tail nor ls are needed, just:

for lf in *.log.0??;do printf '%s %s\n' "$lf" "$(tail -1 "$lf")" | awk '{$2=x}1';done

on HP/UX systems or:

for lf in *.log.0??;do printf '%s %s\n' "$lf" "$(tail -1 "$lf")" | nawk '{$2=x}1';done

on Solaris systems.

Hi Ian,
You can award a gold star (or at least note your thanks) by clicking the "Thumbs Up Thanks" button at the bottom left of any post that you did not submit and that you have not already thanked.

It looks like you have done this two times already (and it is appreciated).

Cheers,
Don