How do i read only last 5 files records from a directory.?

Hello team,

I have a directory containing huge number of files.
I need to go select last 5 files and read it to do a calculation.

Kindly guide, how can i achieve it.

Regards,
Sadique

That highly depends how you define "the last 5" - sorted alphabetically? by date? Other?

The directory keep geeting files continously.
So if i do ls -l | tail -5
I get that latest 5 files,
My main concern is i need to read each files in loop
Then awk few data from the files comeout and do some calculation.

For all the files inside a directory it os working, but i want to implement it only last 5 files of the directory.

---------- Post updated at 04:17 PM ---------- Previous update was at 09:29 AM ----------

here is my code:

#!/bin/sh
DATE=`date +"%d-%m-%Y-%H:%M"`
stream=IUCS
FLAG=LAST
path=/log/INVESTIG/TEST
for  files in $path/*
        do
        for f in $files
        do
        #echo $files
        read -r line || [ -n "$line"];
        TTHEX=`awk -F ',' 'END{print $4}'`
                done < $files
                TIMESTAMP=$( date +'%H:%M:%S' -r $files)
                TRANS_TIME=$(date -d @$(expr `printf "%d" 0x$TTHEX` / 1000) | awk '{print $4}')
                TIME_LAG=$(date +%H:%M:%S -ud @$((`expr $(date -u -d "$TIMESTAMP" +"%s") - $(date -u -d "$TRANS_TIME" +"%s")`)))
 
        echo "${DATE} ${stream} ${FLAG} $(ls -l $files | awk '{print $9}'| cut -d '/' -f5) ${TIMESTAMP} ${TRANS_TIME} ${TIME_LAG}"
        done

Output:

26-10-2017-15:24 IUCS LAST TDR-IU-8-139 11:12:23 10:14:59 00:57:24
26-10-2017-15:24 IUCS LAST TDR-IU-8-127 15:22:33	15:14:59 00:07:34
26-10-2017-15:24 IUCS LAST TDR-IU-8-140 15:22:33 15:14:59 00:07:34
26-10-2017-15:24 IUCS LAST TDR-IU-8-59   15:22:33 15:14:59 00:07:34

My script is reading all the files of the directory.
for files in $path/*

instead of it i want to read only last 5 files of the directory

kindly help.

echo `ls | tail -5`

Looks okay?
Then replace the echo with the awk command.

1 Like

Output:

26-10-2017-15:24 	IUCS 	LAST 	TDR-IU-8-139 	11:12:23 		10:14:59 		00:57:24
26-10-2017-15:24 	IUCS 	LAST 	TDR-IU-8-127 	15:22:33	 	15:14:59 		00:07:34
26-10-2017-15:24 	IUCS 	LAST 	TDR-IU-8-140 	15:22:33 	15:14:59 		00:07:34
26-10-2017-15:24 	IUCS 	LAST 	TDR-IU-8-59 		15:22:33 	15:14:59 		00:07:34

---------- Post updated at 04:24 PM ---------- Previous update was at 04:21 PM ----------

for file in $path/*

its taking all the file in for loop, i need last five files of the directory to do the later calculation

How can i do that, kindly help.

Do exactly what MadeInGermany suggested...

Change:

for  files in $path/*

to:

for  files in `ls $path|tail -5`

Although I must admit that I do not understand why you have the inner for loop in your script??? In what way would the output be different if that entire loop was replaced by:

                TTHEX=`awk -F ',' 'END{print $4}' $files`

Attention: ls omits the path,
so in the loop you must prepend it to the loop variable like "$path/$file"
Or you do

for  files in `printf "%s\n" $path/* | tail -5`

--
Yes, the inner loop looks odd - certainly not yet ready...

1 Like

The odd indentation didn't help in trying to figure out what the inner loop was doing and the fact that the inner loop variable is never referenced is some kind of clue (I think).

I should have just said to cd to the directory before the outer loop and use ls (or printf '%s\n' * ) piped through tail which would also allow replacing:

$(ls -l $files | awk '{print $9}'| cut -d '/' -f5)

with just:

$files

in the final echo in the outer loop.

1 Like

Looking at the overly complicated (and erroneous) script that you posted, making use of the proposals in the handful of your recent threads (none which you neither commented nor thanked, BTW) seemingly pertaining to the same topic - why don't you show the underlying problem and all the input data instead of asking a little question here and a tiny puzzle there? People in here might come up with a consistent proposal that hits the bull's eye.

3 Likes

file have hexadecimal value on column 4 i am selecting that and doing further calculation.

---------- Post updated at 02:24 PM ---------- Previous update was at 02:13 PM ----------

for f in $files
        do
        #echo $files
        read -r line || [ -n "$line"];
        TTHEX=`awk -F ',' 'END{print $4}'`

above loop reading each files lines and printing the column 4th value into the variable TTHEX

so TTHEX keep assigning the value every time it goes into the loop and outer loop i am calculating value using it.

2nd last column is the value in my output.

That is not your complete inner loop. The complete inner loop is:

for f in $files
        do
        #echo $files
        read -r line || [ -n "$line"];
        TTHEX=`awk -F ',' 'END{print $4}'`
                done < $files

more readably written as:

for f in $files
do
	#echo $files
	read -r line || [ -n "$line"];
	TTHEX=`awk -F ',' 'END{print $4}'`
done < $files

For demonstration purposes, let me rewrite this loop as:

for f in $files
do
	echo '$files expands to: '"$files"
	echo '    $f expands to: '"$f"
	read -r line || [ -n "$line"];
	echo ' $line expands to: '"$line"
	TTHEX=`awk -F ',' 'END{print $4}'`
	echo '$TTHEX expands to: '"$TTHEX"
done < "$files"

and let us assume that a file named x.txt contains the five lines:

header,decimal,octal,hexadecimal
line1,100,144,64
line2,200,310,C8
line3,300,454,12C
line4,400,620,190

If we set files to x.txt and run the modified loop, this is the output we see:

$files expands to: x.txt
    $f expands to: x.txt
 $line expands to: header,decimal,octal,hexadecimal
$TTHEX expands to: 190
  1. Note that even though there are five lines in the input file, we only go through the loop once; not five times.
  2. Note that the loop control variable ( $f ) in your loop) is never used in your script.
  3. Note that the contents of the first line in the input file ( $line in your loop) is never used in your script.
  4. Note that the contents of the variable that is used in the outer loop ( $TTHEX in your code) is set to the contents of the 4th field in the last line of the input file.

Note that if we replace that entire loop with the replacement I suggested for it (and an echo to show what it did):

		TTHEX=`awk -F ',' 'END{print $4}' $files`
		echo '$TTHEX expands to: '"$TTHEX"

then we get the output:

$TTHEX expands to: 190

So, I repeat my question from post #6 in this thread:

??? As long as the input file processed by this loop contains at least two lines of text, your loop should produce exactly the same output as the single awk statement I suggested.

If the input file is empty, both loops set TTHEX to an empty string.

If the input file contains a single line (presumably a header similar to the first line in x.txt ), your loop sets TTHEX to an empty string while my suggestion sets TTHEX to the contents of the header for the fourth field. Is that an important distinction for the proper behavior of your script? If it is, change my suggestion to:

		TTHEX=`awk -F ',' 'NR==1{$4=""}END{print $4}' $files`

I assumed that it didn't matter because you don't check whether or not $TTHEX expands to an empty string before passing its value to expr and if $TTHEX does expand to an empty string, bash 's built-in printf will fail with an arithmetic syntax error when trying to use %d to format the string 0x .

PS Note that two missing back-quotes have been added to fix my suggested awk scripts (as noted in post #13 by MadeInGermany).

1 Like

Thank you :slight_smile:

I will change my script as suggested and will check.

Thnx everyone, for the support.

And IF the read would make sense, the only purpose of the inner loop is to form a block. The loop variable is unused, the list always contains one word.
A simple block is

{
	#echo $files
	read -r line || [ -n "$line"];
	TTHEX=`awk -F ',' 'END{print $4}'`
} < $files

Now the purpose of the block becomes obvious: it redirects the input from the file, so both read and awk read from the file.
But then the next question is: what is the || [ -n "$line"]; for??
And then, as Don found out, the awk code reads through all lines, so why steal the first line with read line and do nothing with $line ?
And if we can omit the useless read then we can also omit the block, and let awk read the file directly.
BTW Don's sample misses the closing backtick, should be

TTHEX=`awk -F ',' 'END{print $4}' $files`

Last but not least, the name files (plural) can make you think it holds many files. But it holds only one file (well, a different file in each loop cycle, but always one file). Therefore I vote for file (singular)!

1 Like
read -r line || [ -n "$line"]; 

This line should be commented while pasting the code, i did blunder mistake.

Extremely sorry folks.

Regards,