Tail 86000 lines from 1.2 million line file?

Shell Programming and Scripting
1

I have a log file that is about 1.2 million lines long and about 300MB.
we need a way to clean up this file and only keep the last few thousand lines.

if i use tail command we run our of memory as the file is too big.

I do have a key word to match on.
example, we want to keep every line after the line containing 1104100311229
and this magic number will only be in the log once.

I feel there may be a way to use awk?

I offer 10,000 Bits to the best answer...

2

How about:

sed '1,/1104100311229/d' logfile > logfile.keepers
3

so u need 86000 last lines or magic number (many different numbers?) or both?
P.S. 300mb is weak

4

the total lines we need to keep will be about 86000 lines.

the exact point we need to keep from is the magic number.

5

but u said:

so there 86000 different magic numbers?

6

Turning Scrutinizers sed round so we keep the line containing the search key, we can seek to the start line then start outputting.

sed -n "/1104100311229/,\$ p" filename >filename.new
7

that works for me.

thanks.

8

All fine and dandy, except the requirement was to output everything after the search key, no?

9

same concept but without moving the file to a temp file

sed -ni "/1104100311229/,\$p" logfilename