I have nginx web server logs with all requests that were made and I'm filtering them by date and time.
Each line has the following structure:
127.0.0.1 - [13/Jul/2018:21:51:31 +0000] xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br)
These text files are all compressed and what I'm currently doing to filter them is:
Does it even work the way you've implemented it?
I'd be surprised the way you're comparing dates.......
You need to convert date to the YYYYMMDDHHMMSS format
Learn to use awk time functions. In the BEGIN {} function, convert the start date and the end date to epoch seconds (%s format)- a big sequential number -- seconds since Jan 1 1970.
For each line in the file convert the date to "%s" format, then see if the number is >=start and <=finish epoch seconds.
If it is in the date range print the line.
This will be a few lines of awk.
And I think vgersh is correct - I do not see how your command worked correctly.
I must be missing something here. But, since the starting and ending timestamps in the awk code in the sample pipeline are on the same date and the times are in 24 hour format (not 12 hour with AM/PM), I see no reason why there is any need to convert the two string arguments to Seconds since the Epoch values and perform numeric comparisons on those converted Seconds since the Epoch values instead of comparing the input values as strings. Furthermore, performing the string comparisons should be faster than converting to strings to integers and then performing a numeric comparison. However, if the start and end timestamps are on different dates, the comments made by vgersh99 and jim mcnamara are absolutely correct.
I have never heard of the unpigz command used at the head of the pipeline being used and I have no idea how the files matched by the pattern nginx* are named nor how big they are. If there are lots of huge compressed files and unpigz is being used to produce uncompressed text from all of those files to be used as input to awk (or if unpigz is a typo and the intended utility at the start of the pipeline was gunzip -c or, equivalently, zcat ) and if part of the name matched by the asterisk in nginx* encodes the dates contained in that file, the way to speed up your pipeline might well be to select a smaller set of files to uncompress instead of trying to speed up the awk code when the slow part of your pipeline may well be the time needed to uncompress unneeded data and to then filter that unneeded data in your awk code.
First, like you've said about comparing string should be faster than coverting to time for compare later, it's exactly what I think happens and that's why my code is how it is and it's working just fine. I have one directory per day, and I ran awk in the files of just one day, so I think it works because of that.
pigz is a parallel implementation of gzip , but it's parallel just for compressing files, but it's a little faster than zcat for descompressing because it uses additional threads for reading, writing, and check calculation (I've read this in one answer in stackoverflow but couldn't refer here because I'm new to the forum). So that's why I use it instead of zcat . About selecting files instead of using nginx* that selects all files in the given directory, it's not possible because I can't tell easily what are the content of each file. That's why I thought maybe it would be something I could do with awk to make it a little faster.
How big are your files, how long do they take, and how fast is your disk? If you're hitting throughput limits optimizing your programs won't help one iota. If you're not, however, you can process several files at once for large gains.
Assuming your input is sorted by date (which seems likely, given logfiles), processing files individually makes another really big optimization possible - once the date exceeds the cutoff, quit! You might skip entire files.
File sizes varies, for instance, some files have ~300MB, others ~1GB or ~2GB.
My disk is not operating in full capacity while running awk, so I think your solution to read multiple files at once will benefit me.
Although I can't tell what's the datetime just by file name, I can leverage the fact that log files are ordered to choose what files i'll be reading, like it was suggested here. I'll make an exampe to illustrate.
I have 832 files in one directory totalizing 100GB, let's say nginx1.gz, nginx2.gz, ..., nginx832.gz.
first line of nginx1.gz has [11/Jul/2018:18:00:01 and the last line [11/Jul/2018:21:00:01
first line of nginx2.gz also has [11/Jul/2018:18:00:01 and the last line [11/Jul/2018:21:00:01
The natural would be nginx2.gz start with the same time or later than nginx1.gz
I could do what you've suggested with this code:
, but I would avoid to read just 2 hours of logs. So, I thought of reading the first and last line of each file and decide whether I would read the file or not. For instance,
1st line datetime would be in: zcat file1.gz | head -n 1
last line datetime would be in: zcat file1.gz | tail -n 1
So, there are two cases when I could skip reading the files:
1 - When 1st and last line is before the time I want
2 - When 1st line is after the time I want (like you've suggested)
This way I think it'll be much faster. I've finish this modification and I'm testing it right now.
If everything is allright I'll test your solution of reading multiple files at once and give you the feedback of the results.
Thank you very much for your help, I didn't know about wait and it'll probably help me.
Unfortunately you can't seek to the end of a compressed file without decompressing the entire thing, so it really doesn't save you time. Otherwise that'd be a really good idea.
How big is your data, and how long does it take to process? These numbers will tell us how much improvement is possible worst-case.
decompress the file twice (maybe not completing the first decompression), and if you find that there is some data in that file that you need, you'll then decompress it again for your awk script to process.
I would strongly suggest creating a separate text file that contains the timestamp of the first record in each compressed file and the name of that compressed file. (And, add a new entry to the end of that file each time you create a new compress log file.) Then you can look at that (uncompressed) text file to quickly determine which compressed file(s) you need to uncompress and feed to your awk script to get the records you want for any particular timestamp range.
I have ~120GB of compressed files per day. It's taking many hours to process
real 217m15.559s
user 763m17.030s
sys 87m40.926s
I didn't know using tail -n 1 would descompress the entire file, but it shouldn't take less time because it doesn't compare the 3rd column of all lines? For instance, if I have 100000000 lines, it would be 100000000 comparisons that would me eliminate if the last line is before the time I want to get
Worst-case, you're adding tons more overhead to files which need processing anyway. There's a lot more to be gained by quitting early than doing extra.
12 hours is surprisingly slow, though. awk is definitely slower than gzip, but it can process 50 megs per second on one of my more ancient systems. Assuming an 8:1 compression ratio for text, you're getting closer to 20. Given that, I'm suspicious that you really are hitting disk bandwidth limits.
Are you using an SSD or a spinning disk? A spinning disk will be hit particularly hard if it has to read and write simultaneously. Its bandwidth will be more than halved. And this is a worst-case situation, where your data is so large that cache is simply no help at all. And is your disk physically attached or a NAS, NFS share, USB disk, or some other such thing? The protocol overhead of these can be horrendous in practice.
If you're not hitting disk bandwidth limits though, multiprocessing should be a big gain.
I am not hitting disk limit, I saw with iotop and htop, it's using less than 10mb/s the majority of time, just in the begining it reaches 60mb/s, I guess. It's a spinning disk, SATA. I will write to another hard drive to stop using the same drive to read and write. Besides that, I`ll take a look in multiprocessing too.
By the way, it's 217 minutes. But it has overhead decompressing that shouldn't exist, I'll fix it
Hi, here a perl script that output first and last line, maybe most faster but i'm not sure...
#!/usr/bin/env perl
use strict;
use warnings;
use IO::Uncompress::Gunzip qw(gunzip $GunzipError);
my $line;
for my $input ( glob "./*.gz" )
{
print $input."\n";
my $gz = new IO::Uncompress::Gunzip $input;
my $mm=$gz->getHeaderInfo();
print $gz->getline();
$gz->seek(%$mm{"ISIZE"}-80,0); # here, max size of line is 80
$line=$gz->getline() while(!eof($gz));
print $line;
close($gz)
}
It might, or might not, be faster depending on what hardware you're using, what operating system you're using, what version of perl you're using, and what other tools you're using as a comparison. But, note that even though this is only printing the first and last lines of the compressed file, it can't avoid reading the entire compressed file and uncompressing all of the compressed data to be able to determine the contents of the last line in the file. (Decompression can't start at random places in the file; it must start at the beginning and progress byte by byte from there.)
I know all that, but there may be a significant gain because we only create the process once instead of creating 4 * n file.
Where I have a doubt is that perl does not emulate seek by just passing through a memory buffer.
Perl is not magic, therefore it must decompress a file to seek inside it too.
This is because gzip doesn't have a "main dictionary" of symbols anywhere, just calculates symbols as it passes through the file. It can't skip ahead without losing track of what the data means. Most stream compressors are like this.