Can you extract (remove) lines from log files?

Shell Programming and Scripting
1

I use "MineOS" (a linux distro with python scripts and web ui included for managing a Minecraft Server). The author of the scripts is currently having a problem with the Minecraft server log file being spammed with certain entries. He's working on clearing up the spam.

But in the meantime, I'm wondering if I can run a shell script that will occasionally remove these spam lines (and only these lines) to keep the log file more manageable. Here is an example of the spam:

2013-08-02 20:51:30 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-02 20:51:30 [INFO] /127.0.0.1:52343 lost connection
2013-08-02 20:51:30 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-02 20:51:30 [INFO] /127.0.0.1:52344 lost connection

The consistent piece is the "[SEVERE] Reached end of stream for /127.0.0.1". I would like to remove that entire line PLUS the single line that immediately follows each time (that lists 127.0.0.1:##### lost connection). But I don't want to remove other entries from the log file (named, "server.log" btw).

Is there a way to parse this server.log and selectively delete lines?

2

This deletes the first line and the corresponding line (searched in the next 2 lines in order to allow another log entry in between).

awk '/\[SEVERE\] Reached end of stream for \/127\.0\.0\.1/ {c=2; next} c && c-- && /\[INFO\] \/127\.0\.0\.1:[0-9]+ lost connection/ {c=0; next} 1' server.log
1 Like
3

Hello,

So I guess I need to refine my request.

Here is a real example of the log from recent testing.

2013-08-07 18:13:13 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:14 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:16 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:17 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:19 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:21 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:22 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:24 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:25 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:27 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:28 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:30 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:32 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:33 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:35 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:36 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:38 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:39 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:39 [INFO] /127.0.0.1:42057 lost connection
2013-08-07 18:13:41 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:42 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:44 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:46 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:47 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:49 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:50 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:52 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:53 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:55 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:57 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:58 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:14:00 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:14:01 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:14:03 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:14:04 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:14:06 [SEVERE] Reached end of stream for /127.0.0.1

So I guess what I really need is two separate scripts:

1) A script that parses the server.log file and removes ALL instances of lines that contain "[SEVERE] Reached end of stream for /127.0.0.1"

2) A script that parses the server.log file and removes ALL instances of lines that fit this general pattern: "[INFO] /127.0.0.1:????? lost connection" (where ????? represents a random 5-digit port number)

And of course, the scripts have to leave all the remaining logs so that the log file is useful.

Thank you for any help that anyone can provide!

4 
[root@centosgeek ~]# cat testfile4
2013-08-07 18:13:16 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:17 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:19 [SEVERE] Reached end of stream for /127.0.0.1
SEVERE my stream
2013-08-07 18:13:21 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:27 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:28 [SEVERE] Reached end of stream for /127.0.0.1
INFO is flowing but lost
2013-08-07 18:13:30 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:38 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:39 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:39 [INFO] /127.0.0.1:42057 lost connection
2013-08-07 18:13:41 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:42 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:43 [INFO] /127.0.0.1:42000 messy connection
2013-08-07 18:13:44 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:45 [INFO] /127.0.0.1:22057 under consideration
2013-08-07 18:13:46 [SEVERE] Reached end of stream for /127.0.0.1
SEVERE you forgot the doughnuts
2013-08-07 18:13:47 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:50 [SEVERE] Reached end of stream for /127.0.0.1
[root@centosgeek ~]# awk '(!/SEVERE/ || !/end of stream/) && (!/INFO/ || !/lost connection/)' testfile4
SEVERE my stream
INFO is flowing but lost
2013-08-07 18:13:43 [INFO] /127.0.0.1:42000 messy connection
2013-08-07 18:13:45 [INFO] /127.0.0.1:22057 under consideration
SEVERE you forgot the doughnuts
[root@centosgeek ~]#
1 Like
5

So, if I want to catch "end of stream" and "lost connection" messages for IP addresses other than local host, could I do this for a script?

awk '(!/SEVERE/ || !/end of stream/ || !/127.0.0.1/) && (!/INFO/ || !/127.0.0.1/ || !/lost connection/)' server.log

Thanks for your help!

---------- Post updated at 02:26 PM ---------- Previous update was at 02:05 PM ----------

I modified a few IP addresses and tested with adding the extra bits to your code.

Here is what I came up with for my script:

cp server.log server.log.backup
awk '(!/SEVERE/ || !/end of stream/ || !/127.0.0.1/) && (!/INFO/ || !/127.0.0.1/ || !/lost connection/)' server.log.backup >server.log

Thank you!

6

May be written like this

awk '!/SEVERE.*end of stream.*127.0.0.1/ && !/INFO.*127.0.0.1.*lost connection/' server.log.backup >server.log
7

@ nbsparks -- glad it helped ... just remember always that whatever code shows up on this site, you definitely need to test first prior to using in production ...

@ jotne -- i like the brevity of your code but i seem to be missing something ... i tested on both centos and debian coming out with the same result as posted below ...

root@debiangeek:~# cat testfile4.1
2013-08-07 18:13:16 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:17 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:19 [SEVERE] Reached end of stream for /127.0.0.1
SEVERE my stream
2013-08-07 18:13:21 [SEVERE] Reached end of stream for /192.168.5.167
2013-08-07 18:13:21 [INFO] /198.162.5.167:42057 lost connection
2013-08-07 18:13:27 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:28 [SEVERE] Reached end of stream for /127.0.0.1
INFO is flowing but lost
2013-08-07 18:13:30 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:38 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:39 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:39 [INFO] /127.0.0.1:42057 lost connection
2013-08-07 18:13:41 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:42 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:43 [INFO] /127.0.0.1:42000 messy connection
2013-08-07 18:13:44 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:45 [INFO] /127.0.0.1:22057 under consideration
2013-08-07 18:13:46 [SEVERE] Reached end of stream for /127.0.0.1
SEVERE you forgot the doughnuts
2013-08-07 18:13:47 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:50 [SEVERE] Reached end of stream for /127.0.0.1
root@debiangeek:~# awk '(!/SEVERE/ || !/end of stream/ || !/127.0/) && (!/INFO/ || !/lost connection/ || !/127.0/)' testfile4.1
SEVERE my stream
2013-08-07 18:13:21 [SEVERE] Reached end of stream for /192.168.5.167
2013-08-07 18:13:21 [INFO] /198.162.5.167:42057 lost connection
INFO is flowing but lost
2013-08-07 18:13:43 [INFO] /127.0.0.1:42000 messy connection
2013-08-07 18:13:45 [INFO] /127.0.0.1:22057 under consideration
SEVERE you forgot the doughnuts
root@debiangeek:~# awk '!/SEVERE.*end of stream.*127.0.0.1/ && !/INFO.*lost connection.*127.0.0.1/' testfile4.1
SEVERE my stream
2013-08-07 18:13:21 [SEVERE] Reached end of stream for /192.168.5.167
2013-08-07 18:13:21 [INFO] /198.162.5.167:42057 lost connection
INFO is flowing but lost
2013-08-07 18:13:39 [INFO] /127.0.0.1:42057 lost connection
2013-08-07 18:13:43 [INFO] /127.0.0.1:42000 messy connection
2013-08-07 18:13:45 [INFO] /127.0.0.1:22057 under consideration
SEVERE you forgot the doughnuts
root@debiangeek:~#
8

Need to change to order of the test to make it work correctly.

awk '!/SEVERE.*end of stream.*127.0./ && !/INFO.*127.0.*lost connection/'
SEVERE my stream
2013-08-07 18:13:21 [SEVERE] Reached end of stream for /192.168.5.167
2013-08-07 18:13:21 [INFO] /198.162.5.167:42057 lost connection
INFO is flowing but lost
2013-08-07 18:13:43 [INFO] /127.0.0.1:42000 messy connection
2013-08-07 18:13:45 [INFO] /127.0.0.1:22057 under consideration
SEVERE you forgot the doughnuts
9

thank you for the correction ..