Outputting data from log file to report

Shell Programming and Scripting
1

I have a log file that looks like this. the lines are grouped. 2 lines per entry.

M: 2019-01-25 13:02:31.698 P25, received network transmission from KI4EKI to TG 10282
M: 2019-01-25 13:02:35.694 P25, network end of transmission, 4.3 seconds, 1% packet loss
M: 2019-01-25 13:02:38.893 P25, received network transmission from AE4ML to TG 10282
M: 2019-01-25 13:02:39.595 P25, network end of transmission, 0.9 seconds, 0% packet loss
M: 2019-01-25 13:02:42.747 P25, received network transmission from KI4EKI to TG 10282
M: 2019-01-25 13:02:52.123 P25, network end of transmission, 9.7 seconds, 0% packet loss
M: 2019-01-25 13:03:00.388 P25, received network transmission from AE4ML to TG 10282
M: 2019-01-25 13:03:03.426 P25, network end of transmission, 3.2 seconds, 0% packet loss

I'm trying to pull data into one line , initial date, time and mode (P25) , past "transmission from" , To TG ", " end of transmission" and packet loss.

I have tried to grab everything from " Transmission from " to " loss "

using awk I can only pull what I want from the first line.

awk ' /transmission\ from/,/loss/ {for (I=1;I<=NF;I++) if ($I == "from") {print " CALL: "$(I+1) " TalkGroup "$(I+4)};} ' MMDVM.log
2

Like so?

awk -F, -vOFS=,  '/transmission from/ {getline X; $0 = $0 FS X; print $1, $2, $4, $6}' file
M: 2019-01-25 13:02:31.698 P25, received network transmission from KI4EKI to TG 10282, network end of transmission, 1% packet loss
M: 2019-01-25 13:02:38.893 P25, received network transmission from AE4ML to TG 10282, network end of transmission, 0% packet loss
M: 2019-01-25 13:02:42.747 P25, received network transmission from KI4EKI to TG 10282, network end of transmission, 0% packet loss
M: 2019-01-25 13:03:00.388 P25, received network transmission from AE4ML to TG 10282, network end of transmission, 0% packet loss
3

I don't need "received network transmission from" or the "network end of transmission"

M: 2019-01-25 18:46:00.771 P25, AE4ML to TG 10282, 0.5 seconds, 0% packet loss

Thank you

4

One could try:

awk '
NR % 2 {sub(/ received.*from/, "")
	printf("%s", $0)
}
!(NR % 2) {
	sub(/.*mission/, "")
	print
}' MMDVM.log

which produces the output:

M: 2019-01-25 13:02:31.698 P25, KI4EKI to TG 10282, 4.3 seconds, 1% packet loss
M: 2019-01-25 13:02:38.893 P25, AE4ML to TG 10282, 0.9 seconds, 0% packet loss
M: 2019-01-25 13:02:42.747 P25, KI4EKI to TG 10282, 9.7 seconds, 0% packet loss
M: 2019-01-25 13:03:00.388 P25, AE4ML to TG 10282, 3.2 seconds, 0% packet loss

from your sample input. But, since 0.5 seconds doesn't appear anywhere in your sample input data, I have no idea where the output you specified in post #3 in this thread came from???

1 Like
5

How about adapting the proposal to exactly fit the (fuzzy) spec (which e.g. lacks the seconds)?

6

Don thanks but that didn't work. The output was mixed. It gave me more output than what I needed. it wasn't consistent. Using what you have shown me I haven't been able to do any better.

M: 2019-01-25 12:12:13.929 P25, AE4ML to TG 10282, 1.8 seconds, 0% packet loss
M: 2019-01-25 12:12:20.828 P25, AE4ML to TG 10282, 3.1 seconds, 0% packet loss
M: 2019-01-25 12:12:26.359 P25, AE4ML to TG 10282, 1.6 seconds, 0% packet loss
M: 2019-01-25 12:22:44.126 P25, network end of transmission, 1.6 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:22:55.009 P25, network end of transmission, 4.7 seconds, 1% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:23:02.687 P25, network end of transmission, 2.7 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:23:10.208 P25, network end of transmission, 5.0 seconds, 1% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:23:31.438 P25, network end of transmission, 17.6 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:23:40.070 P25, network end of transmission, 6.1 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:23:44.716 P25, network end of transmission, 2.0 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:24:16.129 P25, network end of transmission, 28.1 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:24:36.629 P25, network end of transmission, 13.3 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:25:16.532 P25, network end of transmission, 36.2 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:25:35.567 P25, network end of transmission, 12.6 seconds, 1% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:26:01.025 P25, network end of transmission, 21.8 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:26:39.710 P25, network end of transmission, 33.5 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:27:01.881 P25, network watchdog has expired, 15.7 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:27:29.827 P25, network end of transmission, 21.6 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:28:09.817 P25, network end of transmission, 1.8 seconds, 1% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:29:15.648 P25, network end of transmission, 5.0 seconds, 1% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:29:21.679 P25, network end of transmission, 2.3 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:29:54.216 P25, network end of transmission, 29.3 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:30:03.202 P25, network end of transmission, 3.2 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:30:55.123 P25, network end of transmission, 43.9 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:31:09.964 P25, network end of transmission, 6.1 seconds, 2% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:31:44.923 P25, network end of transmission, 31.1 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:32:09.982 P25, network end of transmission, 13.1 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:32:49.741 P25, network end of transmission, 35.3 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:33:09.973 P25, network end of transmission, 11.0 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:33:33.505 P25, network end of transmission, 19.4 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:33:46.262 P25, network end of transmission, 6.8 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:33:55.647 P25, network watchdog has expired, 2.7 seconds, 0% packet loss from 0 to TG 0
M: 2019-01-25 12:34:16.471 P25, network end of transmission, 19.3 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:34:40.820 P25, network end of transmission, 15.5 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:35:05.293 P25, network end of transmission, 20.9 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:35:38.192 P25, network end of transmission, 24.7 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:36:11.575 P25, network end of transmission, 28.1 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:36:47.519 P25, network end of transmission, 23.4 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:37:39.276 P25, network end of transmission, 46.6 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:37:54.973 P25, network end of transmission, 8.8 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:39:06.123 P25, network end of transmission, 64.8 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:39:16.175 P25, network end of transmission, 4.9 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:40:07.191 P25, network end of transmission, 46.3 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:40:24.352 P25, network end of transmission, 13.3 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:40:50.022 P25, network end of transmission, 22.3 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:41:01.398 P25, network end of transmission, 3.4 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:41:22.434 P25, network end of transmission, 18.0 seconds, 0% packet loss from AE4ML to TG 10282
M: 2019-01-25 12:41:29.433 P25, network end of transmission, 4.5 seconds, 0% packet loss from KI4EKI to TG 10282
M: 2019-01-25 12:41:34.259 P25, network end of transmission, 2.7 seconds, 2% packet loss, 2.7 seconds, 2% packet loss
M: 2019-01-25 12:54:00.882 P25, AE4ML to TG 10282, 0.4 seconds, 0% packet loss
M: 2019-01-25 12:54:06.856 P25, AE4ML to TG 10282, 0.5 seconds, 0% packet loss
M: 2019-01-25 13:02:23.852 P25, network end of transmission, 0.5 seconds, 0% packet loss, 0.5 seconds, 0% packet loss
M: 2019-01-25 13:02:31.698 P25, KI4EKI to TG 10282, 4.3 seconds, 1% packet loss
M: 2019-01-25 13:02:38.893 P25, AE4ML to TG 10282, 0.9 seconds, 0% packet loss
M: 2019-01-25 13:02:42.747 P25, KI4EKI to TG 10282, 9.7 seconds, 0% packet loss
M: 2019-01-25 13:03:00.388 P25, AE4ML to TG 10282, 3.2 seconds, 0% packet loss
M: 2019-01-25 13:03:06.106 P25, KI4EKI to TG 10282, 15.1 seconds, 0% packet loss
M: 2019-01-25 13:03:24.864 P25, AE4ML to TG 10282, 6.5 seconds, 0% packet loss
M: 2019-01-25 13:03:34.471 P25, KI4EKI to TG 10282, 6.5 seconds, 0% packet loss
M: 2019-01-25 13:03:49.833 P25, AE4ML to TG 10282, 3.8 seconds, 0% packet loss
M: 2019-01-25 13:08:08.229 P25, AE4ML to TG 10282, 0.4 seconds, 0% packet loss
M: 2019-01-25 13:08:44.319 P25, AE4ML to TG 10282, 3.1 seconds, 0% packet loss
7

Does the sample given in post #1 represent the log file's structure one to one? I guess you don't have two alternating consecutive line types but quite some more ("watchdog"), and mayhap the pairs split, like some initial lines, and then some end lines.

Try this adapted - easily done by the interested person - version of my earlier proposal:

 awk -F, -vOFS=,  'match ($0, /, .*transmission from/) {sub (substr ($0, RSTART, RLENGTH), ","); getline X; $0 = $0 FS X; print $1, $2, $5, $6}' file

This would not help if the interesting lines are not adjacent. For that case, you'd need to identify a characteristic connecting the two lines.

2 Likes
8

Hi ae4ml,
I'm afraid I'm a little more pessimistic than RudiC about our chances for finding the proper pairs of lines to join to produce the output you want.

In post #1 you said that your input was grouped in pairs of two lines each. To get the output shown above, you would have to have pairs of lines in a group that are not adjacent.

If you don't provide us with representative sample input and a clear description of how the input actually appears, we can't be expected to write code that will work.

If you look at the first 10 lines of the input file you used to create the above output, you'll see that there is an unmatched pair of lines starting around line 7 or 8 of the input.

The output you've shown us looks like a very simple case of what computer scientists refer to GIGO (garbage in, garbage out) where garbage refers to data that is not in the expected format.

And looking at your sample data in post #1, we can't use time stamps in pairs of records together with the elapsed times given in the 2nd line in each pair because none of the elapsed times even come close to matching the differences between the time stamps.

It also appears that you have input lines that are of a format not included in your description in post #1. For example, to get the output you've shown us, there have to be lines in the input in a format similar to the following:

M: 2019-01-25 12:33:55.647 P25, network watchdog has expired, ...

You didn't mention there being any lines in this format in your input files and my code wasn't expecting to see any lines like these. Having them in your input file will certainly cause output that is not in the format you requested.

If you can come up with a more accurate description of your input file format, we might be able to come up with something that comes closer to the output you want.

If your input doesn't have adjacent pairs of lines that are to be used together to produce a line of output, you have to come up with some way to reliably find the 2nd line of a pair of lines that need to be joined to produce your desired output. Since the second line in a pair doesn't include the "from" and "to" tags, I don't see any good way to pair lines if they aren't adjacent.

Sorry,
Don

1 Like