Fetching address and user details from log file

UNIX for Beginners Q & A
1

Hi All,

I have a requirement to get the address values from a large log file along with the user details.
line1,line2,city,stateCode,postalCode,countryCode . The below code as advised in the earlier post is giving the user data

zgrep -B1 "Failed to calculate  Tax" log.2018-05-23.gz | grep "CartService" | grep -Eo '.{4}-.{2}-[^]]*|.{2}:.{2}:.{8}' | paste - -|sort -u -k3,3

Results.

2018-05-23 11:01:31,252 XX:XX:CC061367
2018-05-23 23:17:44,611 XX:XX:EW976815

The log file is as below.

[2018-05-23 11:01:31,251] app14.xx.xx.net 1527087690803 XX:XX:CC061367 http-nio-8080-exec-6 ERROR Get Price Failed! CC061367 , XX , XX (AbstractLocalP
ricingServiceV1)
[2018-05-23 11:01:31,252] app14.xx.xx.net 1527087690803 XX:XX:CC061367 http-nio-8080-exec-6 ERROR Exception while search for products (CartService)
com.xx.xx.xx.xx.xx.xx: Error: Failed to calculate  Tax. Error Number: '2' Error Description: 'A valid city/state or zip code required.' | Params: Address={
  "line1" : "Haci # 34",
  "line2" : " Vallescondido",
  "city" : " Zaragoza",
  "stateCode" : "MEX",
  "postalCode" : "52937",
  "countryCode" : "US"
} taxType=Sales

Please advise how to get the address values along with the user details with heading like below.

Expected Results

Date    Program    line1    line2    city    stateCode    postalCode    countryCode
2018-05-23 11:01:31,252    XX:XX:CC061367    Haci # 34,     Vallescondido,     Zaragoza,    MEX,    52937,    US

2018-05-23 11:01:31,252    XX:XX:UWA38046    3-5-10 Haruhino,     Asao,     Kawa,    KN,    215,US

Attempted couple of commands ,one sample as below but was not giving any results.

zgrep -B1 "Failed to calculate US Tax" log.2018-05-23.gz | grep "CartService" | grep -Eo '.{4}-.{2}-[^]]*|.{2}:.{2}:.{8}' | paste - -|sort -u -k3,3|zgrep -A2 "Failed to calculate Tax" log.2018-05-23.gz|egrep "line(1|2)"|egrep "city" |sort -u
2

Did you consider that (the first) zgrep output is unzipped, clear text, so the second zgrep will mess that up?

3

Sorry, I didnt get the point

---------- Post updated at 06:09 AM ---------- Previous update was at 06:06 AM ----------

---------- Post updated at 06:10 AM ---------- Previous update was at 06:09 AM ----------

I will put code tag for data as well. Thanks

4

Double checking man zgrep I see it will accept and grep straight text as well - so this is not in error. But by supplying the input file again to the second zgrep as an argument, you lose the results of the first one, because the pipe pipes into nirvana.
Please be aware that

  • neither search pattern will produce a match (there's no "US" in front of "Tax", but TWO spaces).
  • egrep ping "line(1|2)" would match two lines IF above had found a match, but egrep ping "city" will drop both lines.
    So - there are several reasons it "was not giving any results".
5

Understood, there should be keyword US in the search pattern , the log file is having the entry , it was omitted accidentally.
Could you please advise any other alternative solutions.Am not familiar with multiple search patterns

6

As any regex matching is utmost sensitive to the pattern, please provide the correct one.

7

Please find the log

[2018-05-23 11:01:31,251] app14.xx.xx.net 1527087690803 XX:XX:CC061367 http-nio-8080-exec-6 ERROR Get Price Failed! CC061367 , XX , XX (AbstractLocalP
ricingServiceV1)
[2018-05-23 11:01:31,252] app14.xx.xx.net 1527087690803 XX:XX:CC061367 http-nio-8080-exec-6 ERROR Exception while search for products (CartService)
com.b2s.service.pricing.pricingapi.exception.InvalidCartRequestException: Error: Failed to calculate US Tax. Error Number: '2' Error Description: 'A valid city/state or zip code required.' | Params: Address={
  "line1" : "Haci # 34",
  "line2" : " Vallescondido",
  "city" : " Zaragoza",
  "stateCode" : "MEX",
  "postalCode" : "52937",
  "countryCode" : "US"
} taxType=Sales
8

Try

gunzip log.2018-05-23.gz | awk '
BEGIN                           {HD = "Date    Program    line1    line2    city    stateCode    postalCode    countryCode"
                                 MX = split(HD, HDFLDS)
                                }

/CartService/                   {TMP = $1 " " $2 OFS $5
                                }

/Failed to calculate US Tax.*{/ {while (!(T1 ~ /}/))    {getline T1
                                                         gsub (/ *" */, _, T1)
                                                         split (T1, T2, ":")
                                                         OUT[T2[1]] = T2[2]
                                                        }

                                 if (!HDprinted) print HD
                                 HDprinted = 1

                                 printf "%s", TMP
                                 for (i=3; i<=MX; i++) printf OFS "%s", OUT[HDFLDS]
                                  printf ORS


                                 TMP = T1 = ""
                                 split ("", OUT)
                                }

 ' OFS="\t"
9

Created a script file with the given code and redirected the execution to a text file(parseAwk.txt) but there was no records from the output.When the script executed initially it failed to locate the file so added the directory reference to the file as below

gunzip /home/nextStep/log.2018-05-23.gz

Output

 0 May 29 11:05 parseAwk.txt