Prepare file run report

Shell Programming and Scripting
1

I have a requirement to prepare a report.

We validate some incoming data fields and create validation_error reports which will contain records which do not pass validation. Once files are processed they will all be dropped under one folder.

EMPLOYEE_20140915.txt
EMPLOYEE_20140915.txt.error.dat
DEPARTMENT_20140911.txt
DEPARTMENT_20140911.txt.error.dat
PRSNL_INFO_20140911.txt
SAL_HIKE_20140911.txt
SALARY_20140911.txt
SALARY_20140911.txt.error.dat

As seen above, of the 5 files, 3 of them have rejects and 2 were processed fine. I need to prepare a report which looks like:

FILE NAME	  Status
-------------------------------------
PRSNL_INFO	- Processed successfully
SAL_HIKE	- Processed successfully
EMPLOYEE 	- Has error resubmit
DEPARTMENT	- Has error resubmit
SALARY		- Has error resubmit

Can someone help me how I can go about creating the report?

2

Not the slightest attempt from your side? Try

ls *.txt |
awk     'BEGIN          {print "FILE NAME\tStatus\n-------------------------------------"}
                        {gsub(/_[0-9]*$/, "", $1); OUT[$1]}
         /error/        {OUT[$1]++}
         END            {for (i in OUT) print i, OUT?"- has error":"- success"}
        ' FS="." OFS="\t"
FILE NAME     Status
-------------------------------------
SALARY    - has error
DEPARTMENT    - has error
EMPLOYEE      - has error
PRSNL_INFO    - success
SAL_HIKE      - success
3

Thanks RudiC.

I am a beginner in Unix and still learning. I tried a few things but did not get anywhere so wanted to get help.

Can you help me understand what this code is doing? I seem to be getting a success for all the files.

EMPLOYEE_20140915.txt
EMPLOYEE_20140915.txt.error.dat
DEPARTMENT_20140911.txt
DEPARTMENT_20140911.txt.error.dat
PRSNL_INFO_20140911.txt
SAL_HIKE_20140911.txt
SALARY_20140911.txt
SALARY_20140911.txt.error.dat

ls *.txt | awk     'BEGIN          {print "FILE NAME\tStatus\n-------------------------------------"}
                        {gsub(/_[0-9]*$/, "", $1); OUT[$1]}
         /error/        {OUT[$1]++}
         END            {for (i in OUT) print i, OUT?"- has error":"- success"}
        ' FS="." OFS="\t"
FILE NAME       Status
-------------------------------------
PRSNL_INFO      - success
EMPLOYEE        - success
DEPARTMENT      - success
SAL_HIKE        - success
SALARY  - success
4

What's your system and awk version? Please post the output of ls *.txt without editing.

5

Please find below details:

dev/processed>ls *.txt
DEPARTMENT_20140911.txt  EMPLOYEE_20140915.txt  PRSNL_INFO_20140911.txt  SALARY_20140911.txt  SAL_HIKE_20140911.txt

AWK version:
dev/processed>awk -W version
GNU Awk 3.1.7
Copyright (C) 1989, 1991-2009

Linux version: 2.6.32-358.11.1.el6.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux

Thanks!

6

Please use code tags as required by forum rules!

So - where are the error files? No error files - no errors - all success!

7

Error files are also in the same folder:

Processed files:

dev/processed>ls *.txt
DEPARTMENT_20140911.txt  EMPLOYEE_20140915.txt  PRSNL_INFO_20140911.txt  SALARY_20140911.txt  SAL_HIKE_20140911.txt

Error files:

dev/processed>ls *err*
DEPARTMENT_20140911.txt.error.dat  EMPLOYEE_20140915.txt.error.dat  SALARY_20140911.txt.error.dat

I get all files as processed successfully:

dev/processed>ls *.txt | awk     'BEGIN          {print "FILE NAME\tStatus\n-------------------------------------"}
>	{gsub(/_[0-9]*$/, "", $1); OUT[$1]}
>   /error/        {OUT[$1]++}
>   END            {for (i in OUT) print i, OUT?"- has error":"- success"}
>   ' FS="." OFS="\t"
FILE NAME       Status
-------------------------------------
PRSNL_INFO      - success
EMPLOYEE        - success
DEPARTMENT      - success
SAL_HIKE        - success
SALARY  - success
8

OK, make it ls *.txt.* | awk ...

1 Like
9

Aah, that worked. Thanks RudiC. This is really helpful. I will try to break it down and understand the logic.