Multiple bash variables passed into nawk

Shell Programming and Scripting
1

I have a file that has 2 fields called b_file:

11977 DAR.V3.20150209.1.CSV
3295 DAR.V3.20150209.1.CSV
1721 DAR.V2.20150210.1.CSV

I need to search a sftplog using the field 1, but want to maintain the relationship between field 1 and 2. I am passing field 1 as a parameter in a bash loop.

for i in `cat |nawk '{ print $0 }' /tmp/b_file`
do

I am searching a sftplog for the ID for the user, but this is only giving me the field after "user".

nawk -v st=$i '$5 ~ st && /closed/ && /user/ && !/ORG/  && !/LE/ { for (x=1;x<=NF;x++) if ($x~"user") print $(x+1) } ' /var/adm/sftplog >> /tmp/dar3.out

I want the output in /tmp/dar3.out to be

tom DAR.V3.20150209.1.CSV
charlie DAR.V3.20150209.1.CSV
steve DAR.V2.20150210.1.CSV

where tom was 11977, charlie was 3295, and steve was 1721.

complete code:

for i in `cat |nawk '{ print $0 }' /tmp/b_file`
do
nawk -v st=$i '$5 ~ st && /closed/ && /user/ && !/ORG/  && !/LE/ { for (x=1;x<=NF;x++) if ($x~"user") print $(x+1) } ' /var/adm/sftplog >> /tmp/dar3.out
done

Open to alternative methods if recommended. Any assistance is much appreciated.

2

You have shown the output you want, now show the input you have please.

3

Hello smenago,

You was not clear about requirement, based on my assumptions which are (You have 2 files named b_file and c_file as follows.)

cat b_file
11977 DAR.V3.20150209.1.CSV
3295 DAR.V3.20150209.1.CSV
1721 DAR.V2.20150210.1.CSV
 
cat c_file
tom 11977
charlie 3295
steve 1721

Then as per your expected output I can suggest following.

awk 'FNR==NR{A[$1]=$2;next} ($2 in A){print $1 OFS A[$2]}' b_file c_file

Output will be as follows.

tom DAR.V3.20150209.1.CSV
charlie DAR.V3.20150209.1.CSV
steve DAR.V2.20150210.1.CSV

If you are happy with above awk command then you can redirect it's output to a file named > /tmp/dar3.out .

Hope this helps, if my assumptions are not correct, please do let us know the expected output with os details.

NOTE: On Solaris/SunOS systems, you need to change awk to /usr/xpg4/bin/awk , /usr/xpg6/bin/awk

EDIT: Sorry Corona, seems we have posted on same time, please do remove this post if needed until OP comes back with answer.

Thanks,
R. Singh

4

The input from the sftplog looks like this

Feb 11 12:01:46 ftp02 sftp-server[11977]: [ID 800047 local1.info] session closed for local user tom from [65.200.105.28]
Feb 11 20:49:57 ftp02 sftp-server[3295]: [ID 800047 local1.info] session closed for local user charlie from [65.200.105.28]
5

Bash/shell is even enough:

while read uNo fileNm
do
 echo "$( <procedure to translate $uNo to user>) $fileNm
done <in_file >out_file

But if the user is in file c, just join:

sort -u -o c c
sort -u -o b b
join c b | cut -f 2,3

I think you can adjust the output of join with arguments and avoid the cut. See the man page. You can also join with any tool with associative arrays (hash maps), like ksh, bash, awk; putting the smaller set into an array and the looking up the larger one line by line.

6

Hello smenago,

Could you please try following and let me know if this helps.

awk 'FNR==NR{A[$1]=$2;next} ($2 in A){sub(/.*user /,X,$5);sub(/ from/,X,$5);print $5 OFS A[$2]}' b_file FS="[][]" c_file

Output will be as follows.

tom  DAR.V3.20150209.1.CSV
charlie  DAR.V3.20150209.1.CSV

Thanks,
R. Singh

7

Hi R.Singh,

I tried the following, but received a lot more of the content of the source file.

/usr/xpg4/bin/awk 'FNR==NR{A[$1]=$2;next} ($2 in A){sub(/.*user /,X,$5);sub(/ from/,X,$5);print $5 OFS A[$2]}' b_file FS="[][]" /var/adm/sftplog

Here is a sample of what was received.

brcp  DAR.V3.20150209.1.CSV
 received client version 3 DAR.V3.20150209.1.CSV
 realpath "." DAR.V3.20150209.1.CSV
 opendir "outbound/" DAR.V3.20150209.1.CSV
 closedir "outbound/" DAR.V3.20150209.1.CSV
 stat name "outbound/DAR.V3.20150209.1.CSV" DAR.V3.20150209.1.CSV
 open "outbound/DAR.V3.20150209.1.CSV" flags READ mode 0666 DAR.V3.20150209.1.CSV
 close "outbound/DAR.V3.20150209.1.CSV" bytes read 9627450 written 0 DAR.V3.20150209.1.CSV
 stat name "outbound/DDR.V2.20150209.1.CSV" DAR.V3.20150209.1.CSV
 open "outbound/DDR.V2.20150209.1.CSV" flags READ mode 0666 DAR.V3.20150209.1.CSV
 close "outbound/DDR.V2.20150209.1.CSV" bytes read 16393726 written 0 DAR.V3.20150209.1.CSV
brcp  DAR.V3.20150209.1.CSV
jpmcc  DAR.V3.20150209.1.CSV
 received client version 3 DAR.V3.20150209.1.CSV
 realpath "." DAR.V3.20150209.1.CSV
 realpath "/outbound" DAR.V3.20150209.1.CSV
 stat name "/outbound" DAR.V3.20150209.1.CSV
 stat name "/outbound/." DAR.V3.20150209.1.CSV
 opendir "/outbound/." DAR.V3.20150209.1.CSV
 closedir "/outbound/." DAR.V3.20150209.1.CSV
 open "/outbound/DAR.V3.20150209.1.CSV" flags READ mode 0666 DAR.V3.20150209.1.CSV
 close "/outbound/DAR.V3.20150209.1.CSV" bytes read 9627450 written 0 DAR.V3.20150209.1.CSV
jpmcc  DAR.V3.20150209.1.CSV
ORG116  DAR.V2.20150210.1.CSV
 received client version 3 DAR.V2.20150210.1.CSV
 realpath "." DAR.V2.20150210.1.CSV

My original had this to isolate the the specific record on the file. Can I include this anywhere to get the record I need?

$5 ~ st && /closed/ && /user/ && !/ORG/  && !/LE/

---------- Post updated 02-13-15 at 12:04 PM ---------- Previous update was 02-12-15 at 12:12 PM ----------

Hi R.Singh,
The file that contains the records to be evaluated in an another file is

ftp01$ less b_file
11977 DAR.V3.20150209.1.CSV
3295 DAR.V3.20150209.1.CSV

The datafile that needs to be evaluated to extract the records needed is formatted like so. Please let me know if you want me to provide in an attachment.

mdadmin@ftp01$ grep 11977 /var/adm/sftplog
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] session opened for local user tom from [111.11.11.111]
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] received client version 3
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] realpath "."
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] opendir "outbound/"
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] closedir "outbound/"
Feb 11 04:29:14 ftp01 sftp-server[11977]: [ID 800047 local1.info] stat name "outbound/DAR.V3.20150209.1.CSV"
Feb 11 04:29:14 ftp01 sftp-server[11977]: [ID 800047 local1.info] open "outbound/DAR.V3.20150209.1.CSV" flags READ mode 0666
Feb 11 04:30:00 ftp01 sftp-server[11977]: [ID 800047 local1.info] close "outbound/DAR.V3.20150209.1.CSV" bytes read 9627450 written 0
Feb 11 04:30:00 ftp01 sftp-server[11977]: [ID 800047 local1.info] stat name "outbound/DDR.V2.20150209.1.CSV"
Feb 11 04:30:00 ftp01 sftp-server[11977]: [ID 800047 local1.info] open "outbound/DDR.V2.20150209.1.CSV" flags READ mode 0666
Feb 11 04:31:27 ftp01 sftp-server[11977]: [ID 800047 local1.info] close "outbound/DDR.V2.20150209.1.CSV" bytes read 16393726 written 0
Feb 11 04:31:27 ftp01 sftp-server[11977]: [ID 800047 local1.info] session closed for local user tom from [111.11.11.111]

mdadmin@ftp01$ grep 3295 /var/adm/sftplog
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] session opened for local user charlie from [111.11.11.111]
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] received client version 3
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] realpath "."
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] realpath "/outbound"
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] stat name "/outbound"
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] stat name "/outbound/."
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] opendir "/outbound/."
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] closedir "/outbound/."
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] open "/outbound/DAR.V3.20150209.1.CSV" flags READ mode 0666
Feb 11 05:14:22 ftp01 sftp-server[3295]: [ID 800047 local1.info] close "/outbound/DAR.V3.20150209.1.CSV" bytes read 9627450 written 0
Feb 11 05:14:22 ftp01 sftp-server[3295]: [ID 800047 local1.info] session closed for local user charlie from [111.11.11.111]

Just to confirm, I would like to maintain the relationship between the two fields in b_file and print in a new file.

tom DAR.V3.20150209.1.CSV
charlie DAR.V3.20150209.1.CSV
8

Hi all,
Any assistance or suggestions on the above is much appreciated. I have read more and tried different code, but cannot obtain the desired result.

Thanks.

9

As much as I would like to help - I can't. Neither the verbal specification nor the data samples nor the code snippets given - all spread over several posts - give me a chance to understand what you're after.

Mayhap a step back might help; start over with a new, precise, careful specification seconded by accurate data samples, both input and desired output, and improved code snippets that would really work...?

10

From the file content below which is from a sftplog I need extract the user and the file name into a file. I would like to use nawk to do so, but am open to any other menthods. The only way to maintain the relationship between user and file is by the session ID which is in brackets.

  1. The user is identified after the word
    text user
    .
  2. The file appears after the string
    text close "/outbound/
    or
    text close "outbound/
  3. The session IDs below are
    text 11977
    and
    text 3295 
mdadmin@ftp01$ grep 11977 /var/adm/sftplog
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] session opened for local user tom from [111.11.11.111]
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] received client version 3
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] realpath "."
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] opendir "outbound/"
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] closedir "outbound/"
Feb 11 04:29:14 ftp01 sftp-server[11977]: [ID 800047 local1.info] stat name "outbound/DAR.V3.20150209.1.CSV"
Feb 11 04:29:14 ftp01 sftp-server[11977]: [ID 800047 local1.info] open "outbound/DAR.V3.20150209.1.CSV" flags READ mode 0666
Feb 11 04:30:00 ftp01 sftp-server[11977]: [ID 800047 local1.info] close "outbound/DAR.V3.20150209.1.CSV" bytes read 9627450 written 0
Feb 11 04:30:00 ftp01 sftp-server[11977]: [ID 800047 local1.info] stat name "outbound/DDR.V2.20150209.1.CSV"
Feb 11 04:30:00 ftp01 sftp-server[11977]: [ID 800047 local1.info] open "outbound/DDR.V2.20150209.1.CSV" flags READ mode 0666
Feb 11 04:31:27 ftp01 sftp-server[11977]: [ID 800047 local1.info] close "outbound/DDR.V2.20150209.1.CSV" bytes read 16393726 written 0
Feb 11 04:31:27 ftp01 sftp-server[11977]: [ID 800047 local1.info] session closed for local user tom from [111.11.11.111]

mdadmin@ftp01$ grep 3295 /var/adm/sftplog
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] session opened for local user charlie from [111.11.11.111]
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] received client version 3
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] realpath "."
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] realpath "/outbound"
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] stat name "/outbound"
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] stat name "/outbound/."
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] opendir "/outbound/."
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] closedir "/outbound/."
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] open "/outbound/DAR.V3.20150209.1.CSV" flags READ mode 0666
Feb 11 05:14:22 ftp01 sftp-server[3295]: [ID 800047 local1.info] close "/outbound/DAR.V3.20150209.1.CSV" bytes read 9627450 written 0
Feb 11 05:14:22 ftp01 sftp-server[3295]: [ID 800047 local1.info] session closed for local user charlie from [111.11.11.111]

The desired output is

tom DAR.V3.20150209.1.CSV
tom DDR.V2.20150209.1.CSV
charlie DAR.V3.20150209.1.CSV
11

You could try something like this:

awk '/ session .* user /{u=$(NF-2)} / close /{m=split($0,F,/["\/]/); print u, F[m-1]}' logfile

But this requirement seeds to be different from the one in post #1

12

Yes. I tried to simply my issue by putting the session ID and file name in a file and then using the contents of that file to extract the user name for the session ID. I was then struggling to use an array across two files to make sure the user was attributed to the correct file name. In the process I confused everyone so I eliminated the first step.

Unfortunately, The above suggestion is not producing all of the file names.

This is the closest I have come.

  1. Create a file with session ID and file_name -

---------- Post updated at 10:09 AM ---------- Previous update was at 09:59 AM ----------

This is the closest I have come.

  1. Create a file with session ID and file_name -
ftp02$ less /tmp/b_file
11977 DAR.V3.20150209.1.CSV
11977 DDR.V2.20150209.1.CSV
3295 DAR.V3.20150209.1.CSV
  1. Loop through isolated record to get user id
for i in `cat |nawk '{ print $1 }' /tmp/b_file`
do
nawk -v st=$i '$5 ~ st && /closed/ && /user/ && !/ORG/  && !/LE/ { for (x=1;x<=NF;x++) if ($x~"user") print $(x+1) } ' /var/adm/sftplog >> /tmp/log.out
done
  1. Here is the sample of sftlog.
mdadmin@ftp01$ grep 11977 /var/adm/sftplog
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] session opened for local user tom from [111.11.11.111]
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] received client version 3
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] realpath "."
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] opendir "outbound/"
Feb 11 04:29:13 ftp01 sftp-server[11977]: [ID 800047 local1.info] closedir "outbound/"
Feb 11 04:29:14 ftp01 sftp-server[11977]: [ID 800047 local1.info] stat name "outbound/DAR.V3.20150209.1.CSV"
Feb 11 04:29:14 ftp01 sftp-server[11977]: [ID 800047 local1.info] open "outbound/DAR.V3.20150209.1.CSV" flags READ mode 0666
Feb 11 04:30:00 ftp01 sftp-server[11977]: [ID 800047 local1.info] close "outbound/DAR.V3.20150209.1.CSV" bytes read 9627450 written 0
Feb 11 04:30:00 ftp01 sftp-server[11977]: [ID 800047 local1.info] stat name "outbound/DDR.V2.20150209.1.CSV"
Feb 11 04:30:00 ftp01 sftp-server[11977]: [ID 800047 local1.info] open "outbound/DDR.V2.20150209.1.CSV" flags READ mode 0666
Feb 11 04:31:27 ftp01 sftp-server[11977]: [ID 800047 local1.info] close "outbound/DDR.V2.20150209.1.CSV" bytes read 16393726 written 0
Feb 11 04:31:27 ftp01 sftp-server[11977]: [ID 800047 local1.info] session closed for local user tom from [111.11.11.111]

mdadmin@ftp01$ grep 3295 /var/adm/sftplog
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] session opened for local user charlie from [111.11.11.111]
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] received client version 3
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] realpath "."
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] realpath "/outbound"
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] stat name "/outbound"
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] stat name "/outbound/."
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] opendir "/outbound/."
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] closedir "/outbound/."
Feb 11 05:14:11 ftp01 sftp-server[3295]: [ID 800047 local1.info] open "/outbound/DAR.V3.20150209.1.CSV" flags READ mode 0666
Feb 11 05:14:22 ftp01 sftp-server[3295]: [ID 800047 local1.info] close "/outbound/DAR.V3.20150209.1.CSV" bytes read 9627450 written 0
Feb 11 05:14:22 ftp01 sftp-server[3295]: [ID 800047 local1.info] session closed for local user charlie from [111.11.11.111]
  1. With the above the contents of log.out will be
tom
tom
charlie

I want it to be -

tom DAR.V3.20150209.1.CSV
tom DDR.V2.20150209.1.CSV
charlie DAR.V3.20150209.1.CSV

How can I get the file_name or field $2 lncluded in log.out?

---------- Post updated 02-18-15 at 09:56 AM ---------- Previous update was 02-17-15 at 10:09 AM ----------

Hi all,
Is this clear, or am I still confusing everyone/anyone? I realize I am potentially annoying everyone with this problem. Any assistance is tremendously appreciated.

Sean

13

Try

akshay@Aix:/tmp$ cat parse.awk
FNR==NR{
	a[$1]= $1 in a ? a[$1] SUBSEP $2 : $2
	next
}
{
	for( i in a )
	{ 
	 if( $5 == "sftp-server["i"]:" && /closed/ && /user/ && !/ORG/ && !/LE/ )
	 {
		split(a,A,SUBSEP)
		for(j in A)
		{
			print $14,A[j]
		} 
	 }
	}
}

akshay@Aix:/tmp$ awk -f parse.awk b_file sftp_log 
tom DAR.V3.20150209.1.CSV
tom DDR.V2.20150209.1.CSV
charlie DAR.V3.20150209.1.CSV
1 Like
14

Akshay,
Thanks so much!! This is solved.
:b:

15

Scrutinzer's suggestion produced exactly your desired output when I ran it with your samples. Did you use different samples? Why, then, didn't you post them and show us which output lines were missing?

What exactly is this file for? Where/how is it produced? Does it need to persist or is it a temp file?

The confusion is due to the unprecise specification / problem statement, unclear logics as well as missing or incomplete error messages.
Describe the request, backed by input and desired output samples, and the logics behind it in simple words. On top, you should accept/consider proposals that dismiss your approach and present their own with a correct result.