Copying section of file based on search criteria

Hi Guru's,

I am new to unix scripting. I have a huge file with user details in it(file2) and I have another file with a list of users(file1). Script has to search a user from file1 and get all the associated lines from file2.

Example:

fiel1:
cn=abc
cn=DEF
cn=xyx

File 2:

dn: cn=abc,ou=111,dc=111,dc=111,dc=com
uid: abc
userPassword:: lQ0E9sPsdfQ==

dn: cn=ghi,ou=111,dc=111,dc=111,dc=com
uid: ghi
userPassword:: PZHc9PsdfQ==

dn: cn=xyz,ou=111,dc=111,dc=111,dc=com
uid: xyz
userPassword:: PZHc9PsdfQ==

dn: cn=mno,ou=111,dc=111,dc=111,dc=com
uid: mno
userPassword:: PZHc9P23Q==

dn: cn=pqr,ou=111,dc=111,dc=111,dc=com
uid: pqr
userPassword:: PZHc9P23Q==

dn: cn=DEF,ou=111,dc=111,dc=111,dc=com
uid: DEF
userPassword:: PZHc9P23Q==

output file:

dn: cn=abc,ou=111,dc=111,dc=111,dc=com
uid: abc
userPassword:: lQ0E9sPsdfQ==

dn: cn=xyz,ou=111,dc=111,dc=111,dc=com
uid: xyz
userPassword:: PZHc9PsdfQ==

dn: cn=DEF,ou=111,dc=111,dc=111,dc=com
uid: DEF
userPassword:: PZHc9P23Q==

Thanks in advance for the help.
Sam

Try this one liner, It will create a file(file_3) containing the lines from file_2 for id from file_1:

IFS='='; while read cn id;do perl -lne "print if /^dn: cn=${id}.*$/ .. /^$/" file_2; done <file_1 >file_3
awk 'NR == FNR {split($0, a, FS); next} {split($0, c, " |,"); for(x in a) {if(a[x] == c[2]) print $0; continue}}' FS='\n' ORS='\n\n' RS= file1 file2

Hi Srinishoo,

I am getting an empty file when I use the awk that you have provided.

Thanks,
Sam

---------- Post updated at 12:47 AM ---------- Previous update was at 12:45 AM ----------

Hi Spacebar,

Thanks for the update. The script is failing with the below error when I run it against the big file.

Bareword found where operator expected at -e line 1, near "/^dn: cn=n/a"
        (Missing operator before a?)
syntax error at -e line 1, near "/^dn: cn=n/a"
Execution of -e aborted due to compilation errors.

Thanks,
Sam

You would make your life easier if you didn't put spaces in filenames (like File 2 ), the filename fiel1 looks like a typo, and your output show lines matching cn=xyz , but the corresponding line in fiel1 is cn=xyx ; not cn=xyz .

You could try something like:

awk -F '[ ,]' '
FNR == NR {
	cn[$1]
	next
}
$1 == "dn:" {
	copy = ($2 in cn)
}
copy {	print
}' fiel1 "File 2"

which (with the sample input files you provided) produces the output:

dn: cn=abc,ou=111,dc=111,dc=111,dc=com
uid: abc
userPassword:: lQ0E9sPsdfQ==

dn: cn=DEF,ou=111,dc=111,dc=111,dc=com
uid: DEF
userPassword:: PZHc9P23Q==

If you want to use this on a Solaris/SunOS system, use /usr/xpg4/bin/awk , /usr/xpg6/bin/awk , or nawk instead of /usr/bin/awk .

2 Likes

Hi Don,

Thanks for the update. How do I execute this script? I tried by moving the code to a sh file but it failed.

Regards,
Sam

---------- Post updated at 03:15 AM ---------- Previous update was at 02:54 AM ----------

Hi Don,

I was able to execute the script. code works great. How can we ignore case sensitivity in the search. File1 may hold the value in small letter but file2 may have the value in capitol's. The output should be in all small letters.

Example:

File1:
cn=abc
cn=DEF
cn=xyz
File2:
dn: cn=ABC,ou=111,dc=111,dc=111,dc=com
uid: ABC
userPassword:: lQ0E9sPsdfQ==

dn: cn=ghi,ou=111,dc=111,dc=111,dc=com
uid: ghi
userPassword:: PZHc9PsdfQ==

dn: cn=xyz,ou=111,dc=111,dc=111,dc=com
uid: xyz
userPassword:: PZHc9PsdfQ==

dn: cn=mno,ou=111,dc=111,dc=111,dc=com
uid: mno
userPassword:: PZHc9P23Q==

dn: cn=pqr,ou=111,dc=111,dc=111,dc=com
uid: pqr
userPassword:: PZHc9P23Q==

dn: cn=def,ou=111,dc=111,dc=111,dc=com
uid: def
userPassword:: PZHc9P23Q==
Output:
dn: cn=abc,ou=111,dc=111,dc=111,dc=com
uid: abc
userPassword:: lQ0E9sPsdfQ==

dn: cn=xyz,ou=111,dc=111,dc=111,dc=com
uid: xyz
userPassword:: PZHc9PsdfQ==

dn: cn=def,ou=111,dc=111,dc=111,dc=com
uid: def
userPassword:: PZHc9P23Q==

Thanks for the help.
Srikanth

Try this:

awk -F '[ ,]' '
FNR == NR {
	cn[tolower($1)]
	next
}
$1 == "dn:" {
	copy = (tolower($2) in cn)
}
copy {	print ($1 == "userPassword::" ? $0 : tolower($0))
}' File1 File2
1 Like

Hi Don,

Thanks for the update. I am running into a small issue where one line in the file is getting converted to lower case. How can I ignore a line to get converted to a lower case.

File1:
cn=abc
cn=DEF
cn=xyz
File2:
dn: cn=ABC,ou=111,dc=111,dc=111,dc=com
uid: ABC
Xid: abcDEF123dDFV
userPassword:: lQ0E9sPsdfQ==

dn: cn=ghi,ou=111,dc=111,dc=111,dc=com
uid: ghi
Xid: dfsf3242SDS
userPassword:: PZHc9PsdfQ==

dn: cn=xyz,ou=111,dc=111,dc=111,dc=com
Xid: rew32DSDFS234
uid: xyz
userPassword:: PZHc9PsdfQ==

dn: cn=mno,ou=111,dc=111,dc=111,dc=com
uid: mno
Xid: DFDre23SD
userPassword:: PZHc9P23Q==

dn: cn=pqr,ou=111,dc=111,dc=111,dc=com
uid: pqr
Xid: 54we3reWEdfse
userPassword:: PZHc9P23Q==

dn: cn=def,ou=111,dc=111,dc=111,dc=com
uid: def
userPassword:: PZHc9P23Q==

Xid: AB34dSCrevtT3

Output:
dn: cn=abc,ou=111,dc=111,dc=111,dc=com
uid: abc
Xid: abcDEF123dDFV
userPassword:: lQ0E9sPsdfQ==

dn: cn=xyz,ou=111,dc=111,dc=111,dc=com
Xid: rew32DSDFS234
uid: xyz
userPassword:: PZHc9PsdfQ==

dn: cn=def,ou=111,dc=111,dc=111,dc=com
uid: def
userPassword:: PZHc9P23Q==
Xid: AB34dSCrevtT3

The line starting with Xid: should not get converted to lower case.

Thanks,
Sam

Hi Sam,In message #6 in this thread, you said that the output you wanted should be all lowercase. I ignored that request for the userPassword:: lines because I didn't expect password comparisons to work correctly if the (presumably) encrypted password had been converted to single case.

Can you find the line in the script I gave you:

awk -F '[ ,]' '
FNR == NR {
	cn[tolower($1)]
	next
}
$1 == "dn:" {
	copy = (tolower($2) in cn)
}
copy {	print ($1 == "userPassword::" ? $0 : tolower($0))
}' File1 File2

that translates the output to lowercase for every output line except lines that have userPassword:: in the first field on the line?

After looking at the man page for awk, what change do think would be needed to make that line of code convert all input lines except those that have userPassword:: and those that have Xid: in the first field to lowercase?

Hi Don,

Thanks for your time. Below is the line of code that does not convert the user password to lower case.

copy {	print ($1 == "userPassword::" ? $0 : tolower($0))

I have to use a similar code in the script but I am not sure where to add that. I tried it by adding but all the lines were getting duplicated.

copy {	print ($1 == "Xid:" ? $0 : tolower($0))

Sam

Good, you're learning. You found the right line.

Instead of duplicating that line to look for Xid lines, try changing it to:

copy {  print ($1 == "userPassword::" || $1 == "Xid:" ? $0 : tolower($0))

Doing it this way, there is only one print statement for each line that you want to print. Lines with either of those two values in the first field ( $1 == "userPassword::" || $1 == "Xid:" ) are printed as they appeared in the input file ( $0 ) while all other lines are converted to lower case ( tolower($0) ) before being printed.