PERL or SHELL Scrript to search in Directories by taking line by line from a text file

pasam · December 16, 2011, 4:02am

Unix box server version
***********

>uname  -r
B.11.00

>echo $SHELL
/usr/bin/ksh

--> in this server, I have the path like /IMbuild/dev/im0serv1
---> in that directory I have the folders startup(.jsp files nearly 100 jsp's ) and scripts(contains .js files nearly 100 files)
So, in .jsp,.js files there are two methods calling the Application ID like
returnMMCObjectContent("chiru");//
getDCCObjectContent("chiru");
like chiru I need to search for 22,000 articles.
till now I am using one command

find . -name  -exec  grep -l "chiru" {}\; | -exec  grep -n  "chiru" {}\;//

then I vl get the result like that method where ever the chiru was used and the file name with path like ../start/loutout.js,../filenames/login.jsp

so with this I can make a note that chiru was used name bcoz reference files were displayed.
If any reference file not came,that means chiru was unUsed name....

This whole part is MANUAL kind of thing I have to do CTRL+C and CTRL+V ..........for 22,000 times

MY REQUIREMENT IS:
--------------------
I want to take 10,000 names in a text file (test.txt) and directly pass as input file to the script and the script has to read the file line by line and searching in the sub directories(start and scripts) ,if reference has found,then those names has to go into one output file with that name and reference file(.jsp, .js) (out1.txt) , if any reference was not found then those names should go into another output file(out2.txt).
So I am asking for that script ;

either perl script or any UNIX shell script that can work in version:B.11.00

Another Unix box server
***********

>uname -r
2.6.9-89.0.20.ELsmp

>echo  $SHELL
ksh:echo: not found

This is different version of UNIX
The same script for the above server I want to execute in this server....

Thanks in Advance ........................

m.d.ludwig · December 16, 2011, 10:17am

I have to admit some puzzlement from:

This will run man grep (linux) on every file and directory found - one at a time. Also, the output of these greps is then piped to -exec, which I am pretty sure does not exist. Restricting this to 'jsp' and 'js' files would help:

find . -type f \( -name '*.jsp' -o -name '*.js' \) -exec ....

And instead of using find to invoked grep on each file, use man xargs (linux):

find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 | xargs -0 grep -n chiru

The two greps are not needed, because grep -l regex filename will read the entire file it it does not find regex, and grep -n regex filename reads the entire file to find matches. By not using two greps, you are saving a extra (partial) read of files that contain regex.

Now the problem of finding matches of 10,000 patterns. Re-reading each file 10,000 times would be time consuming at best. But given that the pattern can occur anywhere in the file, that is what you are stuck with:

FILE=$(mktemp)
trap "rm -f ${FILE}" EXIT

# built list of files
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 > ${FILE}

cat "${@}" | while read pattern; do
    xargs -0 grep -n "${pattern}" < ${FILE}
done

Now if the patterns occur only in lines like:

then you use some perl to make this run faster.

#! /path/to/perl
# USAGE: patternfind patternlist [filelist]...

my $patternlist = shift @ARGV;
die unless defined $patternlist;

my %OBJECTS;
# key = object
# value = list of file/line pairs

while (<>) {
    chomp;
    unless (-e $_) { print STDERR $_, ': no such file or directory'; next; }
    unless (-f $_) { print STDERR $_, ': not a file';                next; }
    unless (-r $_) { print STDERR $_, ': permission denied';         next; }

    open FH, '<', $_ or die $_;

    $. = 0;

    while (<FH>) {
        my ($object) = m{ObjectContent\s*\(\s*"([^"]+)"\s*\)};

        if (defined $object) {
            push @{$OBJECT{$object}}, { file => $_, line => $. };
        }
    }

    close FH;
}

open FH, '<', $patternfile or die $patternfile;

$\ = "\n";
$, = '';

while (<FH>) {
    chomp;

    if (defined $OBJECT{$_}) {
        foreach my $X (@$OBJECT{$_}) {
            print $_, ': ', $X->{file}, '(', $X->{line}, ')';
        }
    }
}

close FH;

This will read in every file in filelist (or read from stdin), search every file for the 'ObjectContents' function calls - saving the filename and location of every pattern found. Once that is done, the pattern file is read in and checked for matching patterns.

pasam · December 16, 2011, 11:38am

Thanks Ludwig,
Actually I am new to this shell script and PERL and I want the file content which have to print line by line and reference file (.jsp, .js,.xsl )
For Example:
if in the input text file app.txt ,4 names are there like below

then I want the output file containing the content like

Starting the  New Name:******************
pasam
/start/login.jsp
/start/header.js
Starting the  New Name:******************
nithin
/start/logout.jsp
/start/header.js
Starting the  New Name:******************
 anil
Starting the  New Name:******************
eric
/scripts/log/login.xsl

Thanks a lot Ludwig.

pasam · December 19, 2011, 10:15am

Hi Ludwig,
when I tried this command,
Code:

  find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 | xargs -0 grep -n chiru
I got the below errors :
find : missing conjunction
xargs: unknown option: -0

when I tried the below script by creating a file with name Plan.sh
Code:

  FILE=$(mktemp) trap "rm -f ${FILE}" EXIT  # built list of files find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 > ${FILE}  cat "${@}" | while read pattern; do     xargs -0 grep -n "${pattern}" < ${FILE} done

in the above code, I have given Input filename: testfile in place of < $ {FILE} here.

when I tried to run the script by giving >sh plan.sh, I got the following errors:

find: bad option -print0
plan.sh[8]: "${ @ }" the specified substitution is not valid for this command

when I tried the Perl code by replacing the inputfile name (testfile) instead of FH

#!/usr/bin/perl
# USAGE: patternfind patternlist [filelist]...

my $patternlist = shift @ARGV;
die unless defined $patternlist;

my %OBJECTS;
# key = object
# value = list of file/line pairs

while (<>) {
    chomp;
    unless (-e $_) { print STDERR $_, ': no such file or directory'; next; }
    unless (-f $_) { print STDERR $_, ': not a file';                next; }
    unless (-r $_) { print STDERR $_, ': permission denied';         next; }

    open FH, '<', $_ or die $_;

    $. = 0;

    while (<FH>) {
        my ($object) = m{ObjectContent\s*\(\s*"([^"]+)"\s*\)};

        if (defined $object) {
            push @{$OBJECT{$object}}, { file => $_, line => $. };
        }
    }

    close FH;
}

open FH, '<', $patternfile or die $patternfile;

$\ = "\n";
$, = '';

while (<FH>) {
    chomp;

    if (defined $OBJECT{$_}) {
        foreach my $X (@$OBJECT{$_}) {
            print $_, ': ', $X->{file}, '(', $X->{line}, ')';
        }
    }
}

close FH;

Then I got the following errors:

syntax error plan1.pl  line 39 at EOF
Execution of plan1.pl aborted due to compilation errors

when You are writing the solution for this , plz give the input filename with INPUT and Output filename with OUTPUT .

Thanks.

[/COLOR]It is very urgent .Please help me in this .

m.d.ludwig · December 19, 2011, 9:26pm

Hi there.

You wrote:

pasam:

When I tried this command,

find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 | xargs -0 grep -n chiru

I got the below errors :

find : missing conjunction
xargs: unknown option: -0

I made a poor assumption that HP-UX 11 was more similar to a linux system, where find does have a -print0 option. But since you are running a system-V unix, you can use:

find . -type f \( -name '*.jsp' -o -name '*.js' \) -print | xargs grep -n chiru

which will work provided none of your file or directories contains whitespace or quotes -- see this for more information. If your files might contain whitespace or quotes, you will have to use:

find . -type f \( -name '*.jsp' -o -name '*.js' \) -print | sed -e 's/[ "'"'"']/\\&/g' | xargs grep -n chiru

to escape the separators.

Can you please explain what you mean by this?
Also, what version of perl are you running?

pasam · December 20, 2011, 2:28am

Hi, Ludwing,
in one of my UNIX box
which I have the Perl version 5.8.2 built for PA-RISC1.1-thread-multi(with 25 registered patches).In this box I want to search for .jsp and .js reference file(with paths) for the Object names (20000 object names read line by line from a text file and to get the reference file along with Object Name )
in another UNIX box
I have the Perl version, v5.8.5 built for i386-linux-thread-multi
In this box, I want to search for reference files of .xsl, .xml for the ObjectNames.
The requirement is same as above.

I need the PERL code for the above two versions :
the requirement is: The code has to read the text file(which is having 20000 object names ) and print the OUTPUT file with reference file.

The output file have to print like this
ObjectName1
../startup/registration/login.jsp
../script/code/val.js
ObjectName2
../.................jsp
../..................jsp
../...............js

Thanks a lot .

Skrynesaver · December 20, 2011, 4:30am

Something like the following?

#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
# usage $0 <directory list>
open (my $input, '<', "names.txt")
    ||die "Couldn't open names.txt file:\n\t$!\n";
while (<$input>){
    print "$_\n";
    find(sub {wanted($_)} , @ARGV);
}

sub wanted{
    if (/\.jsp?$/){
        my $regex=shift;
        open(my $current_file, '<', $_) || print "Could not read $_:\n\t$!\n";
        FILE:
        while(<$current_file>){
            if(/$regex/){
                print "$File::Find::name";
                next FILE;
            }
        }
        close($current_file);
    }
}

As far as I know File::Find is a core module in both Perl versions you list.

pasam · December 21, 2011, 3:37am

Thanks Skrynesaver. I have tried with Your script as following.

#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
# usage $0 startup
open (my $input, '<', "names.txt")
    ||die "Couldn't open names.txt file:\n\t$!\n";
while (<$input>){
    print "$_\n";
    find(sub {wanted($_)} , @ARGV);
}

sub wanted{
    if (/\.jsp?$/){
        my $regex=shift;
        open(my $current_file, '<', $_) || print "Could not read $_:\n\t$!\n";
        FILE:
        while(<$current_file>){
            if(/$regex/){
                print "$File::Find::name";
                next FILE;
            }
        }
        close($current_file);
    }
}

just I changed "startup" at the place of <directory list> and executed by placing some Object names in names.txt file.

I got the following error
-------------------------------
chiru (the first line in names.txt file)

Invalid top directory at /opt/perl/lib/5.8.2/File/Find.pm, line1

and in other server the error is:
----------------
chiru

invalid top directory at /usr/lib/perl5/5.8.5/File/Find.pm line 585,<$input> line 1

So, please assist me to get out of this errors and You have mentioned only for .jsp, but I want the reference for .js (javascript )files and .xsl files also. and in the directory list plz mention How to give <directorylist>....I am running in the im0anku1(directory) ,in this directory I have the scripts and startup subdirectories.

Please help me .

Thanks.

pasam · December 26, 2011, 4:45am

Thanks Skrynesaver

when I run this script I got following error:
------------------------------
chiru (from input text file)

Invalid top directory at /opt/perl/lib/5.8.2/File/Find.pm line 568 ,<$input> line1.

I have the doubt about this place in the code
<directory list>

pasam · December 29, 2011, 2:30am

:wall: