fast search and replace in all files

Shell Programming and Scripting
1

Hi

I need to find one string in all files and replace tht string with blank space and need to redirect all the files into the same directory again.

now i am using

find ./ -name "*.dmp" | xargs perl -pi -e 's/\\N//g' | sed 's/.$//g'

but now its not redirrecting properly .

its taking too much time for this operation.

Is there any way to search and replace very fast in all thousand files quickly without looping.

2

This removes \N, replaces it with " "and then chops off the last character of each line and inserts a space there. Do you know about dos2unix (or dos2ux)?

find ./ -name "*.dmp" | xargs perl -pi -e 's/\\N/ /g; s/.$/ /g'

If you want it to run faster try using background processes:

cnt=0
find ./ -name "*.dmp" |\
while read file do
  perl -pi -e 's/\\N/ /g; s/.$/ /g'  $file  &
  cnt=$cnt+1
  z=$(( $cnt % 10 ))
  if [[ $z -eq 0 ]] ; then
     wait
  fi
done 
wait

This runs ten process at the same time in background. And then waits for completion. The code you proivided does not replace the characters with a space

3

Hi its giving error message with the use above code
syntax error at line 7: ']' unexpected

4

I've edited the original posting by Jim.

5

Hi

Is it working?

Is it possible to replace the below sed syntax also to the above xargs perl
scenario
sed -e :a -e '/;$/!N;s/\n//; ta' -e 's/;$//' file

6

Really, the requirements are ridiculous. There is no way to search and replace anything in thousands of files quickly. You might find some ways to faster than others, but nothing is going to be a magic bullet. I suggest you try perls inplace editor, it might be faster than sed although I don't really know.

7

Hi kevin

Do u know how to replace sed with perl in the below syntax

sed -e :a -e '/;$/!N;s/\n//; ta' -e 's/;$//' file

8

I am not real good with command line syntax but see if this works:

perl -pi -e '!/;$/ && chomp;' file

9

Hi Kevin,

The above option is not working properly..can u have a look at it

10

"Not working properly" is too vague, describe what happens when you try the code. What output do you get? Do you get error messages?

11

I didnt get any error message but its not replacing file

12

works for me (Linux, not sure what version though) when I tested it with your sample data:

ABC|FGH|HJK|JKK;
BHJ|AAA|BBB|L
NNNN|JJJJ|LLLL;
JJJJJJ;
out put file consists of
ABC|FGH|HJK|JKK
BHJ|AAA|BBB|L NNNN|JJJJ|LLLL
JJJJJ

output:

ABC|FGH|HJK|JKK;
BHJ|AAA|BBB|LNNNN|JJJJ|LLLL;
JJJJJJ;
out put file consists ofABC|FGH|HJK|JKKBHJ|AAA|BBB|L NNNN|JJJJ|LLLLJJJJJ

13

Hi Kevin,

Is it removing the semicolon at the end of each line.For me its not removing semicolon.I am using HP Unix.

14

No it is no removing the semi-colon. I missed that requirement when I wrote the code. Try this instead:

perl -pi -e's/;$// || chomp;' file
15

Hi Kevin,

Thanks .

Is it possible to apply this for multiple files as
find ./ -name "*.dmp" |xargs perl -pi -e's/;$// || chomp;'

can i apply as above for modifying multiple files in the same directory?

16

You don't need any sed or awk or other shell stuff, you can do it with just perl:

perl -pi -e's/;$// || chomp;' path/to/*.dmp

if you are in the correct directory you can just use .dmp instead of path/to/.dmp

17

Hey , first of all sorry for posting here instead of starting the new thread. I could not find the link for starting a NEW thread.

"HERE GOES MY QUESTION"
can anyone please tell me how to search for a particular string in a file (I know the file name) and then accordingly increase the counter which is pre set to ZERO. and hence showing the total number of times the character/string is found.

18

You could do grep -c ...

19 
awk '/pattern/{
    gsub(/pattern/," ")
}
{ 
    print $0 > FILENAME"-new.dmp" 
    f[FILENAME]=FILENAME"-new.dmp" #probably not here
}
END {
  for ( i in f ){
    cmd = "mv " f " "  i
    system(cmd)
  }
}
' *.dmp
20 
perl -ne '$i++ if /pattern/; END{print "pattern count = $i"}' filename