Remove duplicate line starting with a pattern

Shell Programming and Scripting
1

HI,

I have the below input file

  /* ----------------- cmdsDlyStartFWJ -----------------*/
UNIX_JOB CMDS065J
  RUN ANY
  CMDNAME sleep 5
  AGENT CMDSHP
  USER proddata
  RUN MON,TUE,WED,THU,FRI
  DELAYSUB 02:00
  /* "Triggers daily file watcher jobs" */
  ENVAR STDOUT=/home3/cmdsprod/stdout/cmdsDlyStartFWJ
ENDJOB
 
  /* ----------------- cmdsDlyStartFWJ -----------------*/
UNIX_JOB CMDS065J
  RUN ANY
  CMDNAME sleep 5
  AGENT CMDSHP
  USER proddata
  DELAYSUB 02:00
  /* "Triggers daily file watcher jobs" */
  ENVAR STDOUT=/home3/cmdsprod/stdout/cmdsDlyStartFWJ
ENDJOB
 
  /* ----------------- cmdsDlyStartFWJ -----------------*/
 
FILE_TRIGGER CMDS065J
  RUN ANY
  CMDNAME sleep 5
  AGENT CMDSHP
  USER proddata
  RUN MON,TUE,WED,THU,FRI
  DELAYSUB 02:00
  /* "Triggers daily file watcher jobs" */
  ENVAR STDOUT=/home3/cmdsprod/stdout/cmdsDlyStartFWJ
ENDJOB

UNIX_JOB

or

FILE_TRIGGER

marks the beiginning of a job and

ENDJOB

marks the end. If you see 1st one has two entries for RUN..so what i want is to remove the RUN ANY line if there are two RUN statements

output should look like below

  /* ----------------- cmdsDlyStartFWJ -----------------*/
UNIX_JOB CMDS065J
  CMDNAME sleep 5
  AGENT CMDSHP
  USER proddata
  RUN MON,TUE,WED,THU,FRI
  DELAYSUB 02:00
  /* "Triggers daily file watcher jobs" */
  ENVAR STDOUT=/home3/cmdsprod/stdout/cmdsDlyStartFWJ
ENDJOB
 
  /* ----------------- cmdsDlyStartFWJ -----------------*/
UNIX_JOB CMDS065J
  RUN ANY
  CMDNAME sleep 5
  AGENT CMDSHP
  USER proddata
  DELAYSUB 02:00
  /* "Triggers daily file watcher jobs" */
  ENVAR STDOUT=/home3/cmdsprod/stdout/cmdsDlyStartFWJ
ENDJOB
 
  /* ----------------- cmdsDlyStartFWJ -----------------*/
FILE_TRIGGER CMDS065J
  CMDNAME sleep 5
  AGENT CMDSHP
  USER proddata
  RUN MON,TUE,WED,THU,FRI
  DELAYSUB 02:00
  /* "Triggers daily file watcher jobs" */
  ENVAR STDOUT=/home3/cmdsprod/stdout/cmdsDlyStartFWJ
ENDJOB
2

Try this (may need some polishing):

awk 'gsub (/RUN/, "&") > 1 {sub (/\n *RUN ANY *\n/,"\n")}1' RS= file
3 
nawk 'gsub (/RUN/, "&") > 1 {sub (/\n *RUN ANY *\n/,"\n")}1' RS= cmdsDly_JIL.pull_job_esp
nawk: input record `UNIX_JOB CMDJ000J
  ...' too long
 source line number 1
4

The empty lines contain spaces. Try correcting the file with nawk '!NF{$0=x}1' to remove spaces from empty lines, before trying RudiC' suggestion. So it becomes:

awk '!NF{$0=x}1' file | awk 'gsub (/RUN/, "&") > 1 {sub (/\n *RUN ANY *\n/,"\n")}1' RS=
5

It worked fine.. but it got rid of empty lines, I dont want that to happen. I need them for my formating purpose.

every job should have a blank line before next jon starts

6

You can add ORS='\n\n' at the end