Sed: -e expression #1, char 16: unterminated address regex

Shell Programming and Scripting
1

I am trying to grep for a particular text (Do action on cell BL330) in a text file(sample.gz) which is searched in the content filtered by date+timestamp (2016-09-14 01:09:56,796 to 2016-09-15 04:10:29,719) on a remote machine and finally write the output into a output file on a local machine.

Few details of variables passed as a parameter:

server_id= hostname
first_line_log_file=sample.gz
first_line_date_time=2016-09-14 01:09:56,796
last_log_first_line=2016-09-15 04:10:29,719
do_action_on_cell_2=Do action on cell BL330
 
ssh -q -o "StrictHostKeyChecking no" $server_id "cd /intucell/data/logs/app; zcat $first_line_log_file | sed -rne '/$first_line_date_time/,/$first_log_first_line/p'| zgrep -A10 -i '$do_action_on_cell_2';" >> ./output.log

However, on executing the above i get the below error

sed: -e expression #1, char 62: unterminated address regex

I think the problem is with sed expression, Please suggest a way forward.

2

There seem to be several problems with your above code:

  • shell variable assignment doesn't allow for spaces, so remove those around the "=" sign, and use quoting or escaping for the contents.
  • don't expand (using the $ sign) the variable to be assigned to.
  • within single quotes, no variable expansion is performed, so edit your sed and zgrep parameters. you may need to escape the $ sign within the ssh command line.
  • I'm not sure zgrep makes sense after zcat | sed ... | .
3

Hi RudiC, Thanks for quick reply and correcting the code format.

  • Space in assignment of variable was a typo error, I have corrected it now. The variables given are only for providing more information to my question on the parameters used.

  • I did not understand the 2nd point mentioned.

  • There is a double quote in the code pasted, which i believe is taking care of variable expansion. Also below is another example which works perfectly fine and fetches me desired output:

ssh -q -o "StrictHostKeyChecking no" $server_id "cd /intucell/data/logs/app; zcat $first_line_log_file | tail -1| cut -d' ' -f1-2;"
  • I will make changes to see if [CODE] zcat | zgrep and then [CODE] sed works or not.
4

Hi RudiC, Thanks for quick reply and correcting the code format.

  • Space in assignment of variable was a typo error, I have corrected it now. The variables given are only for providing more information to my question on the parameters used.

  • I did not understand the 2nd point mentioned.

  • There is a double quote in the code pasted, which i believe is taking care of variable expansion. Also below is another example which works perfectly fine and fetches me desired output:

ssh -q -o "StrictHostKeyChecking no" $server_id "cd /intucell/data/logs/app; zcat $first_line_log_file | tail -1| cut -d' ' -f1-2;"
  • I will make changes to see if
 zcat | zgrep

and then

 sed

works or not.

---------- Post updated at 08:42 PM ---------- Previous update was at 08:18 PM ----------

Just a quick update:

I have tried the below two options on remote server (hostname) directly on the file and results as below

  1. zcat | sed | zgrep -> This works fine.
zcat sample.gz | sed -rne '/2016-09-14 01:09:56,796/,/2016-09-14 01:46:56,438/p' | zgrep -A10 -i "Do action on cell BL330"
  1. zcat | zgrep | sed -> This does not print any output
zcat sample.gz | zgrep -A10 -i "Do action on cell BL330" | sed -rne '/2016-09-14 01:09:56,796/,/2016-09-14  01:46:56,438/p'

I suspect that the problem is with sed expression and escaping the correct characters.
Please have a look and suggest.

5

Hi,

  1. Worked because sed prints range of lines and in those lines grep looks for " "Do action on cell BL330" and print 10 lines after matched pattern.
  2. Grep first prints only 10 lines . Sed looks for range of lines from the output of grep , may be it is not present.
    Check the output of grep.

Note : you can use
zgrep <string to look> sample.gz
Zcat is not required , IMHO.

6

Hi greet_sed,

Thanks for the reply.

I have tried like you mentioned but does not print any output:
without zcat:

zgrep -A10 -i "Do action on cell BLA330" sample.gz | sed -rne '/2016-09-14 01:09:56,796/,/2016-09-14 01:46:56,438/p'

sed -rne '/2016-09-14 01:09:56,796/,/2016-09-14 01:46:56,438/p' sample.gz | zgrep -A10 -i "Do action on cell BLA330"

The above commands are directly executed on the remote machine and hence the characters to be escaped are not taken into consideration.

My original problem lies in the below piece of line:

 
ssh -q -o "StrictHostKeyChecking no" $server_id "cd /intucell/data/logs/app; zcat $first_line_log_file | sed -rne '/$first_line_date_time/,/$first_log_first_line/p'| zgrep -A10 -i '$do_action_on_cell_2';" >> ./output.log
7

After

(c.f. man zcat ) there's no compressed data any more, so using zgrep is overkill (although it may work as zgrep recognizes and works on uncompressed data).

2nd point: you have
$do_action_on_cell_2=... . The $-sign is incorrect.

For the unterminated address regex error, we need to see what's going on. Try to run the command with the -vx options set to enable xtracing.
For the address range matching, the string you give for the starting address NEEDS to be found dot for dot, comma for comma, char for char, in the target file. Are you sure that's the case?

8

Hi RudiC,

Thanks for the reply.

Regarding point 2 - it is just a typo error which i have corrected in this question.

The below commands when executed gives the followed result with -vx options:

 ssh -q -o "StrictHostKeyChecking no" $server_id "cd /intucell/data/logs/app; zcat $first_line_log_file | tail -1| cut -d' ' -f1-2;" 

+ ssh -q -o 'StrictHostKeyChecking no' deb011 'cd /intucell/data/logs/app; zcat sample.gz | tail -1| cut -d'\'' '\'' -f1-2;'
head -1 first_line_output.txt)
head -1 first_line_output.txt
++ head -1 first_line_output.txt
+ first_log_first_line='2016-09-14 01:46:56,438'

Not working case:

 ssh -q -o "StrictHostKeyChecking no" $server_id "cd /intucell/data/logs/app; zcat $first_line_log_file | sed -rne '/$first_line_date_time/,/$first_log_first_line/p'| zgrep -A10 -i '$do_action_on_cell_2';" >> ./output.log 

+ ssh -q -o 'StrictHostKeyChecking no' deb011 'cd /intucell/data/logs/app; zcat sample.gz | zgrep -A10 -i '\''Do action on cell BLA330'\'' | sed -rne '\''/2016-09-14 01:09:56,796/,/2016-09-14 01:46:56,438/p'\'''
sed: -e expression #1, char 62: unterminated address regex

Let me know in case any other information is needed.

9

Hi,

change this to

sed -rne '/$first_line_date_time/,/$first_log_first_line/p'

double quotes. (as per post#1 these are variables)

sed -rne "/$first_line_date_time/,/$first_log_first_line/p"
10

There's no 62 chars in that regex; try setting the -vx options on the remote node, e.g. just before cd in the ssh command...

11

As per greet_sed changed sed expression to double quotes and output is below - NOK
Change is observed in the time stamp - space is highlighted:

 + ssh -q -o 'StrictHostKeyChecking no' deb011 'cd /intucell/data/logs/app; zcat sample.gz | sed -rne /2016-09-14' '01:09:56,796/,/2016-09-14' '01:46:56,438/p | grep -A10 -i '\''Do action on cell BLA330'\'';'
sed: -e expression #1, char 16: unterminated address regex

As per RudiC, added -vx before "cd in the code and below is the output:

 ssh -q -o "StrictHostKeyChecking no" $server_id -vx "cd /intucell/data/logs/app; zcat $first_line_log_file | sed -rne '/$first_line_date_time/,/$first_log_first_line/p' | grep -A10 -i '$do_action_on_cell_2';"

Output as below:

 + ssh -q -o 'StrictHostKeyChecking no' deb011 -vx 'cd /intucell/data/logs/app; sample.gz | sed -rne '\''/2016-09-14 01:09:56,796/,/2016-09-14 01:46:56,438/p'\'' | grep -A10 -i '\''Do action on cell BLA330'\'';'
OpenSSH_6.0p1 Debian-4+deb7u2, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to deb011 [x.x.x.x] port 22.
debug1: Connection established.
...
...
debug1: Sending command: cd /intucell/data/logs/app; zcat sample.gz | sed -rne '/\033[36m2016-09-14 01:09:56,796/,/\033[32m2016-09-14 01:46:56,438/p' | grep -A10 -i 'Do action on cell BLA330';
sed: -e expression #1, char 62: unterminated address regex
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 2888, received 2000 bytes, in 0.0 seconds
Bytes per second: sent 270866.5, received 187580.7
debug1: Exit status 1

From the output above, i feel the below part is suspicious:

 '/\033[36m2016-09-14 01:09:56,796/,/\033[32m2016-09-14 01:46:56,438/p'

Kindly guide

12

You have "ECMA-48 Set Graphics Rendition" control codes in your variable definition - that's how the 62 char is reached. I think, sed complains about opening but no closing square brackets.

13

Great, so any clue on how do i get out of this ?

14

Remove the control codes when/before assigning the variables.

15

I with help of wolfmilkporc found a way to remove the control color codes.

first_line_date_time=$(echo $first_line_date_time | perl -pe 's/\e\[[\d;]*m//g;')
                                     and
                                    first_log_first_line=$(echo $first_log_first_line | perl -pe 's/\e\[[\d;]*m//g;')

Acknowledgement: Got the Perl regex from the source of the module Term::ANSIColor, see here ("colorstrip"):
lib/Term/ANSIColor.pm - metacpan.org
Special thanks to RudiC, wolfmilkporc, Kurt Starsinic and greet_sed.

16

Where and how did you assign the variables in the first place?

17

Hi RudiC,

The variable is assigned with the date+time of the first line from the text file which contains the content redirected using zgrep.

example:

 sample.gz:2016-09-14 01:09:56,796 [4665:ActionService:22:execute_changes:INFO] Action 126543 was created and sent to execution

I have removed the color escape characters using the below perl regex and the issue is resolved.

first_log_last_line=$(echo $first_log_first_line | perl -pe 's/\e\[[\d;]*m//g;')

However, I have a new requirement - explained as below:

I have to sort only the lines containing the date+time.
Sample text file as below:

========== server_1 ==========
sample1.gz: 2016-09-14 02:55:30,799
sample2.gz: 2016-09-14 01:09:56,796
sample2.gz: 2016-09-14 01:09:57,927
========== server_2 ==========
========== server_3 ==========
========== server_1 ==========
sample1.gz: 2016-09-14 02:55:31,250
sample1.gz: 2016-09-14 02:55:31,777

Desired output:

========== server_1 ==========
sample2.gz: 2016-09-14 01:09:56,796
sample2.gz: 2016-09-14 01:09:57,927
sample1.gz: 2016-09-14 02:55:30,799
sample1.gz: 2016-09-14 02:55:31,250
sample1.gz: 2016-09-14 02:55:31,777
========== server_2 ==========
========== server_3 ==========

Please suggest how this can be achieved.

18

So - you have coloured output in the files? Can't they be produced without colour information?

Do you need the server_n info in the sorted output? If not, try

sort -uk2 file4
sample2.gz: 2016-09-14 01:09:56,796
sample2.gz: 2016-09-14 01:09:57,927
sample1.gz: 2016-09-14 02:55:30,799
sample1.gz: 2016-09-14 02:55:31,250
sample1.gz: 2016-09-14 02:55:31,777
========== server_1 ==========
========== server_2 ==========
========== server_3 ==========
19

Hi RudiC,

I have used the perl regex to avoid the colored output.

Yes, I would need the server_n output as well.
The logs will be only for a single server and on sorting i need the information which server these logs belongs to as well.

20

Try

awk '/^====/ {PRF = $2 = "0"$2} {print PRF, $0}' file | sort -uk1,1 -k3 | awk '{sub ($1 FS, _); sub (/^0/, _, $2)}1'
========== server_1 ==========
sample2.gz: 2016-09-14 01:09:56,796
sample2.gz: 2016-09-14 01:09:57,927
sample1.gz: 2016-09-14 02:55:30,799
sample1.gz: 2016-09-14 02:55:31,250
sample1.gz: 2016-09-14 02:55:31,777
========== server_2 ==========
========== server_3 ==========
1 Like