using substring in shell script

kmanivan82 · October 16, 2008, 12:48pm

This is the data I am having in a file
Just for sample I have given 3 records. The file which I am having consists of n number of records.


ABC123 10 01/02/2008 2008-01-03-00.00.00.000000
DYUU   22 02/03/2008 2008-01-04-00.00.00.000000
RF33   88 03/05/2008 2008-01-05-00.00.00.000000

trackingnum in the position 1-6 6 bytes
trackingnumsuffix in the position 8-9 2 bytes
effdate in the position 11-20
tstmpupdated in the position 22-57 26 bytes
I dont know how to use substring.
Basically I need to extract the trackingnum, trackingnumsuffix and tstmpupdated and pass those values to a query. If it is present in that table, then I need to store the output of query to a file.
I have tried by writing the shell script as follows.

====================
#! /bin/ksh
############################
#   AFI Monitor Script
############################
. /db2/uszlad48/sqllib/db2profile
export mondir=/home/bmwdev1/script/krishna/arc
export monlog=$mondir/rcbl2_`date +%Y%m%d`.log
# connect to DB
db2 connect to r2pdev user bmwdevup using summer08
while read line
do
trackingnum=`expr substr $line 1 6`
trackingnumsuffix=`expr substr $line 8 9`
tstmpupdated=`expr substr $line 22 57`
#db2 "SELECT * FROM ZB_RCBL_ERROR_MSG_MIG WHERE TRACKING_NUM = $TRACKING_NUM AND TRACKING_NUM_SUFFIX = $TRACKING_NUM_SUFFIX AND TIMESTAMP_UPDATED = $TIMESTAMP_UPDATED WITH UR" >> new.log
done < "$monlog" 
# disconnect from DB2
db2 terminate
exit 0
====================

The above shell script is not working. It throws error message.
Can anyone help me to fix this issue.
Krishnakanth

avis1981 · October 16, 2008, 1:50pm

How about using awk, printing it $1, $2..

e.g.
trackingnum=echo $line|awk '{print $1}'

cfajohnson · October 16, 2008, 2:46pm

kmanivan82:

====================
#! /bin/ksh
############################
#   AFI Monitor Script
############################
. /db2/uszlad48/sqllib/db2profile
export mondir=/home/bmwdev1/script/krishna/arc
export monlog=$mondir/rcbl2_`date +%Y%m%d`.log
# connect to DB
db2 connect to r2pdev user bmwdevup using summer08
while read line
do
trackingnum=`expr substr $line 1 6`

trackingnum=`expr substr "$line" 1 6`

trackingnumsuffix=`expr substr $line 8 9`
tstmpupdated=`expr substr $line 22 57`
#db2 "SELECT * FROM ZB_RCBL_ERROR_MSG_MIG WHERE TRACKING_NUM = $TRACKING_NUM AND TRACKING_NUM_SUFFIX = $TRACKING_NUM_SUFFIX AND TIMESTAMP_UPDATED = $TIMESTAMP_UPDATED WITH UR" >> new.log
done < "$monlog" 
# disconnect from DB2
db2 terminate
exit 0
====================

The above shell script is not working. It throws error message.

What is the error messsage?

Do you still get it after enclosing $line in quotes?

kmanivan82 · October 17, 2008, 2:45am

If I give the following code

trackingnum=`expr substr "$line" 1 6`

I am not getting any output/error message.

If I give the following code

trackingnum=`expr substr $line 1 6`

I am getting the following error message

expr: non-numeric argument

expr: non-numeric argument

expr: non-numeric argument

expr: non-numeric argument

expr: non-numeric argument

Please advice how to proceed furthur

Krishnakanth

vidyadhar85 · October 17, 2008, 3:12am

please post the whts there in $line??

kmanivan82 · October 17, 2008, 3:27am

There is a file called rcbl2_`date +%Y%m%d`.log

It consists of the following

ABC123 10 01/02/2008 2008-01-03-00.00.00.000000
DYUU   22 02/03/2008 2008-01-04-00.00.00.000000
RF33   88 03/05/2008 2008-01-05-00.00.00.000000

For sample I have given 3 records. But in production there will be n number of records.
So I am reading line by line in a loop.

The first field trackingnum will be in the position 1-6
The field trackingnumsuffix will be in the position 8-9
the tstmpupdated will be in the position 22-57

while read line
do

	trackingnum=`expr substr $line 1 6`
	trackingnumsuffix=`expr substr $line 8 9`
	tstmpupdated=`expr substr $line 22 57`
	db2 "SELECT * FROM ZB_RCBL_ERROR_MSG_MIG WHERE TRACKING_NUM = $TRACKING_NUM AND TRACKING_NUM_SUFFIX = $TRACKING_NUM_SUFFIX AND TIMESTAMP_UPDATED = $TIMESTAMP_UPDATED WITH UR" >> new.log

done < "rcbl2_`date +%Y%m%d`.log"

Please advice how to proceed furthur.

Krishnakanth

vidyadhar85 · October 17, 2008, 3:51am

try to echo $line inside while loop and check whether you are getting the required line..
and you have to use double quotes if $line has space
because i am getting no errors

> v="ABC123 10 01/02/2008 2008-01-03-00.00.00.000000"
> m=`expr substr "$v" 1 6`
> echo "$m"
ABC123

system · October 17, 2008, 4:22am

Your data looks like well formatted, whitespace separated data, so you can use read to do the parsing:

#!/bin/bash

while read trackingnum trackingnumsuffix date tstmpupdated
do
    echo Tracking: $trackingnum, Suffix: $trackingnumsuffix, Date: $date, Timestamp: $tstmpupdated
done

$ ./parse.sh < data
Tracking: ABC123, Suffix: 10, Date: 01/02/2008, Timestamp: 2008-01-03-00.00.00.000000
Tracking: DYUU, Suffix: 22, Date: 02/03/2008, Timestamp: 2008-01-04-00.00.00.000000
Tracking: RF33, Suffix: 88, Date: 03/05/2008, Timestamp: 2008-01-05-00.00.00.000000

palsevlohit_123 · October 17, 2008, 4:26am

trackingnum=`echo $line|cut -c1-6`
trackingnumsuffix=`echo $line|cut -c8-9`
tstmpupdated=`echo $line|cut -c22-57`

system · October 17, 2008, 4:35am

For a large file, this has very large overhead. Per line of the file, this code does a unwarranted number of fork/execs, and will be very slow.

Always use built-ins when possible, and better yet, use programs designed for fast data processing and scripting such as awk or perl.

kmanivan82 · October 17, 2008, 6:07am

The following code is not working

trackingnum=`echo $line|cut -c1-6`
trackingnumsuffix=`echo $line|cut -c8-9`
tstmpupdated=`echo $line|cut -c22-57`

I didnot get either output or error msg..
F Y I ... we are using Korn shell script

Following is the actual data in production.

BEYJYWG83L                     0 R            902     -                    B       System Exception Occured, please refer to BMW_EXCEPTION_LOG table                                    2008-09-10-00.41.32.787352             
BEYJYXMT4K                     0 R            902     -                    B       System Exception Occured, please refer to BMW_EXCEPTION_LOG table                                    2008-09-11-19.01.23.878551             
BEYEHF93QY                     0 R            003                          B       No message exists in ZBV_BILL_REJCT_RSN table                                                        2008-08-28-16.31.01.881717             
BEYJYTT8FQ                     0 R            013                          B       Invalid Transaction Code                                                                             2008-09-04-17.39.04.005237             
BEYJYTT8FQ                     0 R            016                          B       Invalid Fee Code                                                                                     2008-09-04-17.39.04.008137             
BEYJYTT8FQ                     0 R            016                          B       Invalid Fee Code                                                                                     2008-09-04-17.39.04.011939             
BEYJYTT8FQ                     0 R            016                          B       Invalid Fee Code                                                                                     2008-09-04-17.39.04.012851             
BEYJY2SZVL                     0 R            003                          B       Policy / Contract Account Number are pending in the ZB_MASTER_DATA_LOG                               2008-09-16-09.22.29.120192             
BEYJY2SXNG                     0 R            003                          B       Policy / Contract Account Number are pending in the ZB_MASTER_DATA_LOG                               2008-09-16-09.19.33.868772             
BEYJY2SMS3                     0 R            003                          B       Policy / Contract Account Number are pending in the ZB_MASTER_DATA_LOG                               2008-09-16-08.53.07.872138             
BEYJY2SMJD                     0 R            003                          B       Policy / Contract Account Number are pending in the ZB_MASTER_DATA_LOG                               2008-09-16-08.50.17.307262             
BEYJY2EG2K                     0 R            003                          B       Policy / Contract Account Number are pending in the ZB_MASTER_DATA_LOG                               2008-09-17-15.56.03.418917

I am writing for a sample one. If it works I will customize it according to the production job.

Please advice how to proceed furthur..

Krishnakanth

system · October 17, 2008, 2:10pm

Ok, we can't use whitespace as a column separator, but we can use another trick. We'll use sed in one pass to rearrange the data into whitespace separated columns that read can use:

#!/bin/ksh

sed -E 's/^(.{10}).{36}(.{3}).{34}(.{101})(.{26})/\1 \2 \4 \3/' in2 |
while read trackingnum trackingnumsuffix tstmpupdated text
do
    echo Tracking: \"$trackingnum\", Suffix: \"$trackingnumsuffix\", Timestamp: \"$tstmpupdated\", Text: \"$text\"
done

$ ./parse.sh
Tracking: "", Suffix: "", Timestamp: "", Text: ""
Tracking: "BEYJYWG83L", Suffix: "902", Timestamp: "2008-09-10-00.41.32.787352", Text: "System Exception Occured, please refer to BMW_EXCEPTION_LOG table"
Tracking: "BEYJYXMT4K", Suffix: "902", Timestamp: "2008-09-11-19.01.23.878551", Text: "System Exception Occured, please refer to BMW_EXCEPTION_LOG table"
Tracking: "BEYEHF93QY", Suffix: "003", Timestamp: "2008-08-28-16.31.01.881717", Text: "No message exists in ZBV_BILL_REJCT_RSN table"
Tracking: "BEYJYTT8FQ", Suffix: "013", Timestamp: "2008-09-04-17.39.04.005237", Text: "Invalid Transaction Code"
Tracking: "BEYJYTT8FQ", Suffix: "016", Timestamp: "2008-09-04-17.39.04.008137", Text: "Invalid Fee Code"
Tracking: "BEYJYTT8FQ", Suffix: "016", Timestamp: "2008-09-04-17.39.04.011939", Text: "Invalid Fee Code"
Tracking: "BEYJYTT8FQ", Suffix: "016", Timestamp: "2008-09-04-17.39.04.012851", Text: "Invalid Fee Code"
Tracking: "BEYJY2SZVL", Suffix: "003", Timestamp: "2008-09-16-09.22.29.120192", Text: "Policy / Contract Account Number are pending in the ZB_MASTER_DATA_LOG"
Tracking: "BEYJY2SXNG", Suffix: "003", Timestamp: "2008-09-16-09.19.33.868772", Text: "Policy / Contract Account Number are pending in the ZB_MASTER_DATA_LOG"
Tracking: "BEYJY2SMS3", Suffix: "003", Timestamp: "2008-09-16-08.53.07.872138", Text: "Policy / Contract Account Number are pending in the ZB_MASTER_DATA_LOG"
Tracking: "BEYJY2SMJD", Suffix: "003", Timestamp: "2008-09-16-08.50.17.307262", Text: "Policy / Contract Account Number are pending in the ZB_MASTER_DATA_LOG"
Tracking: "BEYJY2EG2K", Suffix: "003", Timestamp: "2008-09-17-15.56.03.418917", Text: "Policy / Contract Account Number are pending in the ZB_MASTER_DATA_LOG"

cfajohnson · October 17, 2008, 6:53pm

You wouldn't get any output because you are not outputing anything.

Do you mean that $trackingnum is empty?

If I give the following code

trackingnum=`expr substr $line 1 6`

I am getting the following error message

expr: non-numeric argument

expr: non-numeric argument

expr: non-numeric argument

expr: non-numeric argument

expr: non-numeric argument

Of course. Since $line is not quoted, it is being separated into multiple strings, instead of a single string.

You can get the first 6 characters of $line with:

temp=${line#??????}
trackingnum=${line%"$temp"}

kmanivan82 · October 21, 2008, 6:22am

I have tried to execute the following script as per MrC's syntax. It is throwing the error message as follows.

#! /bin/ksh
############################
#   AFI Monitor Script
############################

. /db2/uszlad48/sqllib/db2profile
export mondir=/home/bmwdev1/script/krishna/arc
export monlog=$mondir/rcbl1.log

sed -E 's/^(.{10}).{36}(.{3}).{34}(.{101})(.{26})/\1 \2 \4 \3/' in2 | 
while read trackingnum trackingnumsuffix tstmpupdated text
do
    echo Tracking: \"$trackingnum\", Suffix: \"$trackingnumsuffix\", Timestamp: \"$tstmpupdated\", Text: \"$text\"
done

exit 0

# ksh arc11.ksh
arc11.ksh[6]: ./db2/uszlad48/sqllib/db2profile:  not found
sed: illegal option -- E
Usage:  sed [-n] Script [File ...]
        sed [-n] [-e Script] ... [-f Script_file] ... [File ...]
bmwdev1@laaddb26  /home/bmwdev1/script/krishna/arc
#

what is wrong with this syntax ?

Please help me to fix this problem.

Krishnakanth

system · October 21, 2008, 10:00am

You have "./db2/uszlad48/sqllib/db2profile" and ". /db2/uszlad48/sqllib/db2profile".

Are you trying to source the file (the . <space> <filename> syntax) or are you trying to use a file starting the directory search at ./...?

kmanivan82 · October 21, 2008, 10:37am

Please have my updated code and the error message.


#! /bin/ksh
############################
#   AFI Monitor Script
############################

. /db2/uszlad48/sqllib/db2profile
export mondir=/home/bmwdev1/script/krishna/arc
export monlog=$mondir/rcbl1.log

sed -E 's/^(.{10}).{36}(.{3}).{34}(.{101})(.{26})/\1 \2 \4 \3/' in2 |
while read trackingnum trackingnumsuffix tstmpupdated text
do
    echo Tracking: \"$trackingnum\", Suffix: \"$trackingnumsuffix\", Timestamp: \"$tstmpupdated\", Text: \"$text\"
done

exit 0

# ksh arc11.ksh
sed: illegal option  -- E
Usage: sed [-n] Script [File ...]
           sed [-n] [-e Script] ... [-f Script_file] ... [File ...]
#

where we are giving the input file rcbl1.log in this script.
is "-E " correct ? Because it is throwing error message in -E

Please help me to fix this issue.

Thanks in advance.

Krishnakanth

system · October 21, 2008, 10:48am

The -E option is for sed's Extended Regular expressions. Check sed's man page for your sed's ability to use EREs.

Which platform is this?

kmanivan82 · October 21, 2008, 11:46am

Please let me know how to check the platform ..

Is there any command available...

Krishnakanth

system · October 21, 2008, 11:47am

uname -a

and for sed's man page:

man sed

kmanivan82 · October 22, 2008, 3:03am

Here's the platform details

#uname -a
AIX laaddb26 3 5 00C8FE604C00

Krishnakanth