Insert value to ORACLE table from sqlldr log

aimy · February 17, 2009, 4:09am

This is the sample sqlldr log:

------------------------------------------------------------------------------------------------------------

SQL*Loader: Release 9.2.0.7.0 - Production on Sun Feb 8 23:37:02 2009

Control File: /home/oracle9/dba_area/billed_recurr_charges.ctl
Data File: /tmw/oradata12/bill_recc_charges_bp01_semenanjung.data
Bad File: /tmw/oradata18/bad/bill_recc_charges_bp01_semenanjung.data.bad
Discard File: none specified

(Allow all discards)

Number to load: ALL
Number to skip: 0
Errors allowed: 50
Bind array: 64 rows, maximum of 256000 bytes
Continuation: none specified
Path used: Conventional

Table CRISPADM.BILLED_RECURR_CHARGES, loaded from every logical record.
Insert option in effect for this table: APPEND
TRAILING NULLCOLS option in effect

Column Name Position Len Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
SERVICE_NUM 1:16 16 CHARACTER
BILL_DATE 17:24 8 DATE YYYYMMDD
NULL if BILL_DATE = 0X3030303030303030(character '00000000')
SEF_CODE 25:28 4 CHARACTER
ACCOUNT_NUM 29:41 13 CHARACTER
BILL_PERIOD 42:43 2 CHARACTER
SERVICE_TYPE 44:46 3 CHARACTER
CHARGE_AMT 47:61 15 CHARACTER
CHARGE_RATE 62:76 15 CHARACTER
CHARGE_QTY 77:91 15 CHARACTER
EQUIPMENT_OWNERSHIP_CODE 92:92 1 CHARACTER
CHARGE_EXEMPTION_IND 93:93 1 CHARACTER
SOURCE CONSTANT
Value is 'CASS'

Record 719882: Rejected - Error on table CRISPADM.BILLED_RECURR_CHARGES.
ORA-01400: cannot insert NULL into ("CRISPADM"."BILLED_RECURR_CHARGES"."SEF_CODE")

Table CRISPADM.BILLED_RECURR_CHARGES:
1722967 Rows successfully loaded.
1 Row not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.

Space allocated for bind array: 8128 bytes(64 rows)
Read buffer bytes: 1048576

Total logical records skipped: 0
Total logical records read: 1722968
Total logical records rejected: 1
Total logical records discarded: 0

Run began on Sun Feb 08 23:37:02 2009
Run ended on Sun Feb 08 23:42:19 2009

Elapsed time was: 00:05:17.27
CPU time was: 00:01:30.26

------------------------------------------------------------------------------------------------------------

So, I need to ouput the bolded info into a table:

insert into PROCEDURE_TRACKING_LOG
(procedure_name, table_name, rows_inserted, rows_updated, rows_deleted, rows_inserted_of_deletion, 
rows_inserted_after_deletion, rows_rejected, start_time, end_time, elapse_time, remarks, subject_area)
values
(proc_name,table_name,row_insert,NULL,NULL,NULL,
NULL,row_reject,start_time,sysdate,elapsed_time,v_ErrorText,'NETWORK INVENTORY');

whereby

proc_name = 'the filename which runs the sqlldr'
table_name = 'BILLED_RECURR_CHARGES'
row_insert = 1722967
row_reject = 1
start_time = 'Sun Feb 08 23:37:02 2009'
sysdate = 'Sun Feb 08 23:42:19 2009'
elapsed_time = 317 (after conversion to rounded seconds)
v_ErrorText = 'ORA-01400: cannot insert NULL into ("CRISPADM"."BILLED_RECURR_CHARGES"."SEF_CODE")'

Please help me.

Thank you very much.

All your help are much appreciated..

aimy · February 17, 2009, 4:02pm

Is this impossible to achieve??

aimy · February 17, 2009, 8:57pm

Why there are still no responses to this question?

bakunin · February 17, 2009, 10:41pm

/Moderative Mode ON

First of all, you shouldn't bump up questions, because it is against the rules.

Second: please notice that this is not a help desk. If you want informed, precise answers to be given within a certain timeframe please consider hiring an UNIX expert.

Third: please bear in mind, that all people answering here are VOLUNTEERS. They might be (in fact they are) most times interested in helping you or someone else but they are not obligated to do so. If they do not feel like it (and, frankly, exhibiting a demanding attitude like you did makes them not feeling like it more likely) they do not have to answer at all.

Fourth: all these boards are about is helping one to help himself. Right now you have shown more effort in getting us to work faster than in solving your problems yourself. Take this as an opportunity to think over (and perhaps rearrange) your priorities.

/Moderative Mode OFF, User Mode ON

There is a simple answer to your problem: probably it could be solved with a finite effort and a little sed-scripting.

Longer answer: the real problem lies perhaps in the analysis of the problem and in formulating a solution which is general and reliable enough to let it run unsupervised. For this there are simply not enough data.

Some questions you will have to answer: you have provided a file with exactly one error. How would a file look like when there are two (any arbitrary number of) errors? How does the file look like when there are no errors at all? How will the file look like when there are different types of errors (say, the load stops midways because of some error - file error, disk full, what else)?

A stable and reliable script would be able to parse all these types of output (in fact it should be able to cope with every possible output) therefor you have to take all these different (?) output formats into account.

So please give more details and maybe we can work out a script that does exactly that or try to write one yourself. Provided that the problems mentioned above can be overcome here is a way to isolate your needed values:

You search for a specific value (for the sake of the example we will use start_time) and you know how the line looks like which contains this value:

.... any text ....
....
Run began on Sun Feb 08 23:37:02 2009
... some more text ....

You can search for this line easily with sed. Now replace the fixed text part "Run began on" with the variable name you want to assign it to to form a declaration:

sed -n 's/^Run began on/start_time \=/p'

And you will notice that from your logfile you will get out a single declaration reading:

start_time = Sun Feb 08 23:37:02 2009

This declaration you could directly use to initialize some variables in your script (source the file written this way in).

I hope this helps.

bakunin

aimy · February 17, 2009, 11:33pm

bakunin:

/Moderative Mode ON

First of all, you shouldn't bump up questions, because it is against the rules.

Second: please notice that this is not a help desk. If you want informed, precise answers to be given within a certain timeframe please consider hiring an UNIX expert.

Third: please bear in mind, that all people answering here are VOLUNTEERS. They might be (in fact they are) most times interested in helping you or someone else but they are not obligated to do so. If they do not feel like it (and, frankly, exhibiting a demanding attitude like you did makes them not feeling like it more likely) they do not have to answer at all.

Fourth: all these boards are about is helping one to help himself. Right now you have shown more effort in getting us to work faster than in solving your problems yourself. Take this as an opportunity to think over (and perhaps rearrange) your priorities.

/Moderative Mode OFF, User Mode ON

There is a simple answer to your problem: probably it could be solved with a finite effort and a little sed-scripting.

Longer answer: the real problem lies perhaps in the analysis of the problem and in formulating a solution which is general and reliable enough to let it run unsupervised. For this there are simply not enough data.

Some questions you will have to answer: you have provided a file with exactly one error. How would a file look like when there are two (any arbitrary number of) errors? How does the file look like when there are no errors at all? How will the file look like when there are different types of errors (say, the load stops midways because of some error - file error, disk full, what else)?

A stable and reliable script would be able to parse all these types of output (in fact it should be able to cope with every possible output) therefor you have to take all these different (?) output formats into account.

So please give more details and maybe we can work out a script that does exactly that or try to write one yourself. Provided that the problems mentioned above can be overcome here is a way to isolate your needed values:

You search for a specific value (for the sake of the example we will use start_time) and you know how the line looks like which contains this value:
.... any text ....
....
Run began on Sun Feb 08 23:37:02 2009
... some more text ....
You can search for this line easily with sed. Now replace the fixed text part "Run began on" with the variable name you want to assign it to to form a declaration:
sed -n 's/^Run began on/start_time \=/p'
And you will notice that from your logfile you will get out a single declaration reading:
start_time = Sun Feb 08 23:37:02 2009
This declaration you could directly use to initialize some variables in your script (source the file written this way in).

I hope this helps.

bakunin

Thanks for your response and please do forgive me about my bad behavior..

It's just that I feel my question is not that complicated but yet not one seems to answer..

While waiting for the solution, I take my own iniative to study a bit about awk utility.

So, this is what I got:

table_name = awk '/Table.+[:]/ {print substr($2,10)}' testfile.log
row_insert = awk '/Rows successfully loaded./ {print $1}' testfile.log
row_reject = awk '/not loaded due to data errors./ {print $1}' testfile.log
v_ErrorText = awk '/^ORA-/ {print}' testfile.log
start_time = awk '/Run began on/ {print $5, $6, $7, $8}' testfile.log
sysdate = '/Run ended on/ {print $5, $6, $7, $8}' testfile.log
elapsed_time = awk '/Elapsed/ {print substr($4,1,2)*3600 + substr($4,4,2)*60 + substr($4,7)}' testfile.log

sqlplus -s crispadm/admcrisp <<EOF
set autocommit on;
set echo on;
set serveroutput on;

insert into PROCEDURE_TRACKING_LOG
(procedure_name, table_name, rows_inserted, rows_updated, rows_deleted, rows_inserted_of_deletion, 
rows_inserted_after_deletion, rows_rejected, start_time, end_time, elapse_time, remarks, subject_area)
values
('TEST',replace($table_name,':'),$row_insert,NULL,NULL,NULL,
NULL,$row_reject,$start_time,$sysdate,round($elapsed_time),$v_ErrorText,'NETWORK INVENTORY');

EOF

All the awk command works fine:

$ awk '/Table.+[:]/ {print substr($2,10)}' testfile.log
BILLED_RECURR_CHARGES:

$ awk '/Rows successfully loaded./ {print $1}' testfile.log
1722967

$ awk '/not loaded due to data errors./ {print $1}' testfile.log
1

$ awk '/^ORA-/ {print}' testfile.log
ORA-01400: cannot insert NULL into ("CRISPADM"."BILLED_RECURR_CHARGES"."SEF_CODE")

$ awk '/Run began on/ {print $5, $6, $7, $8}' testfile.log
Feb 08 23:37:02 2009

$ awk '/Run ended on/ {print $5, $6, $7, $8}' testfile.log
Feb 08 23:42:19 2009

$ awk '/Elapsed/ {print substr($4,1,2)*3600 + substr($4,4,2)*60 + substr($4,7)}' testfile.log
317.27

So, where i do went wrong in the script to set all that awk commands as a variable?

Thank you.

bakunin · February 18, 2009, 8:21am

One of the reasons questions get answered is the are "interesting" to solve: that means usually: non-trivial, non-obvious, complicated. Its an ironic fact that questions easy to be answered might take a longer time to actually be answered than the complicated and awkward ones.

That is exactly the right approach.

First off, your solution with awk is about the same line as my suggestion of sed - in this regard these two tools do about the same even if they do it in a somewhat different way.

Secondly there is a subtle syntactical error in your code: a variable assignment in shell scripts must not use blanks surrounding the "="-sign. Notice the difference between the two lines:

x = 100   # wrong
x=100    # correct

The reason is that the shell takes everything up to the "=" to be the variables name, hence when you write "x =" you are assigning a variable named "x " (x followed by a blank) and this is probably not what you intended.

Now for the main problem with your code, which is also of syntactical nature. You do NOT want to assign a string that resembles a command to the variable but you want the output of the command be assigned instead. Therefore you must tell the shell to execute the command separately first and then assign its output. Consider the following lines:

var=date
var=$(date)

The first line will assign the word "date" to the variable, the second one will first execute the "date"-command (whatever is between $(...) ) and execute it (in a separate shell instance, btw.). Whatever output is produced is then replacing the "$(....)" and only then the commandline is executed assigning the variable.

What does that mean to your problem? Edit the lines in the following way:

table_name="$(awk '/Table.+[:]/ {print substr($2,10)}' testfile.log)"

PS: some might suggest using backticks (`command`) instead of the $(command) construct. Backticks are an outdated way of basically achieving the same, but should be avoided because the newer construct is much more flexible and also better readable. It is one of the most prominent goals of shell scripting to produce code that is easily maintainable, easily read, easily understood. Backticks look in some fonts so similar to single quotes that because of this alone they should be abandoned. This being not the only disadvantage they have they should be avoided even more.

I hope this helps.

bakunin

aimy · February 19, 2009, 1:58am

bakunin:

One of the reasons questions get answered is the are "interesting" to solve: that means usually: non-trivial, non-obvious, complicated. Its an ironic fact that questions easy to be answered might take a longer time to actually be answered than the complicated and awkward ones.

That is exactly the right approach.

First off, your solution with awk is about the same line as my suggestion of sed - in this regard these two tools do about the same even if they do it in a somewhat different way.

Secondly there is a subtle syntactical error in your code: a variable assignment in shell scripts must not use blanks surrounding the "="-sign. Notice the difference between the two lines:
x = 100   # wrong
x=100    # correct
The reason is that the shell takes everything up to the "=" to be the variables name, hence when you write "x =" you are assigning a variable named "x " (x followed by a blank) and this is probably not what you intended.

Now for the main problem with your code, which is also of syntactical nature. You do NOT want to assign a string that resembles a command to the variable but you want the output of the command be assigned instead. Therefore you must tell the shell to execute the command separately first and then assign its output. Consider the following lines:
var=date
var=$(date)
The first line will assign the word "date" to the variable, the second one will first execute the "date"-command (whatever is between $(...) ) and execute it (in a separate shell instance, btw.). Whatever output is produced is then replacing the "$(....)" and only then the commandline is executed assigning the variable.

What does that mean to your problem? Edit the lines in the following way:
table_name="$(awk '/Table.+[:]/ {print substr($2,10)}' testfile.log)"
PS: some might suggest using backticks (`command`) instead of the $(command) construct. Backticks are an outdated way of basically achieving the same, but should be avoided because the newer construct is much more flexible and also better readable. It is one of the most prominent goals of shell scripting to produce code that is easily maintainable, easily read, easily understood. Backticks look in some fonts so similar to single quotes that because of this alone they should be abandoned. This being not the only disadvantage they have they should be avoided even more.

I hope this helps.

bakunin

Wow.. I am so impressed with your lengthy explanation to my question..

Thank you very much sir..

I really meant it.

Yeah, I already solved my problem yesterday when I noticed that when I put a space surrounding the '=' sign it doesn't work. But I am not really sure that is the real cause of the problem, so thanks for telling me the fact.

Also, the '`' sign surrounding the command. I used that coz I thought the correct syntax should be both i.e. without a surrounding space between '=' and also have to put that '`' sign. So, once again, thanks a lot.

My question now is about.. If I put this script inside a filename called test_script.sh:

table_name="$(awk '/Table.+[:]/ {print substr($2,10)}' testfile.log)"
row_insert="$(awk '/Rows successfully loaded./ {print $1}' testfile.log)"
row_reject="$(awk '/not loaded due to data errors./ {print $1}' testfile.log)"
v_ErrorText="$(awk '/^ORA-/ {print}' testfile.log)"
start_time="$(awk '/Run began on/ {print $5, $6, $7, $8}' testfile.log)"
sysdate="$(awk '/Run ended on/ {print $5, $6, $7, $8}' testfile.log)"
elapsed_time="$(awk '/Elapsed/ {print substr($4,1,2)*3600 + substr($4,4,2)*60 + substr($4,7)}' testfile.log)"

sqlplus -s crispadm/admcrisp <<EOF
set autocommit on;
set echo on;
set serveroutput on;

insert into PROCEDURE_TRACKING_LOG
(procedure_name, table_name, rows_inserted, rows_updated, rows_deleted, rows_inserted_of_deletion, rows_inserted_after_deletion, rows_rejected, 
start_time, end_time, elapse_time, remarks, subject_area)
values
('$curfile',replace('$table_name',':'),$row_insert,NULL,NULL,NULL,NULL,$row_reject,to_date('$start_time','MON dd hh24:mi:ss yyyy'), 
to_date('$sysdate','MON dd hh24:mi:ss yyyy'), round($elapsed_time),'$v_ErrorText','NETWORK INVENTORY');

EOF

How do I set that variable called $curfile so that it will be returning the script filename itself i.e. test_script.sh

Can anyone please help me?

Thank you very much..

sunpraveen · February 19, 2009, 2:11am

Very simple. Use the below code at the beginning of the script.

curfile=${0##*/}

HTH,

Regards,

Praveen

aimy · February 19, 2009, 2:27am

Wow!! Thank you very much..

But would you mind to explain me what does the code actually?

Coz by 1st look, it seems not so meaningful.. ehehe..

sunpraveen · February 19, 2009, 2:40am

Whenever shell scripts are run, "$0" will always be the name of that shell script.

Generally, shell scripts are run by using the ./ prefix. If you use this, then $0 will also show ./. In order to remove this, ##*/ is used.

HTH,

Regards,

Praveen

aimy · March 1, 2009, 11:53pm

Thank you very much..

aimy · March 1, 2009, 11:57pm

I want to extend my question..

I have this variable set

errors="$(awk '/^ORA-/ {print}' pbsb.cv004.cr260.charnge.log|sort -d|uniq)"

which will produce this as the output:

ORA-00001: unique constraint (CRISPADM.PK_CHARNGE) violated
ORA-01400: cannot insert NULL into ("CRISPADM"."CHARNGE"."CODE_SYS_ID")

The logfile is something like this:

TOTAL_CHANNEL_STUMP_CTR             69:73     5           CHARACTER
START_LEN_NUM                       74:82     9           CHARACTER
END_LEN_NUM                         83:91     9           CHARACTER

Record 3484: Rejected - Error on table CRISPADM.CHARNGE.
ORA-01400: cannot insert NULL into ("CRISPADM"."CHARNGE"."CODE_SYS_ID")

Record 3485: Rejected - Error on table CRISPADM.CHARNGE.
ORA-01400: cannot insert NULL into ("CRISPADM"."CHARNGE"."CODE_SYS_ID")

Record 4719: Rejected - Error on table CRISPADM.CHARNGE.
ORA-01400: cannot insert NULL into ("CRISPADM"."CHARNGE"."CODE_SYS_ID")

Record 4720: Rejected - Error on table CRISPADM.CHARNGE.
ORA-01400: cannot insert NULL into ("CRISPADM"."CHARNGE"."CODE_SYS_ID")

Record 4721: Rejected - Error on table CRISPADM.CHARNGE.
ORA-01400: cannot insert NULL into ("CRISPADM"."CHARNGE"."CODE_SYS_ID")

Record 4722: Rejected - Error on table CRISPADM.CHARNGE.

My question is that how do I output those

ORA-00001: unique constraint (CRISPADM.PK_CHARNGE) violated
ORA-01400: cannot insert NULL into ("CRISPADM"."CHARNGE"."CODE_SYS_ID")

into a single line separated by comma i.e.

ORA-00001: unique constraint (CRISPADM.PK_CHARNGE) violated, ORA-01400: cannot insert NULL into ("CRISPADM"."CHARNGE"."CODE_SYS_ID")

Can I?

Thanks for helping..

vgersh99 · March 2, 2009, 7:04am

errors="$(awk '/^ORA-/ {print}' pbsb.cv004.cr260.charnge.log|sort -d|uniq | sed 's/$/,/')"

aimy · March 2, 2009, 9:32pm

Thank you very much..

But could you please tell me what's the difference between this two:

awk '/^ORA-/ {print}' pbsb.cv004.cr260.charnge.log|sort -d|uniq | sed 's/$/,/'

would output

ORA-00001: unique constraint (CRISPADM.PK_CHARNGE) violated,
ORA-01400: cannot insert NULL into ("CRISPADM"."CHARNGE"."CODE_SYS_ID"),

But if i do this:

errors="$(awk '/^ORA-/ {print}' pbsb.cv004.cr260.charnge.log|sort -d|uniq | sed 's/$/,/')"

and then issuing

echo $errors

would display it in one line as desired

ORA-00001: unique constraint (CRISPADM.PK_CHARNGE) violated, ORA-01400: cannot insert NULL into ("CRISPADM"."CHARNGE"."CODE_SYS_ID"),

And also, is there anyway to trim the last trailing commas in the end?

Thanks.