First time post. I did a search so I didn�t see this specific issue. It seems to be a head scratcher for me.
I have an hourly job that on rare occasions, gets into an endless loop.
I�ve tried different scenarios but the current version does basically the following.
Find all the *.arc files and copy them to a new location. Since this runs every hour, I only copy files that don�t already exist in the new location. There is another process which follows the same logic and deletes from both the source as well as the destination. The only difference is I use the +mtime in the find command because I keep a weeks worth of files on the source and 10 days on the destination.
Every once in a while ( 3 to 4 times in the last 6 months), this script gets into an endless loop and the output fills up the log directory. We have to kill the process and cleanup the files.
Three questions:
1) Any thoughts on what I�m doing would cause an endless loop?
2) Any change in logic that might prevent this problem?
3) Any way to insert a trace to help trap the cause?
Here is the KSH function that does the copy:
# ------------------------------------------------------------------------------
# Copy archive files to the backup locaton. Skip the most current archive file.
# ------------------------------------------------------------------------------
save_files() {
ERR=N
sav_files1="/tmp/save_files1_${DB}.tmp"
sav_log1="/tmp/save_log1_${DB}.tmp"
>$sav_files1
>$sav_log1
find ${dir_dest} -type f -name \*.arc > $archive_list
curr_nbr=$(cat $archive_list | wc -l)
if [[ $curr_nbr -gt 1 ]]; then
#skip the most current file.
curr_arch=$(ls -1t ${dir_dest}/*.arc | head -1
cat $archive_list | while read aname
do
if [[ $curr_arch != $aname ]]; then
base_file=$(basename $aname)
if [[ $OS = "Linux" ]]; then
compress_name="${Backup_dir}/${base_file}.zip"
else
compress_name="${Backup_dir}/${base_file}.Z"
fi
cp_name="${Backup_dir}/${base_file}"
if [[ ! -a $cp_name && ! -a $compress_name ]]; then
# Save the file
if [[ $OS = "Linux" ]]; then
if [[ $COMPRESS = "Y" ]]; then
print "Compressing $compress_name" | $LOGCMD
echo "zip $compress_name $aname" >> $sav_files1
else
print "Coping $aname to $cp_name" | $LOGCMD
echo "cp -p $aname $cp_name" >> $sav_files1
fi
else
if [[ $COMPRESS = "Y" ]]; then
print "Compressing $compress_name" | $LOGCMD
echo "compress -c $aname > $compress_name" >> $sav_files1
else
print "Coping $aname to $cp_name" | $LOGCMD
echo "cp -p $aname $cp_name" >> $sav_files1
fi
fi
fi
else
print "Skipping $aname" | $LOGCMD
fi
done
# Chmod sav_files1 and run it.
chmod +x $sav_files1
$sav_files1 > $sav_log1 2>&1
if [[ $? -ne 0 ]]; then
print "Saving files had error bailing application" |$LOGCMD
cat $sav_log1 |$LOGCMD
abort_pgm
fi
fi
}
What is in the log? Where is the rest of the script? There is no infinite loop here.
You save some overhead doing it on a pipe and not one at a time: 'find . . . | xargs compress'
If you have many CPU and time is important, something that runs N at a time like my bctl tool speeds things up.
Compress is getting old, faster than gzip, bzip2 but with less compression, and slower than lzjb, which might also have a bit better compression, and I have not seen a lzjb command, but if you switch to zfs, you can just turn on lzjb compression and skip the script.
Thanks for all the replies. Here is the entire script.
Because the endless loop issue is so rare, I think it must be some kind of UNIX/KSH bug, but I have no way of verifying that. As you see by the comments, I had this issue since Febuary of 2010.
One thing someone can help is... is there a way to do a conditional copy because this script only copies new files to the destination. If "cp" or "find" has a way so I don't have to re-copy all the files each run, that's the main reason the script is so large. As far a using compress, I know it's old and gzip would be better, currently we don't compress any more because the destination takes care of it. Thanks for pointing it out though.
#!/bin/ksh
# CARIDIAN BCT
# Object Type: Shell Script
# Object Name: ibm_rman_arch_cleanup.ksh
# Subsystem..: Oracle maintenance
# Written by.: Sylvan Creach
# Parameters.:
# Output File:
# External...: /usr/local/scripts/oracle/.dbowner
# /usr/local/scripts/oracle/oracle.login [SID]
#
# Description: Created for IBM production databases that have archiving turned on.
# The tape backup system is not install so this script will maintain
# the archive logs on the system and also copy them to a Netapp drive
# that can be set up to perform the tape backup.
#
# USAGE: ibm_rman_arch_cleanup.ksh [SID] [DAYS]
# SID - DB id
# DAYS - All file older that this will be removed
#
# ------------------------------------------------------------------------------
# M A I N T E N A N C E L O G
# ------------------------------------------------------------------------------
# Date | Programmer | Description
# ------------------------------------------------------------------------------
# 12/15/04 | Sylvan Creach | Initial Release
# 02/25/10 | Sylvan Creach | Modified the logic to save the .arc files to a flat
# file to process. Prior it was just looping through
# a find command to process. I suspected when a database
# was generating excessing *.arc files, this made the script
# go into an endless loop.
# 05/17/11 | Sylvan Creach | Change Backup_dir="/db_dumps/db_archive" to
# | "/db_archive"
# 06/09/11 | Sylvan Creach | Also Inserted a COMPRESS option and set it to NO.
# | | Add the logic to remove archive older that 10 days (Arc_expire)
# ------------------------------------------------------------------------------
#
# ------------------------------------------------------------------------------
# Verify Oracle Sid and Days to cleanup have been specified on the Unix command
# ------------------------------------------------------------------------------
if [[ $# -lt 2 ]]; then
print "USAGE: $0 [SID] [DAYS] {Email Threshold} {Paging Threshold}"
print "SID - Database SID name"
print "DAYS - The number of days Old before removing (will add 1 for overhead)"
print " - (Will add 1 day for overhead)"
print "Email Thres.. - Override the default 79 percent Threshold"
print "Paging Thres.. - Override the default 90 percent Threshold"
exit 1
fi
DB=$1
DAYS=$2
EmailT=$3
PageT=$4
((DAYS+=1))
# ------------------------------------------------------------------------------
# Set the COMPRESS=Y if you want files compress going to the Backup_dir
# ------------------------------------------------------------------------------
COMPRESS=N
jobrunning=$(ps -ef | grep ibm_arch_cleanup | grep -c $DB)
if [[ $jobrunning -gt 1 ]]; then
print "$0 already running for $DB -exiting"
exit 0
fi
typeset -i dest_cnt dest_cnt2
typeset -u usid
usid=$DB
# ------------------------------------------------------------------------------
# Get database owner/password and set Oracle environment variables
# ------------------------------------------------------------------------------
. /usr/local/scripts/oracle/.dbowner
. /usr/local/scripts/oracle/oracle.login $DB
# ------------------------------------------------------------------------------
# Main routine
# ------------------------------------------------------------------------------
main() {
init
get_arch_dir
save_files
remove_files
chk_disk
rm_old
notify
}
# ------------------------------------------------------------------------------
# initialization function
# ------------------------------------------------------------------------------
init() {
# Set deletion of the archive logs in days
Arc_expire=10
archive_list="/tmp/rman_cleanup_files$$.out"
archive_del="/tmp/rman_cleanup_rmfiles$$.out"
OS=$(uname -a | awk '{print $1}')
STAMP=$(date "+%m/%d/%Y %H:%M")
LOG=$HOME/logs/arch_cleanup_${DB}.log
>$LOG
LOGCMD="tee -a $LOG"
if [[ $DAYS -le 2 ]]; then
print "DAYS must be at least a value of 2 or greater \nSetting to 2" | $LOGCMD
DAYS=2
fi
if [[ $DAYS -ge $Arc_expire ]]; then
print "DAYS must be at less than the Archive Expire value - $Arc_expire" | $LOGCMD
tempnum=$Arc_expire
((tempnum-=1))
DAYS=$tempnum
print "Setting to - $DAYS"
fi
if [[ $EmailT -gt 0 ]]; then
E_THOLD=$EmailT
print "Setting Email Threshold to - $E_THOLD"
else
E_THOLD=79
fi
if [[ $PageT -gt 0 ]]; then
P_THOLD=$PageT
print "Setting Paging Threshold to - $P_THOLD"
else
P_THOLD=90 #Paging Threshold
fi
mailusers="$(/usr/local/scripts/oracle/get_dba_email.ksh)"
pageuser="oncall"
print "$0 Script parameters:" | $LOGCMD
print "SID :$DB" | $LOGCMD
print "DAYS :$DAYS\n" | $LOGCMD
}
# ------------------------------------------------------------------------------
# Parse the INIT file and get the archive directory
# ------------------------------------------------------------------------------
get_arch_dir() {
INIT="$ORACLE_HOME/dbs/init${ORACLE_SID}.ora"
SPFILE="$ORACLE_HOME/dbs/spfile${ORACLE_SID}.ora"
if [[ -a $SPFILE ]]; then
SPTMP=/tmp/spfile${ORACLE_SID}.txt
strings $SPFILE > $SPTMP
INIT=$SPTMP
parse_init
elif [[ -a $INIT ]]; then
parse_init
else
echo "SCRIPT: $SN" | $LOGCMD
echo "${STAMP} : ERROR: ORACLE Database \"${ORACLE_SID}\" init or spfile file not found" | $LOGCMD
/usr/local/scripts/oracle/pager.ksh oncall "$ORACLE_SID - init for spfile not found"
echo "${STAMP} : Aborting..." | $LOGCMD
echo "ERROR SPFile not found - $SPFILE"
abort_pgm
fi
Backup_dir="/db_archive/$DB"
if [[ ! -d $Backup_dir ]]; then
mkdir $Backup_dir
ERR=$?
if [[ $ERR -ne 0 ]]; then
echo "$(date) : ERROR Could not create directory in $Backup_dir" | $LOGCMD
fi
fi
}
parse_init() {
typeset -l Linit_dest
dest_cnt2=0
dest_cnt=0
dest_cnt=$(egrep -i LOG_ARCHIVE_DEST $INIT |wc -l|awk '{print $1}')
arch_grep=$(egrep -i LOG_ARCHIVE_DEST $INIT)
print "INIT=$INIT arch grep command: $arch_grep dest_cnt=$dest_cnt"
case $dest_cnt in
1)
init_dest=$(egrep -i LOG_ARCHIVE_DEST $INIT | sed 's/"//g' | sed "s/'//g" |grep -v '^#' | cut -d= -f2)
Linit_dest=$init_dest
if [ $Linit_dest = "location" ]; then
init_dest=$(egrep -i LOG_ARCHIVE_DEST $INIT | sed 's/"//g' | sed "s/'//g" |grep -v '^#' | cut -d= -f3)
fi
print "dest = $init_dest"
;;
0)
echo "${STAMP} : ERROR: ORACLE Database \"${ORACLE_SID}\" Archive Destination not found" | $LOGCMD
echo "${STAMP} : Aborting..." | $LOGCMD
print "Option 0"
exit 1
;;
2)
print "Option *"
init_dest=$(egrep -i LOG_ARCHIVE_DEST_1 $INIT | sed 's/"//g'|sed "s/'//g" | grep -v '^#' | cut -d= -f2)
Linit_dest=$init_dest
if [ $Linit_dest = "location" ]; then
init_dest=$(egrep -i LOG_ARCHIVE_DEST_1 $INIT | sed 's/"//g' |sed "s/'//g"| grep -v '^#' | cut -d= -f3)
fi
dest_cnt2=$(egrep -i LOG_ARCHIVE_DEST_2 $INIT |wc -l)
if [[ $dest_cnt2 -eq 1 ]]; then
init_dest2=$(egrep -i LOG_ARCHIVE_DEST_2 $INIT | sed 's/"//g' |sed "s/'//g"| grep -v '^#' | cut -d= -f2)
Linit_dest=$init_dest2
if [ $Linit_dest = "location" ]; then
init_dest2=$(egrep -i LOG_ARCHIVE_DEST_2 $INIT | sed 's/"//g' |sed "s/'//g"| grep -v '^#' | cut -d= -f3)
fi
fi
print "dest 1 = $init_dest"
print "dest 2 = $init_dest2"
;;
*)
print "invalid number of detinations - $dest_cnt"
exit 1
;;
esac
#init_dest=$(egrep -i LOG_ARCHIVE_DEST $INIT | sed 's/"//g' | grep -v '^#' | cut -d= -f2)
print "Init Dest = [$init_dest]"
if [[ ! -d $init_dest ]]; then
dir_dest=$(dirname $init_dest)
else
dir_dest=$init_dest
fi
print "Archive Directory = [$dir_dest]" | $LOGCMD
if [[ -d $init_dest2 ]]; then
print "Second Archive destination = [$init_dest2]" | $LOGCMD
fi
}
# ------------------------------------------------------------------------------
# Compress archive files older than one day. Skip the most current archive file.
# ------------------------------------------------------------------------------
save_files() {
ERR=N
sav_files1="/tmp/save_files1_${DB}.tmp"
sav_log1="/tmp/save_log1_${DB}.tmp"
>$sav_files1
>$sav_log1
find ${dir_dest} -type f -name \*.arc > $archive_list
curr_nbr=$(cat $archive_list | wc -l)
if [[ $curr_nbr -gt 1 ]]; then
curr_arch=$(ls -1t ${dir_dest}/*.arc | head -1)
#get_curr_archives
cat $archive_list | while read aname
do
if [[ $curr_arch != $aname ]]; then
base_file=$(basename $aname)
if [[ $OS = "Linux" ]]; then
compress_name="${Backup_dir}/${base_file}.zip"
else
compress_name="${Backup_dir}/${base_file}.Z"
fi
cp_name="${Backup_dir}/${base_file}"
if [[ ! -a $cp_name && ! -a $compress_name ]]; then
# Save the file
if [[ $OS = "Linux" ]]; then
if [[ $COMPRESS = "Y" ]]; then
print "Compressing $compress_name" | $LOGCMD
echo "zip $compress_name $aname" >> $sav_files1
else
print "Coping $aname to $cp_name" | $LOGCMD
echo "cp -p $aname $cp_name" >> $sav_files1
fi
else
if [[ $COMPRESS = "Y" ]]; then
print "Compressing $compress_name" | $LOGCMD
echo "compress -c $aname > $compress_name" >> $sav_files1
else
print "Coping $aname to $cp_name" | $LOGCMD
echo "cp -p $aname $cp_name" >> $sav_files1
fi
fi
fi
else
print "Skipping $aname" | $LOGCMD
fi
done
# Chmod sav_files1 and run it.
chmod +x $sav_files1
$sav_files1 > $sav_log1 2>&1
if [[ $? -ne 0 ]]; then
print "Saving files had error bailing application" |$LOGCMD
cat $sav_log1 |$LOGCMD
abort_pgm
fi
fi
}
# ------------------------------------------------------------------------------
# Remove *.arc files from the arch_destination that are older than DAYS
# ------------------------------------------------------------------------------
remove_files() {
print "Removing archive log Files from the archive destination locations found older than $DAYS old..."
rm_archive1="/tmp/rm_archive1_${DB}.tmp"
rm_archive2="/tmp/rm_archive2_${DB}.tmp"
rm_log1="/tmp/rm_log1_${DB}.tmp"
rm_log2="/tmp/rm_log2_${DB}.tmp"
find $dir_dest -name \*.arc -mtime +${DAYS} 2>/dev/null | awk '{print "rm "$1}' > $rm_archive1
fcnt=$(cat $rm_archive1 | wc -l)
if [[ $fcnt -gt 0 ]]; then
print "Removing $fcnt archives from $dir_dest" | $LOGCMD
chmod +x $rm_archive1
$rm_archive1 > $rm_log1 2>&1
rc=$?
if [[ $rc -ne 0 ]]; then
print "Archive delete from $dir_dest had errors..." | $LOGCMD
cat $rm_log1 | $LOGCMD
fi
fi
if [[ -d $init_dest2 ]]; then
print "NOTE: Found a second Archive destination: $init_dest2"
find $dir_dest2 -name \*.arc -mtime +${DAYS} 2>/dev/null | awk '{print "rm "$1}' > $rm_archive2
fcnt=$(cat $rm_archive2 | wc -l)
if [[ $fcnt -gt 0 ]]; then
print "Removing $fcnt archives from $dir_dest2" | $LOGCMD
chmod +x $rm_archive2
$rm_archive2 > $rm_log2 2>&1
rc=$?
if [[ $rc -ne 0 ]]; then
print "Archive delete from $dir_dest2 had errors..." | $LOGCMD
cat $rm_log2 | $LOGCMD
fi
else
print "Warning: Second archive had no archive logs to delete"
fi
fi
}
# ------------------------------------------------------------------------------
# Check disk capacity. Email if disk capacity is greater than E_THOLD.
# Page if disk capacity if greater than P_THOLD
# ------------------------------------------------------------------------------
chk_disk() {
cap=$(df -k $dir_dest | grep -v Filesystem | awk '{print $4}' | sed "s/%//g")
if [[ $cap -gt $E_THOLD ]]; then
do_trim_cleanup 4
fi
cap=$(df -k $dir_dest | grep -v Filesystem | awk '{print $4}' | sed "s/%//g")
if [[ $cap -gt $P_THOLD ]]; then
do_trim_cleanup 3
/usr/local/scripts/oracle/pager.ksh $pageuser "$(uname -n) DIR - $dir_dest is ${cap}% full"
fi
}
do_trim_cleanup() {
DAYS=$1
if [[ $DAYS -eq 4 ]]; then
Tmsg="Email Threshold Exceeded"
else
Tmsg="Paging Threshold Exceeded"
fi
messbody=" $Tmsg \n The $dir_dest directory is more than $E_THOLD - $cap for $DB
Performing an emergency cleanup by reducing the DAYS kept to $DAYS"
remove_files
newcap=$(df -k $dir_dest | grep -v Filesystem | awk '{print $4}' | sed "s/%//g")
messbody="$messbody \nFree space after cleanup - $newcap"
echo "$messbody" | mailx -s "ibm_arch_rman_cleanup.ksh has a space issue $DB - !! Emergency Cleanup performed !!" $mailusers
}
# ------------------------------------------------------------------------------
# This standardized the the email when the Script aborts abnormally
# ------------------------------------------------------------------------------
abort_pgm() {
cat $LOG | mailx -s "$0 script had ERRORS" $mailusers
exit 1
}
# ------------------------------------------------------------------------------
# Notify users of script success
# ------------------------------------------------------------------------------
notify() {
if [[ $curr_nbr -gt 1 ]]; then
cat $LOG | $LOGCMD
else
print "No archives needed to process" | $LOGCMD
fi
# Doing file cleanup...
if [[ $DEBUG = "" ]]; then
print "Removing temp files"
rm -f $archive_list $archive_del
fi
}
get_curr_archives() {
curr_arch=$(sqlplus -s <<!
$DBAUSER/$DBAPASS
set pages 0 feed off
whenever sqlerror exit 1
select name from v\$archived_log where sequence#=(select max( sequence#) from v\$archived_log where name is not null)
/
exit
!)
xcnt=$(echo $curr_arch | wc | awk '{print $2}')
if [[ $xcnt -eq 1 ]]; then
if [[ -a $curr_arch ]]; then
print "Current Archive File= $curr_arch"
else
print "ERROR occured - $curr_arch"|$LOGCMD
exit 1
fi
else
save_curr_arch=$curr_arch
curr_arch=$(echo $save_curr_arch|awk '{print $1}')
if [[ -a $curr_arch ]]; then
print "Current Archive File= $curr_arch"
else
print "ERROR occured - $curr_arch"|$LOGCMD
exit 1
fi
fi
}
rm_old() {
find $Backup_dir -name \*.arc -mtime +${Arc_expire} -type f -exec rm -f {} \;
}
main
conditional on what? if files in one dir are different than another?
how about rsync? rsync's basic usage is like cp, but it avoids recopying files with identical sizes+timestamps. (it can also use checksums if you tell it.)
rsync -r sourcedir/ destdir
Note the ending / in sourcedir. Without that you'll end up with destdir/sourcedir/files instead of destdir/files
Also note that destdir doesn't have to exist yet, so it can be a procedurally generated string or what have you.
rsync also has a plethora of options for how to preserve permissions and so forth. It even has a network protocol, for syncing to files on a different server, but it works fine locally too.
Sorry, a few thoughts without reading the whole thread and script:
Check if backup ... delete input rman command could work for you (so you don't need to code everything by hand).
I mean that you could backup your archive logs with rman to disk (on a network drive, for example) using the delete input clause, so the backupsets could be later backed up to tape.
I suppose the script should run on AIX (given it's for IBM Production databases :)), a part from the powerful rsync, GNU cp, for example, has the -u option:
-u, --update
copy only when the SOURCE file is newer than the destination file or when the destination file is missing
What I meant by conditional copy is to just copy files that don't already exist in the destination. I'll check out resync and see how that works.
The script was created long before Oracle RMAN was used but the idea was we wanted to leave x number of days online and available for recovery and send a copy of them to a NAS drive for offsite backup.
Thanks so far for all the replies. This is a great site.
OK, now it's easy: just backup your database and archive logs to disk with rman using a certain retention (the restore and recovery will be fast from the disk).
Then backup the backupsets to tape (you can do it with rman or whatever system utility you prefer).
Long shot but I would suggest you eliminate the main and init functions and place the code in these functions directly into the shell script to ensure that declared variables have global scope. Depending on the version of ksh you are using, you can run into variable scoping issues declaring variables within a function and then using that variable in another function.
Often, if variables are first loaded or exported in the ksh main flow, then references to them in functions work, unless the function ends up in a pipe chain and so in a child process, in which case it can set it for its own benefit, but that change does not make it back to the parent.
A great part of learing is learning what works and staying away from wha does not, so life is double hard as a newbie: wrong approach, coded wrong, unstable or unworkable, much longer than necessary, usually not nicely formatted, and for many experts trying to help, not like anything they would write. I have an acronym, OPC, for other people's code or children. One obstacle to reusable code is that it may be cleaner, easier, better to write from scratch and discard. I use cscope (on a web server) so I can find bits of code without looking at all that OPC!
You mention that this is an hourly job. I assume cron.
There is potential for the script to misbehave if any run takes longer than one hour because the workfile names in /tmp do not have unique names.
May be worth having a simple interlock mechanism (e.g. a "flag" file) to prevent a second instance starting.
