Need UNIX script to check checksum and dummy file creation.

UNIX for Beginners Q & A
1

Hi Folks,
I need a UNIX script which will copy files(Table wise) from source directory to destination directory (Under table directory) and also creates 2 additional files after getting copied to destination directory with extension .pdy and .ldy , . pdy file will be zero byte file should get created with table name (i.e. Table : xyz should get xyz.pdy file ) and .ldy file should get table_name, checksum of that file, and checksum method used (where comma(,) is field separator.
any help is greatly appreciated.

2

Welcome to the forum.

We're glad to help you if you're stuck, but please show your own efforts first, accompanied with OS, shell, and tools' versions, careful, detailed decription of the problem, meaningful sample data, and error messages if they exist (or description of malbehaviour otherwise).
If what you posted is homework / classwork, please post in the adequate forum filling in the entire form.

1 Like
3

Hi RudiC,

i'm executing this command in linux x86-64 bit server on korn shell.
i tried using below command but it doesn't help me completely.

for file in *.txt
do
  count=$(wc -l < $file)
  echo $file `date +%Y/%m/%d`, $count, cksum $file, checksum method use 
done

(comma(,) is field separator)

source path:

/data/files_to_transfer/xyz.txt

destination path with required output (files):

/data/final_files/
xyz.txt
xyz.pdy
xyz.ldy

content of xyz.ldy file should be as below:

row_count (xyz.txt),timestamp of coping the file,checksum value of file, checksum method used

where comma (,) is field seperator

4

I'm having a hard time to reconcile your specification in post #1 with what you show in post #3 - no cp command, no empty .pdy file creation, no cksum method determination, and the fields echo ed don't match the ones given in spec nor within post #3

How would you get along with this as a starting point:

SRC="/data/files_to_transfer"
TGT="/data/final_files"


(
 cd "$SRC" 

for FN in *.txt
   do    read CKS REST <<< $(cksum $FN)
        > "$TGT/${FN%.*}.pdy"
        echo "$FN", $(date +%Y/%m/%d), $(wc -l < $FN), $CKS, "checksum method used" > "$TGT/${FN%.*}.ldy"
        cp "$FN" "$TGT/$FN"
  done
)

This is executed in a subshell so the cd command won't change your parent shell's CWD. And, yes, you're reading correctly - the redirection without a command will just create an output file but won't put anything in.

5

Do you have sha1sum available? That is very good at generating/checking checksums. Expanding on the suggestion from RudiC:-

SRC="/data/files_to_transfer"
TGT="/data/final_files"
OWN=myappuser                                                                                      # Define file owner
GRP=myappgroup                                                                                     # Define file group
PRM=440                                                                                            # Define permissions
SUMS="${TGT}/sha1sums"

exec 50> "${SUMS}.new"                                                                             # Define the output file for checksums

cd "${SRC}"                                                                                        # Change to the source directory
for FN in *.txt
  do
     CKS=$(sha1sum "${FN}")                                                                        # Generate a checksum
     echo "${CKS}"  >&55                                                                           # Write the sha1sum record to intermediate file descriptor 55
     > "${TGT}/${FN%.*}.pdy"                                                                       # Create en empty file
     echo "${FN}", $(date +%Y/%m/%d), $(wc -l < ${FN}), ${CKS}, "sha1sum" > "${TGT}/${FN%.*}.ldy"  # Still creating this as before if you need it
     cp "${FN}" "${TGT}/${FN}"
     chown "${OWN}:${GRP}" "${TGT}/${FN}"                                                          # Set desired ownership
     chmod "${PRM}" "${TGT}/${FN}"                                                                 # Set desired permissions
  done  55>&50                                                                                     # Catch all the sha1sum output into the new file in one IO operation

grep -vf - "${SUMS}" < <(cut -f3 -d" "  "${SUMS}.new") > "${SUMS}.tmp"                             # Ignore previous checksums of the same filename
sort "${SUMS}.tmp" "${SUMS}.new" > "${TGT}/sha1sums"                                               # Put the older & new file checksums together neatly

This will also give you the file /data/final_files/sha1sums that your can use as input to sha1sum very simply like this:-

sha1sum -c /data/final_files/sha1sums

This command will read the checksum file and then try to confirm the files listed. Note that it will be a relative path in the checksum file, so you will have to be in the correct directory.

Would just collecting the sha1sum output be easier than trying to write your own and build your own checking process? (presumably some similar counting etc., but in reverse)

I hope that this helps,
Robin

3 Likes