Newbie needs help with some bash scripting

Shell Programming and Scripting
1

Please bear with me, I'm a beginner but have had some experience and decent knowledge to understand things as I read them and I'm in the process of trying to learn more. I recently inherited a UNIX server which has a bash script which is part of a crontab schedule that needs to be modified (or potentially replaced at this point)

Here's what I understand
Essentially, the bash script takes the data that is being sent to the server and after it is successfully received, runs a Unix2Dos command against it -- which doesn't seem to work due to an older version, as well are removes line fees in certain files. It then takes the data, renames it to have a unix2dos extension, sends it out via FTP to another server. It then archives the original data, and the data that has the Unix2Dos command run against it, by running a gzip command against it and then moves it to a folder with a YYYYMM format (example /folder1/Data/VENDOR/inbound/archive/format/201702)

Here is what they now need
Since the Unix2Dos process is not working, we have opted to use another server for the process but the data is still coming through this server and will always be coming into here. We just now don't want to FTP over to another one, or run the Unix2Dos command or remove any line feeds. However, we want to do the following now through a crontab job:

Takes the original data that is being sent to the server, without renaming it, and after it is successfully received, copy it over to a folder (/folder1/Data/VENDOR/inbound/CompanyArchive/format/Unix2Dos) for an outside FTP job to pick up. Only specific file names can be placed here. It then needs to gzip the data, move it over to a folder with a YYYYMM format (example /folder1/Data/VENDOR/inbound/CompanyArchive/format/201702)

I know it seems simple enough, but I have no idea how to do this based on my very limited knowledge of Unix, bash scripting and anything else. I'll share the script and hope many can try to help me here. Thanks in advance for any help provided

Here is our crontab schedule

# crontab /var/spool/cron/crontabs/username
# crontab -l
#ident  "@(#)username       1.00    05/08/09"
#
5 2,5,8,11,14,16,17,20 * * * bash /folder1/Data/bin/SplitMoveFiles.sh 2>/dev/null
1 11,21 * * * bash /folder1/Data/bin/MoveFormatFiles.sh 2>/dev/null
#1 * * * * bash /folder1/Data/bin/RenameFileAsSoonAsItIsDownloaded.sh 2>/dev/null

This is the bash script I'm focusing on MoveFormatFiles.sh
(sorry, had to modify this a bit to remove company names and such.. hopefully someone can help me with what I am trying to accomplish)

#!/usr/local/bin/bash
# Created by User on 20170204
#
# we are now receiving files in a  format for client data from vendor
# These files need to have their line feed replaced by carriage return + line feed
# The files will need to be posted to a subfolder in the SFTP site
 
# get date in the form MCDYMMDD
FormattedDate="`date -u +%Y%m%d`"
YearMonth="`date -u +%Y%m`"
# get date formatted with seconds
# log file path
MasterLog="/folder1/Data/FOLDER2/Place_Log/Client_"${FormattedDate}".log"
 
pid=$$
FilesPostedToClient="/d01/Data/ListOfPostedFiles."${pid}
CompanyFTPConnect="/d01/Data/CompanyFTPConnect."${pid}
 
# initiate with commands to connect to FTP site
echo "cd CLIENT_FORMAT" > ${CompanyFTPConnect}
echo "lcd /folder/Data/Vendor/inbound" >>  ${CompanyFTPConnect}
 
##########################################################
#Function that adds all available files to FTP command
##########################################################
funCheckIfThereAreFilesToSend(){
    for sFilesToSend in `cd /folder1/Data/"${1}"/inbound; ls -d "${2}"*.u2d`
    do
         echo "put ${sFilesToSend}" >>  ${CompanyFTPConnect}
    done
}
 
#############################################
# Function that creates a folder if it does not exist
#############################################
funCreateNewFolder(){
    sFolderPath="/folder1/Data/VENDOR/inbound/CompanyArchive/Format/"${YearMonth};
 
    if [ ! -e  ${sFolderPath} ];
    then
        mkdir ${sFolderPath};
    fi;
}
##############################################
#function to look for new files received from VENDOR
# make a copy of original file and store it as a compressed file.
##############################################
funHandleNewFormatFiles(){
    for sNewFile in `cd /folder1/Data/"${1}"/inbound; ls -d "${2}"*|/opt/sfw/bin/gawk '{c=split($0,a,".");if(a[c]!="u2d"){print $0}}'`
    do
        #Create folder that will house the archived informtaion if it does not exist
        funCreateNewFolder
 
        cd /folder1/Data/"${1}"/inbound
        echo "File " ${sNewFile} " was received from VENDOR." >> ${MasterLog}
 
        # compress and store/backup
        /usr/bin/gzip -q -c /folder1/Data/"${1}"/inbound/${sNewFile} >> \
            /folder1/Data/VENDOR/inbound/CompanyArchive/Format/${YearMonth}/${sNewFile}"_"${FormattedDate}.gz
        #Create a quick copy for now (comment the line below after a couple of months without issues)
        cp /folder1/Data/"${1}"/inbound/${sNewFile}  /folder1/Data/"${1}"/inbound/temp/
 
        # convert to DOS format
        #unix2dos /folder1/Data/"${1}"/inbound/${sNewFile}  /folder1/Data/"${1}"/inbound/${sNewFile}.u2d
        #/bin/awk '{printf "%s\r\n", $0}' /folder1/Data/"${1}"/inbound/${sNewFile} > /folder1/Data/"${1}"/inbound/${sNewFile}.u2d
        /opt/sfw/bin/gawk 'BEGIN{ORS="\015\012"}{print $0}' /folder1/Data/"${1}"/inbound/${sNewFile} > /folder1/Data/"${1}"/inbound/${sNewFile}.u2d
        #/bin/sed -e 's/$/\r/' /folder1/Data/"${1}"/inbound/${sNewFile} > /folder1/Data/"${1}"/inbound/${sNewFile}.u2d
        #tr -d "\000" < /folder1/Data/"${1}"/inbound/${sNewFile} > /folder1/Data/"${1}"/inbound/${sNewFile}.tr
        #/bin/perl -pe 's|\n|\r\n|' /folder1/Data/"${1}"/inbound/${sNewFile}.tr > /folder1/Data/"${1}"/inbound/${sNewFile}.u2d
 
        # if for some reason unix2dos does not see the need to convert and leaves as it found it, we'll go ahead and rename
        if [ ! -e /folder1/Data/"${1}"/inbound/${sNewFile}.u2d ];
        then
             mv -f /folder1/Data/"${1}"/inbound/${sNewFile} /folder1/Data/"${1}"/inbound/${sNewFile}.u2d
        fi;
 
        # get rid of orig
        rm -f /folder1/Data/"${1}"/inbound/${sNewFile}  2>/dev/null
 
        # we now want to make sure we remove any duplicate carriage returns (0d) - seen them in the past
        if [ -e /folder1/Data/"${1}"/inbound/${sNewFile}.u2d ];
        then
             tr -s "\015" < /folder1/Data/"${1}"/inbound/${sNewFile}.u2d > /folder1/Data/"${1}"/inbound/${sNewFile}.tr
             mv -f /folder1/Data/"${1}"/inbound/${sNewFile}.tr /folder/Data/"${1}"/inbound/${sNewFile}.u2d
        fi;
 
        # call function that adds any available u2d files
        funCheckIfThereAreFilesToSend $1 $2
    done
}
###########################################################
#Call above function for all dataset names and vendor paths
#FunctionName - VendorFolderName - DataSetName
##########################################################
funHandleNewFormatFiles VENDOR 006620.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 050720.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 111840.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 160800.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 185510.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 200120.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 221310.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 277310.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 277910.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 284370.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 323530.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 338600.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 347130.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 598200.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 730100.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 803220.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 809520.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 821720.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 850410.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 940210.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR A48275.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR AV8887.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR E03989.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR F90855.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR LU1178.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR LY5444.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR NP7727.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR PCBS.S.ONE.C1
funHandleNewFormatFiles VENDOR PCBS.V.ONE.C1
funHandleNewFormatFiles VENDOR QA0052.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR RE1939.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR RX1961.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR RZ7223.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SE3158.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SK9344.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SK9481.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SL0261.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SL1683.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SL3757.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SL5383.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SL9121.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SM2403.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SM2448.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SN6707.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SN8793.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SP7431.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SR2668.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR ST3975.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR ST6484.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR PCBS.I.ONE.C100106.C0
funHandleNewFormatFiles VENDOR PCBS.O.ONE.C100106.C0
funHandleNewFormatFiles VENDOR 000000.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 003070.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 030300.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 142650.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 214500.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 362200.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 452140.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 622900.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 754550.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 754550.VENDOR.VTNS.I7
funHandleNewFormatFiles VENDOR 788700.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 840530.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR DX2038.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR GH3849.VENDOR.SDN.C0
funHandleNewFormatFiles VENDOR RQ2307.VENDOR.SDN.C0
funHandleNewFormatFiles VENDOR SP1967.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SY3025.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SY3151.VENDOR.DCS.C6
 
 
##########################################################
# connect to the FTP site and verify that the files have been successfully posted; only then, archive them.
##########################################################
 
#production ftp site
#echo "password!" > ${CompanyFTPConnect}
 
echo "ls "  >> ${CompanyFTPConnect}
echo "quit" >> ${CompanyFTPConnect}
 
###################################
#Connect to Production server, post files and get list of files posted
##################################
/usr/bin/sftp username@ftp.site.com < ${CompanyFTPConnect} > ${FilesPostedToClient}
 
#cat ${CompanyFTPConnect}                #DEBUG
#cat "${FilesPostedToClient}"           #DEBUG
 
##########################################################
# archive all files that may have been posted
##########################################################
for sUploadedFile in `cat "${FilesPostedToCompany}"|/opt/sfw/bin/gawk '{cLineSplit=split($0,aLineSplit);\
    if((aLineSplit[cLineSplit] ~ ".VENDOR.DCS.C6")||(aLineSplit[cLineSplit] ~ ".VENDOR.VTNS.C0")||(aLineSplit[cLineSplit] ~ "PCBS.S.ONE.C1")||(aLineSplit[cLineSplit] ~ "PCBS.V.ONE.C1")||(aLineSplit[cLineSplit] ~ ".I.ONE.C100106.C0")||(aLineSplit[cLineSplit] ~ ".O.ONE.C100106.C0")||(aLineSplit[cLineSplit] ~ "VENDOR.VTNS.I7")||(aLineSplit[cLineSplit] ~ "VENDOR.SDN.C0")){print aLineSplit[cLineSplit]}}'`
do
    echo "File " ${sUploadedFile} " was successfully posted to Company sFTP site \
and will now get moved to the /archive folder" >> ${MasterLog}
 
    if [ -r /folder1/Data/VENDOR/inbound/${sUploadedFile} ]; then
 
        /usr/bin/gzip -q -c /folder1/Data/*/inbound/${sUploadedFile}  > \
            /folder1/Data/VENDOR/inbound/CompanyArchive/Format/${YearMonth}/${sUploadedFile}.gz
        rm -f /folder1/Data/*/inbound/${sUploadedFile}
    fi
done
 
rm "${CompanyFTPConnect}" 2>/dev/null
rm "${FilesPostedToClient}" 2>/dev/null
2

To help you we need to know a bit more... like your environment:
What architecture, what OS and version etc...
now:

1 11,21 * * * bash /folder1/Data/bin/MoveFormatFiles.sh 2>/dev/null

Why call bash as it is mentioned as shell to be used in your script ?
Unix2Dos is standard but not on all UNIX and may not have that name ( unix2dos, ux2dos...) without more clues about your OS hmmm....

3

I can't see a reason why unix2dos should not work, even an older version. Does it if you run it interactively? Does it work on your data files, or mayhap on highly simplified test files?
For clarity, you might want to simplify your - overcomplicated - script as well, and add some decent error checking and logging.

4

Thank you both for responding. As far as I can see, it's Sun Microsystems on SunOS 5.9 running Solaris 2.6

As I mentioned, I inherited this but not necessarily inherited the server. I inherited the functions and processes it does. To give an idea of it's main function, it's responsible for receiving a lot of data company wide through a NDM connection. It's one of those servers that needs to be up almost all the time.

From what I know, the user that is running this script is not able to run it as shell but can run it as bash, if that makes any sense at all (sorry if it doesn't, I am fairly new to understand this)

With that said, I should be more clear. It's not that the unix2dos command doesn't work. It does work. It just doesn't work in every case for how they want it to work. The person who wrote this code, made it overly complicated because the person who requested this type of data was also making things complicated. In some cases, line feeds were being left in there after the unix2dos command was ran, but they needed to be removed. In other cases, the line feeds needed to stay in there in certain spots so trying to figure out what was and wasn't needed was a challenge.

Myself and one other coworker figured out that the unix2dos command on this server was older, and wasn't doing everything it was supposed to but we weren't able to get a newer version on there. However, he had the latest on a server he had which worked 100% the way they wanted it to and essentially now he just wants to pick up the data from our server and run the command from there (we can't move the NDM process over, unfortunately, as much as I'd love to... mainly because it does other company functions that actually DO work on this one)

Hope this makes sense and thank you both for the responses.