Shell Script to Loop through Quarter dates

Hi,

Trying to automate a Postgres query using Shell script

Every month 1st week has to run a Monthly Queries to generate delimited files.

July 1st start of fiscal yr which has 4 Quarters until next June 30th

Example

If I'm running on Sept 5th it has to generate one file(Becuase it falls under first Quarter), dates should be between 2017-07-01 and 2017-08-31 to Postgres SQL as Input.

  • July - Sept (1 file)
  • Oct - Dec (2 files which should generate above file too)
  • Jan - Mar (3 files)
  • April - Jun (4 files)

Another Example
If I ran the script on Nov 5th, then it has to get me 2 sets of files

  • One from July 1st to Sept 30th
  • 2nd file from Oct 1st to Oct 31st
quarter_start_date=$(date +"%m %Y" | awk '{printf("%d-%02d-01\n", $2, int(($1 - 1) / 3) * 3 + 1)}')
echo $quarter_start_date
quarter_end_date=$(date '+%Y-%m-%d' -d "-$(date +%d) days")
echo $quarter_end_date
fiscal_year=$(date -d "+1 Year" "+%Y")
echo $fiscal_year
mon=$(date +'%m')
if [ $mon -ge 7 -a $mon -le 9 ]; then
     qtr=1
elif [ $mon -ge 10 -a $mon -le 12 ]; then
     qtr=2
elif [ $mon -ge 1 -a $mon -le 3 ]; then
     qtr=3
elif [ $mon -ge 4 -a $mon -le 6 ]; then
     qtr=4
fi
echo $qtr

I would be inclined to create a data file rather than a lot of arithmetic.

1 20170701 20170930
2 20171001 20171231
3 20180101 20180331
4 20180401 20180630

Then you read this file for the number of quarters you are reporting. This is also way easier to modify if the company gets sold and changes its year end.

---------- Post updated 08-22-17 at 10:19 AM ---------- Previous update was 08-21-17 at 10:51 PM ----------

today=$(date +%Y%m%d)
while read quarter start end
do
if [ $start -ge $today ]
then
   exit
fi
if [ $today -gt $end ]
then
  let new_end=$today-$(date +%d)+1        #first day of month
  let new_end=$(date +%Y%m%d --date="$new_end yesterday")   #last day of previous month
fi
if [ $new_end -le $start ]
then
   exit  
else 
   end=$new_end
fi
do sql with $quarter $start $end
done <quarter_file  

What should be in the files if the end of previous month is also the end of previous quarter?

For example, if the process is run on 4-Oct-2017, then:
1) "File 1" has dates of Quarter 1 i.e. 01-Jul-2017 to 30-Sep-2017
2) What dates does "File 2" have?

A few similar cases are below:

Process Date: 05-Jan-2018
File 1 : Dates of Quarter 1 i.e. 01-Jul-2017 to 30-Sep-2017
File 2 : Dates of Quarter 2 i.e. 01-Oct-2017 to 31-Dec-2017
File 3 : What dates does this have?

Process Date: 05-Apr-2018
File 1 : Dates of Quarter 1 i.e. 01-Jul-2017 to 30-Sep-2017
File 2 : Dates of Quarter 2 i.e. 01-Oct-2017 to 31-Dec-2017
File 3 : Dates of Quarter 3 i.e. 01-Jan-2018 to 31-Mar-2018
File 4 : What dates does this have?

Also, what dates should be in the file(s) if the process is run in July of any year?

For example, if the process is run on 4-Oct-2017, then:
1) "File 1" has dates of Quarter 1 i.e. 01-Jul-2017 to 30-Sep-2017
2) What dates does "File 2" have?
We will be generating only 1 quarter file.

A few similar cases are below:

Process Date: 05-Jan-2018
File 1 : Dates of Quarter 1 i.e. 01-Jul-2017 to 30-Sep-2017
File 2 : Dates of Quarter 2 i.e. 01-Oct-2017 to 31-Dec-2017
File 3 : What dates does this have?
We will be generating only 2 quarter files. NO Jan Data

Process Date: 05-Apr-2018
File 1 : Dates of Quarter 1 i.e. 01-Jul-2017 to 30-Sep-2017
File 2 : Dates of Quarter 2 i.e. 01-Oct-2017 to 31-Dec-2017
File 3 : Dates of Quarter 3 i.e. 01-Jan-2018 to 31-Mar-2018
File 4 : What dates does this have?
3 quarters only

Also, what dates should be in the file(s) if the process is run in July of any year?
Generate All 4 quarters files. Because data is not complete yet in July. For that reason, we don't generate data for the month which we are running the script

But script will be running every month 1st week. So most of the time the previous month may not be the end of Quarter. So we have to consider following example also

For example, if the process is run on 4-Sept-2017, then:
Only One file has dates of Quarter 1 i.e. 01-Jul-2017 to 31-Aug-201

Let me know if anything is unclear and thanks for the follow-up questions

---------- Post updated at 02:29 PM ---------- Previous update was at 02:28 PM ----------

Thanks, JGT. the script is not working but i will use that as the base to cover all scenarios.

Try this function with variable "start of fiscal year" in the first argument, the quarter in the second:

QUARTER()       { TMP=$(($1 %10000/100+($2 -1)*3))
                  printf "%d %d " $2 $(( ($1 /10000+TMP/12) * 10000 + TMP%12 *100 + $1 %100))
                  date -d"$(( ($1 /10000+(TMP+3)/12) * 10000 + (TMP+3)%12 *100 + $1 %100)) -1day" +"%Y%m%d"
                }
for Q in 1 2 3 4; do QUARTER 20170701 $Q; done
1 20170701 20170930
2 20171001 20171231
3 20180101 20180331
4 20180401 20180630
1 Like

Would this come close to what you need:

for X in 20170805 20170830 20170915 20171005 20171112 20180522
   do   END=$(date +"%Y%m%d" -d"$X - $(date +"%d" -d$X) days")
        for Q in 1 2 3 4
          do    read _ QSTART QEND <<< $(QUARTER 20170701 $Q)
                printf "$X:  sqlplus $USER%%$PASSWD %d %d " $Q $QSTART
                [ "$END" -gt "$QEND" ]  &&      printf "%d\n" $QEND \
                                        ||      { printf "%d\n" $END
                                                  break 
                                                }
          done
   done
20170805:  sqlplus user% 1 20170701 20170731
20170830:  sqlplus user% 1 20170701 20170731
20170915:  sqlplus user% 1 20170701 20170831
20171005:  sqlplus user% 1 20170701 20170930
20171112:  sqlplus user% 1 20170701 20170930
20171112:  sqlplus user% 2 20171001 20171031
20180522:  sqlplus user% 1 20170701 20170930
20180522:  sqlplus user% 2 20171001 20171231
20180522:  sqlplus user% 3 20180101 20180331
20180522:  sqlplus user% 4 20180401 20180430
1 Like

I noticed a slight error in my original script.

today=$(date +%Y%m%d) 
while read quarter start end
do if [ $start -ge $today ] 
then
    exit
 fi 
if [ $today -gt $end ]
 then 
  let new_end=$today-$(date +%d)+1        #first day of month
   let new_end=$(date +%Y%m%d --date="$new_end yesterday")   #last day of previous month
 #fi
 if [ $new_end -le $start ] 
then
    exit
   else
     end=$new_end 
fi
fi
 do sql with $quarter $start $end done <quarter_file

Also if you need to re-run an old report, just force a value for 'today'.
I would presume that if you run the report in July, you are expecting 4 quarterly reports for the previous fiscal year.

1 Like

Here's my attempt in Perl, in case Perl is an option for you.
The following program accepts a date in "yyyymmdd" format and returns the start/end date pairs for the previous quarters and month as per your post.
If no date is passed, it uses today's date.
Only the core modules are used, so no additional module from CPAN is required.
However, these core modules - Time::Piece and Time::Seconds are available from Perl version 5.8 onwards, so that's the minimum version required.

$ 
$ cat -n get_quarter_dates.pl 
     1    #!/usr/bin/perl -w
     2    # ====================================================================
     3    # Usage: perl get_quarter_dates.pl 20170705
     4    #        perl get_quarter_dates.pl
     5    # Desc : If a date is passed in "YYYYMMDD" format, then it is used.
     6    #        Otherwise, current date is used.
     7    # ====================================================================
     8    use strict;
     9    use Time::Piece;
    10    use Time::Seconds;
    11    
    12    my $date;
    13    my $start_fiscal_yr;
    14    # Fiscal year starts from July. If that changes, change the next line.
    15    my $start_fiscal_month = "07";
    16    
    17    # Use either the date passed or today's date as the working date.
    18    if ($#ARGV >= 0) {
    19        $date = Time::Piece->strptime($ARGV[0], "%Y%m%d");
    20    } else {
    21        $date = Time::Piece->new;
    22    }
    23    
    24    # Determine the start of fiscal year
    25    if ($date->mon > $start_fiscal_month) {
    26        # Go to the fiscal month of current year
    27        $start_fiscal_yr = Time::Piece->strptime($date->year.$start_fiscal_month."01", "%Y%m%d");
    28    } else {
    29        # Go back 12 months to the previous year.
    30        # Then set the fiscal month of previous year as start of fiscal year.
    31        my $prev_yr_dt = $date->add_months(-12);
    32        $start_fiscal_yr = Time::Piece->strptime($prev_yr_dt->year.$start_fiscal_month."01", "%Y%m%d");
    33    }
    34    
    35    # Set up the quarter start/end dates and then loop through the quarters.
    36    my $qtr_start_date = $start_fiscal_yr;
    37    my $qtr_end_date = $start_fiscal_yr->add_months(3) - ONE_DAY;
    38    foreach my $i (1..4) {
    39        # If quarter end is later than working date then determine previous month end
    40        # and print the interval (qtr_start, prev_mth_end) only if qtr_start is earlier
    41        # than previous month end.
    42        if ($qtr_end_date >= $date) {
    43            my $curr_mth_start_date = Time::Piece->strptime($date->strftime("%Y-%m-01"), "%Y-%m-%d");
    44            my $prev_mth_end_date = $curr_mth_start_date - ONE_DAY;
    45            if ($qtr_start_date < $prev_mth_end_date){
    46                printf("%s %s\n", $qtr_start_date->ymd, $prev_mth_end_date->ymd);
    47            }
    48            last;
    49        }
    50        printf("%s %s\n", $qtr_start_date->ymd, $qtr_end_date->ymd);
    51        # Move on to the next quarter.
    52        $qtr_start_date = $qtr_end_date + ONE_DAY;
    53        $qtr_end_date = $qtr_start_date->add_months(3) - ONE_DAY;
    54    }
    55    
$ 
$ 

A few test runs:

$ 
$ # No date passed; uses today's date.
$ perl get_quarter_dates.pl
2017-07-01 2017-07-31
$ 
$ # For July 2017, generate the start/end dates of 4 quarters of previous fiscal year
$ perl get_quarter_dates.pl 20170705
2016-07-01 2016-09-30
2016-10-01 2016-12-31
2017-01-01 2017-03-31
2017-04-01 2017-06-30
$ 
$ perl get_quarter_dates.pl 20170805
2017-07-01 2017-07-31
$ 
$ perl get_quarter_dates.pl 20170905
2017-07-01 2017-08-31
$ 
$ perl get_quarter_dates.pl 20171005
2017-07-01 2017-09-30
$ 
$ perl get_quarter_dates.pl 20171105
2017-07-01 2017-09-30
2017-10-01 2017-10-31
$ 
$ perl get_quarter_dates.pl 20171205
2017-07-01 2017-09-30
2017-10-01 2017-11-30
$ 
$ perl get_quarter_dates.pl 20180105
2017-07-01 2017-09-30
2017-10-01 2017-12-31
$ 
$ perl get_quarter_dates.pl 20180205
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-01-31
$ 
$ perl get_quarter_dates.pl 20180305
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-02-28
$ 
$ perl get_quarter_dates.pl 20180405
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-03-31
$ 
$ perl get_quarter_dates.pl 20180505
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-03-31
2018-04-01 2018-04-30
$ 
$ perl get_quarter_dates.pl 20180605
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-03-31
2018-04-01 2018-05-31
$ 
$ # And back to the same logic for July of next year
$ perl get_quarter_dates.pl 20180705
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-03-31
2018-04-01 2018-06-30
$ 
$ 

The start of the fiscal month is set as "07" (July) at line # 15, but it can be changed.
I changed the start of fiscal month to "10" (October) below:

$ 
$ cat -n get_quarter_dates.pl | sed -n 1,16p
     1    #!/usr/bin/perl -w
     2    # ====================================================================
     3    # Usage: perl get_quarter_dates.pl 20170705
     4    #        perl get_quarter_dates.pl
     5    # Desc : If a date is passed in "YYYYMMDD" format, then it is used.
     6    #        Otherwise, current date is used.
     7    # ====================================================================
     8    use strict;
     9    use Time::Piece;
    10    use Time::Seconds;
    11    
    12    my $date;
    13    my $start_fiscal_yr;
    14    # Fiscal year starts from July. If that changes, change the next line.
    15    my $start_fiscal_month = "10";
    16    
$ 
$ 

And the script works accordingly:

$ 
$ perl get_quarter_dates.pl 20171005
2016-10-01 2016-12-31
2017-01-01 2017-03-31
2017-04-01 2017-06-30
2017-07-01 2017-09-30
$ 
$ perl get_quarter_dates.pl 20171105
2017-10-01 2017-10-31
$ 
$ perl get_quarter_dates.pl 20171205
2017-10-01 2017-11-30
$ 
$ perl get_quarter_dates.pl 20180105
2017-10-01 2017-12-31
$ 
$ perl get_quarter_dates.pl 20180205
2017-10-01 2017-12-31
2018-01-01 2018-01-31
$ 
$ perl get_quarter_dates.pl 20180905
2017-10-01 2017-12-31
2018-01-01 2018-03-31
2018-04-01 2018-06-30
2018-07-01 2018-08-31
$ 
$ 

Thereafter, incorporating this script in a shell script should be easy.
I assume you are connecting to PostgreSQL via the "psql" command-line utility.
If so, the Bash shell script looks like this:

$ 
$ cat -n fetch_pg_data.sh 
     1    #!/bin/bash
     2    # ========================================================================================
     3    # If a date in "YYYYMMDD" format is passed to this shell script, it will be fed to
     4    # the Perl program "get_quarter_dates.pl". The output from the Perl program is
     5    # passed to the PostgreSQL client "psql". The output file is named uniquely according
     6    # to the start and end dates.
     7    # ========================================================================================
     8    perl get_quarter_dates.pl $1 |
     9    while read dt_start dt_end
    10    do
    11        outfile="data_${dt_start}_${dt_end}.log"
    12        echo "Start Date : $dt_start"
    13        echo "End Date   : $dt_end"
    14        echo "Spool file : $outfile"
    15        echo "================================="
    16        psql -d mydb -o $outfile -c "select sum(values) from t_data where dt between '$dt_start' and '$dt_end'"
    17    done
    18    
$ 
$ 

If the script has to be run for some previous date, it can be passed to the shell script.
The shell script, in turn, passes it on to Perl.
The "outfile" name is based on the start/end dates.

Execution of the shell script:

$ 
$ . fetch_pg_data.sh 20170705
Start Date : 2016-07-01
End Date   : 2016-09-30
Spool file : data_2016-07-01_2016-09-30.log
=================================
Start Date : 2016-10-01
End Date   : 2016-12-31
Spool file : data_2016-10-01_2016-12-31.log
=================================
Start Date : 2017-01-01
End Date   : 2017-03-31
Spool file : data_2017-01-01_2017-03-31.log
=================================
Start Date : 2017-04-01
End Date   : 2017-06-30
Spool file : data_2017-04-01_2017-06-30.log
=================================
$ 
$ 
$ cat data_2016-07-01_2016-09-30.log
       sum        
------------------
 50.4268608423881
(1 row)

$ 
$ cat data_2016-10-01_2016-12-31.log
       sum        
------------------
 51.2326508448459
(1 row)

$ 
$ cat data_2017-01-01_2017-03-31.log
       sum        
------------------
 47.1403563958593
(1 row)

$ 
$ cat data_2017-04-01_2017-06-30.log
       sum        
------------------
 44.4815620388836
(1 row)

$ 
$ 

Thanks Tyler!. This was Awesome and complete script. But One issue I was facing, see below input for 20180105

perl get_quarter_dates.pl 20180105
2016-07-01 2016-09-30
2016-10-01 2016-12-31
2017-01-01 2017-03-31
2017-04-01 2017-06-30

perl get_quarter_dates.pl 20180205
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-01-31

Hmm... the output of 20180205 is correct, but the output for 20180105 looks like the one you should've seen for 20170705.
Can't figure out, without further information, why it would work incorrectly for just one value.
The code extracts the month from 20180105 and since that is less than 7, it would:

  • go back 1 year to 20170105
  • extract the year from it, to get 2017
  • add the fiscal month "07" and date "01" to the year, to get 20170701 (start fiscal yr)
  • and then jump quarters from start fiscal year. so it should return, for 20180105:
$ 
$ perl get_quarter_dates.pl 20180105
2017-07-01 2017-09-30
2017-10-01 2017-12-31
$ 

A few questions to get more information:

1) Could you check again if that output is really for 20180105 and not 20170705 ?
2) What output do you get for 20170705 ?
3) Are there any other dates, besides 20180105, for which the output is incorrect ?
4) Could you run the program for all the dates I ran it, in my post ?
5) Did you make any changes in the Perl program I posted ?
6) What is your Perl version ? The following command shows the version:

perl -v

I tested it on my system that has Perl 5.24 on Debian 9.

$ 
$ perl -v

This is perl 5, version 24, subversion 1 (v5.24.1) built for x86_64-linux-gnu-thread-multi
(with 73 registered patches, see perl -V for more detail)

Copyright 1987-2017, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

$ 
$ uname -morsv
Linux 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u3 (2017-08-06) x86_64 GNU/Linux
$ 
$ cat /etc/debian_version
9.1
$ 
$ 

7) What are the versions of your Perl modules Time::Piece and Time::Seconds ? The following commands show the version:

perl -MTime::Piece -le 'print $Time::Piece::VERSION'
perl -MTime::Seconds -le 'print $Time::Seconds::VERSION'

I have version 1.31 for both these modules.

$ 
$ perl -MTime::Piece -le 'print $Time::Piece::VERSION'
1.31
$ 
$ perl -MTime::Seconds -le 'print $Time::Seconds::VERSION'
1.31
$ 

A few questions to get more information:

1) Could you check again if that output is really for 20180105 and not 20170705 ?

Yes I still see same output for both inputs

$ perl get_quarter_dates.pl 20180105
2016-07-01 2016-09-30
2016-10-01 2016-12-31
2017-01-01 2017-03-31
2017-04-01 2017-06-30

2) What output do you get for 20170705 ?

$ perl get_quarter_dates.pl 20170705
2016-07-01 2016-09-30
2016-10-01 2016-12-31
2017-01-01 2017-03-31
2017-04-01 2017-06-30

3) Are there any other dates, besides 20180105, for which the output is incorrect ?
Tried all months except for Jan remaining all looks perfect

4) Could you run the program for all the dates I ran it, in my post?

$ perl get_quarter_dates.pl 20170705
2016-07-01 2016-09-30
2016-10-01 2016-12-31
2017-01-01 2017-03-31
2017-04-01 2017-06-30
$ perl get_quarter_dates.pl 20170805
2017-07-01 2017-07-31
$ perl get_quarter_dates.pl 20170905
2017-07-01 2017-08-31
$ perl get_quarter_dates.pl 20171005
2017-07-01 2017-09-30
$ perl get_quarter_dates.pl 20171105
2017-07-01 2017-09-30
2017-10-01 2017-10-31
$ perl get_quarter_dates.pl 20171205
2017-07-01 2017-09-30
2017-10-01 2017-11-30
$ perl get_quarter_dates.pl 20180105
2016-07-01 2016-09-30
2016-10-01 2016-12-31
2017-01-01 2017-03-31
2017-04-01 2017-06-30
$ perl get_quarter_dates.pl 20180205
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-01-31
$ perl get_quarter_dates.pl 20180305
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-02-28
$ perl get_quarter_dates.pl 20180405
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-03-31
$ perl get_quarter_dates.pl 20180505
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-03-31
2018-04-01 2018-04-30
$ perl get_quarter_dates.pl 20180605
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-03-31
2018-04-01 2018-05-31
$ perl get_quarter_dates.pl 20180705
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-03-31
2018-04-01 2018-06-30
$ perl get_quarter_dates.pl 20180805
2018-07-01 2018-07-31
$ perl get_quarter_dates.pl 20190105
2017-07-01 2017-09-30
2017-10-01 2017-12-31
2018-01-01 2018-03-31
2018-04-01 2018-06-30

5) Did you make any changes in the Perl program I posted ?

Nope, Just copied as it is

6) What is your Perl version? The following command shows the version:

perl -v

This is perl, v5.10.1 (*) built for x86_64-linux-thread-multi

Copyright 1987-2009, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

Thank you for testing it for various inputs and posting the Perl version.
I could not find a working Perl 5.10 version to test it, so am unable to reproduce the bug.
However, after searching on the Internet, it looks like "add_months()" method has a few quirks while working with dates close to the ends of a month.
My best guess is that this line is the culprit:

 31    my $prev_yr_dt = $date->add_months(-12);
 

When you subtract 12 months from "20180105", I think it goes back to some day in December, 2016 instead of "20170105".
Thereafter, it goes takes the year "2016", pads "0701" and arrives at "20160701" as the start fiscal year.
If you print $prev_yr_dt, you should be able to see what date exactly it goes to.

For months 1 through 6 of any year, we only have to jump back a bit enough to reach the previous year. So, instead of 12 a safe jump could be 9.
Change the line # 31 from this:

 31    my $prev_yr_dt = $date->add_months(-12);
 

to this:

 31    my $prev_yr_dt = $date->add_months(-9);
 

That should fix the problem.

A more robust solution would be to extract the year for all dates in months 1 through 6.
Subtract one from the year to get to the previous year.
Then pad month "07" and day "01" to arrive at the start of fiscal year.
That way, the "add_months()" method is avoided altogether.

1 Like

Thanks Tyler. Worked.