Converting filenames from julian day to yyyy-mm-dd and retrieving weekly mean values

Shell Programming and Scripting
1

Hi, I need help to convert the filenames of my 9-year daily files (1999-2007) from a julian day to yyyy-mm-dd format. my original files are patterned likes the ones below.

1999001.txt
1999002.txt
1999003.txt
1999004.txt ...
1999365.txt

desired output:
19990101.txt
19990102.txt
19990103.txt
19990104.txt ...
19991231.txt

Based on the renamed files, I would like to take the average of the values of the 3rd column of the files at 7 days interval for each of the month. The daily files contains sample data like these.

1999001.txt renamed to 19990101.txt
12.3 124.4 2.0
12.1 124.2 1.2
12.4 123.5 1.5

1999002.txt renamed to 19990102.txt
12.3 124.4 1.8
12.1 124.2 2.2
12.4 123.5 1.2

1999003.txt renamed to 19990103.txt ...
12.3 124.4 1.5
 12.1 124.2 2.4
 12.4 123.5 1.5

1999007.txt renamed to 19990107.txt
12.3 124.4 1.2
12.1 124.2 2.6
12.4 123.5 1.8

hence, in Jan and Feb for instance, I would have resulting averaged files like the ones below:

199901_wk01.txt average of files in Jan 1999 from day 1-7
199901_wk02.txt average of files in Jan 1999 from day 8-14
199901_wk03.txt average of files in Jan 1999 from day 15-21
199901_wk04.txt average of files in Jan 1999 from day 22-28
199901_wk05.txt average of files in Jan 1999 from day 29-31

199902_wk01.txt average of files in Feb 1999 from day 1-7
and so on

Many thanks in advance.

2

Easier to deal with using Python. Hope this works for you

#! /usr/bin/python

import datetime
import glob
import os


# $ cal 10 1999
#
#    October 1999
#Su Mo Tu We Th Fr Sa
#                1  2
# 3  4  5  6  7  8  9
#10 11 12 13 14 15 16
#17 18 19 20 21 22 23
#24 25 26 27 28 29 30
#31
#
# 1999-10-28 return 5
def getwk(y,m,d):
        blanks=datetime.date(y,m,1).weekday()
        return (d+blanks)/7+1


average=dict()
for source in glob.glob('1999[0-3][0-9][0-9].txt'):
        year=int(source[0:4])
        nday=int(source[4:7].lstrip('0'))
        dt=datetime.timedelta(days=nday-1)
        d=datetime.date(year,1,1)+dt
        target=str(d)+'.txt'

        os.rename(source,target)

        month=int(target[5:7].lstrip('0'))
        day=int(target[8:10].lstrip('0'))
        week=getwk(year,month,day)

        # 199903_wk03.txt
        target_wk=target[0:4]+target[5:7]+'_wk'+'%02d'%week+'.txt'

        for line in open(target):
                try:
                        average[target_wk].append(float(line.split()[2]))
                except:
                        average[target_wk]=[float(line.split()[2])]

# write out the average to week files.
for i in average:
        avg=float(sum(average))/len(average)
        f=open(i,'w')
        f.write('%.2f\n'%avg)
        f.close()

I create the below script to generate all the sample files:

#! /bin/bash

rm -f 1999*.txt
seq -w 1 365 | while read d
do
        echo "12.3 124.4 $RANDOM
12.1 124.2 $RANDOM
12.4 123.5 $RANDOM" > 1999$d.txt
done
1 Like
3

@chihung: thanks much for your reply. I havent tried running a python script, how do i execute a python script?many thanks

4

Simply cut and paste the python code into a file and chmod 755 on that. Just like other shell scripts

1 Like