Merge files according date

Shell Programming and Scripting
1

Gents,
I have a big list of files and i would like to merge some of them using a select date

In my case i would like to merge all files which are bigger that 2017-05-15

List of files

SG-10_2017-05-10de.csv
SG-10_2017-05-11de.csv
SG-10_2017-05-12de.csv
SG-10_2017-05-13de.csv
SG-10_2017-05-14de.csv
SG-10_2017-05-15de.csv
SG-10_2017-05-16de.csv
SG-10_2017-05-17de.csv
SG-10_2017-05-18de.csv
SG-10_2017-05-19de.csv
SG-10_2017-05-20de.csv

I have try with this code

for i in {15..30}
do
cat "SG-10_2017-05-${i}de.csv" >> merged
done

the problem is that in the loop i have files that are not present.. Then, I will like to use the date 2017-05-15 as the first file to concatenate to the last file present.

Thanks for your help

2

What does it mean for you "bigger tha[n] 2017-05-15". Is 2017-08-10 a possibility?
In that particular example, what would be the ceiling or last file name? Any files you are not showing?

1 Like
3

Hello jiam912,

Considering that the day your file gets created has the same date in it's name, if this condition is TRUE then could you please try following and let me know if this helps you.

find -type f -mtime -$((($(date +%s) - $(date --date "2017-05-15" +%s))/86400)) -exec cat >> merged {} \+

So explanation in short of above code(as it is too late in IST now, will add full explanation in morning) is getting the days difference and searching only those files which are older than(or same day 2017-05-15) and concatenating their values then into merged file name.

I hope this helps you, let me know if you have any queries on same.

NOTE: If your files are always .csv then you could add a condition like -name *.csv into above command and could test with it too.

find -type f -name "*.csv" -mtime -$((($(date +%s) - $(date --date "2017-05-15" +%s))/86400)) -exec cat >> merged {} \+

Thanks,
R. Singh

1 Like
4

Hi Singh,

Any timestamp is not guarantee to have a correlation with the name of the file.

1 Like
5

Dear RaviderSingh13 and Aia.

Mr Alia is correct I would like to merge the files using the name file not the timestamp

The purpose is to select all files with name more than 2017-05-15 and merge all of them.

Input files

SG-10_2017-05-10de.csv
SG-10_2017-05-11de.csv
SG-10_2017-05-12de.csv
SG-10_2017-05-13de.csv
SG-10_2017-05-14de.csv
SG-10_2017-05-15de.csv
SG-10_2017-05-16de.csv
SG-10_2017-05-17de.csv
SG-10_2017-05-18de.csv
SG-10_2017-05-19de.csv
SG-10_2017-05-20de.csv

Merged files

SG-10_2017-05-15de.csv
SG-10_2017-05-16de.csv
SG-10_2017-05-17de.csv
SG-10_2017-05-18de.csv
SG-10_2017-05-19de.csv
SG-10_2017-05-20de.csv

Appreciate your help

6

Please, try the following and see if produces the result you want.

cat SG-10_2017-05-1[6-9]de.csv SG-10_2017-05-[2-9][0-9]de.csv >> merge

or if you know before hand you want up to SG-10_2017-05-20de.csv

cat SG-10_2017-05-1[6-9]de.csv SG-10_2017-05-20de.csv >> merge