Need help with listing file name and modified date on a huge directory

UNIX for Beginners Q & A
1

hi,

We have a huge directory that ha 5.1 Million files in it. We are trying to get the file name and modified timestamp of the most recent 3 years from this huge directory for a migration project.

However, the ls command (background process) to list the file names and timestamp is running for more than 24 hrs and it aborted after sometime.

Is there an efficient way to get the file names and modified timestamp in such huge directory in a relatively quicker time?

Appreciate any options/feedback. Thanks.

2

You don't mention the OS you are using. If you have a GNU find the -printf option will assist with this task.

You could try:

find . -printf "%TY-%Tm-%Td %TH:%TM %f\n"

This will give modification time and files name in this format:

2019-08-21 13:37 .bashrc
2019-10-24 13:09 .bash_history
2019-08-21 13:44 .bash_profile

Other info like filesize, permissions, etc are also available using the -printf option.

As far as feedback goes if you've been managing this huge directory on a system for a while you probably realize it's not a good idea. Many simple commands, shell auto completes etc. will take a long time and usually require processes to be killed. If your migrating this I would consider a sub folder layout, this could be derived from the filename created date etc.

1 Like
3

Hi Chuber,

thanks for the quick response. Its a SUSE Linux.

The command used is:
find -newermt "oct 1, 2016" -ls