I am new to Linux/Scripting and need some assistance in coming up with a script that can move certain amount of files from one directory to other every seconds.
Usercase: We have around 100k files in tmp directory on my server which needs to be moved to another folder to get processed. However I can not move all files at once due to limitations with file parsing as it hangs after seeing 100k files at once and can move around 10 files per seconds.
It would be helpful if someone can suggest a best way to achieve this through a shell script.
Why do you care how many files you can move in a second?
Why not just start moving files from the 1st file you want to move to the last file you want to move and just quit when you're done?
What are your "limitations with file parsing"? You can easily run into maximum argument length limits if you are trying to construct an mv command with 10,000 pathnames as operands; but that isn't file parsing. And, it can easily be worked around with the find utility with a couple of selection primaries and an -exec primary to perform the moves in chunk that will not exceed the arglist length limits on your system.
Depending on what operating system and shell you're using there may also be other ways to do what you need to do; but with the very limited information you have provided, it is very hard to make any reasonable suggestions.
If it really matters, note that trying to guess how many files can be moved in a second varies greatly from operating system to operating system, from filesystem type to filesystem type, how much activity there is on the filesystems involved, and on network characteristics if you're trying to move files to a remote directory as well as depending on the hardware being used and the sizes of the files being moved. You haven't given us any of the information that would be required to meet your requirements