We are taking backup of our application data(cobol file system, AIX/unix) before and after EOD job runs. The data size is approximately 260 GB in biggest branch. To reduce the backup time, 5 parallel execution is scheduled through control-m which backups up the files in 5 different *.gz. The job takes approximately 90 minutes to complete. Backup is done locally and later on it's passed to different location for retention.
Issue:
Each execution takes approximately 10% CPU which is putting heavy load on the server. This is causing the issue as the server is hosting multiple branches.
gtar -cvzf ${DBTBKUP}/${DATADIR}/${DATADIR}${BKUPSEQ}.gz [${FILESET}*
${DBTBKUP}/${DATADIR}/${DATADIR}${BKUPSEQ}.gz - {backup directory}
[${FILESET}* - fileset backed-up
Is there anyway, we can improve the backup further? Is there any new features in GTAR to fasten the backup or any new backup command which can replace GTAR? Any suggestion would be really appreciated.
If computers had a "go faster" setting, everyone would be already using it.
Are your disks organized in such a way that splitting the job in 5 makes it better, or worse?
1 Like
Thanks for your response.
Yes, the splitting is helping to completing the job execution in less than 90 mins(otherwise would have taken 5+ hours). Each job run is consuming 10% CPU, overall 50% CPU for 5 runs, this is putting a lot of load on the server as the same server is shared by multiple branches. My question - is there any change we can do on the GTAR command I provided earlier for faster backup? Is there any faster alternative to GTAR? Any other suggestion pls?
Blocking factor - I used blocking factor option in GTAR thinking that it would give better performance. Trying to zip and backup 5 gb file on local disc. I didnt find any difference between the below execution interms of end result(backup completion time or size)
gtar -cvzb 2 -f ABC3.gz BIGFILE
gtar -cvzb 1024 -f ABC3.gz BIGFILE
gtar -cvzf ABC3.gz BIGFILE
Any suggestion pls?
My suggestion would be to spell it 'please', not 'pls'.
File I/O, to a disk, doesn't really care about block size as much as raw tape I/O would -- especially since compressing it with -z is going to mess up all your block sizes anyway. 1024 bytes in, ???? bytes out... You could try --block-compress to force it to write to the disk in fixed-size blocks instead of arbitrary.
Also try bigger block sizes -- just doubling it isn't going to make much difference. Maybe 4096 or 8192, conveniently the same size as CPU memory pages.
It might help a little, but the difference is unlikely to be that dramatic... Either your disk, or the compression, is liable to be what's slowing it down. More likely the compression if running several in parallel makes it faster. Try writing to a different partition than the source of the files. Try using a more CPU-efficient compressor, like lzop i.e. tar -cf - bigfile | lzop > file.tar.lzop
1 Like
How much does 10% mean on your machine? Is that one full CPU on a ten-CPU machine for example?
You could try using a lower-load compressor. lzop will end up slightly bigger than gzip but its performance is much better CPU wise.
tar -cf - path/to/files | lzop > file.tar.lzop
I suppose the real problem is that "gzip" is a single-threaded application (and probably has to be). So each "gzip"-process will have a natural maximum operation speed which is how fast one CPU can work one single thread. The more "gzip"s you can distribute the backup process to the faster it will be done, but the more CPU resources will be used during this time.
You could try to move often-accessed data (like the work directory of the tar/gzip-processes) to a SSD. This might speed things up.
I hope this helps.
bakunin
1 Like
Thanks Corona688 for your suggestion. I will try lzop option. Regarding CPU - yes, 10% means, 1 full CPU on ten CPU machine as mentioned in your example(we have 22 core CPU, 88 GB RAM)
---------- Post updated at 05:49 PM ---------- Previous update was at 05:11 PM ----------
Thanks bakunin for your response and advise.
Thanks a lot for your help. I used "lzop", it is really faster than "gzip", though the file size has increased.
1 Like