AIX & TAR related stuff

filosophizer · February 22, 2013, 9:37am

This thread is about using tar & other compression utilities on AIX ;

Find out which version of tar you are using
thanks to bakunin

>what $(which tar)
/usr/bin/tar:
        61      1.14  src/bos/usr/ccs/lib/libc/__threads_init.c, libcthrd, bos53  0 7/11/00 12:04:14
        10  1.49.1.75  src/bos/usr/bin/tar/tar.c, cmdarch, bos53A, a2004_38A1 8/  31/04 07:07:20
        92      1.7  src/bos/usr/ccs/lib/libIN/eprintf.c, libIN, bos530 6/10/91   10:23:13

>lslpp -L |grep -i tar
  devices.scsi.tm.rte       5.3.0.10    C     F    SCSI Target Mode Software
  devices.ssa.tm.rte         5.3.0.0    C     F    Target Mode SSA Support
  tar                         1.14-2    C     R    A GNU file archiving program.

>tar --version
tar: Not a recognized flag: -
Usage: tar -{c|r|t|u|x} [ -BdDEFhilmopRUsvw ] [ -Number ] [ -f TarFile ]
           [ -b Blocks ] [ -S [ Feet ] | [ Feet@Density ] | [ Blocksb ] ]
           [ -L InputList ] [-X ExcludeFile] [ -N Blocks ] [ -C Directory ] File ...
Usage: tar {c|r|t|u|x} [ bBdDEfFhilLXmNopRsSUvw[0-9] ] ]
           [ Blocks ] [ TarFile ] [ InputList ] [ ExcludeFile ]
           [ [ Feet ] | [ Feet@Density ] | [ Blocksb ] ] [-C Directory ] File ...

> find / -name tar
/opt/freeware/bin/tar
/usr/bin/tar
/usr/linux/bin/tar

GNU Tar

why it is important to know which version of tar you are using ?

# tar cvf - file | gzip > file.tar.gz
tar: 0511-825 The file 'file' is too large.

AIX tar has a 2 GB limit to the file size. On UNIX tar there are some limits: avg. 12 GB of file archive (may vary depend of UNIX flavor), 6 level of directories (all UNIX, but not GNU tar), 32k directories on one level (this is a limitation of JFS and JFS2) Use tar from the Linux Toolbox CD - can get it alternatively here: https://www14.software.ibm.com/webapp/iwm/web/reg/download.do?source=tbxsrc&lang=en\_US&S_PKG=tar&cp=UTF-8
or
Welcome to Bull AIX freeware site
ftp://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/tar/tar-1.14-2.aix5.1.ppc.rpm

Procedure
Find the native AIX tar location.

#which tar
/usr/bin/tar

Move the native binary tar command:

#cd /usr/bin
#mv tar tar_

Install the GNU tar, which can be obtained from the AIX toolbox site: IBM AIX Toolbox download information
Set the install location to be something similar to /opt/freeware/bin/tar

Create a gnu tar soft link:

ln -s  /opt/freeware/bin/tar /usr/bin/tar

Validate that the tar command is gnu tar:

#tar --version

Popular Tar options

-c = create
-f = read to/from the named file (instead of the device /tape)
-t = list contents of .tar file
-r = append to a .tar file
-v = verbose (tells you everything its doing)
-x = extract contents of .tar file
-z = compress files

tar: 0511-169 A directory checksum error on media ; 804399360 not equal to 12401.

Solution : Gunzip tar A directory checksum error on media

How to find if particular file is present in the tar file ?
How to find the size of particular file in the tar file ?

Skrynesaver · February 22, 2013, 9:52am

5 and 6
supply the filename (including path) to tar

skrynesaver@busybox ~$ ls tmp/
1.txt  2.txt  lac_sum.pl  regex.pl  time_check.pl  tmp.dat  tmp.dat.bak  tmp.pl  tmp.tok
skrynesaver@busybox ~$ tar -cjvf tmp.tbz2 tmp
tmp/
tmp/1.txt
tmp/2.txt
tmp/lac_sum.pl
tmp/regex.pl
tmp/time_check.pl
tmp/tmp.dat
tmp/tmp.dat.bak
tmp/tmp.pl
tmp/tmp.tok
skrynesaver@busybox ~$ tar -tjvf tmp.tbz2 tmp/regex.pl
-rw-r--r-- skrynesaver users 374 2013-02-12 16:20 tmp/regex.pl
skrynesaver@busybox ~$

bakunin · February 22, 2013, 2:59pm

AIX adheres to the "USTAR" standard. The 2GB limit was lifted with AIX 5.1, as shockneck pointed out. AIX 5.1 was out of software service 2005 or so - too long ago, in any case, to still use it in any conclusions. In fact, quoting from the tar man page for AIX 6.1:

Further, you can easily work around some limitations by using <stdin>/<stdout> as input/output destinations:

tar -cf - some_large_dir > /path/to/outfile.tar
cat /path/to/outfile.tar | tar -xf -

IMHO this is not a good way to do it. You change the OS by doing so and that is most probably not what you want. First, you will run into troubles when you update or apply corrective services, because the install routines of these procedures will expect "tar" (the executable, not some spurious link) to be found at "/usr/bin/tar", not at "/usr/bin/tar_". Second, you mislead tools which may rely on a certain behavior of the command they are using. These tools may or may not fail simply because they expect behavior X but experience behavior Y instead.

You may install any tar replacement either in a different location ( /opt/gnutar-version/bin/tar if you think of it as an application, /usr/local/bin/tar if you think it is an additional administrative tool) and set your PATH accordingly (just put the part where GNU-tar resides in front of /usr/bin ) or you can install GNU-tar under a different name ("gtar" for instance) and define an alias for your scripts and/or shells. But you should NEVER, NEVER EVER replace system commands with home-brewn concoctions.

I hope this helps.

bakunin

MichaelFelt · February 23, 2013, 11:48am

If it is just to archive files and directories on AIX - consider the backup file format (bff) and the backupbyname option of backup.

I try to stay away from tar for a couple of reasons (does not know about ACL or most special files, as examples) so instead I use the following:

$ cd /some/dir/needing/backup
$ find . | backup -if - | bzip2 >/my/backup/directory/backupName.bff.bz2

And when I want to unpack, examine, etc

$ bzip2 -dc /my/backup/directory/backupName.bff.bz2 | restore -Tqf -

Don't forget to set fsize to unlimited. To change the fsize default:

chuser fsize=-1 default

otherwise, to change ony for michael and root

chuser fsize=-1 root
chuser fsize=-1 michael

Sure, it is not tar - but it works quite well, also for copying files/directories.

filosophizer · February 23, 2013, 3:53pm

Thank you for contributing to the thread:

@MichaelFelt : tar is recommended by oracle for cold backup. it is widely used for this purpose ( backup ) otherwise what you have mentioned is very reasonable.

@bakunin : thank you for the great valuable tips

How to find out if you have any errors during creating a tar

Suppose you are making a tar file of a filesystem which has several directories and files ( 100GB )
you execute the command
# tar -cvf tarfile_name.tar /filesystem

this is will start creating the tar file and will take several minutes/hours

how can you find out , if you have any errors during creating the tar archive :
errors like => tar: 0511-825 The file 'file' is too large
or any other errors ?

Is there a way to verify if all the files have been included in the tar achieve ?
How to list the number of files in the tar file ?

bakunin · February 23, 2013, 4:05pm

After the tar has finished you can consult the exit code (=errorlevel). As always in Unix, "0" means no error, any other code means various sorts of error conditions. Check the man page for further information.

The answer to these questions is one: you can use

tar -t

to "test" a tarfile. With the "-v" option it will display a listing rather similar to a directory listing. Use this and a "ls -lR" as a reference to check if all files are included (if you haven't access to the error output tar produces in case of errors and/or the above mentioned exit code) and pipe the output in "wc -l" to count files.

Regarding Oracle recommending "tar": any command which backs up many files into one archive is fair game. Don't cling too much to what DB-manufacturers tell you. If their OS knowledge would be so hot stuff they wouldn't do database programming, after all, would they? ;-))

Seriously: MichaelFelt has it absolutely correct: "backup" is a full-blown replacement for whatever you ever used "tar" for. I use "tar" myself, mostly out of habit, but have to acknowledge that "backup" has no drawbacks at all (and possibly some advantages).

I hope this helps.

bakunin

MichaelFelt · February 24, 2013, 6:41am

Nods. TAR is a generic solution and this recommendation has wide applicability. That does not imply that it is a best practice for any platform.

Tar will often be a good choice for saving - I use it frequently - and opensource projects are usually distributed as a tar ball, but TAR is not always so great for restoring. Not sure what the status is of AIX tar atm, but it used to be if you restored a file using tar all the zero bytes that are not on disk actually get written to the "tape" AND these zero bytes get restored. (so called sparse files getting backed up and restored as "full" files. There is a real danger that the cold backup causes some hot tempers when it does not fit where it is supposed to )). In other words, a file/database that fit before the restore does not fit the same file system afterwards. This is what got me started >10 years ago to use backup and restore.

Granted, using backup and restore is less portable than tar , but any platform dependent tool understands everything there is to understand about files living on their system. Think: Backup and Restore are the generic UNIX tools for system backups.
I do not know what Solaris, HPUX, "Linux" recommend for system backups. If it is tar they will have added features to make it able to backup files, directories and device definitions. However, even though it is a "tar" file, it is not really portable.

I diverge - getting back on track - I would recommend taking a look at the AIX savevg script, or for something simpler my "HOWTO: copy directories on AIX". It is not exactly what your are doing for your cold backups, but I hope it is easy to improve - to fit to your needs.

Any questions about either script - Post and I will do my best to answer them here!

Or - make some improvements and post them back! I know there are many new features in backup and restore (working with encryption, e.g.). Much room for improvement.

bakunin · February 24, 2013, 9:10am

This is true. From the AIX 6.1 manual:

IMHO "tar" is a good device to bundle some files up to an archive. If one wants to bundle the source files for a software project or to create a snapshot of ones HOME-directory it is a tool worth considering. "tar" plays in the same league as "cpio", "pax" and probably some other tools, "backup" among them.

As Michael hinted at "tar" is platform-independent to a quite unparalleled degree. This is as much an advantage as it is a limiting factor because of some limitations. And for system-wide backups i would use "mksysb" and "savevg" solely under AIX - neither "tar" nor any of the other aforementioned programs.

Just my 2 cents.

bakunin

alister · February 24, 2013, 2:25pm

tar is a lot like ps. Both tools are universally deployed, but if you cross platforms, only the most fundamental options will work. Attempt something non-trivial and any semblance of platform-independence evaporates.

Regards,
Alister

Skrynesaver · February 24, 2013, 4:27pm

As Alister (amck?) has said above, the problem with tar is that it has been implemented so often and so variously that options taken for granted in the GNU world aren't available everywhere.

However the GNU solution has (OS regardless) become the most useful solution and (in my opinion) the solution to a deficiency in your local tar is to install gnu tar(personal opinion, your mileage may vary, I'm right but you don't realise it yet(there's a culture ship name in there (though that'd be a cool culture ship name in its own right))

filosophizer · February 28, 2013, 3:06pm

Tar and ZIP the directory in one command

# tar cvf -  /oracle | gzip > /backup/oracle_backup.tgz
or
# tar cvf - /oracle | gzip > /oracle.tar.gz

If you are using GNU tar , you can say instead:
# tar cvzf oracle.tar.gz /oracle

v enables verbose out and is not necessary for the command to function. 
'f' states that the output is to go to a file that then is set to '-' 
(stdout) making tar write its output to stdout which is the default behavior without both 'f' and '-'.
# tar c /oracle | gzip > oracle.tar.gz

unTar and uncompress unZIP the directory in one command

gunzip < /backup/oracle_backup.tgz | tar -xvf -

OR
gunzip < /oracle/abc.tar.gz | tar xvf -

bzip2 < /oracle/abc.tar.bz2 | tar xvf -

Removing leading `/' from member names
sometimes

tar -cvf -  /oracle | gzip > /backup/oracle_backup.tgz
tar: Removing leading `/' from member names

which means:

warning message to tell you that it is writing a tar file without absolute paths.
This is because you cannot then relocate the data at a later date when you un-archive the file.
tar takes relative paths. The file paths in a tar archive are stored as relative paths for a good reason.
It gives you more flexibility about where to restore files. You may want to restore files into another location.
The relative path makes this possible.

See more here:

filosophizer · April 7, 2013, 2:03pm

Tar and ZIP -- multiple directories in one command

we have directories
/oracle
/oracle1
/oracle2

tar -cvf -  /oracle /oracle1 /oracle2 | gzip > /backup/oracle_backup.tgz

unTar and unZIP -- one specific directory from the tar archive in one command

Can someone help with this one. How to untar unzip from oracle_backup.tgz one specific folder which is oracle2 ?

thanks

bakunin · April 7, 2013, 11:33pm

gzip -cd <file> unzips from a file to <stdout> This data stream can be fed to "tar".

tar -x <dir> extracts <dir> and all files containing it from a tar archive.

Hence:

gzip -cd oracle_backup.tgz | tar -xf - /oracle2

I hope this helps.

bakunin

filosophizer · August 28, 2013, 3:14am

How to find the total size of the uncompressed tar zipped file oraclebackup.tgz

oraclebackup.tgz was created using the following command, which gave me 40GB file

# /opt/freeware/bin/tar -cvf -  /oracle | gzip > /backup/oraclebackup.tgz

Now, if someone wants to check how much space is needed to untar and uncompress the file oraclebackup.tgz how can he check ?

thanks

bakunin · August 28, 2013, 4:43am

You can't: because of he algorithm used there is no direct relation between the source and the target file size: if file X is bigger than file Y then X.gz might still be smaller than Y.gz. The only way to find out is to actually compress the file. The same is true for the other way round.

I hope this helps.

bakunin

filosophizer · August 28, 2013, 3:12pm

thanks bakunin,

the reason why I asked this question, when I use

gunzip < /backup1/oraclebackup.tgz | /opt/freeware/bin/tar -xvf-
/opt/freeware/bin/tar: oracle/prodappl/au/11.5.0/forms/AR/GMDSTAED.fmb: Wrote only 9728 of 10240 bytes
/opt/freeware/bin/tar: Skipping to next header
/opt/freeware/bin/tar: Archive contains obsolescent base-64 headers
/opt/freeware/bin/tar: Error exit delayed from previous errors

my 155GB filesystems fills up and tar exits,

but when i use the AIX tar to untar , it untars without any problems, and the filesystem after untar is 147GB where as using GNU tar , my filesystem becomes full at 155GB and exits. My filesystem capacity is 155GB ;

thanks.

bakunin · August 28, 2013, 10:21pm

Two minor corrections: first, tar is one of the few commands where "-" does NOT introduce options. Correct is:

tar xvf -

Still, i do not think this is pertinent to your problem.

Second: i am not sure if gunzip < /backup1 really extracts to stdout . I'd use the "-c" flag, which does that (extract to stdout ), but then, this might be working too. RIght now i have no gunzip on AIX at hand to verify.

Regarding your problem: i have no idea. Which "tar"-version is installed in /opt/freeware ?

I hope this helps.

bakunin

filosophizer · August 29, 2013, 4:14am

tar version

/opt/freeware/bin/tar --version
tar (GNU tar) 1.14
Copyright (C) 2004 Free Software Foundation, Inc.
This program comes with NO WARRANTY, to the extent permitted by law.
You may redistribute it under the terms of the GNU General Public License;
see the file named COPYING for details.
Written by John Gilmore and Jay Fenlason.

filosophizer · September 16, 2013, 6:01am

List the contents of a tar file

$ tar -tvf file.tar

$ zcat file.tar.Z | tar tvf -


$ tar -ztvf file.tar.gz

$ tar -ztvf file.tar.gz > files.txt

note: -z will not work on AIX but will work on Linux , equivalent would be

$ gunzip < test.tgz | tar tvf -

$ tar -jtvf file.tar.bz2

bakunin · September 16, 2013, 9:51am

We had that already: "tar tvf ....", without the dash.

This has nothing to do with the OS, only with the tar-version you use: GNU-tar understands "z" (again: not "-z", but "z") andh "j", the AIX-tar (and any other POSIX-conformant "tar") does not. Linux has the GNU-tar installed by default, AIX the AIX-tar.

This is why i would never on "j" or "z" or any other non-standard subcommand and always would use:

$unpackprogram file | tar tvf -

Simply, because the extra feature may or may not be there, but this is guaranteed to work. Part of my work philosohpy is to avoid unnecessary risks. This is one - and one which is easily avoided, for that matter.

I hope this clears it up.

bakunin