Tarring files to remote server

Hi,

I need to tar some files in a directory to a remote server.
I need to exclude some file from this directory and then tar it over.
This is the command suggested by one article (tarring in the same server) :

tar -zcvf /tmp/mybackup.tar.gz -X exclude.txt /home/me

However it does not work as, it throws error that -z is not a valid option.

Is this the correct way to do it to change it to a .gz format :

tar -cvf -X exclude.txt - /bscsbin/lisa/product/UMOBILE/prod/bscs/scripts | gzip > dir.tar.gz

The files that I don't need is listed in "exclude.txt".

How do i do the same to send it over to a remote server?

Also, this command was suggested in an article to include files that I need in a tar format :

% find project -type f -print | \
egrep '(\.[ch]|[Mm]akefile)$' > Include
 

However, when I try this command (to customise according to my need) in my server, there is not output :

# find . -type f -print | egrep '(\.[jar]|\.[bash])$'

I want to list files with *.jar and *.bash extensions here.

Piping into a compressing / zipping command is the / a correct approach if the z is not available. Be aware that the f option needs an archive file - drop it or specify one. For piping it needs to write to stdout.

The f option allows to specify an archive on a remote node - see man tar . But - you can't use that if you want to compress via a piped gzip or similar. Use e.g. scp , then.

Your egrep regex matches files with an .j , .a , .r , .b , .s , or .h ending. Read about "bracket expression"s in your man regex :

Try

egrep '\.jar$|\.bash$'

instead.

Hi,

When I'm doing this I tend just to do the whole thing in a single line - something like.

#> tar cf - | ssh me@server ( cd /destination ; tar xf - )

Obviously you would adjust the create command to exclude the files that you want to exclude as per your example. If the files are very large you may like to pipe through gzip or whatever compression utility, but with the speed of modern networks I seldom see the need now.

Regards

Gull04

2 Likes

These were the commands I used to accomplish what I wanted :

#i=`find . \( -name '*.sh' -o -name '*.jar' -o -name '*.gz' \) -exec ls -l {} \+`
# tar cf - "$i" | ssh emoaigin@10.61.1.58 ( cd /root; /usr/contrib/bin/gzip > my.tar.gz )

I get this error :

I then change the "(" in the command to "", like this :

#i=`find . \( -name '*.sh' -o -name '*.jar' -o -name '*.gz' \) -exec ls -l {} \+`
# tar cf - "$i" | ssh emoaigin@10.61.1.58 "cd /root; /usr/contrib/bin/gzip > my.tar.gz"

This time I get prompted for password, but I still get some error :

# i=`find . \( -name '*.sh' -o -name '*.jar' -o -name '*.gz' \) -exec ls -l {} \+`
# tar cf - "$i" | ssh emoaigin@10.61.1.58 "cd /root; /usr/contrib/bin/gzip > my.tar.gz"
tar: cannot open -rw-------   1 emoaigin   users            0 Mar  5 18:16 ./abc.sh
-rw-------   1 emoaigin   users            0 Mar  5 18:16 ./create.jar
-rw-------   1 emoaigin   users            0 Mar  5 18:16 ./make.gz
Password:
sh: my.tar.gz: Cannot create the specified file.
#

How do I resolve the error?

Two generic (non- HP-UX- specific) comments:

  • why the -exec ls -l action for find ? You want file names, not a "long listing". Try dropping entirely.
  • Does emoaigin have write access to /root - would be somewhat unusual.
1 Like

No. Actually it is lke this (i will do this step by step so that you can understand what is done):

| gzip > dir.tar.gz

This means that gzip is getting data from stdin , packing them and sending the result to stdout , which is pointed to the file dir.tar.gz .

So far, so good, but now you need to to make tar putting its output to (its) stdout , because this is what a pipe does: connecting one process' stdout to the next process' stdin. Therefore you need to tell tar to not create a tar-file, but to direct the output to stdout:

tar <other subcommands> f - |

Notice, btw., that tar is not driven by options but subcommands . Therefore the introducing "-" is wrong. Everything else is still as in the e working example you found, with the exception of the non-standard "z" command (that works only for tar , which have a "built-in" gzip ):

tar cvf - -X exclude.txt /home/me | gzip > /output/file

The same way you alread tried: instead of writing the file to disk ( > /output/file ), you can use another pipeline to move it to another server and eventually unpack it there:

tar cvf - -X exclude.txt /home/me | gzip | ssh user@host "cd /some/place ; cat - | gzip -cd | tar xf - "

Notice that this instance is one of the select few moments where a seemingly UUOC is justified. At i am not aware of any other way of making gzip look at its stdin for incoming data.

I hope this helps.

bakunin

1 Like

@Bakunin, I tried the method you suggested, and this was the error :

# i=`find /home/emoaigin \( -name '*.sh' -o -name '*.jar' -o -name '*.gz' \)`
# tar -cvf - "$i" | gzip | ssh emoaigin@10.61.1.58 "cd /home/emoaigin ; cat - | gzip -cd | tar xf - "
sh: gzip:  not found.
tar: cannot open /home/emoaigin/abc.sh
/home/emoaigin/create.jar
/home/emoaigin/make.gz
Password:
sh: gzip:  not found.
Tar: blocksize = 0; broken pipe?
#

The error gzip not found can be resolved by putting the full path to gzip.
However, the error "tar: cannot open /home/emoaigin/abc.sh" why does it happen?
And what is this error : "Tar: blocksize = 0; broken pipe?" ?

It's more

tar: cannot open /home/emoaigin/abc.sh
/home/emoaigin/create.jar
/home/emoaigin/make.gz

because the i variable contains the entire find output including <LF> line terminators. If there are no white space chars in filenames, try to use $i unquoted.

Might be a consecutive error as the first tar failed and the pipes and thus inputs are empty...

First off: i think RudiC is correct in his analysis as to where the errors come from. But the - real - problem is something completely different:

Try to think like an engineer does and solve problems like an engineer would do. You throw 20 commands together and expect them to work flawlessly hand in hand on first try. If that fails you ask us.

How about trying the procedure i showed you with a single file or a single (small) directory? Now you notice you have gzip errors and correct that. Perhaps now it works. Now, on to the next more difficult problem. And so on....

The advantage this has is: whenever in this process an error comes up you understand immediately where it comes from, because the previous version was working. You put together a monstrous kludge of a command and then have no idea how to isolate the myriad of (maybe, maybe not) interdependent error conditions, some of which may (or maybe not?) corrected.

Instead of writing "the gzip error could be corrected" - why don't you just do it, run again and post what happens then?? In case all the other stuff would have worked the missing gzip would still have caused the rest of the pipeline to fail and you would still get the same tar -error because this is how tar reacts to empty input. But why do we have to take interactions between different unrelated errors into account when you cannot be bothered to do anything on yourself to isolate the persistent errors?

A child would say "toilet isn't working" - and call for adult help. Adults would observe that the water system works in this part of the house, but not in that part. So they would analyse the pipe system and find out that somewhere between here and there must be a problem. They might then proceed to correcting that problem.

I hope this helps.

bakunin

1 Like

One of the few occasions you should not quote a variable in command arguments. To avoid unwanted expansions set -f (noglob) and set IFS to only a newline.

# PATH=/bin:/usr/bin:/path/to/gzip
# i=`find /home/emoaigin \( -name '*.sh' -o -name '*.jar' -o -name '*.gz' \)`
# (set -f; IFS="
"; tar cf - $i) | gzip -c | ssh -x emoaigin@10.61.1.58 "cd /home/emoaigin && gunzip -c | tar xvf - "