Copy down remote files and rename them to include the server name with full path

I need to pull down a good bit of files for another support team for an upgrade project. I have a server.list with all of the server names.

I need to do two parts:

FIRST:
I have this example, but it does not list the server name in front of each line.

#! /bin/bash
for server in $(< server.list);
do
echo "********* $server *********"
ssh $server "sudo find /u01/ -name \"filename\""
done

I need to get an output file that will look similar to this:
SERVER1:/path/path/filename
SERVER2:/path/path/filename
SERVER3:/path/path/filename
SERVER3:/path/path/filename
so on (and I might have multi files on one server)...

SECOND:
Once I have the "new" list of all servers and files I need to be able to take that output and copy the files down to a remote server and rename the file to include the server name and full path.

EX: SERVER1/path/path/filename
Script will use the above to ssh to SERVER1 and copy the file from the full path then rename it to SERVER1_path_path_filename

This is needed so that the support team will know which file belongs to which server and the full path as one server could have multi files with the same name.

This solution uses sed to prepend the server name to filelist and scp to copy the files locally. Most servers that support ssh also support scp so it should work for you.

#!/bin/bash
 
for server in $(< server.list)
do
   ssh $server "find /u01/ -name \"filename\"" | sed "s/^/$server:/"
done > new.list
 
OLDIFS=$IFS
IFS=":"
while read server filepath
do
   scp $server:$filepath $server${filepath//\//_}
done < new.list
IFS=$OLDIFS

THANK YOU VERY MUCH Chubler_XL

I had to add " " around the $server:$filepath for some reason.

OLDIFS=$IFS
IFS=":"
while read server filepath
do
   scp "$server:$filepath" $server${filepath//\//_}
done < new.list
IFS=$OLDIFS

Without the " " I was getting scp SERVER /path/path/file (No such file or directory)

Once I updated it to "$server:$filepath" I would get scp SERVER:/path/path/file

Thanks again for your help

---------- Post updated at 03:19 PM ---------- Previous update was at 11:02 AM ----------

I was getting the following error on some of the servers because I was using sudo find:

sudo: sorry, you must have a tty to run sudo

I just added ssh -t and that appears to have fixed it

Very odd... When I tested this out on Jan 15th it worked just fine... I try to run it today and I get the following error:

scp: ambiguous target

When I check the cmd it appears to be adding a : to the end of the line / filename
I am getting the following:
/path/filename: No such file or directory

My current script is :

#!/bin/bash
set -x

OLDIFS=$IFS
IFS=":"
while read server filepath
do
  scp -t "$server:$filepath" $server${filepath//\//_}
done < new.list
IFS=$OLDIFS

Any ideas ??

You probably have a blank line in the new.list file try:

#!/bin/bash
set -x

OLDIFS=$IFS
IFS=":"
while read server filepath
do
  [ -n "$filepath" ] && scp -t "$server:$filepath" $server${filepath//\//_}
done < new.list
IFS=$OLDIFS

I double checked and saved the original new.list to new.list-backup. I removed all but 15 lines (no blank lines or spaces):

server1.domain.net:/path/path/path/filename
server2.domain.net:/path/path/path/filename
server3.domain.net:/path/path/path/filename
... on to server 15

I tried adding your update

[ -n "$filepath" ] &&

and I am still receiving the "scp: ambiguous target" for each line...

Like I said this worked fine on the Jan 15th. I did not update the script and tried to run it today to check again and am now getting the "scp: ambiguous target" error.

When I run the script with set -x I am getting the following:

Looks like your data file is in DOS format (perhaps you edited it with windows notepad or something) try $ dos2unix new.list to clean it up.

Hmm very odd... I have only used vi to edit the file.

BEFORE:
new.list: ASCII text, with CRLF line terminators

I ran dos2unix on the file (new.list: ASCII text)

I am still getting :

+ '[' -n /path/path/filename ']' There appears to be a space after the /filename, is that an issue ?

scp seems to require a complete path for the destination file try:

scp $server:$filepath ./$server${filepath//\//_}

I think I know what I did... I split this into two scripts, one to create the new.list then another to scp the files down and rename them. When I was testing out the first one to ssh I was getting "sudo: sorry, you must have a tty to run sudo" so added the -t argument to ssh. Looks like for some reason I also added the -t to my scp cmd in my other script for some reason

Thank you very much for your time Chubler_XL. I just did a test and everything is working

current working script is:

#!/bin/bash
#set -x

OLDIFS=$IFS
IFS=":"
while read server filepath
do
  scp "$server:$filepath" ./$server${filepath//\//_}
done < new.list
IFS=$OLDIFS

---------- Post updated 01-30-13 at 04:47 PM ---------- Previous update was 01-29-13 at 11:24 PM ----------

I am sorry to be a PITA...

I am so close...

I wanted to add "sudo chmod 755" as I was running into permissions issues with trying to scp some files.

#!/bin/bash
set -x

OLDIFS=$IFS
IFS=":"
while read server filepath
do
  ssh -t $server  "sudo chmod 755 $filepath"
  [ -n "$filepath" ] && scp "$server:$filepath" ./FILES/$server${filepath//\//_}
done < new.list
IFS=$OLDIFS

I am getting the following error:

I also tried with ssh -tt and get

Anyone ??