The bash stores each uniqueid in an array and then passes them to %q to get the unique path. That seems to work what I am having trouble with is renaming each .png with the unique value in %q . I thought it was working but upon closer inspection, a .png file is being sent to scp.... but only 1 and with the wrong uniqueid . It seems like the first .png is being used by scp , but with the last uniqueid . In this example there are 2, but there may be more or less. Thank you :).
uniqueid1 ---- this is stored in array ----
data.png ---- this is the file in the path ----
uniqueid2 ---- this is stored in array ----
data.png ---- this is the file in the path ---
current
uniqueid1-data.png (but this is really the first png just with the wrong name)
This is a rather roundabout way of sending files! sshpass executing sshpass executing scp. Isn't there a better way?
Anyway, this doesn't do what you think. Each ID only gets used once. The first statement will contain the first and second ID; the second contains the third and fourth. Et cetera.
Quick fix:
for ID in "${array[@]}"
do
echo "( cd /path/to/${ID}*/*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:'$dir'/folder/${ID}*-data.png )"
done
# Test to make sure the output is what you want. Then replace the above line with the below one.
# done | sshpass -f out.txt ssh -o strictHostKeyChecking=no -t xxx@xxx.xx.xx.xx
With the echo the output is correct, but with sshpass added I get the file not foung though I can manually finnd the directory but the .png that is there is cn_results and when it is transferred by scp it is renamed or has $ID append to it. Thank you :).
script.sh: line 39: ( cd /path/to/ID1*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:path/to/destination/ID1-cn_results.png ): No such file or directory
script.sh: line 39: ( cd /path/to/ID2*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:path/to/destination/ID2-cn_results.png ): No such file or directory
I tried to add the loop to scp and rename as such:
I tried it two ways both resulting in file or directory not found, but I can manually ssh to them. This is not the full code but I hope it's enough as I don't want to post code that is not needed. Thank you :).
attempt 1
# match unique run in list and extract samples
run=$(awk -F '\n' -v RS="" -v ref="$s5" '$0 ~ ref {d=split($0, val, " "); for(i=2;i<d;i+=2) printf "%s ",val; printf "\n"}' list)
# create array of samples from $run
readarray -t array <<< "$(printf '%s\n' $run)"
for ((i=0; i<${#array[@]}; i++ )) ; do
echo "${array[$i]}"
done
# loop through array and append with ID
for ID in "${array[@]}" ; do
"( cd /path/to/${ID}*/*/folder && exec sshpass -f file.txt scp xxx@xxx.xx.xx.xxx:path/to/destination/${ID}-cn_results.png )"
done | sshpass -f out.txt ssh -o strictHostKeyChecking=no -t xxxx@xxx.xx.xxx.xx
attempt 2
# match unique run in list and extract samples
run=$(awk -F '\n' -v RS="" -v ref="$s5" '$0 ~ ref {d=split($0, val, " "); for(i=2;i<d;i+=2) printf "%s ",val; printf "\n"}' list)
# create array of samples from $run
readarray -t array <<< "$(printf '%s\n' $run)"
for ((i=0; i<${#array[@]}; i++ )) ; do
echo "${array[$i]}"
done
# loop through array and append with ID
for ID in "${array[@]}" ; do
"( cd /path/to/${ID}*/*/folder && exec sshpass -f file.txt scp "$ID" xxx@xxx.xx.xx.xxx:path/to/destination/"${ID%}"-cn_results.png )"
done | sshpass -f out.txt ssh -o strictHostKeyChecking=no -t xxxx@xxx.xx.xxx.xx
attempt 1 error:
script.sh: line 39: ( cd /path/to/ID1*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:path/to/destination/ID1-cn_results.png ): No such file or directory
script.sh: line 39: ( cd /path/to/ID2*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:path/to/destination/ID2-cn_results.png ): No such file or directory
attempt 2 error: --- in the server these exsist as ID1-xxxx-xxxxxxx-xxxxxx so I added * but thats not expanding it seems ----
ID1: No such file or directory
ID2: No such file or directory
The echo you removed was necessary. Without it, you are attempting to run a file named "( cd /path/to/${ID}*/*/folder && exec sshpass -f file.txt scp xxx@xxx.xx.xx.xxx:path/to/destination/${ID}-cn_results.png )" which quite probably does not exist.
rm -f /tmp/testoutput
for ID in "${array[@]}" ; do
echo "( cd /path/to/${ID}*/*/folder && exec sshpass -f file.txt scp xxx@xxx.xx.xx.xxx:path/to/destination/${ID}-cn_results.png )"
done | tee /tmp/testoutput | sshpass -f out.txt ssh -o strictHostKeyChecking=no -t xxxx@xxx.xx.xxx.xx
The tee is optional but may help you by duplicating the output in /tmp/testoutput for you to check. * won't expand in /tmp/testoutput but ought to do so on the remote host.
So if I understand and am reading the error correctly the * is not expanding on the remote machine? I think this because tee looks good but the terminal doesn't like the * . I am testing this on ubuntu but will use it on centos. Thank you :).
for ID in "${array[@]}" ; do
echo "( cd /path/to/${ID}*/*/folder && exec sshpass -f file.txt scp xxx@xxx.xx.xx.xxx:path/to/destination/${ID}-cn_results.png )"
done | tee /tmp/tmp.txt | sshpass -f out.txt ssh -o strictHostKeyChecking=no -t xxxx@xxx.xx.xxx.xx
terminal output
-bash: line 1: cd: /path/to/ID1*/*/folder: No such file or directory
-bash: line 2: cd: /path/to/ID2*/*/folder: No such file or directory
No reason * shouldn't expand, perhaps you ended up with nonprinting characters like carriage returns in your array or some such. If the commands work when pasted, that's what I'd bet.
By the way, you don't need readarray to make arrays:
# Set an array in one step in bash
array=( $(awk -F '\n' -v RS="" -v ref="$s5" '$0 ~ ref {d=split($0, val, " "); for(i=2;i<d;i+=2) printf "%s ",val; printf "\n"}' list | tr -d '\r') )
I won't know the issue until it works. Nonprinting characters are nonprinting, invisible to copy-paste. If there were nonprinting characters in your array you might not see them. Did you try my code, which removes carriage returns?
/path/to/${ID}*/path/to/folder (always this format)
The ${ID} will always be unique and have some random text after it that the * is used for. At the last folder in the path there is a .png always named cn_results that I am trying to append the unique ${ID} on front of ${ID}-cn_results.png , then scp that .png to xxxx@xxx.xx.xxx.xx:/path/to/destination/${ID}-cn_results.png)
There are two values in array , but only the first is printed:
# Set an array in one step in bash
array=( $(awk -F '\n' -v RS="" -v ref="$s5" '$0 ~ ref {d=split($0, val, " "); for(i=2;i<d;i+=2) printf "%s ",val; printf "\n"}' list | tr -d '\r') ) ## match run with samples
echo "This is array" "$array" > /tmp/out
Contents of out
This is array ID1
I added a loop to iterate though each ID in array, but only the second value is printed:
# Set an array in one step in bash
array=( $(awk -F '\n' -v RS="" -v ref="$s5" '$0 ~ ref {d=split($0, val, " "); for(i=2;i<d;i+=2) printf "%s ",val; printf "\n"}' "$run_dir"/rna.barcodes | tr -d '\r') ) ## match run with samples
for file in "${array[@]}" ; do
echo "This is array" "$file" > /tmp/loop.out
done
Contents of loop.out
This is array ID2
In this example array has two values in it ID1 and ID2 , but there may be more or less it just depends on the run which the values for array come from. Thank you very much
No, it's not. How do you expect people to bug hunt if they don't see what is actually going wrong? Show the directory listing from your target system. If that is confidential, build a dummy one but adapt your code to reproduce the error.
├──/path/to/ ---- common path after ssh ---
│ ├── ID1* --- unique ${ID}* represents random text ---
│ │ └── /this/ID1/folder
│ ├── ID2* --- unique ${ID}* represents random text ---
│ │ └── /this/ID2/folder
Description:
After ssh to the common path on the server, each unique ID is used to further navigate to folder. In each folder there is a png (cn_results), that the unique ID is append to (ID-cn_results) and this
append file is scp to xxx@xxx.xx.xx.xxx:/path/to/destination. It seems that since array is one line both are being used and that is causing the error?
Well that's not going to work. You're just printing the statements to the screen and not giving anything to sshpass at all.
Does your script change the variable IFS? Strings are supposed to split when you do ARRAY=( 1 2 3 ) and not doing so is very weird unless IFS was changed.
That's why you need to be careful to do
OLDIFS="$IFS"
IFS="."
...
IFS="$OLDIFS"
...whenever you alter IFS for anything. If you don't put it back when you're done, that changes how all splitting works later.
