Curl/http 503 error with bash script

kieranfoley · December 4, 2017, 6:22am

I am trying to use REST API and curl in a bash script to generate a http report. The curl command at the end of the script should generate a html file but instead I get an error "HTTP/1.1 503 Service Unavailable".

This is the script

#!/bin/bash

export HOME=/export/home/kfoley/scripts/REST_API
export TEMP_DIR=/export/home/kfoley/scripts/REST_API/Temp

echo "Enter the hostname:"
read HOSTNAME

curl -u username:Password  http://seieadpa01:9004/apollo-api/nodes/?query=name%3d$HOSTNAME > $TEMP_DIR/test_file.txt

HOST_ID=`cat $TEMP_DIR/test_file.txt | grep "<id>"`

cat $HOME/host_query_input_file.out | awk -v ID="$HOST_ID" '{if ($1~/\<id>/) $1=ID ;print}' > $TEMP_DIR/host_query_input_file.out

curl -u username:Password -X POST -H "Content-Type:application/vnd.emc.apollo-v1+xml" --data "@/export/home/kfoley/scripts/REST_API/Temp/host_query_input_file.out" http://seieadpa01:9004/dpa-api/report -o $TEMP_DIR/test_xml.out

LINK=`cat $TEMP_DIR/test_xml.out | grep link | sed 's/<link>//g;s/<\/link>//g'| awk '{sub(/^[ \t]+/, ""); print}'`

#curl -u username:Password $LINK -o /symdata/DPA-REST-API-REPORTS/tester.html

curl -u username:Password $LINK

yet when I run the same command outside the script from the command line I do not get any error.

LINK="http://seieadpa01:9004/dpa-api/report/result/f73356fb-c08c-4ab8-90ec-7caa0bd19d50"
curl -u username:Password $LINK

Anybody know what I am doing wrong here? I'm thinking I need to define a CURL variable within the script?

RudiC · December 4, 2017, 6:29am

Wouldn't it be helpful to show the LINK variable's contents within the script?

kieranfoley · December 4, 2017, 7:23am

Hi RudiC,

I'm not sure what you mean? The LINK variable is defined from this line in the script.

LINK=`cat $TEMP_DIR/test_xml.out | grep link | sed 's/<link>//g;s/<\/link>//g'| awk '{sub(/^[ \t]+/, ""); print}'`

Once the script finishes running (albeit with the 503 service available error) I am able to define the LINK variable from the command line and run the curl GET using $LINK or using the variables contents - either way it works when I run it outside of the script.

If I echo the $LINK variable within the script to get it's contents it is what I expect it to be.

rbatte1 · December 4, 2017, 8:37am

So, we can shorten the definition of LINK somewhat. Can you show us the output you get from

uname -a
ps -f
LINK=$(grep link $TEMP_DIR/test_xml.out)
echo "${LINK}"

Please follow this by the value you would want to convert it to in this case. We will then be able to clearly see what you are trying to get and be better placed to try to work out where it's going wrong. I've included the first two statements to show us your OS and shell so we know what we're able to work with.

Please post all the output in CODE tags.

Thanks, in advance,,
Robin

kieranfoley · December 4, 2017, 10:10am

Hi Robin,

this is the output from the commands you asked me to run.

[root@seiemccli01 REST_API]# uname -a
Linux seiemccli01 2.6.39-400.215.10.el5uek #1 SMP Tue Sep 9 22:51:46 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@seiemccli01 REST_API]# ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
root     14440 14435  0 10:05 pts/1    00:00:00 -bash
root     14590 14440  0 10:08 pts/1    00:00:00 ps -f
[root@seiemccli01 REST_API]# LINK=$(grep link $TEMP_DIR/test_xml.out)
[root@seiemccli01 REST_API]# echo "${LINK}"
   <link>http://seieadpa01:9004/dpa-api/report/result/57ba6675-1aa6-47cc-afdf-205b1d36d49c</link>
[root@seiemccli01 REST_API]#

In the script I strip out the leading and trailing <link>

Thanks.

rbatte1 · December 4, 2017, 10:26am

So a few suggestions to get your script neater only making one external call:-

LINK=$(grep link $TEMP_DIR/test_xml.out)            # Read the line you want from the file
LINK="${LINK#*>}"                                   # Strip first part up to '>'
LINK="${LINK%<*}"                                   # Strip last part up from '<'

Can you run your code both from the command line and scheduled but adding set -x just before you try to curl and see a difference between the trace output.

Obviously, if you post output here, overwrite the credentials so they are not published to the world.

Regards,
Robin

Yoda · December 4, 2017, 12:11pm

Try running your curl manually in command prompt and check if it works:-

curl -u Username:Password "$LINK"

If you still get a 503 - Service Unavailable , then check if you have a proxy server. If yes, try:-

export http_proxy="<proxy_host>:<proxy_port>"

curl -u Username:Password "$LINK"

or

curl -x <proxy_host>:<proxy_port> -u Username:Password "$LINK"

kieranfoley · December 5, 2017, 7:23am

Thanks rbatte1. I tried setting the LINK variable as you suggested and I am still getting the same 503 http error. This is what my script looks like now.

#!/bin/bash

export HOME=/export/home/kfoley/scripts/REST_API
export TEMP_DIR=/export/home/kfoley/scripts/REST_API/Temp

echo "Enter the hostname:"
read HOSTNAME

curl -u username:Password  http://seieadpa01:9004/apollo-api/nodes/?query=name%3d$HOSTNAME > $TEMP_DIR/test_file.txt

HOST_ID=`cat $TEMP_DIR/test_file.txt | grep "<id>"`

cat $HOME/host_query_input_file.out | awk -v ID="$HOST_ID" '{if ($1~/\<id>/) $1=ID ;print}' > $TEMP_DIR/host_query_input_file.out

curl -u username:Password -X POST -H "Content-Type:application/vnd.emc.apollo-v1+xml" --data "@/export/home/kfoley/scripts/REST_API/Temp/host_query_input_file.out" http://seieadpa01:9004/dpa-api/report -o $TEMP_DIR/test_xml.out

LINK=$(grep link $TEMP_DIR/test_xml.out)
LINK="${LINK#*>}"
LINK="${LINK%<*}"

export LINK

curl -v -u username:Password $LINK

echo "curl -u username:Password $LINK"

This is the output from the script and as you can see I am getting the 503 Service Unavailable error.

Thanks for your reply Yoda. When I run the curl from the command line I never have an issue. I get the expected html output.

In the last line of my script I have an echo and if I run that from the command line it works. If I set the LINK variable on the command line and then un just as it looks in the script it work also.

That is if I do this from outside the script it works

curl -u username:Password http://seieadpa01:9004/dpa-api/report/result/9d193c23-db7d-4c0c-aa09-fb4ed490d13a

or if I do this it also works.

export LINK="http://seieadpa01:9004/dpa-api/report/result/9d193c23-db7d-4c0c-aa09-fb4ed490d13a"
curl -u username:Password $LINK | head

This is the "head" of the output I should expect from the script.

[root@seiemccli01 REST_API]# curl -u username:Password $LINK | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5523    0  5523    0     0   512k      0 --:--:-- --:--:-- --:--:--  512k<html>
<head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        <title>All Jobs</title>
</head>
<style>
.g
{
        margin-top:0px;
    margin-left:3px;
[root@seiemccli01 REST_API]#

Also, when I try to do the same as above using wget I am having the same issues. i.e it work outside the script but fails within it.

---------- Post updated at 10:32 AM ---------- Previous update was at 10:31 AM ----------

Sorry I never posted the output from running the curl -v which shows the 503 http error

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   714    0   714    0     0  20638      0 --:--:-- --:--:-- --:--:--  348k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   116  100   116    0     0   2328      0 --:--:-- --:--:-- --:--:--     0
* About to connect() to seieadpa01 port 9004
*   Trying 172.16.120.152... connected
* Connected to seieadpa01 (172.16.120.152) port 9004
* Server auth using Basic with user 'admin'
> GET /dpa-api/report/result/9d193c23-db7d-4c0c-aa09-fb4ed490d13a HTTP/1.1
> Authorization: Basic YWRtaW46U3RvcmFnZTAwMSE=
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: seieadpa01:9004
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Server: Apache-Coyote/1.1
< Pragma: No-cache
< Cache-Control: no-cache
< Expires: Wed, 31 Dec 1969 19:00:00 EST
< X-FRAME-OPTIONS: SAMEORIGIN
< Retry-After: 2
< Content-Type: application/octet-stream
< Content-Length: 116
< Date: Tue, 05 Dec 2017 10:04:29 GMT
< Connection: close

---------- Post updated at 11:03 AM ---------- Previous update was at 10:32 AM ----------

This is the output when I use

set -x

within the script. Note that I have taken out the real username:password in the script in this post but the output shows that the username is actually admin.

+ export HOME=/export/home/kfoley/scripts/REST_API
+ HOME=/export/home/kfoley/scripts/REST_API
+ export TEMP_DIR=/export/home/kfoley/scripts/REST_API/Temp
+ TEMP_DIR=/export/home/kfoley/scripts/REST_API/Temp
+ echo 'Enter the hostname:'
Enter the hostname:
+ read HOSTNAME
seiemccli01
+ curl -u 'username:Password' 'http://seieadpa01:9004/apollo-api/nodes/?query=name%3dseiemccli01'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   714    0   714    0     0  24736      0 --:--:-- --:--:-- --:--:--  697k
++ cat /export/home/kfoley/scripts/REST_API/Temp/test_file.txt
++ grep '<id>'
+ HOST_ID='      <id>560775dc-e949-490d-ae1c-d76ec33b18b5</id>'
+ cat /export/home/kfoley/scripts/REST_API/host_query_input_file.out
+ awk -v 'ID=      <id>560775dc-e949-490d-ae1c-d76ec33b18b5</id>' '{if ($1~/\<id>/) $1=ID ;print}'
+ curl -v -u 'username:Password' -X POST -H Content-Type:application/vnd.emc.apollo-v1+xml --data @/export/home/kfoley/scripts/REST_API/Temp/host_query_input_file.out http://seieadpa01:9004/dpa-api/report -o /export/home/kfoley/scripts/REST_API/Temp/test_xml.out
* About to connect() to seieadpa01 port 9004
*   Trying 172.16.120.152... connected
* Connected to seieadpa01 (172.16.120.152) port 9004
* Server auth using Basic with user 'admin'
> POST /dpa-api/report HTTP/1.1
> Authorization: Basic YWRtaW46U3RvcmFnZTAwMSE=
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: seieadpa01:9004
> Accept: */*
> Content-Type:application/vnd.emc.apollo-v1+xml
> Content-Length: 476
>
> <runReportParameters><report><name>Backup All Jobs</name> <!-- Report template name --></report><nodes><node>      <id>560775dc-e949-490d-ae1c-d76ec33b18b5</id> <!-- scope - node id of the Host --></node></nodes><timeConstraints type="window"><window ><name>Last Week</name> <!-- time period name --></window></timeConstraints><formatParameters><formatType>HTML</formatType> <!-- format type, could be CSV, HTML, PDF, IMAGE, XML. --></formatParameters>  </runReportParameters>HTTP/1.1 201 Created
< Server: Apache-Coyote/1.1
< X-FRAME-OPTIONS: SAMEORIGIN
< Location: http://seieadpa01:9004/dpa-api/report/result/3b33e982-b899-4439-9a8b-ed8a3a89859a
< Content-Type: */*
< Content-Length: 116
< Date: Tue, 05 Dec 2017 10:57:19 GMT
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   116  100   116    0     0   2296      0 --:--:-- --:--:-- --:--:--     0Connection #0 to host seieadpa01 left intact

* Closing connection #0
++ grep link /export/home/kfoley/scripts/REST_API/Temp/test_xml.out
+ LINK='   <link>http://seieadpa01:9004/dpa-api/report/result/3b33e982-b899-4439-9a8b-ed8a3a89859a</link>'
+ LINK='http://seieadpa01:9004/dpa-api/report/result/3b33e982-b899-4439-9a8b-ed8a3a89859a</link>'
+ LINK=http://seieadpa01:9004/dpa-api/report/result/3b33e982-b899-4439-9a8b-ed8a3a89859a
+ export LINK
+ curl -v -u 'username:Password' -H Content-Type:application/vnd.emc.apollo-v1+xml http://seieadpa01:9004/dpa-api/report/result/3b33e982-b899-4439-9a8b-ed8a3a89859a
* About to connect() to seieadpa01 port 9004
*   Trying 172.16.120.152... connected
* Connected to seieadpa01 (172.16.120.152) port 9004
* Server auth using Basic with user 'admin'
> GET /dpa-api/report/result/3b33e982-b899-4439-9a8b-ed8a3a89859a HTTP/1.1
> Authorization: Basic YWRtaW46U3RvcmFnZTAwMSE=
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: seieadpa01:9004
> Accept: */*
> Content-Type:application/vnd.emc.apollo-v1+xml
>
< HTTP/1.1 503 Service Unavailable
< Server: Apache-Coyote/1.1
< Pragma: No-cache
< Cache-Control: no-cache
< Expires: Wed, 31 Dec 1969 19:00:00 EST
< X-FRAME-OPTIONS: SAMEORIGIN
< Retry-After: 2
< Content-Type: application/octet-stream
< Content-Length: 116
< Date: Tue, 05 Dec 2017 10:57:19 GMT
< Connection: close
<report>
   <link>http://seieadpa01:9004/dpa-api/report/result/3b33e982-b899-4439-9a8b-ed8a3a89859a</link>
Closing connection #0
</report>+ echo 'curl -u username:Password http://seieadpa01:9004/dpa-api/report/result/3b33e982-b899-4439-9a8b-ed8a3a89859a'
curl -u admin:Storage001! http://seieadpa01:9004/dpa-api/report/result/3b33e982-b899-4439-9a8b-ed8a3a89859a
[root@seiemccli01 REST_API]#

---------- Post updated at 12:08 PM ---------- Previous update was at 11:03 AM ----------

It must have something to do with the curl calls made earlier in the script that break this. If I just have a small script with just these 2 entries it works.

#!/bin/bash

LINK=http://seieadpa01:9004/dpa-api/report/result/8ffb82ec-c808-4a06-838a-9ac8dc7c93fa

curl -u username:Password $LINK

[root@seiemccli01 REST_API]#

The thing is that I need to generate the report first within the script using POST and then GET the report once I have generated it....

---------- Post updated at 12:23 PM ---------- Previous update was at 12:08 PM ----------

I decided to try to put a "

sleep 5

" before the last curl is executed and this has fixed the issue.

Thanks