wget and xml isssue

Shell Programming and Scripting
1

Hi All,
I need to download with wget all files with "xml" extension for a specifix url say for instance https://www.example.com/xmlfiles/
I need to do this 3 times a day downloading just the new files added since last download and/or files that are changed from the last download. In order to do this i have used the following command :

wget -r -nd -N -A xml --no-check-certificate https://user:password@www.example.com/xmlfiles/

I am successfully authenticate from the server but than I get
403 FORBIDDEN

If fo the following :

wget -r -nd -N --no-check-certificate https://user:password@www.example.com/xmlfiles/file.xml

Than I can successfully download the file/s from the url.

Where is my mistake ?
P.s. I cannot use ftp but just https

Than I need to parse all the downloaded xml files extracting data into some csv file what will be the best way ?
Here in attachment you can find one of my xml files as well as the output csv ( here exported into xls because csv is not allowed) file that I need to abtain after parsing of xml.

Thank you in advance for your help and Merry Christmas to all.
Nino

2

If you point a web browser at:

https://www.example.com/xmlfiles/

does it try and retrieve index.htm or index.html (whatever the web server's default is set to)?
If that is the case then I imagine wget(1) is doing the same and presumably that page does not exist?

3

Did you try using -A.xml as an option ?

4

Hi there, thank you for your reply.
I have tried also the -A.xml but no luck, you are right Tony wget is looking for the index.html page that in this case doesn't exist.
Any idea how solve the problem ?
Thanks you again.
Greetings

5

Either:

  1. Try a wget of index.htm, just in case it then gives you a list of the files in the directory from which you can then extract the names of the files you are interested in and then wget each file in turn.

  2. Just wget each file in turn you are expecting to get.

  3. Get the web server configuration amended so that option 1 works with either index.html or index.htm giving you a directory listing and then doing suggestion 1!