Crontab Wget, downloading a file to a FTP

zYx · November 3, 2010, 7:38am

Hi there,

I've got my own domain, ftp etc.. I'm using cPanel and I want to download a file periodically, every say 24 hours.

I've used this command:

wget -t inf http : / / www . somesite . com / webcam.jpg
ftp : / / i @ MyDomain . net : Password @ ftp . MyDomain . net

^no spaces offcourse... otherwise I wouldn't be able to type urls here on forum...

I was told to use /home/UserName/public_html/ instead of ftp login details, but it doesn't work so I've used ftp login details.

Now, yes, the jpg file is being downloaded to my ftp, but in root folder, not even in /public_html/ folder. I've tried with my main ftp login, which is without the "@" symbol.. I tried adding /public_html/ to an end and still, files are being downloaded to the root folder on my ftp

What am I doing wrong?

I would appreciate help as I am brand spanking new to this

DGPickett · November 3, 2010, 10:03am

Well, even not knowing cPanel or these commands, the spacey URL on the second line lacks a path after the host, s it defaults to the root (unless the user is forced to one subdir as if chroot).

Corona688 · November 3, 2010, 10:10am

I don't think they meant login to /home/user/public_html/, I think they meant login using user's credentials and upload into public_html.

Can you repost, in code tags? It's so mangled I can't tell what you're even trying to do. I think you've got a wget in there, but I don't know what that raw FTP url's supposed to do in cpanel or crontab, it's just a URL. Put your stuff in code tags and it won't get mangled, like this. Quote my post to see how. It's easy.

http://www.url.com/

DGPickett · November 3, 2010, 10:21am

In other words, the second line resembles an ftp URL:

ftp://USER:PASSWORD@host.domain.higher_domain/top_level_dir/next_level_dir/

but has no pathing /top_level_dir/next_level_dir/ inside the host.

zYx · November 3, 2010, 12:12pm

I tried to put it in code and because I had less than 5 posts I wasn't able to use URLs here on forums, so I'm gonna try now:

get -t inf http://www.somesite.com/webcam.jpg ftp://UserName@MyDomain.net:Password@ftp.MyDomain.net

Ok it works now, so it should be clearer for you.

i also tried this:

get -t inf http://www.somesite.com/webcam.jpg ftp://UserName@MyDomain.net:Password@ftp.MyDomain.net/public_html/yyy/

but it would still be saved to the root.

Just so you know that I am not really a unix person, I've always worked with windows, and currently have win7, but it hasn't got anything to do with windows since the CronTab I'm trying to use is in my cPanel on my hosting account. I believe this is unix technology and this is where the problem starts ;]

Another thing, all these JPG files have the same name, so every time a new JPG is being downloaded, numbers are added to the end of the file name. I.e. IMAGE.JPG.1, IMAGE.JPG.2 etc... Is there any chance to change the name to IMAGE.1.JPG or something like that?
Also, with every single JPG file, a HTML files are being downloaded or.. created. Index.html.1, Index.html.2 etc.. which contains the following:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
 <HEAD>
  <TITLE>Index of /yyy/y</TITLE>
 </HEAD>
 <BODY>
<H1>Index of /yyy/y</H1>
<PRE><IMG SRC="/icons/blank.gif" ALT="     "> <A HREF="?N=D">Name</A>                    <A HREF="?M=A">Last modified</A>       <A HREF="?S=A">Size</A>  <A HREF="?D=A">Description</A>
<HR>
<IMG SRC="/icons/back.gif" ALT="[DIR]"> <A HREF="/yyy/">Parent Directory</A>        03-Nov-2010 11:22      -  
</PRE><HR>
<ADDRESS>Apache/1.3.37 Server at MyServer.net Port 80</ADDRESS>
</BODY></HTML>

The /yyy/y/ are just a testing folders in /home/user/public_html/yyy/y/.

Don't know what the "/icons/back.gif" is:confused:

I also get emails every time cron runs:

--01:29:01--  http://somesite.com/webcam32.jpg
           => `webcam32.jpg.24'
Resolving www.somesite.com... 195.173.130.149 Connecting to www.somesite.com|195.173.130.149|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21,470 (21K) [image/jpeg]

    0K .......... ..........                                 100%  165.31 KB/s

01:29:02 (165.31 KB/s) - `webcam32.jpg.24' saved [21470/21470]

--01:29:02--  ftp://user:*password*@ftp.MyServer.net/public_html/yyy/
           => `.listing'
Resolving ftp.MyServer.net... 193.218.153.71 Connecting to ftp.MyServer.net|xxx.xxx.xxx.71|:21... connected.
Logging in as User ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /public_html/yyy ... done.
==> PASV ... done.    ==> LIST ... done.

    0K                                                         589.90 KB/s

01:29:02 (589.90 KB/s) - `.listing' saved [119]

Removed `.listing'.
Wrote HTML-ized index to `index.html.6' [236].

FINISHED --01:29:02--
Downloaded: 21,470 bytes in 1 files

Though, the file is still saved in the root, not in public_html/yyy.

When i used /home/user/public_html/ instead of ftp login details, the email came saying:

/home/user/public_html/: Unsupported Scheme

along with the rest of the emails, similar to the one above... whatever that means.

Tried googling it, but couldn't find anything related to my particular "issue" :]

DGPickett · November 3, 2010, 1:38pm

The message "/home/user/public_html/: Unsupported Scheme" says paths are $HOME relative. Do you have directories: /home/user/public_html/ and /home/user/public_html/folder/ ?

zYx · November 3, 2010, 3:01pm

This is what I see when I log in to my ftp

The "/home/user/public_html/folder/" is just an example name, in fact it's called /yyy/ and yes, it is there, in /home/user/public_html/yyy/

DGPickett · November 3, 2010, 3:33pm

Maybe you need to use the absolute path in the URL. A good URL should work in a browser to show you that dir.

zYx · November 3, 2010, 3:49pm

I'm not sure what do you mean

DGPickett · November 3, 2010, 4:53pm

ftp://id:pw@host/home/user/public_html/folder/ not ftp://id:pw@host/public_html/folder/

URLs are usually absolute to the machine, not relative to the id, but it varies by setup and ftp daemon. Some make the ftp login world look like chroot $HOME so you cannot wander. Try the extended URL in a browser first, to ensure it is valid.

zYx · November 3, 2010, 5:15pm

I tried that too, and this is the log:

--22:11:01--  http://www.somesite.com/webcam32.jpg
           => `webcam32.jpg.173'
Resolving www.somesite.com... xxx.xxx.xxx.149 Connecting to www.somesite.com|195.173.130.149|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21,547 (21K) [image/jpeg]

    0K .......... .......... .                               100%  192.17 KB/s

22:11:01 (192.17 KB/s) - `webcam32.jpg.173' saved [21547/21547]

--22:11:01--  ftp://user:*password*@ftp.myserver.net/home/user/public_html/yyy/
           => `.listing'
Resolving ftp.myserver.net... xxx.xxx.xxx.71 Connecting to ftp.myserver.net|xxx.xxx.xxx.71|:21... connected.
Logging in as UserName ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /home/user/public_html/yyy ... 
No such directory `home/user/public_html/yyy'.

unlink: No such file or directory

FINISHED --22:11:01--
Downloaded: 21,547 bytes in 1 files

Files are still being saved to the very first root folder

The above is using the main ftp user.

I also tried creating an ftp account, which only has access to a particular folder on my ftp server. Still, files were saved to the root folder...

edit:

This command seems to be doing something according to the email below

ftp://user:*password*@ftp.myserver.net/public_html/yyy/

--23:10:01--  http://www.somesite.com/webcam32.jpg
           => `webcam32.jpg.31'
Resolving www.somesite.com... 195.173.130.149 Connecting to www.somesite.com|xxx.xxx.xxx.149|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22,550 (22K) [image/jpeg]

    0K .......... .......... ..                              100%  202.63 KB/s

23:10:01 (202.63 KB/s) - `webcam32.jpg.31' saved [22550/22550]

--23:10:01--  ftp://user:*password*@ftp.myserver.net/public_html/yyy/
           => `.listing'
Resolving ftp.myserver.net... xxx.xxx.xxx.71 Connecting to ftp.myserver.net|xxx.xxx.xxx.71|:21... connected.
Logging in as user ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /public_html/yyy ... done.
==> PASV ... done.    ==> LIST ... done.

    0K .......... .......... .......... .......... ..........    1.82 MB/s
   50K .......... .......... ..........                        112.09 MB/s

23:10:01 (2.88 MB/s) - `.listing' saved [81993]

Removed `.listing'.
Wrote HTML-ized index to `index.html.6' [176981].

FINISHED --23:10:01--
Downloaded: 22,550 bytes in 1 files

Unfortunately, files are still being saved to the root folder :[

DGPickett · November 3, 2010, 7:14pm

Well, I am a bit short on documentation for your 'get'. Does the URL work in IE or Firefox?

zYx · November 3, 2010, 7:24pm

Yes it does.

DGPickett · November 3, 2010, 7:38pm

Maybe you have to wget it to a temp dir and then wput it, or wget|wput with stdout/stdin options. (It might be faster with the pipe, too!) What is your O/S?

zYx · November 3, 2010, 7:49pm

My OS is win7, but the cPanel is installed on my hosting account so I take it it's linux ?

DGPickett · November 3, 2010, 8:22pm

Seems likely.

zYx · November 4, 2010, 12:10pm

I got it working in the end with the following command:

wget http://somesite.com/IMAGE.JPG -P /home/user/public_html/yyy/y/

Is there a way to change the names of IMAGE.JPG.1 to something like IMAGE.001.JPG?

DGPickett · November 4, 2010, 12:58pm

Not have an old one in the way (rename it or remove it) might prevent it. In linux, the rename is move, spelled mv and the delete is spelled rm.

zYx · November 4, 2010, 5:14pm

How can I use it as I am a nob in this

Corona688 · November 4, 2010, 6:02pm

mv oldname newname

Also see man mv