I'm trying to upload a file to a page using curl and after uploading that file i want to store the redirecting page so i can download the results.
I'm using the command:
curl "http://apps.gdgps.net/kag_upload.php?kag_type=static&kag_frequency=dual&kag_latency=accurate&
kag_l1=P&kag_ema=7.5&rinex_interval=300&upfile_0=@CASC2270.09o&upload_button=Upload"
can anybody tell me what is wrong with this command?
The best way to debug this is to Wireshark the web browser packets. It would be easy for me to do this if you would post a simple input file to submit. JavaScript is used to submit the form and Wireshark will show what the actual page submitted is, whether any cookies are used, and whether there are any other hidden variables.
Once that is determined we can figure out what the curl command needs to be and then do a full trace with '--trace <file>'.
JavaScript can really complicate automation and sometimes completely prevent it.
The input of the file is something like this:
2.11 OBSERVATION DATA M (MIXED) RINEX VERSION / TYPE
teqc 2009Oct19 20100212 17:06:33UTCPGM / RUN BY / DATE
EDVD MARKER NAME
EDVD MARKER NUMBER
Ricardo Pinho ENTRE DOURO E VOUGA DIGITAL OBSERVER / AGENCY
REC # / TYPE / VERS
30473663 TRM55971.00 ANT # / TYPE
4774346.5322 -712193.6930 4155295.3307 APPROX POSITION XYZ
0.0000 0.0000 0.0000 ANTENNA: DELTA H/E/N
1 1 WAVELENGTH FACT L1/2
5 C1 L1 P2 L2 C2 # / TYPES OF OBSERV
30.0000 INTERVAL
Forced Modulo Decimation to 30 seconds COMMENT
2009 8 15 0 0 0.0000000 GPS TIME OF FIRST OBS
15 LEAP SECONDS
Linux 2.4.20-8|Pentium IV|gcc -static|Linux|486/DX+ COMMENT
Spider V3,2,1,3225 EDVD 2009 08 15 01:00 COMMENT
THIS RINEX FILE IS CREATED FROM RTCM V3.0 DATA COMMENT
Site Information : COMMENT
S�o Jo�o da Madeira COMMENT
Aveiro COMMENT
Portugal COMMENT
Eurasian COMMENT
Edif�cio Villa Balbina, Parrinho COMMENT
END OF HEADER
09 8 15 0 0 0.0000000 0 12G11G14G17G19G20G23G28G32R03R18R19R20
20204061.046 106173105.736 9 20204055.026 82732302.065 6
24317182.598 127787671.463 6 24317177.518 99574813.950 1
23267139.626 122269661.145 8 23267134.966 95275064.518 3
22988089.228 120803264.966 8 22988082.808 94132424.697 4
20444473.224 107436454.811 9 20444466.824 83716723.899 6
23991862.240 126078089.385 8 23991855.620 98242667.813 1
22921991.588 120455895.437 8 22921985.708 93861740.967 3
20345088.586 106914198.779 9 20345083.826 83309773.986 6
23099342.888 123652633.182 7 23099340.348 96174280.414 5
22347917.172 119294734.537 8 22347917.112 92784802.964 6
19275347.132 103110143.247 9 19275344.132 80196786.689 8
20776460.724 111101100.392 8 20776459.684 86411972.037 7
09 8 15 0 0 30.0000000 0 12G11G14G17G19G20G23G28G32R03R18R19R20
20206296.646 106184854.498 9 20206290.986 82741456.941 6
24329858.198 127854278.930 6 24329852.418 99626715.864 1
That's nice, but we still don't know what the actual page submitted is, whether any cookies are used, and whether there are any other hidden variables.
The page is: Automatic Precise Positioning Service - APPS
I downloaded the page using curl and then run the program named formfind and the result was:
--- FORM report. Uses POST to URL "#"
--- type: multipart form upload [use -F]
Input: NAME="kag_type" VALUE="static" (RADIO)
Input: NAME="kag_type" VALUE="kinematic" (RADIO)
Input: NAME="kag_frequency" VALUE="single" (RADIO)
Input: NAME="kag_frequency" VALUE="dual" (RADIO)
Input: NAME="kag_latency" VALUE="nrt" (RADIO)
Input: NAME="kag_latency" VALUE="accurate" (RADIO)
Input: NAME="kag_l1" VALUE="CA" (RADIO)
Input: NAME="kag_l1" VALUE="P" (RADIO)
Input: NAME="kag_tdp" (CHECKBOX)
Input: NAME="kag_ema" VALUE="7.5" (TEXT)
Input: NAME="rinex_interval" VALUE="300" (TEXT)
Input: NAME="upfile_0" (FILE)
Input: NAME="reset_button" VALUE="Reset" (BUTTON)
Input: NAME="upload_button" VALUE="Upload" (BUTTON)
--- end of FORM
Ran Wireshark (WS) and found out a couple things.
I've attached the WS output as plain text. Sorry but it doesn't include the initial page request, just the submission of defaults with the input file you provided. Also set WS to only display HTTP messages. See attached file 'test15-ws-out.txt'
Using curl to accomplish this function appears to be rather complex. You'll have to simulate the JavaScript random ID to execute the "GET /kag_link_upload.php?rnd_id=1275597861022". You may be able to skip the upload progress "GET /kag_set_progress.php?upload_id=151cf1b0cd09e4cbd4fea32148619618" but you'll have to do MIME encoding on the "POST /cgi-bin/kag_upload.pl?upload_id=151cf1b0cd09e4cbd4fea32148619618" for the file. In each case you must set the proper "http.referer", probably provide a "http.user_agent" value which the site likes, and include the proper cookies.
A couple HTTP packets later my upload fails, so the rest of the trace isn't that relavent except in how to handle errors.
This task to fake this web browser app seems non-trivial to me. On How To Use APPS page it describes a way to upload the file to their FTP server where it will be automatically processed. An FTP transaction would be much easier to implement but you'll have to email them for login details. The other solution is to automate an email and FTP upload which seems to be a bit of a pain compaired to just FTPing a file.
Thanks for your attention, this information was very helpful
I have a new problem in another site, but i think that's the same thing. can you take a look. the command that i'm doing is this:
curl -L "http://webapp.csrs.nrcan.gc.ca/field/Scripts/CSRS\_PPP\_main\_e.pl?user\_name=CGIS0134&customer\_email=dmbl84@gmail.com&session_id=849c1c1630e7523eec0fcf9e4945a58cc851b1818e69" -F rfile_upload=@CASC2270.09o -F process_type=Static -F sysref_system=ITRF -F return_email=dmbl84@gmail.com -F cmd_process_type=std -F return_email=dmbl84@gmail.com -F language=en -F SUBMIT=START > aux.html
and the file that i upload is the same.
Looked at Online Database (CSRS Online Database) which redirected me to a login screen. No surprises there but I can't WireShark (WS) the packets. You can install it for Windows or Linux from the WS download page. Tell WS to filter HTTP or you will get too much data.
I'm guessing, but I think to automate the csrs.nrcan.gc.ca site you'll need to automate the login, grab the session_id and cookies it generates, then send the file and the appropriate cookies. It may be easier or harder than the other site, I can't tell without signing up. PERL with the proper modules could probably do it.
If you're running Windows, maybe the PPP Direct utility would work better for you. Once set up you can drag and drop files to it for processing. It might also work under Wine for Linux.
It seems that both sites could use a decent batching API where you just give it a list of files in a text file to work on. Probably surveyors would find that very useful.
I've never used these services, I'm just happy to get my GPX files to display on Google Maps. What exactly do you get back from these sites, a more accurate file?