Facing difficulty scraping website with curl

UNIX for Beginners Q & A
1

Hi,

As an exercise, I am attempting to scrape the following website to obtain ticket price for a certain journey, so I can capture the cheapest ticket by running the script hourly.

https://www.skyscanner.com.au/transport/d/del/2019-12-14/bkkt/bkkt/2019-12-19/del?adults=2&children=0&adultsv2=2&childrenv2=&infants=0&cabinclass=economy&ref=home#/

I am not able to capture the ticket price via the curl command. Is there any better way to scrape the webpage to get the information I need?

curl -o skyscanner.txt -L https://www.skyscanner.com.au/transport/d/del/2019-12-14/bkkt/bkkt/2019-12-19/del?adults=2&children=0&adultsv2=2&childrenv2=&infants=0&cabinclass=economy&ref=home#/

Thanks.

2

Actually, it looks like this might be more than an exercise. It appears that skyscanner provides us with a method to "Get Price Alerts" if there are fluctuations in the ticket price, however I don't find this option for the flight(s) I intend to go on. (4 flights across multiple cities).

Thanks.

3

I assume you are going to use their API?

API Reference -skyscanner developers

"Screen Scraping" is a thing of the past, really...

JSON APIs like this one are the current "in thing" :slight_smile:

4

Hi Neo,

I didn't know about that - thanks! It looks like this will be useful.

Cheers.

5

Most commercial web sites like Skyscanner have APIs these days.

There is no need so "screen scrape" in most cases these days ... "screen scraping" is a relic of the distance "way of the dinosaur" past.....