Forums

I've written a script which pulls all of the trains departing and arriving at London Waterloo station and puts the value into a CSV file It works fine locally, but when I run it on pythonAnywhere I'm getting a 403 client error. Any idea why please?

code is below...

thanks!

Rupert

import os, requests, bs4, datetime, csv
trainInfo = ['dep','arr']
timeNow = datetime.datetime.now()
trainFile = open('train.csv','a', newline='')
csvWriter = csv.writer(trainFile)

for trainType in trainInfo:
    watPage = requests.get('http://ojp.nationalrail.co.uk/service/ldbboard/%s/WAT' % trainType)
    watPage.raise_for_status()
    watSoup = bs4.BeautifulSoup(watPage.text, "html.parser")
    section1 = watSoup.select('div.results.trains div.tbl-cont table tbody tr')
    for table_row in section1:
        cells = table_row.findAll('td')
        if len(cells) > 0:
            ttime = cells[0].text.strip()
            tdest = cells[1].text.strip()
            tplatform = cells[3].text.strip()
            tstatus = cells[2].text.strip()
            if tplatform == '':
                continue
            csvWriter.writerow([timeNow, trainType, ttime, tdest, tplatform, tstatus])

trainFile.close()

OK, after searching in here, I think its because I'm on a free account with limited outbound access.

that is true.

However, if the national rail endpoint you are trying to access is a public api and they have documentation pointing to the particular url, show us a link to that part of the documentation and we will whitelist that endpoint for you (so free users can also access that)

Hi Conrad,

There is documentation about all of the the endpoints here...

http://www.nationalrail.co.uk/46391.aspx

There is a combined developer doc here which talks about the free developer access to the endpoints

http://www.nationalrail.co.uk/static/documents/NRE%20Feeds_Developer%20Pack%20v.01-04.pdf

Thanks

Rupert

Hi Conrad,

There is documentation about all of the the endpoints here...

http://www.nationalrail.co.uk/46391.aspx

There is a combined developer doc here which talks about the free developer access to the endpoints

http://www.nationalrail.co.uk/static/documents/NRE%20Feeds_Developer%20Pack%20v.01-04.pdf

Thanks

Rupert

There is a slight peculiarity in that I’m attempting to access the LDB (live departure board) which is part of Darwin but the actual URL is prefixed with OJP

A whitelist of *.nationalrail.co.uk

Would be perfect!

Thanks

Rupert

Ok. The only endpoint that I could find mentioned was ojp.nationalrail.co.uk, so I have whitelisted that.