Forums

Australian Government Weather API | HTTP Error 403: Forbidden | Custom Account $30.00 a month

Description: I am developing a webapp for a client and hold a Custom Account at $30.30 a month; and, I get the following error when I run the below code: (API: http://reg.bom.gov.au/fwo/IDN60903/IDN60903.94926.json)

In:

import pandas as pd
df = pd.read_json('http://reg.bom.gov.au/fwo/IDN60903/IDN60903.94926.json')
df = pd.DataFrame(df['observations']['data'])
df = df.head()
print(df)

Out: 

urllib.error.HTTPError: HTTP Error 403: Forbidden

Expected Output:

   sort_order    wmo      name history_product local_date_time  \
0           0  94926  Canberra        IDN60903      27/08:00pm   
1           1  94926  Canberra        IDN60903      27/07:30pm   
2           2  94926  Canberra        IDN60903      27/07:00pm   
3           3  94926  Canberra        IDN60903      27/06:30pm   
4           4  94926  Canberra        IDN60903      27/06:00pm

  local_date_time_full    aifstime_utc   lat    lon  apparent_t  ... rel_hum  \
0       20210627200000  20210627100000 -35.3  149.2         1.6  ...      95   
1       20210627193000  20210627093000 -35.3  149.2         1.8  ...      93   
2       20210627190000  20210627090000 -35.3  149.2         2.0  ...      92   
3       20210627183000  20210627083000 -35.3  149.2         2.3  ...      90   
4       20210627180000  20210627080000 -35.3  149.2         2.2  ...      88

   sea_state  swell_dir_worded swell_height swell_period  vis_km  weather  \
0          -                 -         None         None      10        -   
1          -                 -         None         None      10        -   
2          -                 -         None         None      10        -   
3          -                 -         None         None      10        -   
4          -                 -         None         None      10        -

   wind_dir  wind_spd_kmh  wind_spd_kt  
0       NNE            13            7  
1        NE            13            7  
2         N            15            8  
3         N            15            8  
4         N            19           10

[5 rows x 35 columns]

URL Source

  • Australian Government Bureau of Meteorology | Bureau Home > Weather Services > Data Services for weather information > Public Access Services > Data Feeds

URL: http://www.bom.gov.au/catalogue/data-feeds.shtml

Assumptions

  • I am under the understanding that custom accounts between $5 - $500 have unrestricted web access.
  • As this is a paid account I do not need to connect through proxy.server:3128

Research

Responce: 

"Hmm, we haven't changed anything in that area -- web dev accounts have unrestricted Internet access. What's the site you're scraping? Have you looked at the response contents? It's possible that they're blocking scraping from cloud services."

Responce: 

"Free accounts' internet access goes via a proxy 'whitelist' "
"Paid-for accounts don't have this limitation"

Testing:

  • Other Services: I ran the same code on multiple devices and platforms; and, each time it excited without error.

  • PythonAanywhere bash/terminal:

in:

curl http://reg.bom.gov.au/fwo/IDN60903/IDN60903.94926.json

out:

<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http&#58;&#47;&#47;reg&#46;bom&#46;gov&#46;au&#47;fwo&#47;IDN60903&#47;IDN60903&#46;94926&#46;json" on this server.<P>
Reference&#32;&#35;18&#46;edfda17&#46;1624785879&#46;a072946f
</BODY>
</HTML>
deleted-user-10794023 | 4 posts | June 27, 2021, 10:06 a.m. | permalink

Could you try running the relevant code in a newly created console and check if it reproduces?

Staff pafk | 2973 posts | PythonAnywhere staff | June 28, 2021, 10:27 a.m. | permalink

Thanks, it works; however, it is hit and miss on the many other APIs from the Government Weather website . Below is a small snapshot of the APIs and the code I run:

Origional: (Now Works)

import get_weather as gw
ws = gw.radars["ACT"]["CANBERRA AREA"]["Canberra"]
df = gw.allWeatherStats(ws)
print(df)

Out:

   sort_order    wmo      name history_product local_date_time local_date_time_full    aifstime_utc   lat    lon  ...  sea_state swell_dir_worded  swell_height  swell_period vis_km weather  wind_dir  wind_spd_kmh  wind_spd_kt
0           0  94926  Canberra        IDN60903      28/08:00pm       20210628200000  20210628100000 -35.3  149.2  ...          -                -          None          None     10       -        NE             7            4
1           1  94926  Canberra        IDN60903      28/07:30pm       20210628193000  20210628093000 -35.3  149.2  ...          -                -          None          None     10       -       ESE             6            3
2           2  94926  Canberra        IDN60903      28/07:00pm       20210628190000  20210628090000 -35.3  149.2  ...          -                -          None          None     10       -        SE             9            5
3           3  94926  Canberra        IDN60903      28/06:30pm       20210628183000  20210628083000 -35.3  149.2  ...          -                -          None          None     10       -      CALM             0            0
4           4  94926  Canberra        IDN60903      28/06:00pm       20210628180000  20210628080000 -35.3  149.2  ...          -                -          None          None     10       -         N             9            5

A Different API:

import get_weather as gw
ws = gw.radars["NSW"]["UPPER WESTERN"]["Wilcannia Airport"]
df = gw.allWeatherStats(ws)
print(df)

Out:

urllib.error.HTTPError: HTTP Error 403: Forbidden

Module: (With APIs in a Dictionary)

def allWeatherStats(url):
    df = pd.read_json(url)
    df = pd.DataFrame(df['observations']['data'])
    df = df.head()
    return df

def latestWeatherStats(url):
    df = pd.read_json(url)
    df = pd.DataFrame(df['observations']['data'])
    df = df.head(1)
    return df

radars = {
    "ACT" : {
        "CANBERRA AREA" :{
            "Canberra": "http://reg.bom.gov.au/fwo/IDN60903/IDN60903.94926.json",
            "Braidwood":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.94927.json",
            "Cabramurra":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.95916.json",
            "Cooma Airport":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.94921.json",
            "Goulburn Airport":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.95716.json",
            "Jervis Bay Airfield":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.94751.json",
            "Montague Island":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.94939.json",
            "Moruya Airport":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.95937.json",
            "Mount Ginini":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.95925.json",
            "Narooma":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.95935.json",
            "Nowra":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.94750.json",
            "Perisher Valley":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.94915.json",
            "Thredbo Top Station":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.95909.json",
            "Tuggeranong":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.94925.json",
            "Ulladulla":"http://reg.bom.gov.au/fwo/IDN60903/IDN60903.94938.json",
        },
    },
    "NSW" : {
        "NORTHERN RIVERS" : {
            "Ballina": "http://www.bom.gov.au/fwo/IDN60801/IDN60801.94596.json",
            "Cape Byron": "http://www.bom.gov.au/fwo/IDN60801/IDN60801.94599.json",
            "Casino":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94573.json",
            "Coolangatta":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94592.json",
            "Evans Head":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94598.json",
            "Grafton AgRS":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95571.json",
            "Grafton Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95570.json",
            "Lismore Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94572.json",
            "Murwillumbah":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94582.json",
            "Yamba":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94589.json",
        },
        "MID NORTH COAST": {
            "Coffs Harbour Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95729.json",
            "Dorrigo":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94789.json",
            "Kempsey Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94785.json",
            "Mount Seaview":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94783.json",
            "Port Macquarie Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94799.json",
            "Smoky Cape":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94790.json",
            "Taree Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95784.json",
        },
        "HUNTER": {
            "Cessnock Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95771.json",
            "Gosford":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94782.json",
            "Lake Macquarie - Cooranbong":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95767.json",
            "Lostock":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95772.json",
            "Maitland Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94650.json",
            "Mangrove Mountain":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95774.json",
            "Merriwa":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95754.json",
            "Murrurundi":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94758.json",
            "Murrurundi Gap":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95747.json",
            "Nelson Bay":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95779.json",
            "Newcastle Nobbys":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94774.json",
            "Norah Head":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95770.json",
            "Scone Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95758.json",
            "Singleton (Defence)":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.99738.json",
            "Tocal":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94775.json",
            "Williamtown":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94776.json"
        },
        "NORTHERN TABLELANDS" : {
            "Armidale":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94773.json",
            "Armidale Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95773.json",
            "Glen Innes Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94588.json",
            "Guyra":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94772.json",
            "Inverell":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95541.json",
            "Inverell RS":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94541.json",
            "Applethorpe":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94553.json",
            "Tabulam":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94587.json",
            "Tenterfield":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94556.json",
        },
        "SYDNEY METROPOLITAN" : {
            "Badgerys Creek":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94752.json",
            "Bankstown":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94765.json",
            "Camden":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94755.json",
            "Campbelltown":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94757.json",
            "Canterbury":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94766.json",
            "Fort Denison":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94769.json",
            "Holsworthy":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95761.json",
            "Holsworthy (Defence)":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95684.json",
            "Horsley Park":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94760.json",
            "Kurnell":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95756.json",
            "Little Bay":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94780.json",
            "Lucas Heights (ANSTO)":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95757.json",
            "North Head":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95768.json",
            "Parramatta":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94764.json",
            "Penrith":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94763.json",
            "Richmond":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95753.json",
            "Sydney Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94767.json",
            "Sydney - Observatory Hill":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94768.json",
            "Sydney Harbour":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95766.json",
            "Sydney Olympic Park":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95765.json",
            "Terrey Hills":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94759.json",
            "Wattamolla":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95752.json",
        },
        "ILLAWARRA" : {
            "Albion Park":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95748.json",
            "Bellambi":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94749.json",
            "High Range (Wanganderry)":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.99742.json",
            "Jervis Bay Airfield":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94751.json",
            "Kiama":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95749.json",
            "Moss Vale":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94746.json",
            "Nowra":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94750.json",
            "Point Perpendicular":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95940.json",
            "Port Kembla Harbour":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95745.json",
        },
        "SOUTH COAST" : {
            "Bega":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95931.json",
            "Gabo Island":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94933.json",
            "Green Cape":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94934.json",
            "Merimbula":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95929.json",
            "Montague Island":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94939.json",
            "Moruya Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95937.json",
            "Moruya Heads":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94937.json",
            "Narooma":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95935.json",
            "Ulladulla":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94938.json",
        },
        "CENTRAL TABLELANDS" : {
            "Bathurst":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94730.json",
            "Bathurst Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94729.json",
            "Gulgong":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94732.json",
            "Lithgow":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94741.json",
            "Marrangaroo (Defence)":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95686.json",
            "Mount Boyce":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94743.json",
            "Mudgee":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94727.json",
            "Nullo Mountain":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94754.json",
            "Orange Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95726.json",
            "Springwood":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95744.json",
        },
        "SOUTHERN TABLELANDS" : {
            "Braidwood":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94927.json",
            "Burrinjuck Dam":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94909.json",
            "Canberra":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94926.json",
            "Goulburn Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95716.json",
            "Mount Ginini":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95925.json",
            "Nerriga":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94943.json",
            "Taralga":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94735.json",
            "Tuggeranong":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94925.json",
        },
        "SNOWY MOUNTAINS" : {
            "Bombala":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94928.json",
            "Bombala AWS":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94929.json",
            "Cabramurra":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95916.json",
            "Cooma":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94923.json",
            "Cooma Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94921.json",
            "Perisher Valley":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94915.json",
            "Thredbo Top Station":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95909.json",
            "Thredbo Village":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95908.json",
        },
        "NORTH WEST SLOPES AND PLAINS" : {
            "Barraba":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94761.json",
            "Goondiwindi":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.99468.json",
            "Gunnedah Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95740.json",
            "Moree":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95527.json",
            "Mungindi":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94520.json",
            "Narrabri":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95734.json",
            "Pindari Dam":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94544.json",
            "Tamworth Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95762.json",
            "Walgett":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95715.json",
            "Woolbrook":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94792.json",
        },
        "CENTRAL WEST SLOPES AND PLAINS" : {
            "Brewon":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94793.json",
            "Condobolin Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95708.json",
            "Coonabarabran":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94728.json",
            "Coonabarabran Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95728.json",
            "Coonamble":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95718.json",
            "Cowra":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95721.json",
            "Dubbo":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95719.json",
            "Dunedoo":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95727.json",
            "Forbes":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94715.json",
            "Girilambone":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94794.json",
            "Grenfell":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94725.json",
            "Lake Cargelligo":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95707.json",
            "Parkes Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95717.json",
            "Peak Hill":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94721.json",
            "Temora":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95722.json",
            "Trangie":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95710.json",
            "West Wyalong":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95709.json",
        },
        "SOUTH WEST SLOPES" : {
            "Cootamundra":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94714.json",
            "Gundagai":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94890.json",
            "Khancoban":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94919.json",
            "Tumbarumba":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94918.json",
            "Young":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94712.json",
        },
        "RIVERINA" : {
            "Albury":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95896.json",
            "Deniliquin":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95869.json",
            "Griffith":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95704.json",
            "Hay Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94702.json",
            "Hillston":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94700.json",
            "Hume Reservoir":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94901.json",
            "Kapooka (Defence)":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94944.json",
            "Narrandera Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95706.json",
            "Swan Hill":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94843.json",
            "Tocumwal":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94877.json",
            "Wagga Wagga":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94910.json",
            "Yanco":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95705.json",
            "Yarrawonga":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94862.json",
        },
        "LOWER WESTERN" : {
            "Balranald":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94696.json",
            "Broken Hill Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94691.json",
            "Ivanhoe Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95697.json",
            "Lake Victoria":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94692.json",
            "Menindee":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94694.json",
            "Mildura":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94693.json",
            "Mount Hope":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95682.json",
            "Mulurulu":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94797.json",
            "Pooncarie":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95692.json",
        },
        "UPPER WESTERN" : {
            "Borrona Downs":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94407.json",
            "Bourke":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94703.json",
            "Brewarrina":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95512.json",
            "Cobar":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94711.json",
            "Cobar Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94710.json",
            "Collarenebri":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95520.json",
            "Delta":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95681.json",
            "Fowlers Gap":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94686.json",
            "Lightning Ridge":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94498.json",
            "Noona":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94798.json",
            "Smithville":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94796.json",
            "Tibooburra Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95485.json",
            "Wanaaring":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94497.json",
            "White Cliffs AWS":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95699.json",
            "Wilcannia Airport":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95695.json",
        },
        "ISLANDS" : {
            "Lord Howe Island":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94995.json",
            "Lord Howe Island Windy Point":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.95995.json",
            "Norfolk Island":"http://www.bom.gov.au/fwo/IDN60801/IDN60801.94996.json",
        },
    },
}
deleted-user-10794023 | 4 posts | June 28, 2021, 10:38 p.m. | permalink

What happens when you just use requests to hit the relevant URL?

Staff fjl | 4348 posts | PythonAnywhere staff | June 29, 2021, 9:26 a.m. | permalink

Thanks for your prompt reply.

This time I received a different error in bash with the same error in python.

Bash

In:

curl http://www.bom.gov.au/fwo/IDN60801/IDN60801.95695.json

Out:

Potential automated request detected! We are making changes to our website therefore web scraping is no longer supported. Please contact us by filling in the details at http://reg.bom.gov.au/screenscraper/screenscraper_enquiry_form/ and we will get in touch with you.

Python

In:

import pandas as pd
df = pd.read_json("http://www.bom.gov.au/fwo/IDN60801/IDN60801.95695.json")
df = pd.DataFrame(df['observations']['data'])
df = df.head()
print(df)

Out:

urllib.error.HTTPError: HTTP Error 403: Forbidden
deleted-user-10794023 | 4 posts | June 29, 2021, 9:52 a.m. | permalink

It is clear from the output that you posted in your last message that the site is blocking automated requests. That is not something we can do anything about.

Staff glenn | 9498 posts | PythonAnywhere staff | June 29, 2021, 5:02 p.m. | permalink

Thank you, I missed importance of the error message. As usual, you and your team are amazing! SOLVED :)

deleted-user-10794023 | 4 posts | June 30, 2021, 5:29 a.m. | permalink

Thanks, glad you got that working!

Staff pafk | 2973 posts | PythonAnywhere staff | June 30, 2021, 7:17 a.m. | permalink

I am on custom account $6/month. I got this error:403 Error. Forbidden. while trying to run the sample python script attached to my custom domain

Beloved premium user oluboyede | 3 posts | Aug. 11, 2023, 12:33 p.m. | permalink

What are the contents of the response that you get with that status code?

Staff giles | 11788 posts | PythonAnywhere staff | Aug. 11, 2023, 5:40 p.m. | permalink