Forums

API request stops at the same time, every time. Am I hitting the site too hard?

Hi guys, I am constantly getting this error after i run my query:

    Traceback (most recent call last):
  File "C:\Users\paulg\Desktop\json\apifetch.py", line 16, in <module>
    json_data = requests.get(url).json()
  File "C:\Python\lib\site-packages\requests\models.py", line 896, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Python\lib\json\__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "C:\Python\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

My query is as follows. Any help on this would be appreciated.

import urllib.parse
import requests
import json
import time

main_api = 'http://sc-api.com/?api_source=live&system=accounts&action=full_profile&'

format = '&format=json'

with open('allmembers.json') as f:
    data = json.load(f)

    result = []
    for element in data:
        target_id = element['Handle']

        url = main_api + urllib.parse.urlencode({'target_id': target_id}) + format
        json_data = requests.get(url).json()
        time.sleep(20)
        json_status = json_data['request_stats']['query_status']

        handle = json_data['data']['handle']
        moniker = json_data['data']['moniker']
        citizen_number = json_data['data']['citizen_number']

        # -----preview if no need to write to file-----
        print('target_id: ' + handle + ',', 'Moniker: ' + moniker + ',', 'Citizen Number: ' + citizen_number)

        result.append({'target_id': handle, 'Moniker': moniker, 'Citizen Number': citizen_number})

    with open('output2.json', 'w') as f:
        json.dump(result, f)
deleted-user-4429329 | 2 posts | Sept. 5, 2018, 9:31 p.m. | permalink

It's hard to say what's causing that, but you can probably find out by looking more closely at what requests.get is returning. So, for example, instead of doing this:

json_data = requests.get(url).json()

...which assumes that the request was successful, you could look at the status like this:

response = requests.get(url)
if response.status_code != 200:
    print("Server didn't return an 'OK' response.  Content was: {!r}".format(response.content))
else:
   json_data = response.json()
   time.sleep(28)
   # The rest of your code to handle the response goes here.

That will print out what the server returns for all unsuccessful requests, which should give you some idea as to what's going wrong.

Staff giles | 11800 posts | PythonAnywhere staff | Sept. 6, 2018, 11:18 a.m. | permalink

I understand what you are trying to say. However, it will only pull and then print the response from the last 'handle' in my allmembers.json file (there should be 5 total). Am i missing something?

deleted-user-4429329 | 2 posts | Sept. 6, 2018, 12:40 p.m. | permalink

If you make that change then it will process all of the members in the file for which it gets a response from the other server, but it won't do anything for the ones where it doesn't get a response. But the important thing is that it will give you some debugging information so that you can work out why the other server isn't giving you data for those people. Here's the full modified code, just for clarity:

import urllib.parse
import requests
import json
import time

main_api = 'http://sc-api.com/?api_source=live&system=accounts&action=full_profile&'

format = '&format=json'

with open('allmembers.json') as f:
    data = json.load(f)

    result = []
    for element in data:
        target_id = element['Handle']

        url = main_api + urllib.parse.urlencode({'target_id': target_id}) + format
        response = requests.get(url)
        if response.status_code != 200:
             print("Server didn't return an 'OK' response.  Content was: {!r}".format(response.content))
        else:
            json_data = response.json()
            time.sleep(20)
            json_status = json_data['request_stats']['query_status']

            handle = json_data['data']['handle']
            moniker = json_data['data']['moniker']
            citizen_number = json_data['data']['citizen_number']

            # -----preview if no need to write to file-----
            print('target_id: ' + handle + ',', 'Moniker: ' + moniker + ',', 'Citizen Number: ' + citizen_number)

            result.append({'target_id': handle, 'Moniker': moniker, 'Citizen Number': citizen_number})

    with open('output2.json', 'w') as f:
        json.dump(result, f)
Staff giles | 11800 posts | PythonAnywhere staff | Sept. 6, 2018, 3:51 p.m. | permalink