Forums

Hi all,

I am having an issue HTTP POSTing GPS data from my Arduino to the my Django server so I can save that data in my database. The Arduino Libraries are already set up, and all I need to do is input the host, port, resource, and data i want to send. What I have is:

host [] = "marcoprouve.pythonanywhere.com";

port = 80;

resource [] = "/sensors4G/sensor_log";

data [] = "\"Name\" = \"John\"";

My project (sensorLogger) urls.py looks like:

from django.conf.urls import include, url
from django.contrib import admin
import sensors4G.views

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^$', sensors4G.views.index, name="index"),
    url(r'^sensors4G/', include('sensors4G.urls')),
]

My app (sensors4G) urls.py looks like:

from django.conf.urls import url

from . import views

urlpatterns = [
    url(r'^$', views.index, name='index'),
    url(r'^sensor_log/', views.sensor_log, name='sensor_log'),
]

Finally my App (sensors4G) views.py looks like:

from django.shortcuts import render
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_protect, csrf_exempt

def index(request):
    return HttpResponse("Hello, you're at the sensors4G index!")

@csrf_exempt
def sensor_log(request):
    return HttpResponse("Hello, you're at sensors4G sensor_log!")

The reason my views look empty is because I am currently just testing to see if the POST reaches the URL and gets a response. When I try to Post to the server I get this response:

Http code: 404 Server response: Server: openresty/1.9.15.1

If you have a solution or can provide me with any advice on how to better approach POSTing from an Arduino to my Django Server I would greatly appreciate it. Thanks!

Weird- how are you doing the post to the server? And are you posting to the correct url? Do you see the post in your server access logs?

I am solely using 4G data connection to POST to the Django server. I cannot see the post in my server access logs when trying to connect. I believe the URL is correct it should be :

"marcoprouve.pythonanywhere.com/sensors4G/sensor_log"

Correct? is it a problem with my account not being a paid account and for some reason I may not have access to these functions?

Below is my Arduino code for reference:

#include "arduino4G.h"

// APN settings
///////////////////////////////////////
char apn[] = "phone";
char login[] = "";
char password[] = "";
///////////////////////////////////////

// SERVER settings
///////////////////////////////////////
char host[] = "marcoprouve.pythonanywhere.com";
unsigned int port = 80;
char resource[] = "/sensors4G/sensor_log";
char data[] = "\"Name\": \"Marco\"";
///////////////////////////////////////

// variables
int error;


void setup()
{
  error = _4G.ON();
  if (error == 0)
    Serial.println(F("module ON..."));
  else
    Serial.println(F("module OFF"));

  Serial.println(F("Start program"));

  //********************************************************************
  // POST method to the Libelium's test url                             
  // You can use this php to test the HTTP connection of the module.    
  // The php returns the parameters that the user sends with the URL.   
  //********************************************************************

  //////////////////////////////////////////////////
  // 1. sets operator parameters
  //////////////////////////////////////////////////
  _4G.set_APN(apn, login, password);

  //////////////////////////////////////////////////
  // 2. Show APN settings via USB port
  //////////////////////////////////////////////////
  _4G.show_APN();
}


void loop()
{
  //////////////////////////////////////////////////
  // 1. Switch ON
  //////////////////////////////////////////////////  
  error = _4G.ON();

  Serial.println(F("Start program"));

  if (error == 0)
  {
    Serial.println(F("1. 4G module ready..."));

    ////////////////////////////////////////////////
    // 2. HTTP POST
    ////////////////////////////////////////////////

    Serial.println(F("2. HTTP POST request..."));

    // send the request
    error = _4G.http( POST_METHOD, host, port, resource, data);

    // check the answer
    if (error == 0)
    {
      Serial.print(F("Done. HTTP code: "));
      Serial.println(_4G._httpCode);
      Serial.print("Server response: ");
      Serial.println((const char *)_4G._buffer);
    }
    else
    {
      Serial.print(F("Failed. Error code: "));
      Serial.println(error, DEC);
    }
  }
  else
  {
    // Problem with the communication with the 4G module
    Serial.println(F("4G module not started"));
    Serial.print(F("Error code: "));
    Serial.println(error, DEC);
  }

  ////////////////////////////////////////////////
  // 3. Powers off the 4G module
  ////////////////////////////////////////////////
  Serial.println(F("3. Switch OFF 4G module"));
  _4G.OFF();

  ////////////////////////////////////////////////
  // 4. Sleep
  ////////////////////////////////////////////////
  Serial.println(F("4. Enter deep sleep..."));
  delay(1000);

  Serial.println(F("5. Wake up!!\n\n"));
}

You are correct that having a non paid account is the issue. Your https server requires your username and password to allow access and you need to pass that information in as an HTTP post request. I had the same access problem trying to access my website when creating a mobile web app. in java the solution is : private String username="admin"; (not your username) private String password="the password to access the server using https"; public void onReceivedHttpAuthRequest(WebView view, HttpAuthHandler handler, String host, String realm) {

    if(username != null && password != null) {
        handler.proceed(username, password);
        return;   
    } else {
        handler.cancel();
}

    // By default handle 401 like we'd normally do!
    super.onReceivedHttpAuthRequest(view, handler, host, realm);

} public void setUsernamePassword(String username, String password) { this.username = username; this.password = password; }

Hope this helps

Thank you for your response! Because I dont want to pay for no reason, I just want to make sure that if I simply pay, the HTTP post request should successfully find the URL? I am not trying to implement any user authentication on my website, simply trying to send data to my database to post on a Google Maps API (No Login Necessary). Would I still have to authenticate even though I include the @csrf_exempt token before my view?

Not got that far but I'm pretty sure once you have a paid account your hosting server becomes public.

Thank you, I will pay for the month and see if that works!

im not 100% convinced that's the problem, (pythonanywhere blocks outgoing external Internet access but not incoming for free users) but by all means try and then downgrade if it doesn't work.

Paying didnt work, does anyone have any other ideas?

You're getting a 404 because you don't have a view at /sensors4G/sensor_log. Your view is defined for /sensors4G/sensor_log/

I changed the urls.py for my app to:

from django.conf.urls import url

from . import views

urlpatterns = [
    url(r'^$', views.index, name='index'),
    url(r'^sensor_log', views.sensor_log, name='sensor_log'),
]

But I still get the 404 error...

Did you reload your web app after making the change?

Yes I reloaded my web app after making the change. I just recently tested different URLs with a web sniffer and found the posts to go through fine to my server, so I think it must be something to do with the way Arduino handles the resource[]. Any thoughts? I will post my results after more debugging

Just a quick note, I ran into a nearly identical problem with pythonanywhere. 404 on a POST from an arduino where the exact same POST from postman succeeds well. Unfortunately there is limited logging available. Basically able to watch the Serial Monitor within the Arduino app which has at least gotten me as far as a 404. Pythonanywhere server/access/error logs themselves don't show the transaction as I think it is never actually getting to my domain prefix.
Could it be that pythonanywhere is not interpreting origins coming from IOT type devices?

That seems unlikely -- unless the Arduino isn't sending the HOST header? That's been required since HTTP/1.1 (released 20 years ago this year), so it seems unlikely, but I suppose it's possible...? One other possibility -- are you using HTTP or HTTPS?

http for my traffic. The base config I'm using is the happy-bubbles/nfc reader and it has no https ability. I'll probably have to modify/rebuild the firmware to add in support for https, but I've tried making the same call using postman with both http and https and it has succeeded every time. One other thing I noticed. The version of happy-bubbles/nfc that I'm using is pre-built firmware from before they added in the correct Content-Type header (application/json vs xml)
On a local flask config, I can force interpretation of the payload sent as .json by using .get_json(force=True) I'll ping this forum post in case I find something, but has anyone had any success in getting any logs outside of the standard Server/Error/Access logs?
I've not seen as much as a single ping from my arduino device. Point it to my local flask and it comes right through.

No, don't worry about the HTTP/HTTPS thing -- any problems with that would happen if you were using HTTPS. HTTP should be fine so long as the HOST header is correctly set.

Is there anything in the URL apart from the hostname that I might be able to use to identify requests from your device? For example, is there something pretty unique in the path portion of the URL? For example, the other person with this problem had "sensor_log". If there is something like that, and you can give me a time span when you're sure you sent a request, I can grep through our loadbalancers' logs and see if I can find any requests that for some reason weren't being routed to your PythonAnywhere web app.

Oh, and if you're seeing nothing in your access log, the content-type thing is probably not the cause.

I am able to log all the headers from my arduino device if it helps:

Content-Length: 123
User-Agent: ESP8266
Connection: close
Host: 192.168.1.164:5000 (my local flask instance)
Content-Type: application/x-www-form-urlencoded

ESP8266 is the specific wifi chipset of the arduino module I'm using. Purpose built for IOT type functions. I can provide the model of the module in question, but there are so many built around this chipset at this point...

I should also mention the 404 that is returned is the standard pythonanywhere "Coming Soon!

This is going to be another great website hosted by PythonAnywhere."

That request header log looks like it was one that was sent to your local machine -- no? If you could get the headers from one that was sent to PythonAnywhere, that would be useful.

(If it was one that was sent to PythonAnywhere, the problem is the Host header, which needs to match the domain you're sending the request to -- probably something like lndngpg.pythonanywhere.com)

Excellent question. Unfortunately I can't get any logging of this type from the system yet. I've got a few different updates to the firmware I need to make, one of them is definitely building in logging output.

The example sent is an exact duplicate of the headers sent, the only difference being the target location. There is a simple webUI to edit target URL to post against. If I change it to my pythonanywhere site, any call immediate goes from returning 200's to this 404 While I can only infer, I've a pretty good bit of confidence that the headers would be:

Content-Length: 123
User-Agent: ESP8266
Connection: close
Host: <mysite>.pythonanywhere.com
Content-Type: application/x-www-form-urlencoded

Identical with the exception of the host url.

OK. Could you give us the "path" part of the URL so that we can check through the loadbalancer logs to find the requests? For example, if you're hitting http://yoursite.pythonanywhere.com/log_data, then the bit we need is the "/log_data"

absolutely - I'll set the path so it is referring to a completely unique string. Hold please :)

I just triggered exactly 3x runs to http://<myprefix>.pythonanywhere.com/a12a44a77zz232 Please let me know if there is anything I can do to provide additional data.
Much appreciated

Thanks! This is very strange. I used the IP that those requests came in on to track down requests to your site.

I can see that, for example, on 6 March at around 4am UTC you made four POSTs with the user agent "ESP8266" to a particular URL (I won't mention it here in case it's secret), the first one looking correct, then the next three with a control character at the end of the path. All four returned 404s.

You then made a POST with a normal desktop user agent a minute or so later (specifically, at 06/Mar/2017:04:01:54), and that came back with a 200.

The last of these is in your access log, but the first four are not.

I just made a manual GET request to the URL in question, using nc to talk HTTP directly to it and spoof a request that looked like it came from your Arduino; you can see that in in your access log timestamped 08/Mar/2017:12:58:11 -- because I was making a GET request to a view that you presumably have set up for POST only, your website sent me back a 405 method not allowed error.

So, that means that routing is working for normal desktop user agents, and for the kind of stripped-down HTTP that we would think the Arduino is using. And because your desktop request was made less than a minute after the failed requests from the Arduino, I think we can rule out the idea that routing to your site might have been broken at that point but recovered later.

However, the fact that you're getting a "coming soon" page and that it's not winding up in your access log suggests that the host header isn't been recognised. It's strange that the loadbalancer's access log is showing a host header that looks entirely correct, but the accesses from the Arduino aren't winding up in your site's own access logs.

One possibility -- maybe the host header you're sending is not exactly right, but is wrong in some manner that isn't visible in the loadbalancer's log file. For example, if there were a non-printable control code in the host header, so instead of <mysite>.pythonanywhere.com it was <mysite>.pythonanywhere.com\a (C syntax), I can imagine that we'd see what we are seeing: the loadbalancer would strip out the \a when writing to its own logs, but would use it when it does its "which webserver is running this website" lookup, so it would look up a non-existent site and return the 404 that you're seeing. Similarly, the logging subsystem would try to work out which site's logfiles to write the access to, wouldn't be able to find one because of the rogue control character, and would drop the access log on the floor.

I'll see if I can find anything relevant in the logging system's own logs to either support or disprove that theory... but in the meantime, if you could check with your own logging exactly what you're sending (keeping an eye out for nonprintable control characters), that would definitely be helpful.

Hmm, nothing in the logging system that points either way on that -- there are no errors relating to your domain, but now I think about it, I'm not sure it would log anything in the situation I'm suspecting. However, I'm becoming pretty much sure that something about the host header you're sending is stopping the request from being properly routed to a web server.

A bit of background:

• When a request comes in to a loadbalancer, it's processed by nginx.
• Some custom lua code running in the nginx processing loop uses the host header to look up which web server is currently handling requests for that website.
• It then proxies it over to the appropriate web server, which is also running nginx
• The web server sends a message off to the logging system to tell it to write to the access logs
• ...and then sends the request down to uWSGI to actually run your code and serve up the response.

I've confirmed that when you sent those requests last night with the a12a44a77zz232 in them, the loadbalancer sees one from a desktop, followed by three from an Arduino, then one more from the Arduino shortly after you made your forum post.

However, the web server sees only the one from the desktop. This, at least, explains why it's not winding up in your access logs.

So, clearly, the routing system on the loadbalancer isn't able to identify a web server for the Arduino requests, and that's why it's returning the 404 with a "coming soon" page -- that's what it does by default if it receives a request for a host it doesn't know how to route for.

Given that when I used nc to send a request based on the one in your earlier post:

GET /redacted-path/ HTTP/1.1
Content-Length: 123
User-Agent: ESP8266
Connection: close
Host: <mysite>.pythonanywhere.com
Content-Type: application/x-www-form-urlencoded

...some junk to pad it out to 123 characters...

...it worked fine and got routed to your website, which then correctly rejected it, there's nothing fundamentally wrong in the routing for non-desktop browsers. (That's not to say there's nothing subtly wrong, of course -- but I've not identified anything yet :-)

But the best guess I have at the moment is still that there might be some non-printable control character, or something similar, in the host header that you're sending. One possibility: maybe there's an extra newline character or something at the end? That would be easy to introduce, and hard to spot.

Just discussed this with a colleague -- the hypothesis also seems to make sense even though it's working when you POST to a local Flask server -- Flask servers ignore the Host header, so a broken one wouldn't cause problems.

Actually, I'm completely with you on the non-printable control character. Looking at the example URL that comes coded in the base project, the default config property is as follows: .nfc_url = "http://example.com/api/rfid\0", that "\0" is used to represent a null character in C. I'm definitely not adding that to my value, but it is entirely possible that it is being appended in code after I set my value and I just haven't seen the place where that might happen yet.

I may try throwing a pointless query string at the end to see adding some content after the path helps by not having "\0" right at the end of the path

Interesting! It's odd to see the null terminators being explicit like that.

Remember, though, that the problem (assuming I'm diagnosing it correctly) is with unprintable stuff in the host header -- I'd expect that adding stuff to the path is unlikely to change that. Though, of course, with embedded systems, anything is possible :-)

No joy with the addition of a query string parameter. I'm still working on attempting to get a functional modified build to increase logging but it has been slow going.

I very much appreciate the level of effort taken to date.

No problem! Let us know when you have more information.

My IoT project is suffering from this too! My sensor hardware vendor has a built-in POST system to upload data to my site.

When I run the Django site on a local machine it works fine (using DEV sever). But when I point the sensors at my PAW site - which is the same code - nothing.

No record in my access logs. This product has a SYSLOG server option, using that I find the sensor is getting a 404 error.

Like above, when I inspect the headers of a working local incoming message, host header is set like so:

'HOST': '192.168.2.16:8000'

Now when I target my PAW system, I use my own domain, which has a CNAME set to the PAW specified name, like

webapp-bob.pythonanywhere.com

But what are the rules for HOST header under HTTP/1.1?

Because I am thinking they are doing a lookup on the URL, getting an IP address, and then using THAT for the HOST name!

Which is going to get a 404 every time......

Hi there, I know you said it was a 404, but can we just rule out that it's not related to this common 400 error: https://help.pythonanywhere.com/pages/Django400BadRequest/ ?

I want to say something cheeky. But. I went back and double checked the error. And I enabled DEBUG and changed my allowed hosts to '*' before doing another test. Just To Be Sure.

And I am now relieved to say that it really is a 404 error! The relevant syslog messages I get from the failing device are:

RSV: URL:'my.customdomain.ca' IP:34.206.101.184
REQ :post: 33836
REQ :resp: 33592
[Hp]HTTP/1.1 404 Not Found
[Hl]Server: openresty/1.9.15.1
[Hl]Date: Mon, 22 Jan 2018 15:19:02 GMT
[Hl]Content-Type: text/html
[Hl]Content-Length: 2919
[Hl]Connection: keep-alive
[Hl]ETag: "5a1eced2-b67"
[Code]-404 len:2919
No response!

And nothing in any of my PAW logs.

Just a note to say that we're talking with @visrico over email -- if this turns out to be something generally-applicable, we (or @visrico) will post back here.

Someone find the solution?

The solution discussed here is to set the host header in your request so that it matches your web app's host name.

How do I get my web app host name. I am having the same issue and I need a solution.

sorry- what is the issue that you are running into? your hostname would be something like aggregator.pythonanywhere.com or whatever your webapp "domain name" is.

It post requesg with my host name in the headers return 404. The C code resolve the name of an ip that point to the default a site is coming up here page.

the C code?

what url are you trying to access, are you doing this from the browser or from some code that you have written?