Forums

ssh & GitPython

I am trying to automatically pull changes from my origin in github to the web app in pythonanywhere using webhooks. Repo was cloned using ssh. ssh keys are added to ssh-agent.

I have my pythonanywhere public SSH key stored with github as deploy key. When pushes are made to the origin on github, the webhook sends POST request to my /update_server/ url. The view function looks like this:

@csrf_exempt
def update(request):
    if request.method == "POST"

        repo = git.Repo("/repo_dir/")
        origin = repo.remotes.origin
        origin.pull()

        return HttpResponse("Updated code on PythonAnywhere")
    else:
        return HttpResponse("Couldn't update the code on PythonAnywhere")

The webhook works fine with simple ping, so that functionality is working. The url also works ok. But I keep getting the below 500 Internal Service Error when I try to execute the process from github.

2020-04-28 14:04:13,397: Internal Server Error: /update_server/
Traceback (most recent call last):
  File "/home/deasunod/.virtualenvs/myenv/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/home/deasunod/.virtualenvs/myenv/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/deasunod/.virtualenvs/myenv/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/deasunod/.virtualenvs/myenv/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/deasunod/ppr_corona/main/views.py", line 228, in update
    origin.pull()
  File "/home/deasunod/.virtualenvs/myenv/lib/python3.7/site-packages/git/remote.py", line 812, in pull
    res = self._get_fetch_info_from_stderr(proc, progress)
  File "/home/deasunod/.virtualenvs/myenv/lib/python3.7/site-packages/git/remote.py", line 676, in _get_fetch_info_from_stderr
    proc.wait(stderr=stderr_text)
  File "/home/deasunod/.virtualenvs/myenv/lib/python3.7/site-packages/git/cmd.py", line 408, in wait
    raise GitCommandError(self.args, status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(1)
  cmdline: git pull -v origin
  stderr: 'fatal: Could not read from remote repository.
**NO MATCH**
Please make sure you have the correct access rights
and the repository exists.'

Any help or ideas would be greatly appreciated!

What directory are you using where you have "/repo_dir/" in your posted code? I'm wondering if it's a working directory issue. If you use a relative path like "mysite/" then it will be resolved relative to the current working directory of the running Python code, which might not be what you think it is -- better to use a full absolute path like "/home/deasunod/mysite", or even better to build an absolute path based on the "magic" variable __file__ as described on this help page.

Thanks Giles.

That's my mistake in terms of replicating the code for the snippet. I tried both relative path "mysite/" and "/home/deasunod/mysite" where the .git directory is. Same error as above - so I ruled out a working directory issue.

OK, thanks for the details -- you should definitely use an absolute path, but clearly that's not the issue!

Just to check -- when you said earlier that the webhook works with a simple ping, did you mean that you POSTed to it somehow, and confirmed that you got back the response "Updated code on PythonAnywhere"?

Sorry Giles - yes.

I removed the Deploy Key from github and sent a simple post request with the webhook (removed GitPython functionality from the view) just to test it. Github webhook confirmed it was 200 status response

Ah- I'm pretty sure your webapp doesn't have access to ssh-agent.

@deasunod How you exactly solved the issue can you share the corrected code or steps to be done.I am facing the same problem

@deasunod's issue was that they were not correctly authenticating to have access to a private repository. Have you configured GitPython so that it authenticates correctly? Does the repository that you're trying to access actually exist?

My repository is also private . What I have done is that I cloned the repo on pythonanywhere and hosted it there and have created a GitHub webhook for syncing.

django view function performing the task :

@csrf_exempt
def update(request):
if request.method == "POST":

    repo = git.Repo('Neuroly') 
    origin = repo.remotes.origin

    origin.pull()

    return HttpResponse("Updated code on PythonAnywhere")
else:
    return HttpResponse("Couldn't update the code on PythonAnywhere")

Error that I get in error log at pythonanywhere

 raise GitCommandError(self.args, status, errstr)
 git.exc.GitCommandError: Cmd('git') failed due to: exit code(1)
 cmdline: git pull -v origin
 stderr: 'fatal: Could not read from remote repository.
 **NO MATCH**
 Please make sure you have the correct access rights
 and the repository exists.'

What directory are you running it from? Is it configured anywhere?

Can anyone help with getting this working? I have a private repo on github and generated ssh keys and when i do a manual git pull origin master everything is working fine. Now im trying to do this with a webhook from github. I have a view in my django project with this code:

@csrf_exempt
def update(request):
     if request.method == "POST":
        repo = git.Repo('') 
        origin = repo.remotes.origin
        origin.pull()
        return HttpResponse("Updated code on PythonAnywhere")
    else:
        return HttpResponse("Couldn't update the code on PythonAnywhere")

When this didnt work i tried this:

@csrf_exempt
def update(request):
    if request.method == "POST":
        DIR_NAME = '/home/filozof595/'
        git_ssh_identity_file = os.path.expanduser('/home/filozof595/.ssh/id_rsa')
        git_ssh_cmd = f'ssh -i {git_ssh_identity_file}'

        with git.Git().custom_environment(GIT_SSH_COMMAND=git_ssh_cmd):
            repo = git.Repo('')
            origin = repo.remotes.origin

            origin.pull()

        return HttpResponse("Updated code on PythonAnywhere")
    else:
        return HttpResponse("Couldn't update the code on PythonAnywhere")

This also didnt work. Always the same error: Cmd('git') failed due to: exit code(1) cmdline: git pull -v origin

Everything is in my root dir /home/filozof595/ (all folders: .git, .ssh and my project folder).

I think its a issue with ssh authentication but im not sure and i dont know how to fix this.

I also tried ssh-copy-id filozof595@ssh.pythonanywhere.com but im getting this error: ERROR: ssh: connect to host ssh.pythonanywhere.com port 22: Connection refused

The ssh-copy-id is to set things up for SSH-ing into PythonAnywhere, so that's not necessary for this (and would not work anyway in a free account).

The error that you're seeing is a weird one -- it doesn't seem to include the information that you need to fix it, which is frustrating!

What happens if, instead of using the git module, you just use subprocess to shell out and run the git command? I'm thinking of something like this:

import subprocess

...

@csrf_exempt
def update(request):
     if request.method == "POST":
        subprocess.check_call("cd  /home/filozof595/; git pull", shell=True)
        return HttpResponse("Updated code on PythonAnywhere")
    else:
        return HttpResponse("Couldn't update the code on PythonAnywhere")

Now everything is working as expected. I just changed the git command to git pull origin master.

Just a side note for anyone who will read this in the future...:
If you change any file on Pythonanywhere this wont work because git wont be synced and you will get an error. If you don't care about the changes in Pythonanywhere and just need to sync everything on Pythonanywhere with your local files (files on your PC) then you will need a different command:

subprocess.check_call("cd  /home/<your_username>/; git fetch --all; git reset --hard origin/master; ", shell=True)

WARNING: This will delete all changes that you have made on Pythonanywhere.

Thank you for your help Giles!

No problem at all, glad I could help! And that's good advice regarding handling changes on the PythonAnywhere side. I would recommend that to avoid losing changes like that, you should commit and push them back up to GitHub as soon as you've made them.

Right now im working on a SPA project that uses Django rest framework for API and VueJS for my frontend. I would like to host my app on Pythonanywhere so right now im testing some of my code and settings as a free user.

Because of that i make only one change after uploading my code on Pythonanywhere.

I have a variable inside my settings.py -> PYTHONANYWHERE = False / True.

With this i can easily switch my database settings or static file paths etc...

I managed to automate this variable change using subprocess, similar to github pull. This is inside my update view:

import subprocess
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt

@csrf_exempt
def update(request):
    if request.method == "POST":
       try:
            subprocess.check_call("cd  /home/filozof595/; git fetch --all; git reset --hard origin/master; ", shell=True)
            subprocess.check_call("sed -i 's/PYTHONANYWHERE = False/PYTHONANYWHERE = True/g'  /home/filozof595/django/core/settings.py", shell=True)
    except subprocess.CalledProcessError as e:
            raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))

        return HttpResponse("Updated code on PythonAnywhere")
    else:
        return HttpResponse("Couldn't update the code on PythonAnywhere")

Right now i would like to automate app refresh. I have used your API example but im getting server error 502 and the Pythonanywhere "Something got wrong" page. Do you know what could be wrong? I'm using this code:

import subprocess
import requests
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt

@csrf_exempt
def update(request):
    if request.method == "POST":
        try:
            subprocess.check_call("cd  /home/filozof595/; git fetch --all; git reset --hard origin/master; ", shell=True)
            subprocess.check_call("sed -i 's/PYTHONANYWHERE = False/PYTHONANYWHERE = True/g' /home/filozof595/django/core/settings.py", shell=True)
            message = "Updated code on PythonAnywhere"
        except subprocess.CalledProcessError as e:
            raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))

        username = "filozof595"
        api_token = "my spi token" #token from the api section in my account details
        domain_name = "filozof595.pythonanywhere.com"

        response = requests.post(
            'https://www.pythonanywhere.com/api/v0/user/{username}/webapps/{domain_name}/reload/'.format(
                username=username, domain_name=domain_name
            ),
            headers={'Authorization': 'Token {token}'.format(token=api_token)}
        )
        if response.status_code == 200:
            message += " & App refresh done."
        else:
            message += "Got unexpected status code {}: {!r}".format(response.status_code, response.content)

        return HttpResponse(message)
    else:
        return HttpResponse("Couldn't update the code on PythonAnywhere")

What errors do you see in the error log now?

That's the strange part, I don't see any errors in my error.log. The only log that gets changed after github hook is the server.log, this is the content:

2021-09-22 10:30:21 Fetching origin
2021-09-22 10:30:22 HEAD is now at b3134a6 
2021-09-22 10:30:22 Wed Sep 22 10:30:22 2021 - received message 0 from emperor
2021-09-22 10:30:22 SIGINT/SIGQUIT received...killing workers...
2021-09-22 10:30:23 worker 1 buried after 1 seconds
2021-09-22 10:30:23 goodbye to uWSGI.
2021-09-22 10:30:23 VACUUM: unix socket /var/sockets/filozof595.pythonanywhere.com/socket removed.
2021-09-22 10:30:32 *** Starting uWSGI 2.0.19.1 (64bit) on [Wed Sep 22 08:30:30 2021] ***
2021-09-22 10:30:32 compiled with version: 9.3.0 on 27 May 2021 21:02:35
2021-09-22 10:30:32 os: Linux-5.4.0-1038-aws #40 SMP Thu Mar 4 18:32:33 UTC 2021
2021-09-22 10:30:32 nodename: blue-liveweb10
2021-09-22 10:30:32 machine: x86_64
2021-09-22 10:30:32 clock source: unix
2021-09-22 10:30:32 pcre jit disabled
2021-09-22 10:30:32 detected number of CPU cores: 4
2021-09-22 10:30:32 current working directory: /home/filozof595
2021-09-22 10:30:32 detected binary path: /usr/local/bin/uwsgi
2021-09-22 10:30:32 *** dumping internal routing table ***
2021-09-22 10:30:32 [rule: 0] subject: path_info regexp: \.svgz$ action: addheader:Content-Encoding:gzip
2021-09-22 10:30:32 *** end of the internal routing table ***
2021-09-22 10:30:32 chdir() to /home/filozof595/
2021-09-22 10:30:32 your processes number limit is 128
2021-09-22 10:30:32 your memory page size is 4096 bytes
2021-09-22 10:30:32 detected max file descriptor number: 123456
2021-09-22 10:30:32 building mime-types dictionary from file /etc/mime.types...
2021-09-22 10:30:32 567 entry found
2021-09-22 10:30:32 lock engine: pthread robust mutexes
2021-09-22 10:30:32 thunder lock: disabled (you can enable it with --thunder-lock)
2021-09-22 10:30:32 uwsgi socket 0 bound to UNIX address /var/sockets/filozof595.pythonanywhere.com/socket fd 3
2021-09-22 10:30:32 Python version: 3.9.5 (default, May 27 2021, 19:45:35)  [GCC 9.3.0]
2021-09-22 10:30:32 PEP 405 virtualenv detected: /home/filozof595/.virtualenvs/venv
2021-09-22 10:30:32 Set PythonHome to /home/filozof595/.virtualenvs/venv
2021-09-22 10:30:32 *** Python threads support is disabled. You can enable it with --enable-threads ***
2021-09-22 10:30:32 Python main interpreter initialized at 0x564552f8dc70
2021-09-22 10:30:32 your server socket listen backlog is limited to 100 connections
2021-09-22 10:30:32 your mercy for graceful operations on workers is 60 seconds
2021-09-22 10:30:32 setting request body buffering size to 65536 bytes
2021-09-22 10:30:32 mapped 334256 bytes (326 KB) for 1 cores
2021-09-22 10:30:32 *** Operational MODE: single process ***
2021-09-22 10:30:32 initialized 38 metrics
2021-09-22 10:30:32 WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x564552f8dc70 pid: 1 (default app)
2021-09-22 10:30:32 *** uWSGI is running in multiple interpreter mode ***
2021-09-22 10:30:32 gracefully (RE)spawned uWSGI master process (pid: 1)
2021-09-22 10:30:32 spawned uWSGI worker 1 (pid: 3, cores: 1)
2021-09-22 10:30:32 metrics collector thread started
2021-09-22 10:30:32 spawned 2 offload threads for uWSGI worker 1

It looks like the server has done the reload...?

But every time I try to redeliver i get the same response code: 502

and the PythonAnywhere: something went wrong :-( page (html in response).

That will always happen if you reload your web app from within your web app. If you call reload in a request, then your web app will be reloaded and the pending request will be interrupted by the reload. Don't try to reload your web app with code that is running in your web app.

This makes sense...

I need to find a way to trigger my restart scrypt after git push....right now im thinkig about a git hook...

Thank you for your assistance this far. When i find a way to automate this i will post here.

I am getting a 502 error while trying to pull the code from GitHub. Did anyone encounter this problem?

We need more details to help you. If you do not want to publish it here, send us an email to support@pythonanywhere.com

I received an answer via email to this query. Thank you!

Glad we could help!