Forums

SSLError 'Stale file handle'

My Django site stopped working suddenly today and I am seeing my error logs fill with SSLErrors (see below). I bounced the site, but that didn't resolve it. Any suggestions on what to try?

2023-08-01 11:35:09,479: Error running WSGI application
2023-08-01 11:35:10,466: Error running WSGI application
2023-08-01 11:35:11,421: Error running WSGI application
2023-08-01 11:35:13,523: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:15,255: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:15,273: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:15,374: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:15,378: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:15,641: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:15,690: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:15,690: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:16,869: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:16,900: OSError: [Errno 116] Stale file handle: '/var/www'
2023-08-01 11:35:16,908: OSError: [Errno 116] Stale file handle: '/var/www'
2023-08-01 11:35:17,037: OSError: [Errno 116] Stale file handle: '/var/www'
2023-08-01 11:35:17,109:   File "/var/www/api_bhmc_org_wsgi.py", line 14, in <module>
2023-08-01 11:35:17,111:   File "/var/www/api_bhmc_org_wsgi.py", line 14, in <module>
2023-08-01 11:35:17,231:   File "/var/www/api_bhmc_org_wsgi.py", line 14, in <module>
2023-08-01 11:35:17,659: 
2023-08-01 11:35:18,047: 
2023-08-01 11:35:18,049: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:18,058: 
2023-08-01 11:35:18,242:   File "/home/mpgagolf/.virtualenvs/bhmc/lib/python3.8/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
2023-08-01 11:35:18,349: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:18,458: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:18,512: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:19,166: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:19,319:   File "/home/mpgagolf/.virtualenvs/bhmc/lib/python3.8/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
2023-08-01 11:35:19,319: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:19,320: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:19,331: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:19,651: 
2023-08-01 11:35:19,789:   File "/home/mpgagolf/.virtualenvs/bhmc/lib/python3.8/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
2023-08-01 11:35:20,323: 
2023-08-01 11:35:20,521: 
2023-08-01 11:35:20,545: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:20,553: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(OSError(116, 'Stale file handle'))': /api/5577280/store/
2023-08-01 11:35:20,583:   File "/home/mpgagolf/.virtualenvs/bhmc/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
2023-08-01 11:35:21,182:   File "/home/mpgagolf/.virtualenvs/bhmc/lib/python3.8/site-packages/django/__init__.py", line 24, in setup

I'm getting this same issue.

We had issues with one of our storage servers yesterday, and both of your accounts use the affected one. Sorry for the inconvenience!

It doesn't appear that this is completely resolved:

2023-08-02 12:57:27,949: Error running WSGI application 2023-08-02 12:57:29,130: OSError: [Errno 116] Stale file handle: '/home/mpgagolf/bhmc-api/templates/500.html'

And now this (site is completely down):

Code:
This directory does not exist.

Virtualenv:
Warning: No virtualenv detected at this path. Do you need to create it?

Hi there, we had a pretty severe outtage yesterday evening that affected a proportion of our users. I'm sorry that your sites were affected. Our team was working nonstop throughout the night and today to get the problem sorted