Flask+nginx+uwsgi: Fixing Occasional Bad Gateway 502 + / Sqlalchemy SSL Error: decryption failed or bad record mac
This took a non-trivial amount of time to resolve, so I decided it be worth writing it up in the hopes of helping some poor soul (be it my future self or not :P).
I recently deployed a flask + angularjs app to a staging server serving the app from nginx and uwsgi. Upon deployment I noticed that the requests to the app’s restful api from the front-end would occassionally result in 502 errors. Looking in the app’s uwsgi logfile, I noticed errors such as these:
sqlalchemy.exc.DatabaseError: (psycopg2.DatabaseError) SSL error: decryption failed or bad record mac
Since this had never happened previously in my local dev environment, it was a bit puzzling as to why it was happening in staging.
To cut a long story short, the problem was that my app had an initialize_before_first_request routine which executes a bunch of database calls (queries and updates), like so:
@app.before_first_request def init_before_first_request(): ... bunch of db calls initializing some initial content (user roles, etc) ... return
In the dev environment this was never a problem because the dev server is single threaded, however in the staging environment the flask app, being deployed to nginx + uwsgi was now naturally in a multi-worker / threaded environment, and it turns out that the standard uwsgi configuration does not work well at all when this is combined with a non-trivial before_first_request call.
And I can’t complain. The init_before_first_request routine in my case was essentially due to developer laziness. It sets up some initial data in the db, which can and should have been done during the installation process of the app as a one step procedure, and not dynamically before each ‘first’ request, especially not in production. Particularly so because here ‘first’ request refers to the first request received by a given worker processes / thread, and not the first request in the history of the app, so when the worker / thread is recycled, you will go through the init_before_first_request loop again and again. This will impact performance and in our case, lead to 502 errors, rendering the app quite unreliable from the perspective of the end-user.
However, for those who do insist on doing a bunch of db operations inside an @app.before_first_request routine, this stackoverflow answer describes how to do that without ever incurring 502 bad gateway errors. The trick is to add the following two lines at the end of your app’s uwsgi config file (ie: /etc/uwsgi/apps-enabled/app.ini):
[uwsgi] ... standard config lines here ... # the fix lazy = true lazy-apps = true
This rather obscure configuration choice changes something about how the master and child processes interchange db sessions in such a way that the bad gateway 502 error is avoided (see the above referenced stackoverflow answer for more details as to how).
However, there is mention of performance being sub-optimal with this config choice, so it is best to be used only as a last resort, and not as your default go-to config.
Finally, another answer in the same post suggests invoking ‘engine.dispose()’ at the end of the initialization routine, thereby sidestepping the need for a ‘lazy’ config. I have not yet verified that this works, however, it would be interesting to try.
Note: Flask v0.11.1. Sqlalchemy v1.0.14. psycopg2 v2.5.4.