3

I am using a Flask app with SQLAlchemy and a uWSGI server. It is a known issue that uWSGI's duplicates connection through all the processes during forking. The information about how to fix this issue is a bit scatted in the web, but the two main options seems to be:

  • Using lazy-apps = true within uWSGI's config (Which is not recommended because it consumes a lot of memory)
  • Using uWSGI's @postfork decorator to close the connection after forking to start new fresh connections for each new process, but it's unclear for me when and how should be used within the Flask app.

This the sample of my app:

# file: my_app/app.py db = SQLAlchemy() def create_app(): app = Flask(__name__) app.config.from_pyfile(f'../config/settings.py') db.init_app(app) db.create_all(app=app) return app

Sample of the run.py file:

# file: run.py from my_app/app import create_app app = create_app() if "__main__" == __name__: app.run(debug=app.config["DEBUG"], port=5000)

So the question is where and how should the postfork executed to properly setup uWSGI's server to use isolated connections on each process without using lazy-apps = true?

Improve this question

2 Answers 2

Reset to default
7

The SQLAlchemy manual provides two examples how to approach this: Using Connection Pools with Multiprocessing.

The first approach involving Engine.dispose() can be approached using uwsgidecorators.postfork as suggested by Hett - simple example that should work if there's only the default binding involved:

db = SQLAlchemy() def create_app(): app = Flask(__name__) app.config["SQLALCHEMY_DATABASE_URI"] = "postgres:///test" db.init_app(app) def _dispose_db_pool(): with app.app_context(): db.engine.dispose() try: from uwsgidecorators import postfork postfork(_dispose_db_pool) except ImportError: # Implement fallback when running outside of uwsgi... raise return app

The second one involves sqlalchemy.event and discarding connections from different PIDs - quoting:

The next approach is to instrument the Pool itself with events so that connections are automatically invalidated in the subprocess. This is a little more magical but probably more foolproof.

If you want the latter magical solution, see the docs.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I have chosen this as the right answer because it clearly explains the available options and how postfork can be directly applied to my provided code. Thanks a lot @Arne Welzel
0

In general you need to use @postfork decorator provided by uwsgi. We are not using SQLAlchemy, but I think you can adapt our solution to your case as well.

Here's an excerpt from our uwsgi configuration:

[uwsgi] strict = true ; process management master = true vacuum = true pidfile = /tmp/uwsgi.pid enable-threads = true cpu-affinity = 1 processes = 4 threads = 2 mules = 2 ...

And here's the application initialization code using @postfork to create process safe DB connections.

from flask import Flask from uwsgidecorators import postfork from app.lib.db import DB # DB object will be replaced by SQLAlchemy on your side app = Flask(__name__) # due to uwsgi's copy-on-write semantics workers need to connect to database # to preserve unique descriptors after fork() is called in master process @postfork def db_reconnect(): # Open the DB connection and log the event app.db = DB(config.get('db', 'host'), config.get('db', 'user'), config.get('db', 'pass'), config.get('db', 'name'))

In our case all initialization code is placed in __init__.py or app folder, but I don't think you'll have any difficulties to migrate it into your app.py file.

Let me know if you'll encounter any other questions on this.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.