1

I have a Flask app running on Heroku with uwsgi server in which each user connects to his own database. I have implemented the solution reported here for a very similar situation. In particular, I have implemented the connection registry as follows:

class DBSessionRegistry(): _registry = {} def get(self, URI, **kwargs): if URI not in self._registry: current_app.logger.info(f'INFO - CREATING A NEW CONNECTION') try: engine = create_engine(URI, echo=False, pool_size=5, max_overflow=5) session_factory = sessionmaker(bind=engine) Session = scoped_session(session_factory) a_session = Session() self._registry[URI] = a_session except ArgumentError: raise Exception('Error') current_app.logger.info(f'SESSION ID: {id(self._registry[URI])}') current_app.logger.info(f'REGISTRY ID: {id(self._registry)}') current_app.logger.info(f'REGISTRY SIZE: {len(self._registry.keys())}') current_app.logger.info(f'APP ID: {id(current_app)}') return self._registry[URI]

In my create_app() I assign a registry to the app:

app.DBregistry = DBSessionRegistry()

and whenever I need to talk to the DB I call:

current_app.DBregistry.get(URI)

where the URI is dependent on the user. This works nicely if I use uwsgi with one single process. With more processes, 

[uwsgi] processes = 4 threads = 1

sometimes it gets stuck on some requests, returning a 503 error code. I have found that the problem appears when the requests are handled by different processes in uwsgi. This is an excerpt of the log, which I commented to illustrate the issue:

# ... EVERYTHING OK UP TO HERE. # ALL PREVIOUS REQUESTS HANDLED BY PROCESS pid = 12 INFO in utils: SESSION ID: 139860361716304 INFO in utils: REGISTRY ID: 139860484608480 INFO in utils: REGISTRY SIZE: 1 INFO in utils: APP ID: 139860526857584 # NOTE THE pid IN THE NEXT LINE... [pid: 12|app: 0|req: 1/1] POST /manager/_save_task => generated 154 bytes in 3457 msecs (HTTP/1.1 200) 4 headers in 601 bytes (1 switches on core 0) # PREVIOUS REQUEST WAS MANAGED BY PROCESS pid = 12 # THE NEXT REQUEST IS FROM THE SAME USER AND TO THE SAME URL. # SO THERE IS NO NEED FOR CREATING A NEW CONNECTION, BUT INSTEAD... INFO - CREATING A NEW CONNECTION # TO THIS POINT, I DON'T UNDERSTAND WHY IT CREATED A NEW CONNECTION. # THE SESSION ID CHANGES, AS IT IS A NEW SESSION INFO in utils: SESSION ID: 139860363793168 # <<--- CHANGED INFO in utils: REGISTRY ID: 139860484608480 INFO in utils: REGISTRY SIZE: 1 # THE APP AND THE REGISTRY ARE UNIQUE INFO in utils: APP ID: 139860526857584 # uwsgi GIVES UP... *** HARAKIRI ON WORKER 4 (pid: 11, try: 1) *** # THE FAILED REQUEST WAS MANAGED BY PROCESS pid = 11 # I ASSUME THIS IS WHY IT CREATED A NEW CONNECTION HARAKIRI: -- syscall> 7 0x7fff4290c6d8 0x1 0xffffffff 0x4000 0x0 0x0 0x7fff4290c6b8 0x7f33d6e3cbc4 HARAKIRI: -- wchan> poll_schedule_timeout HARAKIRI !!! worker 4 status !!! HARAKIRI [core 0] - POST /manager/_save_task since 1587660997 HARAKIRI !!! end of worker 4 status !!! heroku[router]: at=error code=H13 desc="Connection closed without response" method=POST path="/manager/_save_task" DAMN ! worker 4 (pid: 11) died, killed by signal 9 :( trying respawn ... Respawned uWSGI worker 4 (new pid: 14) # FROM HERE ON, NOTHINGS WORKS ANYMORE

This behavior is consistent over several attempts: when the pid changes, the request fails. Even with a pool_size = 1 in the create_engine function the issue persists. No issue instead is uwsgi is used with one process.

I am pretty sure it is my fault, there is something I don't know or I don't understand about how uwsgi and/or sqlalchemy work. Could you please help me?

Thanks

Improve this question
2
  • 1
    Hey, it seems that you need to share memory between workers. Check this question, it does not have a solution but it does explain ther reasons stackoverflow.com/questions/54133550/…. Commented Apr 23, 2020 at 22:46
  • @univerio I have been fooled by the unique id of the registry. Commented Apr 24, 2020 at 6:42

1 Answer 1

Reset to default
2

What is hapeening is that you are trying to share memory between processes. There are some exaplanations in these posts.

(is it possible to share memory between uwsgi processes running flask app?).

(https://stackoverflow.com/a/45383617/11542053)

You can use an extra layer to store your sessions outsite of the app.

For that, you can use uWsgi's SharedArea(https://uwsgi-docs.readthedocs.io/en/latest/SharedArea.html) which is very low level or you can user other approaches like uWsgi's caching(https://uwsgi-docs.readthedocs.io/en/latest/Caching.html)

hope it helps.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

thanks, I have tried to use the lazy-apps, so each process starts with "its own app" (I see in the log message that the app starts for each process and I have different current_app ids). However, the problem persists, but why? Aren't the apps independent in this case?
It happens because your apps is independent. If you make a request for the first time and it is handled by worker1, it will be created a session on worker1's app, if your second request is handled by worker2, that does not have acess to worker's1 data, you'll be creating another session, this time in worker2.
I understand the conflict with multiple processes and ONE single app (as in my initial case), but with lazy_apps we have different processes --> different apps --> different registry --> different sessions so here it should not be a problem, as there is no session sharing, and the data is on the DB.
Are you closing your sessions? i often have problems with this. It seems like a deadlock is hapeening under the hood.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.