How to paginate an aggregation pipeline result in pymongo?

Question

I have a web app where I store some data in Mongo, and I need to return a paginated response from a find or an aggregation pipeline. I use Django Rest Framework and its pagination, which in the end just slices the Cursor object. This works seamlessly for Cursors, but aggregation returns a CommandCursor, which does not implement __getitem__().

cursor = collection.find({}) cursor[10:20] # works, no problem command_cursor = collection.aggregate([{'$match': {}}]) command_cursor[10:20] # throws not subscriptable error

What is the reason behind this? Does anybody have an implementation for CommandCursor.__getitem__()? Is it feasible at all?

I would like to find a way to not fetch all the values when I need just a page. Converting to a list and then slicing it is not feasible for large (100k+ docs) pipeline results. There is a workaround with based on this answer, but this only works for the first few pages, and the performance drops rapidly for pages at the end.

Tom Slabbaert · Accepted Answer · 2020-07-06 14:10:03Z

Mongo has certain aggregation pipeline stages to deal with this, like $skip and $limit that you can use like so:

aggregation_results = list(collection.aggregate([{'$match': {}}, {'$skip': 10}, {'$limit': 10}]))

Specifically as you noticed Pymongo's command_cursor does not have implementation for __getitem__ hence the regular iterator syntax does not work as expected. I would personally recommend not to tamper with their code unless you're interested in becoming a contributer to their package.

hhharsha36 · Accepted Answer · 2020-07-06 14:16:07Z

The MongoDB cursor for find and aggregate functions in a different way since cursor result from aggregation query is a result of precessed data (in most cases) which is not the case for find-cursors as they are static and hence documents can be skipped and limitted to your will.

You can add the paginator limits as $skip and $limit stages in the aggregation pipeline.

For Example:

command_cursor = collection.aggregate([ { "$match": { # Match Conditions } }, { "$skip": 10 # No. of documents to skip (Should be `0` for Page - 1) }, { "$limit": 10 # No. of documents to be displayed on your webpage } ])

Collectives™ on Stack Overflow

How to paginate an aggregation pipeline result in pymongo?

2 Answers 2

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Linked

Related