Custom Pagination in python with offset and limit

Offset Pagination

Recently I was stuck with optimising pagination which we use in our REST APIs. Before this, we used to use python’s Rest Framework — PageNumberPagination. This pagination takes page number and limit and then queries whole query set instead of just querying for that particular page. This drastically affects the performance of the API specially when the scale of data is very high.

The other alternative which I considered was — LimitOffsetPagination of Rest Framework. This also worked fine for the use case but it lagged taking page number and limit and then setting the whole context for the function. Thus, I thought of making our own custom pagination which will not only fulfil my purpose but also would be generic enough so that it was be used project wide with other contexts as well. Also, while doing it I had few things in mind- things which we always do while returning a paginated response in an API —

1. Getting data by page number rather than offset.
2. Having freedom of sorting data by any order.
3. Accepting any model object and query.
4. Accepting serializer and serializer context for returning serialized data directly.
5. If serializer is not passed then have an mechanism in place that would serialize the data for the given model object.

Considering all the above points, I could come up with the below Implementation of pagination

class CustomPaginationOffset:
    """
    Custom pagination class which can be used when we get page number
    and ideally don't want to query the whole data, here we get offset
    queryset and count of whole data.

    It expects current page number, model serializer, model in which
    query has to be made, query itself in form of dict and order_by
    which the result would be sorted.

    It returns the count of whole model data and serialized data.
    """

    page_number = 1
    page_size = 20
    order_by = '-created'

    def get_offset(self):
        """It returns limit when it gets page number according to page size."""
        upper_limit = self.page_number * self.page_size
        lower_limit = upper_limit - self.page_size
        return lower_limit, upper_limit

    def get_serialized_data(self, model, queryset, serializer=None, context=None):  # noqa
        """It returns serialized data if serializer is present, else creates a
         serializer from the model given."""
        if serializer:
            serialized_data = serializer(queryset, context=context, many=True).data
        else:

            class Serializer(serializers.ModelSerializer):

                class Meta:
                    fields = '__all__'

            Serializer.Meta.model = model
            serialized_data = Serializer(queryset, many=True).data
        return serialized_data

    def __call__(self, model: object, query: dict, serializer=None, context=None):
        """
        Returns paginated data.
        @param model:
        @param serializer:
        @param query:
        @param context:
        @return: {"results": serialized data, "count": total_objects_count,
                  "total_pages": total_pages, "is_next_page": True/False}
        """
        lower_limit, upper_limit = self.get_offset()

        queryset = model.objects.filter(**query).order_by(
            self.order_by)[lower_limit: upper_limit]
        total_objects_count = model.objects.filter(**query).count()
        serialized_data = self.get_serialized_data(model, queryset, serializer, context)

        total_pages = math.ceil(float(total_objects_count/self.page_size))
        is_next_page = True if total_pages > self.page_number else False

        return {"results": serialized_data, "count": total_objects_count,
                "total_pages": total_pages, "is_next_page": is_next_page}

This returns list of serialized data, total count of objects according to the given query set, total pages and a boolean field — is_next_page.

Usage

query = {'created__date__range': [start_date, end_date]}
context = {'user_profile_uuids': user_profile_uuids}

paginator = CustomPaginationOffset()
paginator.page_number = self.page
# I am not setting paginator.page_size here and using the default
response = paginator(User, query,
                     UserSerializer, context)
# And if we do not have a serializer in place then we can directly do -
# response = paginator(User, query, None, context)

This was a very naive approach to the pagination problem which I was facing. I tried making the code as modular as possible and include few useful and redundant things which might make my overall lines of code come down in my project. I would not be shy in updating this in future if I get suggestions about improvements and adding new functionalities.