In some cases a deep paginated search on an Elasticsearch or OpenSearch cluster is required. In this power tip we compare the various available options.

OpenSearch & Elasticsearch Scroll and Deep Paging Methods Compared

Deep Paging in Elasticsearch and OpenSearch refers to retrieving a large number of search results beyond the default page of 10 and often also beyond the default limit of 10,000 documents. Since Elasticsearch is designed for speed and efficiency, fetching deep results can be resource-intensive and impact cluster performance. For when this is needed still, Elasticsearch provides several mechanisms to handle deep paging efficiently.

While Elasticsearch was designed to respond quickly to aggregation queries or top few documents, some scenarios are valid for retrieving thousands or more of documents, such as:

  • Analytics and Reporting: Generating reports that require processing large (potentially filtered) datasets.
  • Full Data Export: Extracting large sets of data for external storage or processing.
  • Data Synchronization: Syncing search indices with external databases.
  • Pagination for Large Applications: Serving large-scale applications with extensive pagination needs.

Following are the various options of performing deep-paging operations, with

Using from and size

Elasticsearch provides the from and size parameters for paginated searches. It can be used to retrieve even thousands of documents, up to a defined (configurable) limit, but it has performance costs associated with it.

The syntax is as follows:

POST my_index/_search
  {
    "from": 1000,
    "size": 50,
    "query": {
      "match_all": {}
    }
  }

Pros:

  • Simple to implement.
  • Suitable for small-scale pagination.

Cons:

  • Performance Issues: Using this pagination method, Elasticsearch must compute and discard skipped records as search starts from the very first position, including scoring, sorting and so on.
  • Memory Intensive: Large offsets increase memory usage and slow queries.
  • Unstable Pages: If not sorting by so-called "stable" field, meaning a field that is not going to receive new out of sort-order values during the pagination operation, it is possible to receive the same element twice during pagination or to completely skip some items. This is in particular applicable when no sort is applied and results sorting is done based on the result score.

Using Scroll API

The Scroll API is designed for retrieving large numbers of documents efficiently. It maintains a snapshot of results for repeated fetching, hence a fully stable paging and cheaper operation.

To initiate a scroll, with a 1 minute timeout (1m):

GET my_index/_search?scroll=1m
  {
    "size": 1000,
    "query": {
      "match_all": {}
    }
  }

Retrieve the next batch:

GET _search/scroll
  {
    "scroll": "1m", 
    "scroll_id": "DXF1ZXJ5QW5..."
  }

Pros:

  • Efficient for bulk data retrieval.
  • Avoids re-executing the search query.

Cons:

  • Requires maintaining an open scroll session that has performance and memory applications.
  • There is a timeout defined, that when expires could throw an "expired scroll ID" error.
  • Not suitable for real-time pagination due to resources usage.

Using Point in Time (PIT) API

The PIT API improves upon the Scroll API by creating a lightweight search context.

POST my_index/_search
  {
    "size": 1000,
    "query": {
      "match_all": {}
    },
    "pit": {
      "id": "<pit_id>",
      "keep_alive": "1m"
    }
  }

Pros:

  • More efficient than the Scroll API.
  • Suitable for paginated requests without keeping state.

Cons:

  • Requires explicitly managing PIT lifetimes.
  • Still holds resources on the cluster.

Using search_after

search_after provides a stateless approach to deep paging using a sort key.

GET my_index/_search
  {
    "size": 100,
    "query": {
      "match_all": {}
    },
    "sort": [{ "timestamp": "asc" }],
    "search_after": ["2024-01-01T00:00:00"]
  }

Pros:

  • Avoids performance issues of from and size.
  • Suitable for continuous pagination.
  • Provide "stable sorting" while still avoiding heavy resource usage.

Cons:

  • Requires consistent sorting.
  • No direct random access to pages.

Comparison of Deep Paging Methods

Method Performance Use Case Statefulness
from & size Poor Small pagination Stateless
Scroll API Moderate Bulk export Stateful
PIT API Good Large-scale pagination Semi-stateful
search_after Best Real-time paging Stateless

Frequently Asked Questions

What is deep paging in Elasticsearch and OpenSearch?

Deep paging is retrieving a large number of search results beyond the default page of 10, and often beyond the default limit of 10,000 documents. It is resource-intensive and can impact cluster performance, so Elasticsearch and OpenSearch provide dedicated mechanisms for it.

Why is from and size bad for deep pagination?

With from and size, the search starts from the first position and must compute, score, sort and then discard every skipped record, so large offsets are slow and memory intensive. Pages are also unstable unless you sort on a stable field, so items can appear twice or be skipped.

What is the difference between the Scroll API and search_after?

The Scroll API keeps a snapshot of the results in an open, stateful session, which makes it efficient for bulk export but costs memory, can expire with an "expired scroll ID" error, and is unsuitable for real-time pagination. search_after is stateless and pages using a sort key, giving stable sorting with the best performance for real-time paging, but it requires consistent sorting and offers no random access to pages.

What is the Point in Time (PIT) API?

The PIT API creates a lightweight search context that improves on the Scroll API, making it more efficient and suitable for large-scale paginated requests. It still holds resources on the cluster and requires you to manage the PIT lifetime through keep_alive.

Which method should I use to export all documents from an index?

Use the Scroll API for bulk exports, search_after for real-time pagination, and the PIT API as a hybrid that balances efficiency and usability. Avoid from and size for deep pagination whenever possible.

Do these deep paging methods work in OpenSearch as well as Elasticsearch?

Yes. from and size, the Scroll API, the Point in Time API and search_after are all available in both Elasticsearch and OpenSearch, and the same trade-offs apply to both.

Conclusion

Deep paging in Elasticsearch requires choosing the right method based on your use case. For bulk exports, the Scroll API is suitable. For real-time pagination, search_after is the best choice. The PIT API is a hybrid approach that balances efficiency and usability. Avoid using from and size for deep pagination whenever possible to maintain cluster performance.

Struggling with pagination performance or query efficiency in your cluster? Our OpenSearch consulting team can audit your setup and help you choose the right approach for your workload.