Itamar Syn-Hershko
The Nested field type in Elasticsearch and OpenSearch is not as innocent as one would think. Let’s look at what they are used for, the challenges with them, and possible alternatives.
Elasticsearch is a document-oriented data store. Trying to use relational models like in-use in RDBMS systems like Postgresql and MySQL, with Elasticsearch is prone to failure.
Yet, Elasticsearch allows you to model complex data structures using nested fields, which are useful for preserving relationships within arrays of objects. Unlike standard object fields, nested fields store each object in an array as a separate hidden document, enabling precise querying. However, this approach comes with significant downsides.
Let’s review the common issues and challenges with nested fields, then look at solutions and alternatives given the common use-cases where they are being used.
What are Nested Fields?
Nested fields are designed to allow querying on lists of objects that are internalized on the actual document (or object) which can be considered a parent of that list. For example, a list of key / value pairs that serve as properties for a product on an eCommerce website:
PUT products/_doc/1
{
"title": "Product 1",
"attributes": [{
"attribute": "color",
"value": "blue"
}, {
"attribute": "size",
"value": "m"
}]
}
In Elasticsearch and OpenSearch, this is completely equivalent to the following, because indexes have no notion of structure or sub-documents:
{
"title": "Product 1",
"attributes.attribute": ["color", "size"],
"attributes.value": ["blue", "m"]
}
So, you could query for color:blue and get this product in the result set; but also size:blue would return it, which obviously doesn’t make any sense. To solve this problem, the Nested field type helps by keeping this key-value pairing also in the index level, not without some trade-offs.
Nested Fields Shortcomings
1. Performance Overhead
Since each nested object is treated as an independent document internally, queries on nested fields require expensive join operations, leading to increased query execution time and higher memory consumption.
2. Complex Queries
Using nested fields means you must use nested queries, which are more cumbersome to write and understand. Unlike simple object fields, which allow direct queries, nested structures demand a specialized syntax, adding unnecessary complexity.
3. Indexing Costs
Nested fields can bloat your index size, as each nested object is stored separately. This leads to increased storage requirements and slower indexing performance, especially when dealing with large datasets.
4. Limited Aggregations
Aggregating data within nested fields is more complicated. Since nested documents exist separately, standard aggregations won’t work as expected unless explicitly wrapped in a nested aggregation, further increasing query complexity.
Alternatives to Nested Fields
In too many cases, nested fields are being used where they are not really necessary. You should carefully consider your use-case and whether the query requirements indeed demand the use of nested objects.
If it does, maybe there are alternatives and instead of using nested fields, consider:
- Flattening the data model where possible.
- Denormalization by duplicating data in parent documents.
If nested fields are used solely for key-value pair look ups, consider the following scheme to flatten your schema. Index mapping that is defined like this:
PUT product
{
"mappings": {
"properties": {
"title": {"type": "text"},
"attributes": {"type": "keyword", "doc_values": false}
}
}
}
Allows for text attributes to be defined and used like this:
PUT products/_doc/1
{
"title": "Customer 1",
"attributes": ["color|blue", "size|m"]
}
PUT products/_doc/2
{
"title": "Customer 2",
"attributes": ["color|green", "size|s"]
}
PUT products/_search
{
"query": {
"term": {
"attributes": "color|blue"
}
}
}
Admittedly, this only allows for exact lookups (no sorting or aggregations) and only text values (no numeric range searches, or dates or other specialized data types). But in many cases, this is more than enough and can result in huge performance and cost improvements.
Frequently Asked Questions
What is a nested field in Elasticsearch?
A nested field is a field type that stores each object in an array as a separate hidden document, preserving the pairing between keys and values inside that object. With a standard object field the array is flattened, so a product with color: blue and size: m would also match a query for size: blue.
Why are nested fields slow in Elasticsearch?
Each nested object is an independent document internally, so queries on nested fields require expensive join operations that increase query time and memory consumption. Nested fields also bloat index size and slow indexing because every nested object is stored separately.
Do I need a nested query to search nested fields?
Yes. Nested fields require the specialized nested query syntax, and aggregations on them only work when wrapped in a nested aggregation, which adds complexity compared with querying simple object fields.
How can I avoid nested fields in Elasticsearch?
Flatten the data model where possible or denormalize by duplicating data into parent documents. For key-value lookups, index the pairs as a single keyword field with values like "color|blue" and "size|m" and query them with a term query.
What are the limitations of the flattened key|value alternative?
It only supports exact lookups: no sorting, no aggregations, and only text values, so numeric range queries, dates and other specialized types are not possible. In many cases this is more than enough and results in huge performance and cost improvements.
Do nested fields behave the same in OpenSearch?
Yes. Elasticsearch and OpenSearch flatten arrays of objects identically, so the same key-value mismatch problem, the same nested query requirement and the same flattening alternatives apply to both.
