Quick configuration tip for Elasticsearch versions 7.9 and above

Hi, just your friendly neighborhood data engineer chiming in to mention an issue we've seen in several Elasticsearch clusters, and a recommended solution.

The short of it: Unless you're planning to use the Elastic Agent, before upgrading to Elasticsearch 7.9 and above or creating a cluster with those versions, remember to set stack.templates.enabled to false in elasticsearch.yml .

ES versions 7.9 and above (not including OpenSearch) come with index templates in place that create data streams for Elastic's Elastic Agent. Unfortunately those index templates apply for patterns such as logs-*-* and metrics-*-*, which are quite popular for other uses for obvious reasons. The templates apply dynamic rules such as defaulting all dynamic strings to keywords. While we encourage customers to avoid dynamic mapping in general, having an unintended overlapping index definition is definitely not a good idea, and should be regarded as a breaking change. An even worse scenario is one where you don't have existing templates for one of those index patterns, and you expect the default dynamic mapping to take place and find that it didn't (we do recommend customizing your schema and avoiding the default dynamic mapping as well).

Elasticsearch documentation does mention a method to disable the built-in index templates, by setting stack.templates.enabled to false. This setting is dynamic since ES 7.11, meaning you can set it without a restart, and then after deleting the templates they won't reappear . However by the time the node is up before changing the setting, the data streams may have already been created, which means you may be writing data to data streams with irrelevant schema definitions, and also can't delete the templates without taking care of the data streams first.

Therefore, unless you specifically have a use case for the built in templates and data streams, it is recommended to set stack.templates.enabled to false in elasticsearch.yml (or via the environment for the container if you're using docker, for instance) prior to starting a node with version 7.9 and above. You can always set it back if you ever have use for those templates (and specifically in 7.11 and above, without restarting nodes).

bootstrap.memory_lock: true
node.name: ${HOSTNAME}
stack.templates.enabled: false
action.destructive_requires_name: true
indices.fielddata.cache.size: 1% # default is unbounded
cluster.name: ...

At BigData Boutique we routinely provide support for many customers with Elasticsearch installations, sharing best practices, solving technical issues and optimizing cluster performance. Contact us for help with optimizing and stabilizing your Elasticsearch cluster(s).