Elastic Cloud and Amazon OpenSearch Service (fka Amazon Elasticsearch Service) by AWS are managed solutions for search. Both are based on the same Open Source technology. Which one is the right one for your business?

In the market for a managed service for search and analytics? These two contenders well deserve your attention. Many have found themselves stuck at these crossroads, unable to find which has the edge over the other. In this post, we’ll explain some of the pros and cons to each managed solution. By the end, hopefully you’ll gain enough insight to choose your ideal search solution.

Introduction

While you can deploy Elasticsearch on AWS, many of our customers prefer to use a managed service.

OpenSearch is an open-source software licensed under the Apache License. OpenSearch is a fork of Elasticsearch version 7.10.2, the last version of Elasticsearch available as open-source software under the Apache 2.0 software license. OpenSearch Dashboards is the GUI for OpenSearch which was forked from Kibana version 7.10.2 as well. OpenSearch is available as a managed service by AWS (on AWS of course), and on other clouds through other vendors. Previously was called "OpenDistro for Elasticsearch".

Elasticsearch was formerly an open-source product built on top of Apache Lucene, but has since switched to using their own Elastic license and the SSPL. The base code (excluding enterprise features) is still downloadable and usable by your everyday user but there are a lot more restrictions in place to prevent usage by SaaS providers. Thus, the SaaS offering by Elastic is the only managed service nowadays offering managed Elasticsearch and Kibana, and the rest of the solutions in the Elastic Stack.

Elastic Cloud is the fully managed Elasticsearch and Kibana on their latest versions, and the entire Elastic Stack, hosted directly by Elastic themselves. It offers a multi-cloud, multi-region solution, giving you your choice between AWS, Azure or GCP. The managed Elastic Stack gives you access to a wide breadth of software solutions, from Machine Learning to Kibana Maps, all powered by Elasticsearch and managed in a single interface via Kibana (REST API management is available as well).

Both technologies are widely used, for anything from log analytics to full-text search engine, including various machine-learning, anomaly detection and natural-language processing methods. They are still pretty similar, although we anticipate to start seeing differences as time goes by and the OpenSearch Project roadmap diversifies from that of the original ELK Stack's.

Features

Let’s dive right into a comparison between both providers:

    *Though not a true auto scaling feature, OpenSearch does offer auto-tune for memory related suggestions for your clusters.

    **Support covers mostly AWS infrastructure, but the depth of support depends on the account's AWS Support level.

    ***CloudWatch monitoring only. For a more in-depth, detailed monitoring solution, see our Pulse solution

Can you notice a pattern in the table above? Elastic Cloud clearly offers a lot of additional software solutions and proprietary features. There is a lot more bundled together in one seamless package (note that some features are only available to the higher end licenses). OpenSearch, while still fully-featured as a search solution, does not bundle nearly as many additional features. In some cases this is a good thing. Why pay for these additional features when all you need is basic search engine functionality? OpenSearch tends to be a lot more cost-effective because of this less-is-more approach. It can be much more suitable for a logs/metrics use-case while Elastic Cloud can be a better platform for more elaborate, advanced use-cases.

That being said, all of the additional features provided by Elastic Cloud can be done with an OpenSearch instance (aside from things like multi-cloud support), it just requires the use of plugins and/or additional applications outside of the OpenSearch realm. The beauty of Elastic Cloud is, all of this functionality is bundled together in one simple to use solution. Additionally there are quite a few of missing/disabled levers and knobs in Amazon OpenSearch, which often can be tuned by opening a ticket to AWS Support - but there's no direct control over them. A lot of the system logic is concealed from the end-user, thus you do not have as much control. The managed Elasticsearch service by Elastic hides/blocks much less and allows much more flexibility to change advanced configuration (though future versions of Elasticsearch may hide some of this functionality as well).

Cross-cluster replication is a feature supported by both Elasticsearch and OpenSearch - but we highly recommend avoiding it where possible.

Data Tiering

Elastic Cloud supports several templates for data tiering - hence supporting hot / warm / cold architectures. For large workloads, this is extremely important since data that is less frequently queried (e.g. older data) can be kept on cheaper or more densely populated nodes, thus reducing costs significantly.

The Amazon Elasticsearch Service does not support this - you can only use one type of nodes and storage (SSD vs EBS) for your entire cluster.

However, a "frozen" tier exists on both solutions, and works pretty much in the same way. Elastic Cloud offers a "frozen tier" where search is executed on index snapshots (aka backups) stored on S3. Amazon OpenSearch has the same offering but there it's called "UltraWarm". In both cases, searching this very cold data storage will not be as fast, and it's mostly suitable for older time-series data that is never updated after ingestion.

Automated snapshots

Both solutions offer automated frequent and infrequent snapshots (aka backups), to an S3-backed snapshot repository. The Amazon managed service will require you to create a dedicated repository for manually triggering more frequent snapshots.

Upgrades

Both services offer blue/green deployment schemes, where upgrading a cluster spins up a new cluster and migrates all your data into it. It's a little bit more annoying to keep up with future versions when on Amazon Elasticsearch Service since the migration experience is sometimes rough on the edges. Therefore, upgrading to new versions is a bit more of a hassle with AWS's managed service.

Integration

One place where OpenSearch shines is its interoperability with AWS cloud services such as Kinesis, Firehose, and many more. A lot of these features are simple plug and play integrations. Additionally its auto-tune feature allows you to optimize memory usage by auto adjusting caches. If your organization already uses AWS as its cloud provider of choice, you may want to take this ease of use into consideration.

Machine Learning

Both managed services offer Machine Learning capabilities, ranging from anomaly detection and vector search to natural language processing capabilities. Both OpenSearch and Elasticsearch diverge here a little bit already, so it really boils down to what exactly you need. Elasticsearch and Kibana have very impressive support for anomaly detection with very easy to use UI; while the Open-Source OpenSearch is better with vector-based search as it provides several different computation engines for them.

Documentation

Elastic has a wealth of documentation, segmented into easily digestible sections. The documentation is versioned, so you can easily switch to your version of Elasticsearch to see documentation that is specific to that version. It is very simple to find documentation on the available APIs which include excellent examples for easier readability. There are release notes that include known issues and breaking changes, troubleshooting guides, migration guides to migrate to the next major version of Elasticsearch, etc. To top it all off, it is all powered by an excellent search experience that makes it simple to find what you are looking for. Lastly, there is a great integration with Kibana. You can begin typing out a search query in Dev Tools and then select the option to view documentation. Kibana will open a link to the Elasticsearch documentation that corresponds to the query you are writing, making the entire experience seamless.

OpenSearch on the other hand does not have such a rich and vast set of documentation available. It is not nearly as easy to find the information you are looking for, and there are not as many examples that help provide additional clarity. The good news is, you can refer back to the Elasticsearch documentation for a lot of OpenSearch APIs and queries. A lot of functionality works the same in both applications. Though this is true for now, eventually the two products will differentiate themselves from one another more and more. At that point, Elasticsearch documentation will no longer be a usable reference for OpenSearch.

Support

Elastic Cloud has varying levels of support included in their licensing. Different licenses include different SLAs for support. They range from web-based support only with a 3 business day response time with the Standard license up to web and phone support with a 1 hour SLA for the Platinum and Enterprise licenses. Elastic support is a great service that is very easy to use. There is a support portal where you can send messages back and forth between the support engineers. If needed, they will set up a call or run a screen share session in order to get the issue resolved. The only real draw-back for this support service is its price.

The support packages are bundled into the different license levels, so in order to get the highest level of support, you have to have a Platinum or Enterprise Elastic license. The price differences between licensing tiers on the Elastic Cloud managed service is not insignificant. The cost of a cluster under the Enterprise license can be nearly double that of the same hardware under the Standard license. Granted, the Enterprise license has a lot more software and functionality bundled together than the Standard license, but if these additional features are not needed by your enterprise then you are paying a significant premium for support without getting the benefits of the additional features.

Amazon OpenSearch Service includes basic AWS support with its offering. You can create support tickets for issues relating to your OpenSearch infrastructure only. Questions on best practices, OpenSearch queries, Machine Learning, OpenSearch Dashboards, etc are not covered. If you require any additional support, you are forced to turn to 3rd party consulting services.

Pricing

In most circumstances, when you need a managed solution Amazon OpenSearch Service will cost you less than Elastic Cloud. This makes sense since Elastic does not host using their own cloud services like Amazon does. With AWS, Amazon essentially cuts out the middleman and can therefore offer better pricing on the Cloud infrastructure. This is even more apparent if you decide to utilize Amazon’s reserved instance pricing and make a commitment for several years.

One exception to this comes with use cases that have a significant amount of storage requirements (e.g. hundreds of terabytes of data) where the majority of the data does not need to be queried at fast speeds. The searchable snapshots available within Elastic Cloud allows for frozen tier storage which is significantly cheaper than your traditional hot nodes. It is a trade-off between cost and query speeds. OpenSearch provides for cold storage options as well, but it begins to become prohibitively expensive the more storage is needed.

Conclusion

At the end of the day, the choice between Amazon OpenSearch Service (fka Amazon Elasticsearch Service) and Elastic Cloud will come down to your organization's business requirements and price considerations. Since both are managed services, there is no meaning for the Elastic License vs Open Source Apache License discussion.

If you are looking for a managed high performance search service at a lower cost, and you don’t need any of the additional features available within the Elastic Stack – Choose Amazon OpenSearch Service. On the other hand, if you do require some of the additional features available within the Elastic Stack and don’t mind the added premium – Choose Elastic Cloud. Both services provide an excellent search experience, so either choice is a winner!