Extending Elastalert and Migrating from Elastic Watcher

Am I a Whatsapp, Telegram, or Signal person? That's what I've been asking myself recently. Of course, questions like this about software products are just as relevant in software architecture. The specific type of product we'll discuss this time is Alerting software.

There are several different products that allow alerting on top of Elasticsearch. The existing alerting feature by Elastic Co., Watcher, as well as its new Kibana Alerting feature, offer most of their capabilities only as part a of paid subscription. An open-source alternative, Elastalert, is a robust and extensible alerting product, allowing various types of rules to determine when to send an alert, as well as a plethora of built-in destination types for the alerts. 

In this post we will go through the process of extending Elastalert by adding specific features to it that exist in Watcher and were previously missing in Elastalert. Besides making a move from Watcher to Elastalert simpler, this will serve as an example of how to extend Elastalert. As a bonus, we'll also provide a way to incorporate exact-time scheduling into Elastalert, without any coding effort. We won't go into a detailed comparison of the offerings, but you can find one here. 

In order to read on you'll need to have some knowledge of Python and an IDE (such as Pycharm). We assume general familiarity with Elastalert, although we may go into a detailed introduction in a future post. 

Extending an Elastalert rule

Adding a relation to the frequency rule: Elastalert has several types of rules for deciding on when to send alerts, such as Spike or Frequency. The Frequency rule works by comparing the amount of events within a time period to a preset number, and alerting when the first is greater than the second. Watcher doesn't have the built-in complex conditions of Elastalert. However, it does have a functionality similar to the Frequency rule, which also allows to freely determine the comparison relation  - "smaller than", "equal to", etc. While Elastalert allows a smaller-than or greater-than-equal comparison with Frequency, it's missing the other comparison types. So why don't we extend the Frequency rule to support those?

This can be done by creating a class for a new rule type. Since this rule would behave almost the same as the Frequency rule, all we need to do is extend the original rule class, and override its "check_for_match" method with a method that supports all of the comparisons. To define the comparison relation in the Elastalert rule, we will use a new property in the rule definition, which we'll call 'frequency_comparison_relation'. We'll also use a helper method to perform the relevant comparison based on the definition. 

Start by creating a folder called elastalert_modules in the path from which elastalert is run. Within that folder, create a file, say bdb_rules.py, and insert the following code into it:

from elastalert.ruletypes import FrequencyRule
class FrequencyComparisonRule(FrequencyRule):
    @staticmethod
    def compareValues(actualValue,comparator,compareValue):
        if comparator == "gte":
            return actualValue >= compareValue
        elif comparator == "eq":
            return actualValue == compareValue
        elif comparator == "gt":
            return actualValue > compareValue
        elif comparator == "lt":
            return actualValue < compareValue
        elif comparator == "lte":
            return actualValue <= compareValue
    def check_for_match(self, key, end=False):
        # Match if, after removing old events, the comparison returns true for num_events.
        # the 'end' parameter depends on whether this was called from the
        # middle or end of an add_data call and is used in subclasses
        if self.compareValues(self.occurrences[key].count(),self.rules['frequency_comparison_relation'], self.rules['num_events']):
            event = self.occurrences[key].data[-1][0]
            if self.attach_related:
                event['related_events'] = [data[0] for data in self.occurrences[key].data[:-1]]
            self.add_match(event)
            self.occurrences.pop(key)

Within the rule itself, we'll have to tell Elastalert that we're using a new rule. In this example the "rule type" property points to path to the rule file, followed by the class name. Other than that and the comparison relation, the rule definition is the same as that of a Frequency rule.

#general properties
is_enabled: true
name: test rule
type: "elastalert_modules.bdb_rules.FrequencyComparisonRule"
index: testindex
timestamp_field: timestamp
timestamp_type: unix_ms
use_strftime_index: false
#rule-specific properties
num_events: 2
frequency_comparison_relation: gte
timeframe:
  minutes: 1
#alert properties
alert:
- slack
slack_webhook_url:

Enhancing an Elastalert alert

Adding priority to the e-mail alert: In addition to adding a new rule condition, you can also add information to the alert content. We'll add two types of information that are covered by Watcher, but not available by default in Elastalert: The first is the e-mail priority. You might actually not be familiar with e-mail priority, as Gmail doesn't offer this feature, but it's still available in Exchange, for instance, and helps to alert the receiver to an a message requiring immediate action.

Elastalert does have a priority property which can be used within the alert text, but it's not being used for the e-mail priority. To add the new property, we'll have to extend the class `EmailAlerter`. 

We'll use the same terms used by Watcher to determine the priority value, but we'll have to override the alert method. It's a pretty long method which we'll have to completely copy and only modify a specific line. The entire code is accessible here, but in the interest of brevity I'll include the relevant helper method and only the relevant code from the overridden method. We'll create the file in the same folder as before, and call it bdb_alerts.py .

import ...
class PriorityEmailAlerter(EmailAlerter):
    def get_alert_priority(self):
        priority = self.rule['email_priority']
        if priority == "lowest":
            return 5
        elif priority == "low":
            return 4
        elif priority == "normal":
            return 3
        elif priority == "high":
            return 2
        elif priority == "highest":
            return 1
        return None
  def alert(self, matches):
  .
  .
    email_msg['Date'] = formatdate()
    priority = self.get_alert_priority()
    if priority != None:
       email_msg['X-Priority'] = priority

Within the rule file, we'll have to include the following properties:

alert: "elastalert_modules.bdb_alerts.PriorityEmailAlerter"
email_priority: "highest"

Where, again, the alert property is composed of the folder with the relevant python file, the name of the file without the extension, and the class name.

Adding the triggered time to the alert text: But that's a static property that we're adding to the rule itself. What if we want to affect the alert content dynamically? The second way we'll enhance the alert is adding the triggered time into the alert text. That ability is available in Watcher, but does not come as a standard in Elastalert. It is, however, extremely easy to implement.

This time we will create a file called bdb_enhancements.py, again in the elastalert_modules folder, with the following content:

from elastalert.enhancements import BaseEnhancement
from datetime import datetime
class TriggeredTimeEnhancement(BaseEnhancement):
    # The enhancement is run against every match
    # The match is passed to the process function where it can be modified in any way
    # ElastAlert will do this for each enhancement linked to a rule
    def process(self, match):
        match['triggered_time'] = datetime.now()

Now, getting this to appear in the alert requires familiarity with the alert text syntax, but isn't much more complex than the relevant watcher syntax. First, we'll have to tell the rule that it's using the enhancement:

match_enhancements:
- elastalert_modules.bdb_enhancements.TriggeredTimeEnhancement

Then, we need to format the alert text to use this property, in the following way:

alert_text_type: alert_text_only
alert_text: "hits are {0}, time is {1}"
alert_text_args: ['num_hits', 'triggered_time']

In this example, the sent alert will show the number of hits and the time. The same can also be done in the subject by using alert_subject and alert_subject_args.

Adding cron-based scheduling to Elastalert

Finally, one feature Watcher has that Elastalert is lacking is a way to schedule alerts to specific times. This is unfortunately not something that's possible by using the enhancement points offered by Elastalert. However at BigData Boutique we've created a fork of Elastalert where we add various additions and extensions, and a cron-based scheduling mechanism is one of them. We also offer a docker image of Elastalert that already contains this fix. In order to use the image, assuming you have docker installed, you can use:

docker pull bigdataboutique/elastalert

Then, within the rule, you need to use the new scheduling mechanism, rather than interval-based scheduling:

cron_schedule: "0 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 * * *"

As a reminder, the cron syntax goes like this: minute (0-59), hour (0-23), day of month (1-31), month (1-12), day of week (MON-SUN) . The fields are separated by space, values separated by comma, * for all values. Also notice this mechanism currently only works with rules using use_count_query.

That's all on Elastalert for this time. If you require advice on choosing an alerting application, or with the installation and use of Elastalert, you're welcome to contact us in those matters. As for the Whatsapp/Telegram/Signal choice, I think I'll go with one that satisfies the requirements of my family members, wouldn't want to miss on any important alerts.