Alert Severity Context [2024-02-22]

Earlier, I was having a discussion with some friends about what feeds into alert severity. Given our background in cybersecurity, we've seen our fair share of security alerts, but also a fair amount outside of the security domain (think IT, SRE, DEVOPS, Compliance, Business risk, etc.). So, what goes into the severity of alerts typically?

What are Alert Severities?

Let's start by defining what a severity is for those who may not know. A severity is typically a rating:

• Critical, High, Medium, Low
• P0, P1, P2, P3, P4

given to an event to help the end user understand the importance of said event or, in this case, an alert. While this can be helpful, over time, it can lead to a few issues:

• Overwhelming number of alerts
• Need to prioritize/decide which alert to resolve first with the same priority
• Not enough context
• Too much context

For the most part, these events, or alerts rather, don’t come tuned for your situation. So, you are getting what the upstream providers consider should be important for you, leaving it up to you to figure out the rest.

Tuning

Some providers, of course, allow you to tune these alerts for many reasons, such as false positives, false negatives, and so forth. Unfortunately, this takes some time to curate to make sure you are able to reduce the noise or volume of these alerts.

Tuning example non-security related

A simple, non-security way of thinking about this as an example: I have two Great Pyrenees, which are used as livestock guardian dogs (LGD) for protecting our chickens and goats. They both have their own personalities; one loves to roam while the other likes to hang around the property. Despite their differences, they both are fiercely protective of the property and family, actively chasing away threats.

This behavior, however, presents a challenge. If one roams too far, they might not find their way back. To address this, I've spent two years training them in what I call an LGD partnership. They've learned to understand the commands I give, but distance can sometimes put them out of sight or sound.

To mitigate this risk, I invested in dog trackers. These devices are a solid combination of software and hardware, though they still have room for improvement. With these trackers, I can monitor the whereabouts of my dogs at any given time.

This brings me to the crucial point of tuning:

• Tracking Anywhere: While I can track them anywhere, the real challenge is managing when they wander off too far.
• Creating a Virtual Perimeter: I needed to set up a virtual boundary that would alert me if they went beyond the property lines.

By creating this virtual perimeter, I now have the context needed to understand the priority of my actions (High), along with clear direction. This analogy perfectly mirrors the need for tuning in cybersecurity or any alert-generating tooling. Without setting your parameters or "virtual perimeter," the flood of alerts can be overwhelming and, ultimately, less effective.

The same applies to cybersecurity or any other tooling that generates alerts. You must tune or apply similar controls to help aid in understanding the context and severity of the alert. Otherwise, you will have an unpleasant time in many ways, as stated above, ultimately leading to distasteful trust in tooling, products, and companies.

So Why Am I Talking About This...

In order to effectively respond to an alert, you must have the right context. The context leads to higher repeatability with potentially a better outcome to be proactive.

So, what I am really posing to anyone who made it this far in the post,

What context do you need in order to understand a severity?

Drop me a line or share your feedback to see how common or uncommon others' answers may be.