Tim Bass
Thu, 20 Dec 2007 00:24:39 +0000
Detection-oriented technologies generally fall into two broad areas, signature-based detection and anomaly-based detection.*** Complex event processing (CEP) is also a detection-oriented technology, sowe can readilyunderstand that CEP applications must also fall within the same two general areas.
Signature-based detection is sometime referred to as static detection because thetechnology relies on pre-defined rules, filters, and signatures to match known patterns. At the most fundamental level, a virus checking program is anexample of a signature-based system.
On the other hand, anomaly-based detection systems striveto maintain a baseline of what is considered normal and then matches patterns outside normal operating parameters, often usings adaptive or artifical intelligence techniques.
Experts know that both anomaly and signature-based detection methodsare important and each have their unique challenges and engineering tradeoffs.* For example, signature-based systems tend to generate false negatives because it is not possible to write all possible rules and filters to match every pattern, especially in dynamic real-time environments. Anomaly-based detection, on the other hand,tends to generate false positives because it is quite difficult to create a perfect profile of normal behavior.
The challenge in most, if not all, detection-oriented systems is finding the right balance between false positives and false negatives.* In some situations, a system should error toward false positives.* In other applications, the system should error toward false negatives.*
CEP is, by defination, a technology to detect both opportunities and threats in distributed networks, in real-time, so it goes without saying that CEP is challenged by the same engineering tradeoffs that affect other detection-oriented systems.
A few weeks ago, I was discussing CEP with a CTO of one of Thailand�s largest telecommunications companies and he was very bullish on neural-based anomaly detection andfrom Esphion.
First generation detection systems rely on determinism, which is generally rule-based, and known to be insufficient for more complex real-time problems. Esphion uses neuralagents to gathering information on network activity and then creates a unifying situational infrastructure to protect against previously unknown threats.** For example, a fast spreading threat, such as theSQL/Slammer worm, will have reached all possible targets faster than any signature can be published or rule can be written, as mentioned in Worm detection - You need to do it yourself.
Since CEP is designed and marketed as a technology that brings real-time advantages to thedetection of both opportunties and threats, we must ask ourselves the question why do all the current CEP software vendors fail to provide non-determistic methods*thatare proven toadapt to a rapidly changing world?**
In Anomaly Detection 101, Esphion does a great job of describing how they do not rely on any pre-specified rules, baselines, models, signatures, or any other apriori knowledge.** They claim, and my highly respectedtelecommunications CTO colleague confirms, that there is absolutely no prior knowledge required and their customers are no longer adversely affected by zero-day anomalies orchanging network conditions.
The technology behindEsphion does is what I call real complex event processing.
