Tim Bass
Sat, 17 Nov 2007 04:11:25 +0000
In my earlierpost, A Model For Distributed Event*Processing, I promised to address grid computing, distributed object caching and virtualization, and how these technologies relate to complex event processing.Some of my readers might forget my earlier roots in networking if I continue to talk about higher level abstractions! So, inthis follow-up post I willdiscuss virtualizationrelative to database clustering.
In typicalclustered database environments there are quite afew major performance constraints.* These constraints limit our capability to architect and design solutions for distributed, complex, cooperativeevent processingproblems and scenarios.* Socket-based interprocess communications*(IPCs)within database clusterscreate a performancelimitation contrained by low bandwidth, high latency, and processingoverhead.
In addition, the communications performance between the application layer and the database layer can be limited by both TCP and operating system overhead.* To make matter worse, hardware input-output constraints limits scalability for connecting database servers to disk storage.** These are standard distributed computing constraints.
The*physicalarchitecture to address scalability in emerging distributed CEP solutions require alow-latency network communications infrastructure (sometimes called a fabric).* This simple means that event processing agents (EPAs)require virtualization technologies such as Remote Direct Memory Access (RDMA).**CEP agents (often called CEP engines) should*have*thecapability towrite data directly to the memory spaces of a CEP agentfabric (sometimes called an event processing network, EPN).** This is similar tothe concept of shared memory*as*anIPC in UNIX-based systems applied to distributed computing, so all �old hat� UNIXsystems engineers will easily grok these concepts.
RDMA virtualization helps improve performance by bypassing operating-system and TCP overhead resulting in*significantlyhigher bandwidth and lower latency in the EPF (Event Processing Fabric - I justminted a new three letter acronym, sorry!). This, in turn, improves thecommunication speedbetween event processing agents in an event processing network (EPN), or EPF (depending on your taste in acronyms).
Scheduling tasks such as a distributed semaphore checking and lock management canalso operatemore efficiently and with higher performance.Distributed tablesscans, decision tree searches, rule-engine evaluations, Bayesian and neural analytics can all be performed in parallel, dramatically improvingboth performance andscalability of distributed event processing applications.
In addition, by adopting transparent protocols with existing socket APIs, the CEP architect canbypass both operating-system and TCP protocol overhead.In other words, communicationsinfrastructures for CEPthat optimize networking, interprocess communications, and storage, provide architectswith the underlying tools tobuild better solutions to computational complexproblems.
Many of the communicationsconstraints of earlier distributed architectures for solving complex problems, such as blackboard architectures,* can be mitigated with advances in virtualization.* So, in a nutshell,virtualization technologies,*areone ofthe most important underlying capabilities required for distributed, high performance CEP applications, in my opinion.
The article, Virtualization hot; ITIL, IPv6 not,**appears to indicate thatsome of the top IT managers at InteropNew York might agree with me.**
Unfortunately for a few software vendors, virtualization threatens to dilute their market share for ESB and message bus sale. (OBTW,*SOAis DOA.)** �Old hat� UNIX system programmers will recall how the UNIX IPC called �message queues� lost favor to sockets, pipes and shared memory.** A similar trend is happening in the virtualization world withRDMA as a distributedshared memory technology versus message-based communications technologies. I*will opine more on this topic later.