“Streaming processing” is the ideal platform to process data streams or sensor data (usually a high ratio of event throughput versus numbers of queries), whereas “complex event processing” (CEP) utilizes event-by-event processing and aggregation (e.g. on potentially out-of-order events from a variety of sources – often with large numbers of rules or business logic). CEP engines are optimized to process discreet “business events” for example, to compare out-of-order or out-of-stream events, applying decisions and reactions to event patterns, and so on. For this reason multiple types of event processing have evolved, described as queries, rules and procedural approaches (to event pattern detection). The focus of this article is on stream processing.

A stream processing solution has to solve different challenges:

  • Processing massive amounts of streaming events (filter, aggregate, rule, automate, predict, act, monitor, alert)

  • Real-time responsiveness to changing market conditions

  • Performance and scalability as data volumes increase in size and complexity

  • Rapid integration with existing infrastructure and data sources: Input (e.g. market data, user inputs, files, history data from a DWH) and output (e.g. trades, email alerts, dashboards, automated reactions)

  • Fast time-to-market for application development and deployment due to quickly changing landscape and requirements

  • Developer productivity throughout all stages of the application development lifecycle by offering good tool support and agile development

  • Analytics: Live data discovery and monitoring, continuous query processing, automated alerts and reactions

  • Community (component / connector exchange, education / discussion, training / certification)

  • End-user ad-hoc continuous query access

  • Alerting

  • Push-based visualization

Stream processing means processing data record by record as they arrive and incrementally updating all results with each and every new data record. Therefore each updated result is available is available in real-time, typically with a latency of a few milliseconds or less. Stream processing queries run continuously, never ending, processing data as they arrive, usually over time or record-based windows.

Use Cases

  • Network monitoring

  • Intelligence and surveillance

  • Risk management

  • E-commerce

  • Fraud detection

  • Smart order routing

  • Transaction cost analysis

  • Pricing and analytics

  • Market data management

  • Algorithmic trading

  • Data warehouse augmentation