Real-time business intelligence
Real-time business intelligence (RTBI) is an approach to business intelligence that reduces the delay between events and the information used in decisions.[1][2] This is different from conventional business intelligence (BI), where reporting is usually from historical data. By making data available closer to the time that the activity occurs, RTBI can support monitoring and operational response.[2][3]
RTBI systems use refreshed analytical stores, event streams, event logs, complex event processing, business activity monitoring, or related architectures.[4][5][6] Their output may appear through dashboards, alerts, process monitoring tools, analytical views, or responses.[2][5]
“Real time" depends on the context. Some systems process events within seconds, while others are better described as near-real-time. RTBI therefore involves trade-offs[1] between speed, data freshness, cost, complexity,[7] and the reliability of the information available at the time of analysis.[4]
Latency and data freshness
[edit]Traditional BI has often focused on improving the reporting of historical data.[3] RTBI, however, places greater emphasis on reducing delays from data capture through to the response.[2][8]
All real-time business intelligence systems have some latency, but their aim is to reduce the delay between the use of information from when the event happened. Analyst Richard Hackathorn describes three types of latency:[8]
- Data latency: the time taken to collect and store the data;
- Analysis latency: the time taken to analyse the data and turn it into actionable information;
- Action latency: the time taken to react to the information and take action.
Recent work on real-time analytics describes both speed and freshness as important features of real-time analytical systems. Data freshness is not the same as latency.[1] A system could quickly process a query, but with data that is already out of date. Conversely, operational data collected soon after an event, could still require further processing before it can support a decision.
What "real time" means depends on the context. Some systems process events within seconds, while others are described as near-real-time, because data is made available as soon as possible to support a timely decision. Hackathorn[8] notes "right-time" as an alternative to "real-time", but argues the more important issue is whether the timing of information created will create business value.
Architectures
[edit]RTBI architectures vary according to how data is collected, processed, stored, and presented. Conventional business intelligence systems include operational sources, data integration processes, analytical storage, query processing, and tools for presenting it.[3] In RTBI, those functions remain important, but the architecture is designed to reduce the delay between a business event and the use of information from it.[2]
Event processing and streams
[edit]In event-driven architectures, changes in state are represented as events and processed as they happen. Event stream processing and complex event processing systems process information continuously from multiple sources. These systems may filter, aggregate, correlate, or detect patterns in event streams before all data has been loaded into a conventional database.[6]
For RTBI, event processing supports dashboards and alerts, when current business conditions meet defined rules. Streaming systems also need to handle data that does not arrive in sequence. Records may also arrive late and some cases incomplete. Techniques such as continuous queries and time windows are used to calculate results over set periods of an otherwise continuous stream.[6][1]
Event logs and event sourcing
[edit]Some RTBI architectures use event logs to pass changes to other systems. An event log is an append-only record that can be consumed by other systems to update dashboards, analytical stores, alerting systems, search indexes, and materialised views.[4] Related approaches include change data capture and event sourcing.
Change data capture is one way of producing streams from existing databases. It records changes made to database tables and makes those changes available to other systems without reloading the full data set. In RTBI, this can help keep analytical stores or event-processing systems closer to current operational data.[1][4]
Event sourcing is an architectural pattern where changes to an application's state are stored as a sequence of events, rather than just a current state record. The current state of an entity can be reconstructed by replaying all the stored events in order.[9] In RTBI, an event-sourced system can provide an event history that supports reporting and analysis through derived read models. These views can then be used by dashboards or other analytical tools.[9] It can support auditability, reconstruction of earlier states, and the creation of multiple views from the same event history. It is often used with Command Query Responsibility Segregation (CQRS),[9] because querying directly over an event store can be inefficient and separate read models built for reporting or analysis. Event sourcing may introduce design issues, including event schema changes, replay, storage growth, and eventual consistency between the event store and derived views.[10]
Near-real-time analytical stores
[edit]Another approach is to refresh a data warehouse or analytical store more often. This keeps the historical comparison and reporting features associated with conventional BI, while reducing the delay between activity and the availability of the data.[3]
Some cloud data warehouses and hybrid transactional or analytical processing systems are described as supporting near-real-time analytics. In practice, this depends on how current the available data is, the complexity of the queries being run, and how the source and analytical systems are designed. Event-driven systems and analytical stores can be used together. For example, selected events may be sent directly to dashboards or alerts, while the same operational data is also kept in an analytical store for later review.[4]
Business activity monitoring
[edit]Business activity monitoring (BAM) is closely related to RTBI because it follows business processes while they are still in progress.[2][11] Rather than waiting for later reporting, BAM uses events from operational systems to show whether a process is running as expected or requires attention.[5] Some literature describes this overlap between BAM and event-processing technologies as event-driven BI or operational intelligence.[5] In this context, operational intelligence refers to using current operational data to support monitoring and response during business activity.[5]
In a BAM architecture, operational systems produce events that are processed against predefined rules or patterns. The resulting information can then be shown in dashboards or used to initiate responses in workflow and enterprise systems.[5]
Application areas
[edit]RTBI is used where delayed information may reduce the usefulness of a decision or response. Event-processing research discusses examples in areas such as fraud detection, network security, financial-market analysis, sensor monitoring, inventory tracking, and manufacturing control.[6] There, data is processed close to the time it is produced, allowing changes to be identified before a later reporting cycle.[6]
Business activity monitoring systems use events from enterprise applications or process engines to follow work while it is still underway.[5] This can make process exceptions visible through dashboards or alerts, rather than only through later reports.[5]
RTBI extends uses already associated with business intelligence, including operational support in manufacturing, retail, financial services, transport, telecommunications, utilities, and healthcare.[3] Its main feature is the shorter time between operational activity and the information available for monitoring or response.[2]
In a retail environment, RTBI can support stock availability in areas like forecast and replenishment, by making sales and stock movements available closer to the time of store activity. For instance, The Co-operative Group has reported using real-time supply-chain systems from SAP to provide data in those areas.[12]
Financial cooperatives and mutuals have also been documented using real-time or event-driven data systems. Rabobank[13] has been described as moving away from batch-oriented processing to real-time, event-driven operations,[14] while Nationwide Building Society has been reported as using Apache Kafka and change data capture to provide near-real-time access to transaction data.[15]
Limitations
[edit]RTBI limitations include late our out of order data processing. Even early results may need to be updated as more complete information becomes available.[6][1] These issues are present in streaming systems, where continuous data flows are processed before all the information is known.[7]
For event-driven and log-based systems, additional design work may be required for event schemas, replay, monitoring, and consistency between source and derived views.[4] Event sourcing supports reconstruction of earlier states, but requires attention to any changes to the schema, and the management of read models derived from the event store.[9][10]
Immediate analysis is not always necessary. Hackathorn argues that the value of business intelligence depends on whether information is available at the right time, rather than at the earliest possible moment.[8]
See also
[edit]References
[edit]- 1 2 3 4 5 6 Kuznetsov, S. D.; Velikhov, P. E.; Fu, Q. (2023-02-01). "Real-Time Analytics: Benefits, Limitations, and Tradeoffs". Programming and Computer Software. 49 (1): 1–25. doi:10.1134/S036176882301005X. ISSN 1608-3261.
- 1 2 3 4 5 6 7 Azvine, B.; Cui, Z.; Nauck, D.D.; Majeed, B. (2006). "Real Time Business Intelligence for the Adaptive Enterprise". The 8th IEEE International Conference on E-Commerce Technology and the 3rd IEEE International Conference on Enterprise Computing, E-Commerce, and E-Services (CEC/EEE'06). p. 29. doi:10.1109/CEC-EEE.2006.73. ISBN 0-7695-2511-3. S2CID 15523928.
- 1 2 3 4 5 Chaudhuri, Surajit; Dayal, Umeshwar; Narasayya, Vivek (August 2011). "An overview of business intelligence technology". Communications of the ACM. 54 (8): 88–98. doi:10.1145/1978542.1978562. ISSN 0001-0782.
- 1 2 3 4 5 6 Kleppmann, Martin; Beresford, Alastair R.; Svingen, Boerge (2019-04-24). "Online event processing". Communications of the ACM. 62 (5): 43–49. doi:10.1145/3312527. ISSN 0001-0782.
- 1 2 3 4 5 6 7 8 Schmidt, Werner (2013), "Business Activity Monitoring (BAM)", in Rausch, Peter; Sheta, Alaa F.; Ayesh, Aladdin (eds.), Business Intelligence and Performance Management: Theory, Systems and Industrial Applications, London: Springer London, pp. 229–242, doi:10.1007/978-1-4471-4866-1_15, ISBN 978-1-4471-4866-1, retrieved 2026-06-03
- 1 2 3 4 5 6 Cugola, Gianpaolo; Margara, Alessandro (June 2012). "Processing flows of information: From data stream to complex event processing". ACM Computing Surveys. 44 (3): 1–62. doi:10.1145/2187671.2187677. ISSN 0360-0300.
- 1 2 Akidau, Tyler; Bradshaw, Robert; Chambers, Craig; Chernyak, Slava; Fernández-Moctezuma, Rafael J.; Lax, Reuven; McVeety, Sam; Mills, Daniel; Perry, Frances; Schmidt, Eric; Whittle, Sam (2015). "The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing". Proceedings of the VLDB Endowment. 8 (12): 1792–1803. doi:10.14778/2824032.2824076.
- 1 2 3 4 Hackathorn, Richard (2004). "The BI Watch Real-Time to Real-Value". New York: DM Review.
- 1 2 3 4 Millett, Scott (2015). Patterns, principles, and practices of domain-driven design. WROX professional guides. Indianapolis, IN: wrox, a Wiley Brand. ISBN 978-1-118-71470-6.
- 1 2 MSLearnEventSourcing. "Event Sourcing Pattern". learn.microsoft.com. Retrieved 2026-06-03.
- ↑ Vera-Baquero, Alejandro; Colomo-Palacios, Ricardo; Molloy, Owen (August 2016). "Real-time business activity monitoring and analysis of process performance on big-data domains". Telematics and Informatics. 33 (3): 793–807. doi:10.1016/j.tele.2015.12.005.
- ↑ News Centre, S. A. P. (2024-06-13). "Co-op Puts Customer Experience First with RISE with SAP". SAP UK News Center. Retrieved 2026-06-08.
- ↑ Official, Rabobank (15 May 2026). "Home Rabobank Group". Rabobank.
- ↑ Axual, Rabobank (15 Jun 2026). "Building a real-time banking platform with governance". Axual.
- ↑ Carey, Scott (20 May 2019). "Nationwide's 'Speed Layer' is built to compete with challenger banks". CompuerWorld.