Outlook Ventures first reported in 2002 on Real-time enterprise as one of its Top 5 Trends for the Enterprise. In this article, Outlook's Principal, Edgard Capdevielle, explores in greater depth this trend which is now driving venture investment decisions and new start-ups.

The Real-Time Enterprise (RTE) is not a specific IT initiative or technology product, but more a long-term strategic issue. RTE is a strategic vision that does not relate to a single technology component, but a combination of pieces that may already be in place or are in process of being deployed. It also rides on several business and technology trends that indicate an increased focus on managing more dynamic business environments that provide little information to work with (uncertainty) and rapid change. Past definitions have focused mostly on software and process integration. Today, the term is more closely associated with the sensing of business information and the shortening of reaction time through decision support, execution, and automation.

There are market pressures driving RTE adoption in both the Consumer and Enterprise worlds. In the Consumer world, on-line retailers figured out very quickly that websites that were disconnected from back-end systems backfired, frustrating consumers and discouraging cost-effective web experiences. These companies integrated their systems and now consumers and small business customers are able to get immediate access and end-to-end process visibility. Financial institutions have followed suit, if not co-led this effort. Consumers now expect and demand at least this level of real-time interaction. Similarly, in the Enterprise sectors, it is frustrating to corporate planers and buyers that they cannot get the same level of service and visibility from their systems and those of their partners.

The heart of the matter is that despite all of the technology in place today, there are plenty of process and data buffers in the enterprise. These buffers or delays in flow can be observed every time there is a batch of information waiting to be processed or a broken flow. At a high-level, RTE is (among other things) a philosophy that focuses on reducing or eliminating information and process bottlenecks by applying project management & operations principles, as well as an infrastructure for integration, decision support and automation.

With this article we want to share our understanding of RTE technology and business issues as well as bring clarity to the long-term trends and choices that enterprises will have to make.

What is a Real-Time Enterprise?

The term "Real-Time" comes from the computing world and is generally used to describe systems or programs that must run and finish before their users and environment need the output results. In business, real-time processes must happen within a tolerance time-window that is determined by the market environment. For example, the anti-lock brake system of a car must activate and start controlling the brakes immediately after receiving the "brakes-locked" signal (very short time window). In business, the supply chain system must report logistics delays before it is too late for the company to take a corrective action and incur additional costs.

Beyond this definition, RTE is also a business philosophy that adds a new variable to all process metrics (this variable is elapsed time). Under this philosophy enterprise systems need to leverage existing time-stamps or event logs and generate new ones. The goal is to detect and gain awareness of delays and events earlier, reducing the reaction-time to generate a response and eliminate bottlenecks. RTE also helps create closed-loop decision support and automation systems. It enables users or applications to view and get relevant information from business processes and act on that information immediately. The Holy Grail would be to gain visibility from event data (about a problem or delay) and support a business decision with analytical and automation tools, pushing execution to the right people and systems across the enterprise network in a "tolerable" amount of time.

The architecture for an ideal RTE can be classified as follows:

Sensors: We define this category as technology used to receive input from users, partners, or the environment, feeding an enterprise application. In today's IT environments there is plenty of event logs and time-stamp technology. This technology, however, tends not to be used or stay within a functional silo (not shared across applications), making it very difficult to detect problems, events, or delays in a centralized, standard fashion across enterprise applications. A good analogy can be found in the network management space: Every piece of equipment (routers, servers, etc) should be configured to report status, events, logs or simple heartbeat signals to a centralized monitoring application that can represent a comprehensive view of the network. In the "software application" level, there is no such solution for business processes to immediately report delays or events to a central application. With current sensing infrastructure it is not easy to map delays, processes, or product flows across the organization or enterprise network. A new technology (RFID: Radio Frequency Identification) is getting a significant amount of attention because it promises to provide immediate information about the physical location and flow of assets (inventory, equipment, etc) through wireless signals. Technologies like RFID bring new sensing infrastructure to the enterprise and enable early detection of information, events, and trends, allowing for real-time response.

Workflow Support: This technology, also known as Business Process Management (BPM), was explored in detail in the October issue (See BPM Article). In short, this technology handles processes supported by current software solutions (ERP, CRM, etc) plus any other business process that is relevant and worth automating with a workflow engine. In addition to mapping and supporting business processes, BPM technology must enable process changes and process monitoring. A qualified user or application must be able to monitor processes and change them (for improvement) across different systems and functional silos. An end-to-end BPM platform should be able to capture process data from all applications and allow monitoring and troubleshooting of problems or delays, as well as detecting and analyzing trends. There are several innovations for monitoring business processes and transactions classified under the category of Business Activity Monitoring (BAM- yet another acronym in fashion these days). In a BAM application, processes that are being monitored are associated with performance indicators. When a relevant deviation happens, the BAM application notifies or sends an alarm to the appropriate person. Under the RTE philosophy, BPM and BAM solutions need to be an increased focus on logging elapsed time for processes as well as identifying buffers and bottlenecks.

Collaboration in its many forms is also a workflow component of RTE. Instant Messaging (IM), for example, is a real-time technology that is being used in the enterprise to dynamically address issues and solve exceptions, avoiding delays for relevant urgent matters.

Back-end Integration: Another technology that was covered in detail in our BPM discussion (because it is a necessary ingredient of BPM) was Enterprise Application Integration (EAI). It is still worth mentioning as a separate component of RTE because today functional application silos are still hardly integrated (e.g. sales data is passed to finance systems once a day and in an aggregated format, making it impossible to analyze in real-time). Working closely with workflow technologies mentioned above (BPM and BAM), integration solutions need to provide process and data integrity as transactions are executed or changed. New developments in this area include Web Services, which basically allow a more dynamic and flexible method for real-time integration. For RTE, it is important that integration solutions, in addition to fully supporting business workflow, provide and ensure fast access to legacy information (ERP, but also mainframe and archives).

Decision Support: Traditional Data Warehousing (DW) technology and reporting solutions tend not to be useful to RTE because of their batch nature. Instead of extracting data from business applications (making it stale) for analysis, new analytical tools provide ad-hoc queries on fresh data (real-time data or close). The ideal business intelligence for RTE must have the flexibility to support a dynamic workflow natively, must incorporate the new information being sensed by the enterprise (e.g. RFID intelligence), and it must also include elapsed-time and additional process-centric information to its traditional data-centric approach.

For example, a national discount retailer re-designed its DW infrastructure to get data from all the stores immediately after they close every day. The information is aggregated and processed in a central location, making business reports available for review before the next business day, which is real-time enough for them (tolerable) and an improvement from the usual week-long process. Other companies are applying web analytics solutions (traditionally used to monitor consumer click-through activity on the web) to enterprise applications in order to monitor process latency and detect areas for improvement or automation. A new breed of analytical apps also combine archived and fresh information as well as unstructured data that may exist in users desktops.

Automation: Not necessarily a separate software category, Automation is a component of an RTE system. RTE requires that enterprises eliminate delays and process bottlenecks. Beyond integrating applications that discard the batch model, one of the most common delays in the enterprise is human processes and decisions. Business managers and knowledge workers tend to be afraid of automation in the enterprise because in a way they lose control, but not accountability. However, as automation technology matures, enterprises are able to trust automation to a greater extent. In fact, scenarios where criticality and complexity are high, we already use a lot of automation (automatic system fail-overs in disaster recovery architectures). We have already become more comfortable with automatic authentication of online business users, bank transactions, and online purchases (without human verification delays). Automation is an important component of RTE when a traditional delay is the critical factor making a process longer that the "tolerable" limit. Many applications and workflow solutions already have basic Rules Engines and they tend to work well in a small, fixed context. New Automation Engines are more dynamic and manage a larger, more complex environment. To make intelligent decisions, these new engines are incorporating a lot of the real-world context into their knowledgebase. They are also using case/scenario building to understand acceptable and recommended responses to derived scenarios (that arise through alarms and events sensed throughout the enterprise).

RTE Closes the Loop

A visionary example of these five parts working together can be devised in the security world. Enterprises commonly buy and implement firewalls, intrusion detection systems, VPN servers, client-based firewalls and policy managers, etc. All these products do their work and report findings to the Administrator. However, they rarely work together in an organized fashion to counteract a complex security attack. New developments in the security space allow for centralized monitoring and gathering of logs, alarms, and events. A correlation engine decides, based on case/scenarios and rules, when a complex attack is happening (For example if "normal" web traffic is coming in through the firewall and launching a disguised attack from several unprotected laptops or servers). The engine may decide to raise the security level of the firewall for a fixed duration, verify the attack, and counteract by completely separating contaminated machines into quarantine mode to stop the attack. If this is done automatically, as the attack starts, most enterprise systems would likely never be affected, and recovery will likely happen within seconds. Administrators could then analyze what happened and try to improve on the next attack by enhancing automation rules and updating security applications.

This is an example of a closed-loop. The system used specific inputs and metrics, and executed from goals to actions. The cycle went from performance measurement to correction fast enough to satisfy the business requirement (no down time).

A less critical example (this time at the application/user level) can be illustrated in an advanced call-center. Everyone has experienced call center delays and waiting queues, and even decided to end a call due to a long waiting period. In fact, call center demand is not very predictable and capacity allocation (number of available agents) is often optimized for average call loads. If a call center supervisor realized that a peak of calls were waiting in the queue, he does not have time to call and activate new agents, or negotiate with another departments for borrowing personnel. An RTE-enabled system could detect the extra load and activate pre-authorized rules for reacting to this problem. The system could simply redirect the call volume to a partner that can handle the overload, or use pre-authorized contracts for "borrowing" personnel from other departments. The system could activate virtual agents through instant messaging (agents sitting at home with desire for extra work or trained knowledge workers that can assist in these cases) and redirect calls to respective homes or offices for the duration of the peak. In this fashion, the enterprise gets to keep customer satisfaction indicators and react smoothly and in real-time to sudden events in the business.

RTE Implementation: In Phases

As seen in the examples above, RTE solutions can be used to increase revenue (avoid loss of telephone orders) or reduce costs in a business. We believe that pressure from competition, environment, business partners, and customers will drive enterprises to adopt the technologies and philosophy of RTE. The adoption of RTE technology will continue to happen in phases. First, an enterprise systems or groups of systems need to be good at gathering and sharing relevant information. Then these systems must develop intelligence through analytical features (prediction, decision support, trend analysis). Finally, as users and executives learn to live with and trust these systems, automation and dramatic reduction of delays will naturally happen, making these systems real-time systems.

In the average enterprise there are several pieces of the RTE architecture in place today or in process of being implemented. Most companies have several business systems (ERP, CRM, SCM, generic workflow tools, etc) and have an integration framework or solution in place (EAI) and/or may be considering a BPM/BAM solution for process automation and monitoring (from vendors like Savvion, Celequest, WellFound Technologies, firstRain, etc). The first phase is then to enhance and centralize relevant logs, events, and alarms into comprehensive sensing systems (likely to already be part of a solution as opposed to a separate piece). Companies in the supply chain or logistics business, and even retailers should likely consider RFID technology. Some vendors of RFID-enabled solutions include (Savi Technologies, Vizional Technologies, Globe Ranger, etc). NetBrowser Communications (an Outlook Ventures portfolio company) also provides an enhanced sensing infrastructure to data centers and facilities, by enabling and centralizing signals from critical equipment that are not normally monitored and lie below the IP connectivity layer (HVAC, power, fire alarms, etc).

The next step is to implement some of the analytical RTE pieces:

The last step for bringing processes to the real-time world is to automate. Automation features are now being deployed in embedded form (inside specific applications), making automation of fixed context and small scope. Other automation solutions use software agents, which are programs specifically created to navigate a decision or a process and reach a desired goal or state. Start-ups like Agentis Software are good examples of this concept.

Conclusion

Real-time monitoring and management of business processes is important, but not for all processes. When deploying RTE technology solutions, successful companies tend to isolate specific processes that are critical to the business and require real-time behavior. Traditional ROI assessment for real-time projects should include the benefits improved response time and quality.

Aside from many of the companies mentioned in this article, we expect to see more investment opportunities and interesting start-ups to enter the marketplace in the near future. In the coming years, real-time technologies will continue to appear in the enterprise to the point that they will become pervasive. For us, it has certainly been very exciting to track real-time solutions from the very beginning of the wave and hope that you will come to us with your new and exciting real-time idea or company.