Analytical layer – serve it to the end user

The analytical layer is the most creative and interesting of all the components of an NRT application. So far, all we have talked about is backend processing, but this is the layer where we actually present the output/insights to the end user graphically, visually in the form of an actionable item.

The most crucial aspect of the data visualization technique that needs to be chosen as part of the solution is actually presenting the information to the intended audience in a format that they can comprehend and act upon. The crux here is that just smart processing of the data and arriving to an actionable insight is not sufficient; it has to reach the actors, be they humans or processes.

Before delving further into the nitty-gritties of business intelligence and visualization components, first let's understand the challenges big data and high velocity NRT/RT applications have brought to the mix of the problem statement.

A few of the challenges these visualization systems should be capable of handling are:

• Need for speed: The world is evolving and rationalizing the power of now—more and more companies are leaning towards real-time insights to gain an edge over their competition. So, the visualization tools should complement the speed of the application they are part of, so that they are quick to depict and present the key actors with accurate facts in meaningful formats so that informed decisions can be taken.
• Understanding the data and presenting it in the right context: At times, the same result needs to be modulated and presented differently, depending upon the audience being catered for. So, the tool should provide for this flexibility and for the capability to design a visual solution around the actionable insight in the most meaningful format. For instance, if you are charting vehicle location in a city, then you may want to use a heat map, where variance in color shows the concentration/number of vehicles, rather than plotting every vehicle on the graph. While you are presenting an itinerary of a particular flight/ship, you may not want to merge any data point and would plot each of them on the graph.
• Dealing with outliers: Graphical data representations easily denote the trends and outliers, which further enable the end users to spot the issues or pick up the points that need attention. Generally, outliers are 1-5% of the data, but when dealing with big data and handling a massive volume and velocity of data, even 5% of the total data is huge and may cause plotting issues.

The following figure depicts the overall application flow and some popular visualizations, including the Twitter heat map:

The figure depicts the flow of information from event producers to collection agents, followed by the brokers and processing engine (transformation, aggregation, and so on) and then long term storage. From the storage unit, the visualization tools reap the insights and present them in form of graphs, alerts, charts, Excel sheets, dashboards, or maps, to the business owners, who can assimilate the information and take some action based upon it.