How To Integrate Big Data Technologies With Decision-Making Architectures?

More and more companies are now seeking to evolve their business intelligence architectures to incorporate big data technologies, which are synonymous with greater flexibility and processing potential.

This is no longer a fashion effect: big data technologies are being installed in the DSI roadmaps as relevant tools to address their data processing issues. Without completely disrupting their architectures, many IT departments are questioning the possibility of using these technologies to evolve their decision-making infrastructures.

Faced with the explosion of data volume, information has become the nerve of the economic war. It helps to gain competitiveness and find growth drivers. For this reason, companies are looking for the most efficient means of data processing and management in order to position themselves in a market where competition is becoming fiercer and where the one with the information is the most strong.

Today, organizations must act with respect to the exponential increase in the amount of data produced (structured, semi-structured and unstructured). In addition, they must ensure good governance of these data while analyzing them as quickly as possible, in order to obtain information on their current environment, its evolution and thus to obtain a strong and unique competitive advantage.

Business intelligence & Big data, two technological families serving the same uses?

Business Intelligence refers to all the infrastructures, applications, tools and good practices that will enable business managers to make decisions based on reliable information. KPI tracking, reporting, and dashboarding data are key business intelligence needs.

The term “big data”, which is much more generic, does not only include technologies serving the decision-making IS. It includes all the technologies, tools, good practices and infrastructures that will allow processing and/or store massive data.

By “massive data” is meant: the large volume of data, injected at high speed, source, and varied nature. It can be structured or unstructured data. In fact, decision-making is just one of the use cases for which big data technologies can be used.

How to take advantage of the benefits of big data in decision-making architectures?

The model of decisional architecture, prowl, shows today its limits, for example in the establishment of data warehouse. Their setup requires a very structured and top-down approach, which involves defining upstream the customer’s needs in order to design the most appropriate data warehouse architecture. This structure is generally not very scalable and does not handle a larger volume of data than that for which it was designed. Conversely, the data lake offers greater flexibility in both the design and the volumes of data processed.

More and more companies are now seeking to evolve these architectures to integrate big data technologies, synonymous with greater flexibility and potential for processing.

Modifications can occur at several levels of the architecture:

The architecture of the decision-support applications is evolving and converging on a new, more modern, agile architecture with more features. These evolutions are characterized by intermediate architectures integrating technologies and Big Data patterns that aim to meet the limitations of previous architectures.

Towards a disappearance of BI architectures?

We now note the appearance of 100% big data architecture for exclusively decision-making purposes. But between the integration of big data technologies and the replacement of the decision-making chain by a purely big data architecture, we have identified these different intermediate possibilities.

The first type of architecture implemented, the Lambda architecture, consists in adding a layer of stream processing and realizing OLAP processing in real time, something impossible before. This answers all the requests of the decision-maker – but it is relatively complex to set up and maintain, in particular, because it requires adding a lot of application bricks.

The second type of architecture to emerge to meet this type of use case is the Kappa technical architecture. It allows real-time data processing. Very little data is stored at the data store. We only work in streaming, more in batch.

The Kappa architecture is designed to simplify the Lambda architecture. Indeed, it makes it possible to merge the real-time and batch layers into a single real-time layer. Unlike the Lambda architecture, the Kappa architecture does not allow the permanent storage of data (ie the data storage system is restricted and unsustainable, it must be a log file system and can not be modified) and does not support not the advanced analyzes on the history.

These new architectures are certainly richer but also more complex to put in place.

As such, they will never completely replace decision-making architectures and BI. The future is probably halfway between the two approaches, in a generalization of hybrid architectures.

Leave a Reply

Your email address will not be published. Required fields are marked *