We have developed a targeted system integration scheme for a manufacturing enterprise. We implemented ESB technology and launched 48 streams

Client

Electroresheniya LLC (EKF trademark) develops, manufactures and sells electrical equipment and integrated energy-efficient solutions for industrial enterprises. EKF cooperates with major Russian oil and gas corporations, manufacturing, logistics and transport companies, and its products are sold in 20 countries. The scale of the business is constantly increasing, which inevitably affects its IT landscape: the amount of data traveling between systems and the number of integrations are growing.

Objective: to ensure up-to-date data in all systems and the resiliency of the architecture

By scaling the IT architecture, the EKF team wanted to prevent the problems of scaling its IT architecture. As the amount of data used increased, the following challenges might arise:

• long upload of updates to data — the lag between updating information and its receipt by all consumer systems has begun to have a negative impact on the company's business processes;
• the delay in detecting, analyzing and solving problems in data transfer between systems;
• loss or untimely updating of data.

The EKF team came to the conclusion that it was necessary to introduce the Kafka message broker, which was to be entrusted with the transfer of data between systems and control the quality of information delivery.

EKF turned to KT.team with this task.

When interviewing the EKF team during the pre-project stage, KT.team clarified the client's request: to establish a process for ensuring the relevance of data in all systems and to achieve the resiliency of the IT architecture.

This problem can really be solved by introducing an ESB layer (more than one message broker). However, in order for the new architecture to avoid these problems, it is necessary to plan the integration logic correctly.

Therefore, together with the client, we divided the project into two stages:

• the analysis and planning stage;
• implementation phase.

Solution: design an ESB implementation safely, controllably and with clear business results

Result 1: identified the areas of responsibility of the systems

KT.team's previous experience on ESB implementation projects has shown that companies often lack an up-to-date and complete map of flows and integrations as is. Each department manager and leading IT specialist has an expert understanding of how relationships are built specifically in his area and specifically with his system. They have a general idea of the other parts of the integration map, which allows them to interact with other departments.

The absence of a complete as is map is not critical for the organization. However, its availability will give the IT department a single reliable source of knowledge about possible weaknesses in the current architecture, ways to prevent problems, and possible ways to improve the exchange architecture.

Therefore, we devoted the first 10 meetings with the EKF team to building and refining the as is scheme. As part of this scheme, the areas of responsibility of all systems included in the EKF circuit were recorded.

During the pre-project study, the EKF team, together with KT.team, identified which system is the source for a particular type of information and which is the recipient. This made it possible to identify places where information could be duplicated or contradicted by information in the source system. A decrease in the speed of information exchange could cause additional problems.

In addition, KT.team consultants have identified processes with a mechanism for information to enter the recipient system through several third-party systems. Such processes were more prone to failures and errors than others, and the analysis of related incidents required considerable time.

Result 2: the client's team synchronized their understanding of the state of the company's IT circuit

At the time of the start of the project, each of EKF's key IT specialists had in-depth expertise in their area of work, but there was no complete, reliable and publicly available map of the company's IT contour.

Following 10 meetings, the KT.team and EKF teams jointly built an integration map that was relevant at the time of the start of the project, combining and enriching local schemes previously drawn up by EKF employees.

Having formed an overall picture of the IT landscape, EKF's key IT specialists have gained an identical understanding of what the company's IT landscape looks like and what potential problems it poses — for the company as a whole, and not just for individual areas.

The created scheme helped to capture what complicates the data exchange process:

• There is a single point of failure for customer services (distribution portal, electronic catalog, etc.). );
• data is duplicated in several systems and proxied (i.e., the system acts as an intermediary in the transfer of information between other systems), there is no single area of responsibility for the information domain;
• the main ERP system is overloaded, including with data loading tasks, which causes errors and slowdowns in uploading data;
• Different systems and cases have different exchange mechanisms, which increases the complexity of supporting integrations and dealing with incidents, given the scale of the IT circuit;
• Due to the strong connectivity of systems, some business processes in some systems cannot be implemented if other systems are unavailable (for example, a distributor cannot place an order when ERP is not available).

As is analysis and recording, the dependence of the IT circuit on central systems was confirmed. The complexity of current integrations has made it almost impossible to replace any of the central elements due to the high risk of data loss and the loss of performance of the entire IT circuit. Such a replacement would have to take into account hundreds of nuances of existing integrations.

Result 3: we designed a new integration map with weak connectivity and high resiliency

Even in the early stages of communicating with KT.team, the EKF team talked about a number of common case studies from their practice that they had to work out regularly. For example, if ERP was not available for any reason, distributors could not place an order on the portal because they did not see the range.

Such problems can be prevented by asynchronous data exchange and the separation of repositories for different types of information.

The new integration scheme was designed in accordance with these principles. Each system has its domains — its areas of responsibility — and the types of data it is a source of.

Information is transferred asynchronously: first to the storage layer, then to the receiving system. Thus, it was possible to get rid of the mutual dependence of systems included in the IT circuit.

When designing the new IT circuit together with the EKF team, we discussed exactly how data storage would be organized, how microservices would be divided and how their interaction would be structured.

• The systems have been removed from the function of mutual data transfer (including intermediary data), as a result, the upload of updates is greatly accelerated, and the lag between updating information in the source system and uploading it to consumer systems is reduced.
• The monitoring and logging systems included in ESB make it possible to record incidents that occur (what happened to the data and at what point) and promptly report their occurrence to technical support staff. This makes it possible to work out detected problems in a timely manner.
• Data is loaded and uploaded via triggers or at a set frequency (depending on the business process). Each message's path is automatically tracked so that no information will be lost.

At the same stage, the consulting team:

• clarified the requirements for the customer's IT specialists for the independence of services (why they are formulated this way; how the independence of services will affect the company's business processes and the workload of IT specialists);
• compiled a step-by-step description of the transformation process for each of the connectors and flows, specifying the reasons for all the described changes and the expected effect as a result of their implementation;
• explained the logging scheme, the organization of monitoring and technical support.

The consulting report outlines the principles for monitoring the state of systems and flows, a mechanism for reporting problems, and a list of information sent to IT specialists to correct errors. EKF specialists got a common idea for all teams about how the IT circuit will work, how teams will learn about errors, and how undelivered messages will be saved.

Result 4: we provided a comparison of four ESB solutions, which allowed the client to choose a system based on his key parameters

The KT.team has prepared a comparative analysis of tools that can be used to customize the integration the client needs. At the first stage, we compared four solutions: DATAREON, Mule, Talend, WSO2 — according to 50 key parameters, including: adapter development language, availability of a library of ready-made connectors, fault tolerance, support for various formats and protocols, distribution of roles and access, scalability, etc.

At the second stage, two finalist applications were given a score for each of the parameters. According to the overall rating, the Mule tool was chosen for the project.

Result 5: we divided the process of implementing the ESB layer into stages with an understandable duration and value of each of them

The KT.team has developed a roadmap for implementing the ESB layer, showing the reasonable duration of all stages in hours — for each connector and stream separately. The implementation was divided into several stages, each of which resulted in EKF receiving clear value and a working IT circuit. When planning the implementation stages, we took into account the capabilities of the EKF internal team and assessed the budget for attracting outsourcing teams.

The stages are defined so that the input of each of them is safe for the company's IT circuit and previous implemented flows.

The complexity of each stage is indicated in hours — EKF can compare this assessment with offers from other integrators and choose the right contractor.

Together with the EKF team, we have identified a key stage, by the time of implementation of which the introduction of the tire converges with the ROI from the implementation. To this end, we have identified a block of integrations that are critical for the main business processes, the change of which will immediately bring tangible value.

This allowed the EKF team to determine the required budget and priorities for further work.

Results

• EKF IT specialists, together with KT.team consultants, translated an understanding of the current architecture into a service map format.
• The team's understanding of how and why the integration architecture will change has been synchronized.
• The team knows what steps ESB will take and how much value EKF will receive at each of them.
• The EKF team has determined the required budget and priorities for further work.
• The implementation of the ESB layer was started by two teams: EKF and KT.team.

‍