Why are techies against data buses: middleware, ESB, message brokers?

Imagine you want to open sales through Wildberries. You have already built a sales chain, warehouses, a chain of stores — your business is operational. Let's say your directory is in 1C. You are contacting 1C developers with a request: “We have already signed an agreement, we must ship to Wildberries, we need to integrate.”

It seems that the task is simple and straightforward... Let's look at the pitfalls of the “traditional way” of solving such problems, how middleware integrations can be simplified, and why techies are often against using data buses.

When integration becomes a problem

So, your developers took the task to work and gave a rating: “We'll do it next month.” Or they estimate what they will do tomorrow, but in fact, it still takes a month before the project is completed. You're running out of time, but there's nothing you can do about it.

After a month of work and half a month of debugging, you think you'll finally start selling your product on Wildberries. After all, it seems that all the products have been published and nothing prevents sales.

But a week later, it turns out that some products on the marketplace are not being updated, some are basically missing, and some have incorrect balances.

Let's make a small remark: we're not talking about cases where you process orders yourself — you would lose a couple more orders there, this would result in a drop in your site rating, and so on and so forth.
How can developers justify themselves?

“This is a difficult task; this is the first time we are integrating with Wildberries.

(Or not for the first time, but it used to be better!)”

“Wildberries API is poor and their tech support is slow to respond.”

- Resources are scarce.
- If it weren't four people, it would be forty, then it would be nice!

In this example, replace Wildberries and 1C with any other systems, for example, SAP and electronic document management, or CRPT, or anything else—the scenario will be about the same from time to time.

What do you see? You promised to launch sales this season. Everything seemed to be ready, but there are some problems. The development team says you're putting a lot of pressure on them that it's impossible to work. “We have been recycling. Give me an award!”

Or maybe they'll just quit: developers will find jobs on the “red” market. And you'll be left with an unlaunched, somehow working project that it's not clear who will support.

A system that knows

Why does the simple task of integrating systems quickly and reliably become a disaster in our super-automated time?

For almost 20 years, the market has had tools that make it possible to manage integrations between systems quickly, reliably and transparently for everyone. Moreover, such tools are a fundamental concept service-oriented architecture. We'll talk about why many teams don't or are using this approach incorrectly at the end. Let's start by solving the problem.

Collectively, such tools are called middleware, i.e., the intermediate layer. At first glance, this layer alone is of no value. But in the same way, a flat road between your home and the restaurant is not of direct value.

More precisely, such solutions are called tires, that is, a set of software that allows you to collect data from one system (for example, goods from 1C), put it in an intermediate store and send it to another, for example, Wildberries.

This layer “knows” that products from 1C cannot be transferred directly to Wildberries and that the “color” property, for example, needs to be transformed in a special way. This layer can also “know” that 1C does not store all product information, and photos of goods must be picked up from some of your other systems or even from the Internet before sending them to the marketplace.

How do you ensure speed, reliability and transparency?

Speed is ensured by the fact that your 1C developer does not need to write logs, monitor, learn how Wildberries works, etc. A 1C developer initially knows that if his system is a product master system, then his area of responsibility already includes the bus connector and the actual high-quality data in 1C. That is, he can actually set up the transfer long before the upload task appears.

Strictly speaking, every system “knows” what data it is responsible for. For example, your 1C may be responsible for goods and prices, and your warehouse program may be responsible for balances. It is also responsible for the appearance and updating of this data in an intermediate layer — for storage in the broker's appropriate queue.

Middleware integration: don't reinvent bicycles

When the task comes to integrate with the marketplace, the technical team responsible for sales on the site analyzes the Wildberries API and coordinates the fields. Then, in the graphical interface, he assembles the integration into the final service.

You can only have one developer responsible for everything, including building the integration into the bus interface. But in most cases he won't have to develop anything in a tire. It is enough to use ready-made integration designers in a tool that has been developed for many years only for this task. First of all, this is the first thing.

Tires have a huge range of tools for all popular activities, and they often have pre-assembled connectors for popular services. A developer who uses a bus takes less time coding and more time thinking about integration.

Second, developers don't have to “reinvent bicycles” to monitor systems and log actions. Native support for these procedures in tires makes it easier and faster for developers to understand that there is a problem somewhere. As developers spend less time on mechanical work, they have more time debugging and testing systems.

Third, changes in connectors to end systems are faster than changes in development. According to our statistics, while it takes at least four hours to make minor code integration changes, it takes 15 minutes in the tire designer.

On average, when creating integrations using tires, developers' labor costs are reduced fivefold.

In addition, sometimes your 1C team is overloaded, but you won't be able to hire a technician who is completely unfamiliar with your 1C to do this task. This will end up being more expensive for you than waiting for a team.

With tires (provided you follow a service-oriented approach), you can easily scale your team.

Using a tire also has a positive effect on your 1C. 1C does not have special connectors for Wildberries, but they are not needed either, since the bus uses your 1C's standard boxed API.

As a result:

• your 1C developers will not be distracted by supporting the integration;

• your sales channel will not be idle just because a higher priority task has arrived at 1C for implementation;

• There is less code in end systems.

Don't forget that writing an integration isn't a difficult or interesting task in and of itself. This means that the 1C team, spared from this task, does not burn out.
It seems that there are only advantages. But things are usually not the way we describe in this article...

Why are developers against data buses?

We have implemented tires many times both in our team and in other organizations, and we are well aware of the developers' arguments. Here are the main ones.

“Our company doesn't have the right tech stack”

Developers believe that without knowing the right technical stack (mainly Java), it will not be possible to work with the data bus.

Yes, your company may not have a Java developer. But the tire already has many ready-made designer components, and writing new ones requires a limited area of knowledge. The absence of Java developers is not an obstacle.

Developing from scratch in any language, let alone overloading current systems with integration logic, is, any way you slice it, a more labor-intensive path, which is even worse in terms of architecture.

“A tire is not true”

Developers often say that writing integration in code is “more difficult”: faster, more reliable, and more understandable. The other version is “I don't want to work in the tire” because “the tire doesn't have tools that are very necessary for development, like versioning.” The developer will tell you that it takes a long time to understand the tire, and “I'll do it with code now! It'll be ready in a couple of days.”

A couple of days usually turn into problems for a month. And yes: learning is a physically painful process. Ask yourself, will your team do better integration in a month than tire developers have done in 20 years? How many thousands of integrations have gone through tire tools, and how many problems have been identified and resolved during all this time?

In our experience, looking at the developer's excuses always led to the real reason: he just didn't find the tools and thought they didn't exist.

“ESB is out of date”

This statement is often supported by the presence of microservices.

The idea that ESB is outdated stems from a misunderstanding of the fundamental principles of building service and microservice architectures. We will discuss this topic separately in one of our next publications. For now, let's just say that Thomas Earle, founder of SOA and author of the double-reprinted book SOA: Analysis and Design of Service and Microservice Architecture, doesn't think so, nor do other great minds in Computer Science. We highly recommend that both colleagues and businesses read this book.

“I don't believe in low-code”

Mistrust comes from fear. Some developers may not believe in low-code solutions, but the market, on the contrary, shows confidence in this approach. Just look at the sales of Salesforce, one of the biggest low-code solutions in the world.

It is important to inform the team that using low-code tools, and in particular data buses, their work will become more informative and interesting. Why waste time on tasks that someone has already solved before you when you can do something fundamentally new?

“I've had or heard of a negative experience”

Let's be honest: implementing a tire correctly is not an easy task. In order to successfully solve it, you need to understand SOA principles and low-code principles, and you need to want to do well in the tool paradigm, not as usual. We've seen a lot of misused tires that created more problems than they solved.

Well, to drive a car, you must first learn how to drive. And the first attempts at the wheel can also result in negative emotions. But that doesn't mean driving is actually a bad idea, right?

Overcome resistance

All these objections boil down to ignorance of fundamental principles, fear of new things and fear of responsibility for results.

When a developer has fewer opportunities to hide behind technological complexity, when their work is visual and easier to interpret, when they have a clear area of responsibility, when they work with tools that have been sharpened only for integrations for two decades, their first reaction will inevitably be self-defense. He will try to explain why it is better not to implement a data bus and avoid any changes.

But in reality, the introduction of a tire benefits everyone.

Assembling integrations from components is a good entry into the profession for young developers. They start to benefit the team almost immediately, gradually increasing the level of immersion in the project. At the same time, it's harder for them to make a mistake.

After introducing a tire, an experienced developer focuses on the most complex tasks: writing new components and analyzing the system's behavior in depth. The integration customer receives a self-documenting system whose behavior he understands, and in some cases he can even take part in the configuration.

Everything we talk about in this article has been tested on our management experience. Working with our own team, training our clients' teams, consulting and managing our clients' IT infrastructures.

We assume that it is difficult for an untrained manager to introduce tires. We have to overcome resistance, ignorance, and the “Italian strike”. Thus, on one of our own projects, we did not pay enough attention to educational work with a new developer. As a result, we saw integrations that were de jure in the bus, but were de facto indistinguishable in their properties from the “traditional” integrations we talked about at the very beginning of the article.

‍Without sufficient technical authority in the team, it is even more difficult to push for these changes, but this is the role of a manager — to think through and implement changes.