Make it easier to work on integrations with Kafka
Make it easier to work on integrations with Apache Kafka
Apache Kafka message broker is a distributed streaming platform that can process millions of events daily. Kafka guarantees easy integration into the project infrastructure, the reliability and scalability of the system.
Kafka Connect
Kafka Connect is an Apache Kafka framework that provides scalability and flexibility to move data between Kafka and other repositories. This framework allows the broker to act as an ESB service bus.
Apache Kafka features
1
Streaming data processing
Kafka's streaming processing works with data in real time at the speed of message generation. Messages are processed continuously and without blocking. Many business processes are also ongoing and do not require a response for processing. Streaming data processing is necessary for business processes such as alerting suspicious transactions or tracking mail delivery.
2
App activity tracking
Any messages that appear in the app can be published in a special Kafka topic. For example, every ERP system document or every user action on the site: clicks, adding to favorites, adding/removing from the “Cart”, filling out forms, page views (and its depth) — can be sent and distributed according to specially defined Kafka topics. Thus, other topics (consumers) can subscribe to the topics they need for various purposes — monitoring, analysis, reporting, personalization, etc.
3
Logging and log monitoring
Kafka allows you to keep logs and monitor logs. You can publish logs to Kafka topics, and logs can be stored and processed in a cluster for as long as you need. If you have a dedicated monitoring app, it will be able to get real-time data from Kafka topics.
4
Message storage
Kafka adds each message to the journal (saves it to disk) and stores it there until the log is cleared of old messages, which the user assigns in advance. This allows Kafka to be used as a reliable data source (unlike RabbitMQ, which deletes messages immediately after delivery).
The benefits of Kafka Apache
Scalability
Apache Kafka allows you to process data of any size. You can start working with one broker to try out Kafka's features and then increase the number of brokers to fully exploit the system. It is also possible to increase the volume while the current number of brokers is running — this will not affect the system as a whole in any way.
System reliability
One of Kafka's advantages is its reliability. For example, if one of the Kafka brokers “falls” for some reason, it will switch the entire data flow to other brokers and automatically distribute the load between them, and the system will continue to operate normally.
Productivity
Due to its high throughput, Apache Kafka is able to process more than a million events per second. This makes Kafka the most popular message broker when working with big data.
ESB system implementation cases
Watch all
A supplier portal was developed for the Fix Price network and automated the work with product data
Learn more
We have developed a single API to quickly connect 200+ 1C:Retail systems
We have developed a single API to quickly connect 200+ 1C:Retail systems
Learn more
Integration of Kafka into the project's enterprise infrastructure
Integration of Kafka into the project's enterprise infrastructure
Learn more
Watch all
Contact us and we will calculate the time and cost of implementing the ESB system
YouTube
We have collected all the mistakes in integrations,
for you not to
make them
We work with systems
We use systems
Project Calculator
- Preparation of a map of systems and data flows (SOA scheme);
- We will work out exchanges on key entities;
- Creating connectors for exchanging data for each stream on 3 stands (test, preprod, prod);
- Setting up to three dashboards per connector within a ready-made monitoring circuit;
- Documentation on copying integration, reusing, and maintaining;
- Demonstration of the implemented functionality.
- Preparing the infrastructure for connectors to operate;
- Setting up the monitoring and logging circuit;
- Creating connectors (storage - receiver) for exchanging data on each high-load stream (>100 messages per minute) on 3 stands (test, preprod, prod);
- More than 15 attributes for each stream;
- The cost of computing resources (subscription only);
The calculator calculates using an accurate but simplified formula. The scope of work for your project and the final cost may vary. The final calculation will be made by your personal manager.
Калькулятор считает по точной, но упрощенной формуле. Состав работ по вашему проекту и конечная стоимость могут отличаться. Итоговый расчет сделает ваш персональный менеджер.
1
Calculation example
To transfer data between systems, we create a “stream”. Some streams are needed to send data, while others are needed to receive data. Orders, goods, or other entities may be transferred in a separate stream.
For example, on the diagram:
1. The “Merchandise Management System” sends goods. “Warehouse management system” is the fact that an order has been shipped. “Order Management System” — orders. In total, the systems will send 3 streams;
2. The Warehouse Management System accepts goods and orders. “Order management system” — goods and the fact that the order has been shipped. In total, the systems will receive 4 streams.
2
Scope of work in the calculator
Included in the calculation
Additionally
Preparing a map of systems and data flows (SOA scheme)
Preparing the infrastructure for connectors to operate
Development of object logic (connector business process diagram)
Setting up a monitoring and logging loop
Creating connectors for exchanging data for each stream on 3 stands (test, preprod, prod)
Creating connectors (storage - receiver) for exchanging data on each high-load stream (>100 messages per minute) on 3 stands (test, preprod, prod)
Set up to three dashboards per connector within a ready-made monitoring circuit
Over 15 attributes per stream
Documentation on copying integration, reusing, and maintaining
Demonstration of the implemented functionality
Included
Preparing a map of systems and data flows (SOA scheme)
Development of object logic (connector business process diagram)
Creating connectors (source - storage, storage - receiver) for exchanging data on each object on 3 stands (test, preprod, prod)
Set up to three dashboards per connector within a ready-made monitoring circuit
Over 15 attributes per object
Additionally
Preparing the infrastructure for connectors to operate
Setting up a monitoring and logging loop
Creating connectors (storage - receiver) for exchanging data on each high-load object (>100 messages per minute) on 3 stands (test, preprod, prod)
Over 15 attributes per object
Your application has been sent successfully
Submit again
Let's discuss your project
Personal managers will contact you
Submit again
Would you like to wait? Contact us right now
We write articles for our blog and for specialized publications
Read more
Point to point, broker, ESB: what integrations will help build a loosely coupled IT architecture
Learn more
The properties of a flexible and scalable IT infrastructure: an educational program on basic concepts and the main evaluation criteria
Learn more
Watch all
