Using Kafka on Event-driven Microservices Architectures - Apache Kafka Meetup

Using Kafka on Event-Driven Microservices Architectures.

Who am I? Oscar Gómez Soriano What I Do? Software Architect / Developer @Stratio BD How can you contact me? ogomez@stratio.com Oscar Gómez Soriano @oskarflesh

1. Event Driven Architectures 2. Real Use Case: Appointment Creation System 3. Architectural Solution 4. Improving the Architecture AGENDA

“An enterprise application is a system that reacts to events from the outside world”. Martin Fowler

Event-driven Architectural Patterns We can find 4 great types of event-driven architectural patterns depending on the way the events are used: ● Event Notification - This happens when a system sends event messages to notify other systems of a change in its domain. A key element of event notification is that the source system doesn't really care much about the response. ● Event-Carried State Transfer - The event sended contains all the info needed to update the “recipient system” in such a way that it doesn't need to contact the source system in order to do further work.

Event-driven Architectural Patterns ● Event-Sourcing - The core idea of event sourcing is that whenever we make a change to the state of a system, we record that state change as an event, and we can confidently rebuild the system state by reprocessing the events at any time in the future. The event store becomes the principal source of truth, and the system state is purely derived from it. For programmers, the best example of this is a version- control system. The log of all the commits is the event store and the working copy of the source tree is the system state.

Event-driven Architectural Patterns ● CQRS - Command Query Responsibility Segregation (CQRS) is the notion of having separate data structures for reading and writing information. Strictly CQRS isn't really about events, since you can use CQRS without any events present in your design. But commonly people do combine CQRS with the earlier patterns here, hence their presence at the summit.

Implementing Event-driven Architectures “A event-messaging-based application typically uses a message broker, which is an infrastructure service through which the services communicates. However, a broker-based architecture is not the only messaging architecture. You can also use a brokerless-based messaging architecture in which the services communicate with one another directly”. Chris Richardson - Microservices Patterns

Broker-Based Messaging Benefits & Drawbacks There are many advantages to using broker-based messaging: ● Loose Coupling - a client makes a request simply sending a message to the appropriate channel. The client is completely unaware of the service instances. It does not need to use a discovery mechanism to determine the location of a service instance. ● Message Buffering - the message broker buffers messages until they can be processed. This means, for example, that an online store can accept orders from customers, even when the order fulfillment system is slow or unavailable. The Order messages will simply queue up. ● Explicit Inter-Process Communication - Messaging makes differences between remote and local services very explicit so developers are not lulled into a false sense of security.

Broker-Based Messaging Benefits & Drawbacks There are some, however, downsides to using messaging: ● Potential performance bottleneck - there is a risk that the message broker could be a performance bottleneck. Fortunately, many modern message brokers are designed to be highly scalable. ● Potential single-point of failure - its essential that the message broker is highly available otherwise system reliability will be impacted. Fortunately, most modern brokers have are designed to be highly available. ● Additional operational complexity - the messaging system is yet another system component that must be installed, configured and operated.

REAL USE CASE: APPOINTMENT CREATION SYSTEM 2

The Challenge The basic idea is to construct an Appointment Creation System. On matter of the performance & functional requirements we got 3 different scenarios depends on the user who interacts with the system: ● Final user ● Admin User ● BI User ● AI User

The Challenge: Final User Requirements 1. The user can consult calendars to be able to see the available appointments & create it. 2. The user can check their created appointments and reschedule/cancel it. 3. By default, a same hole can not be reserved by two users at the same time, an overload algorithm determines if this restriction is disabled. 4. An AI model determines the probability that the patient attends the appointment and an algorithm determines the overload capacity.

The Challenge: Admin User Requirements 1. A admin user can consult existing appointments. Using filters they can adjust their search. 2. The Admin user can create appointments for patients.

The Challenge: BI User Requirements 1. The BI users can calculate business metrics from visits events in near real time. 2. The BI user can visualize the values of these metrics through dashboards.

The Challenge: AI User Requirements 1. A data scientist has an advanced analytical environment in which they can develop models from the information obtained through the system. 2. The data scientist can do data discovery using Stratio Discovery

The Challenge: Performance Requirements 1. Response time for more complex operations less than 250 ms. 1. Transactional process supporting 10,000 online users interacting appointment API (creation/cancel/Rescheduling) and consulting calendars in a two minute window. 1. Replication of the data to the query model must respond to the principle of eventual consistency with a delay of less than 1 minute, regardless of the size of the HDFS block.

Architectural Solution: The Big Picture

Architectural Solution: Microservice Architecture

Architectural Solution On Detail : Appointment Service

Architectural Solution On Detail : Calendar Search

Architectural Solution On Detail : KPIs (BI Users)

Performance Test Results: Test Scenario Infrastructure: ● Stratio DataCentric Platform deployed on Azure Cloud ● 500,000 pre-reserved appointments. ● > 84,000 agendas registered in the system. ● Production Masters Data ● Test Distribution by functionality: ○ 37% Calendar Search ○ 16,67% Appointment Inserts ○ 15,15% Appointment Reschedule/Cancel ○ 30,30% Created Appointment Search ● Microservices Resource Configuration: ○ Calendar: 4 cores y 4GB, Autoscaler Config from 2 to 5 instances. ○ Appointment: 1 Core, 1GB, Autoscaler Config from 2 to 5 instances. ○ Masters: 1 Core, 1GB Autoscaler Config from 2 to 3 instances.

Performance Test Results Performance Test Results: 2 Hour Performance Test 88 tps (10483 2 minute window) Microservice Request Distribution Request/ s Media Calendar Search 37,88 % 33 20 ms Appointment Insert 16,67 % 15 64 ms Reschedule / Cancel 15,15 % 14 74 ms Appointment Search 30,30 % 26 26 ms 2 Hour Performance Test 178 tps (10483 1 minute window) Microservice Request Distribution Request/s Media Calendar Search 37,88 % 67 19 ms Appointment Insert 16,67 % 30 106 ms Reschedule / Cancel 15,15 % 28 91 ms Appointment Search 30,30 % 53 46 ms

Knowing the Gaps With this numbers in mind we identify two possible bottlenecks on this Architecture: ● Created Appointment Search: As the search and insertion go against the same service and database model both operations are penalized. In addition, the model applied to the insertion is not optimal for the search since it forces the front to make auxiliary requests to obtain the data it needs. ● Calendar Service Responsibilities: Another gap we found is that Calendar service has to receive & process the kafka events (overbooking calculations & update the cache model) and at the same time attend to the get calendar requests so in times of high demand we will have a lot of kafka events to process which could slow down the search speed of the calendars. ● Another consideration to take is that despite having demonstrated a high performance, Memcached is not a solution that can scale well having a series of limitations such as not having a persistence mechanism of its own, having a high availability solution or not being a distributed system Once we know our service needs in matter of performance and scalability it’s time to take advantage of the decoupled way of development we use and separate those subdomains on separate services.

Using Kafka on Event-driven Microservices Architectures - Apache Kafka Meetup

More Related Content

What's hot

Similar to Using Kafka on Event-driven Microservices Architectures - Apache Kafka Meetup

More from Stratio

Recently uploaded

Using Kafka on Event-driven Microservices Architectures - Apache Kafka Meetup