0

I've been at my current job for about 4-5 months, mainly working with Go, and I have no prior experience with Kafka. Before this, my background was in JavaScript, Node.js, React, etc. I recently got a simple task: write a Kafka-based agent that listens to a topic and stores all events from this topic in a database. I believe I've done it correctly and have sent it for code review. However, our architect pointed out several flaws in my code, especially disagreeing with my architecture.

I divided my code into three layers:

  • The first is a thin wrapper around the kafka-go consumer for logging, tracing, etc.
  • The second listens to Kafka messages and sends them via a channel to the user.
  • The third is the processor, which receives the channel, listens to events, and processes each one or in batch. It's designed so you can pass any handler; in my case, it's one that writes to a database.

Our architect argues that my design is overly complex and could be simpler. He suggests there's no need for go routines and channels and that I could use kafka-go's built-in ReadBatch. I found that this method uses minimum and maximum batch sizes and doesn't work with atomic messages, so I would need to handle this myself. My research suggests it's unusual to use this low-level approach due to its pitfalls, with most users preferring ReadMessage, FetchMessage, and Commit. I believe my approach is simpler, especially for batch processing. I aggregate messages in a batch using FetchMessage and process the batch by storing it in DB.

So, my questions for those experienced with kafka-go are:

  • Is my approach valid, or should I reconsider it?
  • Is the solution proposed by my architect more complex than it seems, or am I being lazy? Is there another method in kafka-go for reading messages in batches that I missed?
  • If the architect's assumption is correct, I'm willing to make the effort to change it. But if my solution is better, how can I argue this, especially since he seems a bit emotional about it?

Thanks in advance.

Improve this question
9
  • "and processes each one or in batch" Can you elaborate/clarify how an event results in a [something] in batch? Commented Dec 28, 2023 at 20:16
  • When there's a built-in 'batch' function, it typically has some sort of performance benefit. Have you considered this w/ regard to your expected volume? Commented Dec 28, 2023 at 20:19
  • 1
    Having multiple functions/layers to separate responsibilities might be sensible, but introducing channels here is not. The typical Kafka processor must work synchronously: receive the next message, process it, and then save/commit the offset before touching the next message. Batch processing can work, but just reduces communication overhead and doesn't really provide opportunity for concurrency. If you want to process multiple messages in parallel (without order), then you need multiple topic partitions and can start multiple instances of your tool in a consumer group. Commented Dec 28, 2023 at 21:02
  • 1
    @Anatoly You might want to check this out: Batch Processing for Efficiency "To avoid this, the Kafka protocol is built around a “message set” abstraction that naturally groups messages together. This allows network requests to group messages together and amortize the overhead of the network roundtrip rather than sending a single message at a time. The server in turn appends chunks of messages to its log in one go, and the consumer fetches large linear chunks at a time." Commented Dec 29, 2023 at 16:57
  • 1
    @Anatoly Correct, Kafka uses batching internally but the Go module doesn't expose this to you. What you can do instead is to read more messages with a very low timeout until your own batch is full. There is currently no ReadBatch() function (but the underlying librdkafka has a consume_batch() function). But it seems your real problem is on the database side, not the Kafka side, so batching Kafka messages might be irrelevant. Commented Dec 29, 2023 at 18:14

2 Answers 2

Reset to default
4

As an architect that works with developers at my job, and as a former software developer that found myself in your situation many times in the past, I can certainly understand this scenario you are in and would be happy to give you my thoughts and recommendations here.

One of the hardest parts of being a software developer, at least from my personal experience, wasn't so much the technology or the coding but the ability to take constructive criticism during code/design reviews. I used to get quite heated and exasperated when I proudly displayed my elegant forward thinking approach to my peers and seniors, only for it to be ripped apart, criticized, and ultimately rejected. I admit my arrogance got the better of me many times and I would get quite emotional and judgmental of my peers and seniors.

The thing that I eventually had to learn was that I was not designing a solution or programming for my own benefit and use, but for someone else. The architect here is ultimately responsible for the technical quality of the solution here, and your peers are going to be responsible for supporting this solution. The architect has to consider all of these stakeholders as it relates to the quality attributes of the entire system when guiding solutions. These are all things that you should care about as well, so it should not be an adversarial relationship, you both ultimately want high quality, reliable, resilient and maintainable software after all. You both should have the same goal in mind. Sometimes we have to set aside our ego though and embrace humility as a default when entering code reviews.

With this being said, the most important quality of a good architect isn't so much being "correct" or having good outcomes more often than not. Being a good architect means being able to effectively communicate with peers and rationalize the decisions and recommendations that are made. Making decisions is the easy part. Pointing out flaws or potential improvements is easy. Being able to effectively rationalize potential quality issues or suggested improvements is the hard part. Harder still is communicating the rationale effectively to developers and other stakeholders so that they see the benefit and understand why a particular aspect of your approach is potentially problematic or could be improved upon.

I cannot speak to the rationale here for why the architect suggested using ReadBatch over your current approach, as you kind of hand wave over his explanation a bit. Perhaps the architect did not do a sufficient job of explaining how the complexity of your solution is problematic? Regardless when a concern is raised over the complexity of a solution, then it is usually rooted in a deeper concern over the maintainability of the solution over time. Is the design you presented hugely inconsistent with similar solutions? Are there other cross-cutting concerns that would be better served with a more standardized approach? Can you effectively argue for not only the benefits of your solution versus the standardized Go solution for this problem, but also the drawbacks? These are all questions you should ask yourself of your design approach and attempt to answer for yourself if you want to convince a more senior member of your team that what you presented in review IS the best approach.

A third opinion, on the nature of this design from an architect not involved in nor knowledgeable of your systems is not going to be helpful to you. It is your job and your responsibility to build consensus around your design to your peers and seniors. That either means professionally and effectively strengthening your rationale for your approach, or adjusting your approach to be in alignment to the rationale provided by your peers or architect.

Improve this answer
1

While I do agree with @maple_shaft's answer on a higher level, I would argue that the whole endeavor is somehow flawed because it is reinventing the wheel.

What you describe is a common use case and Confluent Inc, the creators of Kafka addressed it via Kafka Connect.

Since you will have to use a service for the use case anyway it boils down to: "Do we expect to have a similar use case in the future?" setting aside the necessity to keep an in-house solution up to date, patched and what have you.

As for your setup: I tend to agree with your approach.

  1. Contrary to what has been said in the comments, channels do not automatically make an operation asynchronous, so you are good there provided you provide a chan err back to your consumer to let it decide whether the message should be acked or not.
  2. With a proper coding style, channels are a feature, not a problem. Especially if you apply SOLID, as laid out by Dave Cheney's application of the principles to Go
  3. Since apparently, you have built your database "sink" as an interface (or at least a GORM or sql.DB), which makes the whole thing easier to maintain, test(!!!) and extend.
Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.