How to manage depolarization of completed work on Agile Scrum

When I take a glimpse at what you present here my first thought is:

What about shorter sprints? Say weekly sprints. The Tasks E1T2 and E2T2 in their current state were not refined enough for weekly sprints. But if you break them up it should make planning easier. And at the end of the sprint everything would be done.

My second thought on this:

What about feature toggles?

Of course this is to be designed carefully or you end up with dozens of toggles. But the basic idea is that you are able to release a feature which is not complete by this sprint but is fully tested to the state of the current sprint. And the switch is flipped once it is ready to go live. And the switch is removed in the next release. So you decouple development from visibility. No extra branches. Of course this demands more discipline because you have to stay "backwards compatible" with the other "active" features and you have to remove the toggle afterwards to not mess up your configuration with hundreds of temporary options.

But on the other hand every work is released each sprint and nothing has to be held back from a development perspective.

The third take on this:

Why are you having independent teams working on seemingly independent features sitting (at least virtually) at one table for planning? I would have expected there were separate teams for each service with each having a separate agenda of which tasks to tackle next. The question raises whether scrum is the road you want to go down. If so, your product development has to change in such a way that the teams could work more independendly.

tl;dr

Your development method doesn't fit your actual work process therefore causing pain. How to solve that is up to your company.

There are quite a few places that I would recommend looking.

It's not exactly clear the breakdown from "Epic" into "Task". Based on my experience, it's unusual because it's missing a level of granularity. There are multiple definitions of Epic, but it seems like you are using it in the sense of a container for work. If Tasks are development tasks, you may be missing a thin slice of end-to-end, value-adding functionality that is at least demonstrable, if not deliverable. Sometimes, these are called Stories. Having this level that cuts across all of the involved services and estimating at this level may help you plan better.

The estimates also seem a bit off, with differing levels of granularity. If you're going to estimate, I am a fan of ideal time estimation. However, there's a difference between "2 weeks" and "10 days" in terms of what the estimate means. Consider what happens if you delay by 1 or 2 days. You're onto a 3rd week of development, but the 11th or 12th day. Try to be consistent with the units.

In addition to levels of granularity, a common practice is to decompose a deliverable slice of value into something that can be done in days and technical tasks as things that can be done in hours. If you use the decomposition that I mentioned above, a story that takes 3 or 4 days is on the larger side, and anything bigger should be decomposed if it's feasible. Similarly, a task that takes 12 to 16 hours is on the long side and should be broken down if it can be.

If you're going to estimate in time, I'd also make sure that you are considering ideal versus elapsed time. If you are using 2 week iterations, you don't have 10 days of productive work. There are a few different measures. Capers Jones uses something around 84-85%. In the US, with a traditional 5-day work week, 8-hour day and 40-hour week, a 2 week iteration would have 8.5 days or about 68 hours. I prefer to be a little more conservative and use 70-75%, so a highly productive 2-week iteration would be about 7.5 days or 60 hours. If you have historical data for your team and organization, using that would probably be more useful.

Once you have these tasks, map out dependencies and make sure all of the involved teams are aware of them. Then, get these teams talking on a regular basis. If a team realizes that they aren't going to get something done or if they have capacity to do more, get that information out there so the other teams can adjust.

All of this addresses planning. Making work into smaller pieces that can be aligned better to understand when things are likely to be done and helping teams know when to start or integrate work with a mainline branch so you don't have partially integrated work. However, you may still want to take extra precautions from a technical end to keep your system fully stable and integrated.

Isolating work into branches is one option. If your unit of delivery is an Epic, then you would branch per Epic. Every time the mainline branch is updated, be sure to merge it into the Epic branch to resolve conflicts and ensure all tests continue to pass. When the Epic is fully complete, it can be merged in to the mainline branch and fully integrated with the rest of the system.

You could take a slightly different take on branch per Epic by deciding where each change should go. This could even be done on a service-by-service basis. Instead of waiting for the Epic to be done, decide if integrating each story or task would keep the service in the desired state and integrate as soon as possible.

Using feature toggles or keystone interfaces can also help you integrate sooner while maintaining each service and the whole system in a deployable state. You can integrate partially-finished work while making sure it's not exposed to end users until you're ready. There is some added complexity in making sure that there are no unintended side-effects and to manage the state of the toggles, but you do get more frequent integration and less issues with resolving merge conflicts. This approach also supports trunk-based development.

You can even combine branching, feature toggles, and keystone interfaces together to control exactly when changes go in.

Some plans are just too ambitious for scrum/agile or any development method. You have to choose your battles in a convenient order and selectively do parallel work or risk to do go down on overhead and integration troubles.

A strong and capable product manager can make or break this. But the teams should also push back if it looks like you are set up for failure. You ultimately commit to the tasks so when it looks too complicated or dangerous you either say no or claim more time to work out any integration issues.

The combination of these two should make you super agile and deliver easily in small iterations by independent teams:

• Epics would in principle be broken down into small, independent and valuable stories, that are each implementable in a sprint. Prioritization at story level should help to decide what's in the sprint, independently of the epic's value.

• Microservices are meant to be loosely coupled and independently deployable. So, if E1T1 is completed, it should be releasable, even without E1T2 is not ready.

If you are in the complex situation that you describe, your microservices might not be as independent as they should. And deciding on the team's sprint backlog based on global epic ambitions instead of individual and independent stories, might simply be less agile than desired.

I'd recommend to address the root cause: revise your approach to make best use of independence and self-management of each team.

understand whether this is happening in the industry as a practice

That reminds me of something I saw somewhere, "I used to think Dilbert was satire, now I know that it's a documentary." Yes, this does happen other places, quite frequently, but not because we've figured out how to make it work, just because we also have confused management and bad ideas.

It sounds like in this case, though, Scrum is working as intended. Scrum (or any methodology) doesn't fix problems. What Scrum does is give visibility into problems, so that you can recognize the root causes and fix them with real fixes.

With shorter iterations and dedicated time for retrospection you can make incremental changes, test them, see if they're helping in the way that you want, adjust as necessary. It also makes certain problems much easier to document, such as in your case--it's easy to see that you have too much interdependence between teams/services and as a result work is being frequently blocked. Being able to document this is a step towards getting management changes and a step towards being able to demonstrate if changes are/aren't helping (e.g. when management says they will change and then they don't).

So this isn't failed agile quite yet, failed agile is when you've documented all of the problems and it's still impossible to change them...then it's better to go back to whatever patches and bandages work best with those problems in place. (In this case, that would probably be treating all of your services like they are one monolith and your teams are all one big team.)