4

Enterprises are increasingly adopting DevOps methodologies for development of software. A couple of common outcomes of this are that:

  • Application teams are more cross functional in nature, with operations and support staff working in self-contained teams alongside developers.
  • Developers have an increasing support role... the "you wrote it, you fix it" culture. Increasingly they're on-call.

I'm seeing this creating a challenge. Development teams are structured this way primarily because it delivers more rapid software innovation, and hence value to the company and to the customer (new features, new products, etc). They're working to an evolving backlog of feature desires. But at the same time, support issues are arriving.

Facing a queue of two tickets, or ten tickets, or one hundred tickets, the application team has to decide what to do first. If you're working in a structure like this, how do you establish what to prioritise, what to defer?

Improve this question
5
  • 1
    Your second bullet point is extremely contradictory. I think it's true that there's an increasingly blurred line between development and operations in some organizations, but a "you wrote it, you fix it" culture is inherently against the agile principles and I could easily argue against the lean principles as well - shared ownership of the product (all levels of the product - infrastructure through application and services) is essential. If you have a "you wrote it, you fix it" culture in your organization, the answer to your question is different than if you have an agile or lean culture. Commented Nov 25, 2016 at 0:50
  • 2
    The further removed maintenance is from development, the less accountability there is, less feedback, less credibility, and more politics. I often see specialized roles that are not full-time development pick up responsibility for support, including dev ops and QA. Dev ops is supposed to be part of development. If you say that dev writes code and dev ops fixes code, you don't have dev ops. You have a bug creation team and bug fixing team. Commented Nov 25, 2016 at 2:40
  • 4
    @ThomasOwens, my take on the "you wrote it, you fix it" culture is not a case of "here's a bug, let's figure out who wrote that code and make them fix it". That's taking it too literally. It's simply a culture of developers treating (them doing) maintenance and support as important as new feature development, ie one of the core principles to creating a dev-ops culture. This is orthogonal to agile/lean; not in opposition to it. Commented Nov 25, 2016 at 6:42
  • @ThomasOwens: I see no mention of agile or lean in the question? Commented Nov 25, 2016 at 8:20
  • @RemcoGerlich DevOps, to be successful, is lean. As you make your processes leaner, you do end up moving toward what is called DevOps. It's one strategy to eliminate waste (in communication between development and operations) and delivering faster. Commented Nov 25, 2016 at 13:21

5 Answers 5

Reset to default
4

If you're working in a structure like this, how do you establish what to prioritise, what to defer?

The same way you prioritize anything else: you always pick the action that maximizes the benefit over cost. If there's a bug that has an easy resolution that lots of users are running into, fix that first. If there's an exciting feature on the horizon that wouldn't take very long to implement, do that next. Then consider rare bugs and difficult to implement features that aren't terribly exciting.

Improve this answer
3

Facing a queue of two tickets, or ten tickets, or one hundred tickets, the application team has to decide what to do first.

I'd argue that it's not the applications teams' job to do this except in a very low-level tactical way e.g. "Critical issue, drop everything now." "We'll do these two bugs together because they're in the same module." "We'll fix these defects at the same time as adding this functionality because it's in a related area."

If you're working in a structure like this, how do you establish what to prioritise, what to defer?

You talk to the business. There should be a regular (how frequently depends on the volume of incoming work - try to have it frequently enough to finish in around an hour) prioritisation meeting with representatives of the business with enough authority to approve and provide priorities to bugs and features.

Improve this answer
3
  • Prioritization meetings sounds like micromanagement. Of course, the business side should be setting the roadmap. But when you get down to the level of individual bugs and feature knobs, the development team should use their own initiative. The higher-ups don't (and don't have a reason to) understand the impacts of a particular bug. Commented Nov 25, 2016 at 3:49
  • 1
    @DepressedDaniel No matter how well integrated a development team is with the business it cannot be fully appraised of the strategy and priorities (unless the CEO is still a developer). I've encountered a lot of developers who think they know better than the business and that's a very dangerous path to tread. I don't think that a weekly/fortnightly/monthly (depending on rate of raising defects/new functionality) meeting is micromanagement. You have to be pragmatic too - there's no point in spending 5 minutes discussing the priority of a spelling error. Commented Nov 25, 2016 at 6:14
  • This is about moulding the general shape of agile around the support process and feature development. Let the business decide the priority of every significant piece of work, based on your estimates of effort. 15 minute jobs get fitted in as & when. Anything significant gets estimated and prioritised on the general backlog and that goes for new feature as well as bugs. Make sure that even the "insignificant" bugs get accounted for somewhere, because 32 15 minute bugs is still a full days effort - so business needs to know nothing is "free" Commented Nov 25, 2016 at 6:22
0

I think you have perfectly highlighted the problem with the 'you wrote it, you fix it' approach. (although I'm not sure that its completely the same thing as 'devops')

Of course in businesses of any size it is not usual for the developers to be able to choose which tickets they work on. Normally you have a project manager or similar who prioritises tasks in order of importance to the business, not in order of important to the developer.

Thus if you 'write a bug' you cant fix it, unless it happens to have been prioritised and you happen to be picking up the next task in the queue.

Although this approach is supposed to motivate developers to write easy to maintain systems, in fact it simply means they also do the job of the helpdesk/support/operations, rebooting boxes running systems they haven't worked on and don't have the power to change.

Obviously the business pays for their time, and there are always some developers who want the extra on call money and are happy to take shifts.

Improve this answer
2
  • I disagree with the claim that the priority way is good. It considers developers a homogeneous resource. While the project manager can assign the task, it is a bad idea for the project manager to put all tasks into one queue so that the next developer who has fixed his current task picks the next from the queue. Instead, the project manager should consider the differing skills of the developers, assigning tasks in an intelligent way. This can mean the bug fixing time can in the best case be an order of magnitude faster. It usually means "if you wrote it, you fix it" but not always. Commented Dec 31, 2017 at 17:06
  • well, there is an argument for that. However, its generally considered that turning devs into units of labour, is good for the business as a whole. Precisely because you don't have to worry about that one guy being available/overloaded. Maybe I should add 'In the context of agile...' Commented Dec 31, 2017 at 17:27
0

Operations and Maintenance (O&M) mixed with new development is not a new problem. DevOps doesn't fundamentally change how the company prioritizes work. However, a good DevOps approach will provide you more feedback to intelligently triage problems and discuss how to properly architect the problem out of your project.

You've got different levels of O&M events that require a different response:

  • Application is Dead: This is an all hands on deck event that requires a "war room" to collaborate and get the application back up and running as quickly as possible. The solution here is never optimal, but the company needs to understand what happened so that they can architect a better solution and prioritize it in the back log. When the application is down, the company isn't making money.
  • Major Impact: When a problem affects a large group of users, you might need to roll-back the release that introduced the problem and go back to the drawing board. If you are practicing DevOps this should be possible.
  • Lots of error messages in the logs: Many times this can happen and users are not directly affected. It could be that the log messages are reporting things as errors that are not errors, or it could be that the system is dropping information because of the errors. The problem needs to be identified, added to the backlog and prioritized next to the new features.
  • Intermittent loss of performance: Gather as much information as possible, characterize the problem, add it to the backlog and prioritize it along with the new features.

One thing that is quite possible, particularly with applications that self report crashes, is that there are several reports of the same problem. It's the O&M staff's responsibility to escalate the actual problem and report some statistics as to the number of users it affects. Again, this gets prioritized along with new features.

Getting to the point where this is a smooth process takes time. You will likely have to beef up your O&M staff, and possibly introduce more monitoring tools. Having multiple tiers of support you can specialize some of the support staff to analyze the monitoring tool reports and provide feedback on the health of the system and where it is weak. The information provided from this group of O&M staff highlight areas where the design is not well matched with the application as a whole. The problems are deeper and require a bit more thinking on how to solve appropriately. They would work closely with the development team to make sense of the information and to assist with planning.

Improve this answer
-1

One method is to use error reporting software. Like testing software, it tracks errors and provides reports which can be used to prioritize bugs to develop a way to track defect fixes. This can provide you with the tracking you need and also allows you to choose errors to defer due to limited impact or resources. You can incorporate the reports into a work order system or your resource planning system for assignment. Examples of this type of software are Applications Manager, Datadog and Bugsnag.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.