SupportOps

The concept of DevOps, which focuses on improving interaction between Software Development and IT Operations teams, has made the rounds in tech circles for the past five years. At Acquia I try to foster interactions between my team and Hosting Engineering to that end; there’s some room for improvement, but we certainly have made progress.

As tech companies grow from an MVP and a handful of staff to a mature product and hundreds of people spanning many teams, the task of developing the product and running the product can be handled by separate groups of people. Therefore it makes sense to develop process and workflows to ensure that the needs of both groups are met (hence DevOps).

However, there’s a third party in the business that isn’t yet mentioned: the Support organization. They are the first responders when something goes awry in the customer experience, and in tech organizations (like Acquia), can be separate entirely from Operations.

The separation makes sense; it allows for specialization and therefore efficiency since a team no longer has to juggle infrastructure deep dives and a customer conference call at the same time. Support can deflect the common problems, freeing Operations to tend to the more difficult ones.

However, there is a cost. With such specialization comes separate values, language, and culture. Communication breaks down: Support, representing the customer by proxy, gets frustrated when Operations doesn’t align with the percieved urgency due to their own documented processes. Operations gets equally frustrated when Support doesn’t understand how the product works or how to provide key information when escalating customer issues. The situation gets worse as the customer base grows: depending on the staffing levels of either team, one can become a bottleneck. The common scenario is that Operations is understaffed and directly suffers from the company’s success: the rate of issues entering the ticket queue increases to the point where it is impossible to handle them all in a timely manner, and if Support doesn’t have the expertise/means to assist with infrastructure problems, Operations eventually collapses under all of the weight. Ultimately, the customer suffers.

Conflicts between Support and Operations can be similar to conflicts between Operations and Development; just as dev teams can throw broken software over the fence while ops teams are left with a constantly-beeping pager, ops teams can make it difficult for support personnel to get the customer what they need in a timely fashion, leaving them with angry phone calls.

I therefore propose the concept of SupportOps, which aims to realign Support and Operations such that they effectively work in tandem to delight the customer base. The primary aim is to improve communication between the two groups to better equip them to do their jobs.

I don’t have all of the answers since not every tech company is the same. However, I provide the following insight/values:

The Support Organization must:

• Have a basic understanding of the product (the application, the hosting stack, etc) and its components
• Be able to gather basic information about the product state (server status, app status, etc)
• Escalate issues to Operations effectively (customer impact, detailed information about the issue, steps to reproduce)
• Understand and enforce Operation’s processes and scope (reference: http://www.opsreportcard.com/section/2)

The Operations Organization must:

• Be agile to prioritize/complete their work according to customer impact
• Communicate details regarding customer issues clearly and timely
• Create automation and tools to increase efficiency in issue throughput
• Provide Support documentation/training resources/feedback to improve product understanding

SupportOps is an interesting concept to me and hopefully isn’t just another buzzword to throw around and argue about what it actually means for the next five years. :)