In "Why Doctors Hate Their Computers," Atul Gawande describes the frustration medical professionals experience when the requirements imposed by the electronic health records (EHR) system they must use to annotate their patient interactions get in the way of those same patient interactions. Sumit Rana, a senior vice president of EHR company Epic, called one of the more frustrating problems "the Revenge of the Ancillaries." Gawande wrote:
"In building a given function—say, an order form for a brain MRI—the design choices were more political than technical: administrative staff and doctors had different views about what should be included. The doctors were used to having all the votes. But Epic had arranged meetings to try to adjudicate these differences. Now the staff had a say (and sometimes the doctors didn't even show), and they added questions that made their jobs easier but other jobs more time-consuming. Questions that doctors had routinely skipped now stopped them short, with 'field required' alerts. A simple request might now involve filling out a detailed form that took away precious minutes of time with patients."
This problem can manifest in any system with sufficient automation combined with poor visibility into the overall burden that its requirements impose on users, including DevOps implementations.
A "Revenge of the Ancillaries" problem can arise in any system with the following criteria:
- Automation in the system ensures requirements are enforced before an action can be completed
- It's easy for users of the system to add a new requirement
- The composite burden imposed on other users is not made known
DevOps practices force organizations to embrace new disciplines of automation. These facilitate adding pre-deployment checks, such as automated unit tests, which can halt deployment. Organizations typically have individuals or departments dedicated to ensuring that security, compliance, or business requirements are met.
DevOps pipelines encourage expansion to activate DevSecOps, DevComplianceOps, or DevBizOps pre-deployment checks. But most pipelines provide little, if any, visibility into the burden that a system's cadre of pre-deployment checks impose on developers and operators working to deploy their applications. This makes DevOps pipelines susceptible to a Revenge of the Ancillaries problem, and steps must be taken to prevent it.
A key observation in Gawande's description was that doctors frequently did not attend design meetings. Thus, the first step in preventing this type of problem is to ensure open communication among all stakeholders involved in the maintenance and use of the system. Most DevOps guides, including The DevOps Handbook, make recommendations to achieve this level of communication.
Second, individuals responsible for ensuring code compliance must meet with developers, operators, and other compliance representatives to understand and negotiate the overall burden imposed. Site Reliability Engineering advocates for a technical specialist dedicated to working with all stakeholders to enhance the overall system to ensure specific levels of reliability while addressing compliance concerns.
Third, novel ways of providing Fast Feedback, a key DevOps discipline, must be developed. Before a user deploys a new pre-deployment check, they must be notified about the burden it will impose on other users. This feedback can come in a variety of forms, such as receiving a list of other applications that the new pre-deployment check would take offline or prevent from updating.
Another form of Fast Feedback can come in the form of metrics of all types of risk, as advocated by Site Reliability Engineering. These metrics would measure the risk posed by non-compliant software remaining in production, the risk of lost revenue or productivity imposed by an outage, and the opportunity costs associated with preventing the flow of new features for an application. They could be made visible in dashboards, and used to develop error budgets based on service level objectives tailored to each application. The deployment pipeline could ensure that developers and operators can continue to push out new features until a certain error level is reached. It would then prevent further deployments until developers bring the error level to an acceptable level by addressing technical debt. Additional fast feedback is achieved when developers and operators can identify the individual(s) responsible for a specific pre-deployment check and negotiate a temporary (and instant) reprieve for the specific application.
Finally, it is important for all users in a DevOps pipeline to realize that everyone is on the same team. They all want to ensure the deployment of high-quality software that delights their users, provides important services, or increases user productivity while mitigating risk to the institution as a whole. This is where the DevOps discipline of Blameless Post Mortems can play a critical role. Users must be willing to provide useful, constructive feedback and hear feedback without getting defensive. This will ensure that everyone's needs are met by the DevOps pipeline.
Comments are closed.