The Three Ways of DevOps with Clear Explanations and Real-Life Examples | Alpacked

It’s impossible to enjoy the benefits of DevOps without applying this software development approach right. The Three Ways of DevOps explored in this article might be your guiding star in the journey towards DevOps maturity.

The main problem with DevOps is that there’s no one-size-fits-all strategy to implement it. As a result, the concept is surrounded by lots of myths: a whopping 78% of organizations fail to get DevOps right. 

In his novel (yes, it is a novel) “The Phoenix Project," Gene Kim introduces three core rules of DevOps, which he calls the Three Ways. These three principles are a detailed representation of the main characteristics of DevOps maturity. They build upon each other and can serve as a guide to DevOps transformation. 

Ironically, most articles dedicated to these three principles are too theoretical, and you won’t get a clear idea of what DevOps actually is. You may think, “I've seen so many articles about this before, why should I care?” Well. This one isn’t just another generic about Three Ways of DevOps. Armed with our extensive DevOps experience, we’ll spruce up every Kim’s statement with clear explanations, examples, real-life cases, and research.

In the technology value stream, work typically moves from the left (Development) to the right (Operations) side, from the conception of the feature to its deployment in production. 

According to Gene Kim’s Principles of Flow, mature DevOps organizations overcome this route in the fastest way possible, delivering value to their customers quickly. Lead Time—the total time between the request from the business to the functionality deployed and available in production to the customer—is the key metric here. To keep it to a minimum, you should have a lean and consistent project delivery pipeline. Here’s how you can achieve this.

Reduce the size of your work batches

In software development, a batch is a piece of work that moves between 3 stages, such as development, testing, and deployment. In waterfall development, where the batches are huge, weeks or even months of persistent work separate these stages. Too often, serious bugs made at the beginning of development are discovered only during deployment when release is looming over the horizon. 

This can be easily avoided if you shrink your batches, meaning that even the smallest change your team commits to the version control system is integrated, tested, and deployed. Case in point, Amazon does at least 50 million deployments per year (the estimation was made back in 2014, that’s why “at least”)—the principle of small batches in action! As a result, they don’t stumble upon critical bugs that affect the entire solution late in the development process.

If problems arise, they are easier to fix, sparing development teams from the technical dept in the future.

No doubt, keeping your work batches small isn’t that easy. There are still lots of dependencies that can slow down the work. For example, you may have all tests running for each PR. Or the build time can time a while if you use Java or similar languages. Or you are updating resources using AWS CDK can also make things slower. But in any case, the main idea here isn’t making small batches where it’s impossible. It’s more about the mindset and helping your developers overcome the old era of “wait-till-Thursday-and-push-all.”

Limit work in process (WIP)

This rule is closely related to the previous one: large batches equal a large number of undone tasks. As a result, when problems arise (and they always arise in the development process),  they break your processes: you have to rush between multiple unplanned tasks, getting into prioritization traps. In this case, technical debt is just inevitable. 

Avoid multitasking to prevent a catastrophe resulting from such a regular thing as problems during development. “Stop starting, start finishing,” as the Kanban method states. This means: instead of adding a new task, think twice. Maybe, there’s something that needs to be done? It’s easier to control a smaller number of tasks, after all. And if the delay is happening, try to figure out why. 

Give visibility

You can’t control what you don’t see. If you don’t know who is responsible for what, it’s impossible to control and limit your WIP. Unfortunately, visibility can be challenging in software development, where work can be done or reassigned to the other team with a click. 

To achieve the needed level of visibility, you can use kanban boards where work is presented on cards, moving from the left to the right based on its status. Thus, everyone involved in the project can see which tasks are currently in progress at each phase, where everything goes smoothly, and where the work gets stalled. Besides, the tasks become easier to prioritize, manage (you can assign and reassign them to the other team), and measure, increasing your market speed. 

Reduce handoffs

Throughout the development process, the work moves from one contributor to the other, from one department to the other, for functional testing, environment creation, server administration, networking, and other tasks. Each handoff requires clear communication, ticket creation, scheduling, prioritization, and so on, potentially creating delays that remain the top constraint to successful project completion. 

So, your task is not just to make each handoff seamless. You should also reduce the number of times the work is passed between team members. This can be achieved through reorganizing teams and automating processes. For example, Etsy automated its deployments with the help of Deployinator—a tool that offers one click-deployments. As early as 2015 (!), it helped them deploy code about 40 times a day. 

But how should you reorganize teams to reduce the number of handoffs? Build smaller teams comprising a few developers, a QA expert, and an operations engineer like successful companies do. It helps them remove silos and thus avoid unnecessary handoffs and eliminate the it-works-on-my-machine situations. 

So, whether you are a startup or a big enterprise with multiple departments, it’s important easy to stick to this everyone-works-with-everyone setup. Or if you are too big for this arrangement, you should at least monitor the communication between departments.

Eliminate waste

In DevOps, waste is any action that doesn’t bring value to the development process and just hinders the workflow. It can be anything: manual work that can be automated, monolithic infrastructure that interrupts the flow, tools that are never used, unnecessary features in a product, unfinished work, incomplete documentation, motion waste (resulting from too many handoffs), and beyond. So, you have to identify and remove those bottlenecks from the value stream to make the process more efficient. 

You may read and wonder, how can I make sure in the cloud setup that, say, some manual work is done? Well, in one of our projects, we restrict any CRUD-type operations on the infrastructure unless User-Agent is Terraform. It took few months for people to get used to it, but believe us, it’s worth waiting.

Hunt your constraints down 

Eliminating the system’s constraints will help you considerably reduce waste. Based on our experience the most common things, teams might be dealing with the following ones: 

• Environment. You can’t deploy code on-demand since creating the needed environment takes weeks or months. A countermeasure can be self-sufficient environments available to you whenever necessary. 
• Manual deployments. You can’t deploy fast since you do it manually, or the process involves 1,500 error-prone, manual steps. Automated deployments can be the way out. It may sound crazy at first, what we usually do is try to make automated deployments according to the following schedule: first, once a week; then, once in 3 days; and so on (based on how it goes.) This allows the upper management to get used to the schedule and see all the benefits.
• Manual testing. You can’t deploy fast because each deployment requires setting up a test environment from the ground up, which can take weeks. You need to automate your tests. Without this, the tip above is useless. Yes, automation engineers cost more money, but there is a reason for that: if you have your tests automated and code deployed more often, you are far ahead of your competitors in terms of improving your product
• Tight architecture. Every change made in your code affects the entire system, making any update a challenge and blocking fast deployments. That’s why you need to go for a more flexible architecture. If you look at every successful DevOps journey, it begins with migration to the cloud and microservices architecture.

Once you identify a constraint, you need to figure out how to eliminate it, then actually eliminate it, scan your system for other constraints afterward, and repeat the process if necessary.

While the flow of our value stream (according to the Principles of Flow) goes from left to right, the feedback moves in the other direction, from right to left, forming a loop. It’s known as a feedback loop. 

The central idea behind the Principles of Feedback is to detect and solve every single problem in a product as quickly as possible. It’s a continuous process. This way, problems are easier to fix and in most cases, can’t affect the entire system that much. Here’s how you can achieve this. 

Swarm the problem as soon as possible

Once a problem (even the smallest one) occurs, don’t wait until it becomes huge and puts the entire system on hold. Instead, mobilize whoever you need to resolve the issue as soon as possible. 

Let’s take Amazon, for example, which uses Toyota’s Andon Cord principle. If their customer representatives recurrently receive complaints on the same problem, they immediately notify management (pull the cord). If the issue can’t be resolved soon enough, the production is paused, and the product is removed until the problem is fixed. This way, the tech giant prevents the problem from spreading and avoids the technical debt.

Learn from problems

Solving problems once they occur is a great practice. But it’s not enough if you do not learn from these problems. You need to identify what caused the issue and find a way to improve the process to prevent it from happening again. Thus, you reduce the waste, keeping your feedback loop as small as possible. 

The key point here are the actions you take to mitigate the issues in the future. For example, when testing the code or infrastructure, we stick to the following principle: if it happened more than one time, there should be a test case for that. Does it need more manpower to achieve this? For sure. Is it really that useful? Oh yes. Just try recalling how many situations you had like “Oh, this issue again, do we remember how we fixed it last time?”

Make your quality control efficient and error-free

It’s impossible to build a high-quality solution without reviewing and approving every single line of code before deployment. At the same time, multiple inspection steps might increase the likelihood of future failures and even decrease time-to-market. So, how do you prevent quality control from working against you? Here’s what Gene Kim suggests: 

• Keep quality control as close to the source as possible. Avoid requiring approvals from a manager two or three levels above who doesn’t have enough expertise in what he’s approving. A peer review of development can be much more efficient and less error-prone.
• Avoid submitting large batches of work for review. 
• Keep your documentation clear and up-to-date. 
• Minimize manual approvals and automate everything you can. This way, you’ll less depend on other people and reduce the number of handoffs, making things move faster. For example, Etsy automated the entire testing phase. Besides, they developed Kale to detect outliers within data.
• Check early. If you set control early on the value stream, you’ll get feedback and identify problems sooner. 

The third way of DevOps  is all about learning from the insights produced in a feedback loop. This also implies improving the product and workflow based on the lessons learned. 

In fact, continuous learning and improvement is the key ingredient of DevOps maturity. No wonder 78% of IT professionals in Atlassian’s survey learned new skills on their DevOps journey. But what does Gene Kim imply when talking about continual learning and experimentation in a DevOps environment? Here are the key takeaways. 

Eradicate blame 

There are still organizations where those who make suggestions or highlight problems are viewed as whistle-blowers, while mistakes are the reason for punishment. Such a culture of fear and mistrust makes organizational knowledge sharing and learning impossible. You should avoid this if you are pursuing DevOps maturity. 

Make sure everyone in your team feels comfortable raising their hands to ask a question, make a suggestion, share their insights, as well as to easily say they have a problem.

In our projects, for example, we try to detect situations where people are afraid to take responsibility and help them speak. Eventually, they see that there is no issue in healthy discussions, and it usually turns into small brainstorming sessions that allow engineers to find even more useful details and ideas to be implemented.

Analyze and improve daily processes

“Improving daily work is more important than doing daily work,” Mike Orzen, the author of “Lean IT,” once observed. The trick is that when you don’t change anything in your processes, they don’t remain the same but degrade over time. The technical debt builds up, problematic areas in your code aren’t going anywhere, and new tools keep mushrooming—here’s how doing nothing creates collapse in your processes one day. 

Given that, it’s paramount to regularly step away from your routine to analyze how you approach tasks, fix defects, and pay your technical debt. For example, Gene Kim’s engineers have regular kaizen blitzes—special meetups where they self-organize to address any issues they want. Eliminating these minor issues allows them to see bigger problems that remain otherwise invisible. 

When we do the discovery for our projects, we don’t conduct knowledge transfer sessions in the beginning. We first reverse-engineer what clients have. As a result, find a lot of technical debt since we don’t know why things were done in a way they are done. This allows us to find chunks of work that allow to guide the client’s team in the right direction.

Rehearse failures

According to Gene Kim, truly mature DevOps teams are prepared for failures of any kind. Even if total prevention is impossible, they have at least a backup plan at hand. But how do you achieve that level of resilience? 

By failing continuously. Analyze every piece of your system and challenge it to test and improve its resilience. Case in point, Netflix uses “Chaos Monkey” to test the stability of its IT infrastructure. The software tool simulates failures of random service instances to see how a system shifts its resources. 

Our DevOps engineers know how to quickly act in different failure scenarios. And we train them to do so right from the doorstep. For example, when they learn, say, Kubernetes during their probation period, we use “Chaos Monkey” to disrupt some services, see how a person eliminates the issues, and learn from them.

Foster experimentation culture

If it ain’t broke, don’t fix it. Chances are you’ve heard this rule million times. And it works—but to some extent. In fact, if we use the same tools and stick to the same processes, we can’t expect higher product quality or shorter lead times. Or as Vaas from “Far Cry” said, “Insanity is doing the exact same thing over and over again and expecting things to change.”

It all comes to change management in organizations. For people, it may be stressful enough to try changing the workflows unless some big issues are on the horizon. We deal with such things on a monthly basis, and it’s part of our work to show people how brighter and easier the future might be if we try experimenting a bit. 

That’s why constant experimentation and looking for a better way of doing things are a must. Anyone involved in a project should chip in, from executives through team leaders to junior engineers. And by the way, we are preparing an article about change management in companies, so stay tuned.