Guide to Engineering Operations

What is Engineering Operations?

Engineering operations is putting together tools and processes to help engineering organizations solve the right problems and solve them efficiently.

Creating a high performing engineering organization requires building tools and processes iteratively, pragmatically and at the right time. If engineering operations investments are not done iteratively, pragmatically or in a timely way, these investments will likely result in negative returns. Your organization's effectiveness and efficiency will not be improved.

Too vague? Let's talk about a concrete example: Do you remember your engineering onboarding experience?  

This example might resonate a bit more with people who were fortunate enough to join a startup that ended up hiring more engineers after they joined. One of the areas falling under engineering operations is managing engineering onboarding. Ramping up new engineers quickly to start solving your customers' problems has a direct impact on your business' bottom line.

One onboarding experience I remember vividly is the day I joined a startup with less than 10 people as their first engineering hire. Over the next 10 years the engineering team grew from 2 to 500+. Here is a summary of how engineering onboarding progressed over time:  

I remember having to assemble my own computer on my first day. I  was eager to get things up and running and eager to start learning the codebase. As you can imagine there was no onboarding process to speak of. The founders built some stuff and there were code repositories, but there were no instructions for setting up the development environment, or for testing and deploying code. There were no design or architecture diagrams. Lots of tribal knowledge was hidden in places which I would figure out slowly over the next few months. As we started hiring more engineers I took turns walking new hires through setting up their development environment, giving them an overview of the architecture and getting them familiar with the tribal knowledge. We were already stretched thin, working  long hours and shipping product features as fast as possible. Onboarding new engineers was happening more frequently and taking many hours from our days. It was a pain and it had to be mitigated!

We started putting together some documentation (a checklist) for new engineers. The motivation was to  help new engineers ramp up quickly and become productive but also to reduce the load on ourselves. Organizing the information around what the new engineers needed was an iterative improvement for our onboarding process and definitely helped both the new engineers as well as the engineers responsible for onboarding them. These checklist documents evolved and got more detailed over time. We built some tools to automate setting up local environments. We created some documents outlining design decisions, and the architecture of our systems.

Our improved onboarding process still had a few shortcomings:

• Our onboarding checklists and presentations got stale pretty quickly, despite our best efforts. We added to presentations and asked new engineers to update checklists, but keeping onboarding documents up-to-date was not our primary focus, and as at most startups, things changed very quickly.
• Our onboarding was not centralized. As the engineering team grew each code-centric team (api team, ui team) created its own onboarding checklist. Engineers who were just joining the company didn't know about common processes. They didn't know how different systems were interdependent, and they had no view of the overall system design and architecture. They had no knowledge of common engineering expectations, scary production incidents from the recent past, or land mines to watch out for.

We eventually assigned someone to own engineering operations and put together an onboarding curriculum for new engineers joining our organization. Onboarding evolved into a multi-week process that involved not only getting an overview of our engineering organization, walking through systems owned by different groups, and joining a deep-dive presentation, but also running through a product exercise to understand how our customers used the systems our engineers built. It was certainly far from the humble beginnings where I had to build my own computer and figure out all of this myself, but this story is not so uncommon and is certainly a fun one to reminisce about.

Not all startups get to a phase where they have to hire someone full-time to create an onboarding process as part of an engineering operations effort, but I believe the first iteration of onboarding documents and tools should start as soon as there is an engineer . Even if you have a team of one and don't plan to have a new hire for a year, the first iteration of onboarding documents should be done. This will not only start the iteration cycle to keep improving the onboarding process, but it is also important from the "bus-factor" perspective.

Areas Under Engineering Operations:

Engineering onboarding is one area that falls under engineering operations but there are many others. Here are a few to keep in mind:

• Defining engineering roles, levels and expectations: This means defining a set of responsibilities that you will hold every engineer in your organization to. It will not only help you coach and grow engineers in your engineering organization but will also help you hire the right people at the right levels.
• Creating an engineering hiring and interview process: Hiring good help is never a secondary task for a good engineering organization. Invest in collecting the right signals during the interview process, evaluating candidates and ensuring candidates walk away with good experiences and thoughts about your company regardless of the outcome of the interview.
• Implementing tools and processes around on-call rotations and reliability management: Teams who don't take reliability seriously can't build good software. Define expectations for how teams manage on-call rotations, production incident postmortems, and prioritization of follow-up action items.
• Establish/Improve engineering roadmap planning & execution speed
• Organize and contribute to community events such as tech talks, and conferences

Why Should You Invest In Engineering Operations?

Investing in engineering operations is usually deferred at startups. This is mainly because anyone who can work in engineering operations is usually busy with many other engineering related responsibilities, but a journey of a thousand miles begins with a single step and the first step must be taken as early as possible.  

Starting and iteratively improving your engineering operations will have a huge impact on what your engineering organization will look like in years to come. On day one it will set the right tone for anyone who is just joining your team. If done right your engineering operations will ensure that every engineer understands your engineering philosophy, your values and your expectations for them.

How to Start Your Engineering Operations

What you do and how you do it will really depend on the current stage of your engineering organization. As mentioned at the beginning of this post, your engineering operations processes need to be iterative, pragmatic and timely. These attributes help ensure that you are implementing a process with the right reasoning, and with a bottom-up approach that is actually solving a real problem. A process mandated from the top down which does not solve a real problem won't get adopted by engineers in your organization.

If you have multiple engineers joining your company every month I highly recommend hiring someone full-time to oversee you engineering operations. Hire someone who is technical, who understands how engineering teams work and what engineers do day-in and day-out, and who likes working with people. If they are passionate about helping teams, then that person can bring so much value to your engineering org.

If you are not hiring enough people to justify a full-time position, I suggest having an engineering manager or a technical project manager take ownership of engineering operations part-time. They can implement chosen areas of engineering operations.

Finally, you can also consider getting outside help if your team does not have the bandwidth to start iterating on areas of engineering operations. This person should be someone who has gone through the growing pains of engineering organizations in the past. It should be someone who is technical and values improving engineering operations iteratively, in collaboration with you and engineers on your team.

Key Takeaways:

Engineering operations is putting together tools and processes to help engineering organizations solve the right problems and solve them efficiently.

You should figure out engineering operations areas you need to invest in and start implementing them asap. Iterate, iterate, iterate as your engineering org grows, with re-orgs and changes over time.

Assign an owner to oversee engineering operations. This should be someone who is technical, who understands engineering teams, who is experienced with building and maintaining systems, and who is passionate about help people.

Your investment or lack of investment in engineering operations will define the type of engineering culture you end up with. Be in control of it.