How can you create a set of shared beliefs that steers everyone in the right direction?
In early April 2018, we launched Skyscanner’s first set of guiding principles for engineering, internally known as ‘Our Engineering Principles’. Today, this allows all of our engineers to follow a shared set of beliefs and collaborate effectively across our teams. In this article, I’d like to give insight into why we needed them, what they are, and explore some secret sauce ingredients that keep them alive.
What are Skyscanner’s engineering principles?
Principles are defined as: a fundamental truth or proposition that serves as the foundation for a system of belief, behavior, or a chain of reasoning.
The following principles work because we collectively agree on what good looks like as an engineering team. Adhering to these principles results in us providing the best in class experience for our users.
- We have a clear definition of success for every piece of work.
- We ship multiple times a day and deliver customer value week in, week out.
- We use design reviews to validate every significant change.
- We deliver our products using our defined technology standards.
- We peer review every change.
- We cover all changes with automated tests responsibly.
- Our definitions of done include being live in production, responsibly.
- You build it, you run it.
- We own and are responsible for the data we produce.
- We write with inclusivity, diversity, and accessibility in mind.
If you’d like to read more about our principles we have them open sourced on GitHub and they can be found here.
Why create engineering principles?
My worst nightmare would be turning up every day to systematically review, scrutinize, micromanage, and generally ruin the hopes and dreams of those trying to deploy something extraordinary for our users, among what can only be described as chaos.
So, how do we avoid that nightmare and have hundreds of people all on the same page?
First, let’s think about the following question:
As a leader, manager, or individual contributor, how do you gain mutual, two-way trust with your engineers on what ‘good’ looks like?
Very quickly, I’d guess you’re likely to be somewhere in the order of magnitude of having about 30 things just off the top of your mind.
Do you have those things written down?
If your answer is yes, fantastic! You are already ahead of where we were at the start of 2018. Once we recognized this gap, we put our fingers to keyboards, wrote the principles, and obtained alignment of hundreds of people in a heartbeat.
No – not quite. It was and continues to be a journey today – a journey of reminding, measuring, learning, adapting, debating, and ultimately continually collaborating to commit to our shared beliefs.
What’s the secret sauce to alignment on engineering principles at scale?
I’ve tried to boil this down to my three favorite ‘ingredients’ that create the perfect sauce.
These ingredients were used at the core of the initial inception in 2018, and they’re still simmering away today, helping us ensure we don’t stand still.
Ingredient 1: collaboration and consultation on ‘What does “good” look like?’
At Skyscanner, there are always people willing to help no matter what. Of course, creating an in-person working group that includes most of the organization has its challenges, such as finding an appropriately-sized meeting room, acquiring several sharpies and post-its, and then there’s the queue for the coffee machine to contend with.
To make things easier, we formed a smaller working group of about eight people, more commonly referred to as a ‘guild’; this guild still exists today.
A small working group is a vital aspect of the overall success of our engineering principle adoption. Consultation is an essential tool in any rollout like this in creating an inclusive environment; failing to do it well leads to people not feeling heard, which leads to disengagement and insufficient buy-in.
Initially, the team spent around three or four months gathering feedback and listening. By doing so, patterns emerged. It became easy to start grouping things. Finally, a draft of principles was formed and agreed upon. The working group shared a draft with the wider organization to ensure that everyone could once again offer feedback. Over time, some of the details have changed; the core principles, however, remain the same. It is a continual feedback and learning process where anyone in the organization can make suggestions. Feedback is gathered during these sessions and passed to the current group that looks after the principles today.
Our onboarding and training practices cover the principles in-depth with tangible examples. For example, ‘peer review every change’ sounds like a no-brainer – but what about simple config changes? One of Skyscanner’s most significant outages was due to a simple config change. We went through this particular example’s learnings, and the outcome we got was that it wasn’t peer-reviewed. The change didn’t have the same level of diligence applied as a more complex code change. Today, we now regard config as code, and we use the principles as we do with any change going to production.
Ingredient 2: accountability – how do you ensure everyone applies them practically?
Crucially, for economies of scale, you have to enable people.
When hiring lots of engineers at the same time, globally, you want to set them up for success and give them agency to get on with things. We’ve been investing heavily in our enablement platform by offering ‘off the shelf’ adherence to our production standards for any new or existing services. Doing this gives our engineers a head start, and it’s one less thing to think about.
That’s the technical side of things, but there’s also, of course, the human side to it all. Everyone (we’re all leaders) plays a crucial role in ‘empowerment’ rather than ‘enforcement’. Biasing towards our principles and values every day, we all create role model behavior that we want to see reflected in each other.
An easy example of empowerment is if something does not have sufficient test coverage or quality, teams have the power to delay shipping the project to ensure they can correct this. In our case, this did not mean that our production line slowed down. It was the opposite: we failed faster as a result of growing test coverage. In balance with the principle to ensure we are ‘delivering value to our travellers [users]’, testing it appropriately made sense to avoid their disappointment and ensure teams can balance quality vs. quantity sufficiently.
To hold people to account is to afford them the trust, space, help, and encouragement to do what’s right. When someone pushes back, we ‘seek to understand’ why that is.
Ingredient 3: understanding success
I’ve opted to title this ingredient ‘understanding’ rather than ‘measuring’ as principles are about how well you understand and adhere to them.
There are many practices for helping to drive understanding; I’ll share a few of my personal favorites which we use widely today.
I think this quote encapsulates one such practice:
‘The engineering principles are a ubiquitous language that we use across engineering at Skyscanner, which means we’re all on the same page in terms of what good looks like. If you examine most incident learning debriefs, you’ll generally see a gap or misunderstanding in applying engineering principles. Of course, mistakes happen, but the principles give us a lens to look through when examining incidents and help to inform actions and key learnings.’ – Graham Martin, Senior Engineering Manager.
Whenever we have an incident, we run a process called ‘Incident Level Debrief’ (ILD, internally), where our teams self-validate the change made against the engineering principles. In effect, they check the change’s adherence against the principles, highlighting how they can grow their understanding and learning.
Another practice (and principle) is the design review (DR). Even in the most trivial changes, a DR can allow you to see so many things you hadn’t considered in the fraction of a second warranted to think about this task before moving on to think about how quickly you can get a coffee.
My favorite story to tell is about a team that pushed on a ‘simple’ change to our login flow with, ‘If it’s that easy, then it will take us no time at all to do the DR?’
‘Awesome response,’ I thought at the time, with a feeling of pride and admiration.
A point well made by the team: the most trivial of changes still warrant diligence if we adhere to our beliefs.
The team (6-8 engineers) avoided 12 weeks of working full-time on the wrong thing. A solid victory for our engineering principles.
Key takeaways
To have genuinely autonomous teams at scale and an effective set of engineering principles, you need to consider the following:
- Have a well-established and documented understanding of what ‘good’ looks like: a shared set of beliefs.
- To allow for accountability, you have to empower people and afford them time to do the right thing when identifying something that doesn’t adhere to those beliefs.
- Iterate on improving the understanding of those beliefs every day, regardless of your level within the organization, and feedback when you learn something new.
