00:00:00.000
Our next speaker is Michał Łęcicki. You can find him after his talk and ask him about his last name—it’s really difficult! He has studied psychology but is actually a longtime developer with experience in various teams and projects, regardless of their size. Today, he will talk about how to manage mistakes in teams and projects using his knowledge from university. Welcome, Michał!
00:00:41.280
Yes, so my last name is Łęcicki, but you don't have to worry about pronouncing it. I am a software engineer at Visuality, but today I'm going to focus on psychology. Before we start, I have a story for you. In April 2019, a new trend emerged in Europe: electric scooters on the streets. Perhaps you remember that various mobile apps were developed to use them. I happened to be a developer at one of those startups, and as you can imagine, the startup needed to grow rapidly. We were adding new functionalities every week as new developers joined our team. We were expanding to new cities and markets every month.
The project was challenging due to its microservices architecture. At that time, we had about 20 to 25 microservices, each written in different programming languages and maintained by various teams in different time zones. The way microservices communicated with each other varied, adding to the complexity. To make things even more challenging, we used trunk-based development instead of the traditional Git flow, where you create a pull request for a feature branch before merging it into the main branch. In trunk-based development, you commit directly to the main branch, which saves time. If you waste time opening a pull request and waiting for a review, it could mean losing valuable opportunities, especially in startups. Therefore, it's often better to ship code quickly and fix issues later.
Additionally, we used Golang instead of Ruby because Golang is faster. Consequently, we were working on a microservice that was part of a feature called automation. The idea was simple: you enable the configuration and set conditions, and if those conditions are met, something should happen automatically. For example, if a scooter is placed in a warehouse in the middle of the night, the maintenance mode should activate for software updates or other maintenance.
00:02:41.560
On a Wednesday, in the middle of the day, my colleague created the final commit for this feature. I quickly reviewed it on his computer, and he pushed it to the master branch, where the continuous integration checks were green, indicating everything was fine. Therefore, it was deployed to the test server, and automatic tests ran successfully. It looked good and was ready for production, so it was time for lunch. Standing in line, we noticed unusual traffic on Slack—something was wrong. We then received a flurry of emails complaining about issues, followed by phone calls asking what was happening.
It turned out that in one of the largest European cities, all the scooters suddenly stopped functioning—immediately. Guess what? It was due to our microservice. Fortunately, we managed to mitigate the issue right away; we abandoned the kebab queue and returned to the office to fix it. We turned off our service and debugged the issue, which boiled down to one line of code: one incorrect if statement. The condition was wrong, and the tests didn't catch this because we were mocking dates. Instead of running the automation in the middle of the night as planned, it triggered in the middle of the day. The coincidence was that this issue occurred just after our deployment. Despite the situation being resolved quickly and deploying a fix the same day, the trust that clients had in our team was irrevocably damaged, and eventually, the project was abandoned.
00:05:16.880
Why am I sharing this with you? Because we all make mistakes—some are trivial, some hilarious, and some can even be tragic. However, we all make them at some point. In this presentation, I want to show you how to handle mistakes. Let’s start with a definition: what is a mistake? We intuitively know it’s an unintended outcome of our action. Our intention was to run the automation in the middle of the night, but it executed during the day, resulting in a mistake. In both the software world and everyday life, mistakes occur frequently. For instance, last year there was an incident involving Crowdstrike, which led to approximately 5,000 flight cancellations, disrupting numerous hospitals and businesses due to a faulty update. Mistakes can even be more serious; extensive research on medical errors shows a 5% chance that if you go to a hospital, something might go wrong due to human error, whether by a nurse or a doctor.
A prominent example is the Post Office scandal in the UK, which is the largest scandal in British government history, where reliance on buggy software for 20 years led to wrongful accusations against numerous employees who lost their jobs, reputations, and even faced imprisonment due to faults in the software instead of admitting and fixing the errors. However, the interesting part lies not in the mistakes themselves, but in what happens after a mistake has occurred and how we handle such situations.
00:07:42.920
The answer to this question was explored by a famous psychologist, Leon Festinger, in the 1950s, who asked: what happens when people realize they were wrong? To investigate this, he observed a group of cultists who believed an apocalypse would occur on a specific date. On that date, when the world didn't end as predicted, there was a split in the group.
One smaller group expressed disappointment and facedpalm moments, coming to terms with their mistake—perhaps the idea of being a cult member was naïve. The larger group, in stark contrast, claimed that their prayers had saved humanity, reinforcing their belief that they had done something right. This phenomenon is known as cognitive dissonance, a term introduced by Festinger, which describes the feeling one experiences when reality doesn't match expectations. The automation should have worked, but it didn't; or, in the cult's case, the apocalypse should have happened, but it didn’t. This leaves people feeling discomfort, which they want to resolve.
There are two primary ways to do this, although often unconsciously. The first option is to acknowledge reality: "I was wrong. I need to adjust my behavior." In the cult example, a member might decide to stop participating or even leave the cult. In everyday terms, if I am a smoker and learn about its dangers, I might say, "Alright, I will try to quit smoking." Of course, it’s easier said than done.
The second option is to reshape reality. Instead of admitting fault, one might think, "No, it wasn’t my mistake; let’s downplay it." For smokers, it might be rationalized that smoking has benefits, such as stress relief. Cognitive dissonance is well-studied in various contexts, including smoking, where individuals often try to justify their harmful habits instead of acknowledging their mistakes.
00:10:19.040
This concept can also apply to medical professionals. If a surgery has complications and isn’t successful, doctors might justify it by saying unexpected factors influenced the outcome—diverting blame from themselves. As developers, we tend to say, "Writing documentation is boring; my code is self-explanatory; there’s no need for documentation." However, a concerning trend is that seasoned professionals often fall into the trap of the second option concerning cognitive dissonance. The discomfort of admitting a mistake can be overwhelming, particularly if one is an expert with years of experience. Admitting to a trivial bug feels contradictory to their expert identity.
So what can we do about this? Building a healthy culture around mistakes is crucial. This forms the basis for my decision to submit my talk to this conference, as addressing cognitive dissonance in a constructive way impacts how decisions are made, which influences teams, and ultimately affects the company culture. One key takeaway is cultivating a growth mindset, as introduced by psychologist Carol Dweck. She categorizes individuals into two groups: those who believe their abilities are fixed versus those who think skills can be developed. If you believe you can improve at math, even if you struggled in primary school, this belief will shape how you cope with errors.
00:12:38.480
In a growth mindset, mistakes are seen as learning opportunities; they are natural and contribute to growth. Conversely, a fixed mindset encourages avoidance of mistakes, viewing them as reflections of personal failure. This results in seeking out easier tasks to prevent exposure to failure. The mindset also influences how people think about learning and collaborations. In a fixed mindset, an individual may compare themselves to others and fear judgment, while a growth mindset sees the environment as supportive and conducive to personal growth.
Another important aspect is acting like a scientist. In software development, there is a natural inclination to test solutions in practice, especially in Agile methodologies where rapid delivery is key. The ideal scientist tests theories and then validates them against reality. However, it’s crucial that when one has a theory to prove, they are not exclusively looking for scenarios where the theory succeeds; they should also seek out situations where it fails, to refine and improve it. Moreover, having control groups is essential, as in A/B testing, to assess different solutions effectively.
00:14:51.360
It is imperative, particularly for team leaders and CEOs, to foster a culture where team members feel safe discussing their mistakes. In my company, we have weekly software engineering meetings where we share experiences and challenges from the past week, covering both successes and failures. Admitting to mistakes isn't a source of shame; it's viewed as an opportunity for the team to learn. Furthermore, it’s essential that team members don’t perceive mistakes as threats to their careers. Everyone, even experts with extensive experience, can make mistakes, and it is crucial to normalize this aspect.
00:17:03.520
From these experiences, it’s not just about acknowledging mistakes but also about drawing conclusions from them. Research suggests that the learning derived from mistakes must be straightforward, simple to implement, and accessible to others. If I make a mistake and learn from it, it is vital that others can learn from my experience too. Thus, it’s important to appreciate efforts, especially when teams have worked hard even if they didn’t succeed. Encouraging experimentation is key to promote a culture where team members are not afraid to try new approaches.
00:19:56.640
To finish the story I began earlier about the electric scooters in April 2019, let’s revisit the three challenges we faced: using Golang, microservices, and handling code reviews. Initially, I didn’t mention our mistakes in detail, but we did make several, some of which were quite silly. For example, we used technology we weren’t familiar with, which isn’t something you should do for a client project. Never experiment when delivering a working solution; always use technologies you know. Another mistake we made was abandoning code reviews, which I mentioned was a consequence of our approach to trunk-based development, where we failed to provide adequate context for our code during reviews. Therefore, we should always prioritize code quality—perhaps find tools to facilitate this process.
The biggest mistake, however, was that we didn't understand our feature's context within the broader company framework. With many microservices, it was essential for major developers to grasp how our feature impacted others and the business as a whole. The lesson I learned is to be bold and willing to admit mistakes while viewing them as learning opportunities instead of threats. By embracing a growth mindset and recognizing cognitive dissonance, we can take valuable steps to address this discomfort.
00:24:07.600
If this topic resonated with you, I recommend two insightful books: 'Black Box Thinking' about fostering a constructive mindset towards mistakes and 'Mistakes Were Made' to help build a healthy culture in your teams. You can find a link to my full presentation, including notes and resources, at the end. Thank you!
