LOOK WHAT I HAVEN’T DONE
By Dan Toma & Tendayi Viki
– Dan Toma is a mentor at the Academy for Corporate Entrepreneurship –
Traditionally regarded as the father medicine, Hippocrates of Cos was one the most prolific figures in the history of medicine. He is most revered for the ethical standards he created that are known as the Hippocratic Oath. A key principle in the Hippocratic Oath is that of Primum non nocere, or First, do no harm. This is the first principle upon which medical doctors around the world have to swear during their commencement day. It is the important notion that their actions, first and foremost, will do no harm to others.
Unlike medical doctors, corporate managers do not have to take an oath before they start their jobs. Outside of Google’s motto of “Don’t Be Evil” there are very few corporations that make explicit to their employees that they should be ‘doing no harm’. It is true that corporations cannot be regarded as the equivalent of medicine. However, the global financial crisis demonstrated that corporate decision-making has a real impact on human lives. As such, the way these decisions are made has to be scrutinized for its capacity to do harm to the organization and its customers.
It is now hard to deny that the global business environment is not as stable as it used to be. The pace of change in technological, social and economic factors is staggering. In today’s volatile business environment the traditional practice of executing on long term business plans is no longer adaptive. And yet, one hundred years after the glory days of the industrial revolution enterprises still measure the performance of their employees on the delivery of certain outcomes based on the business plan (i.e. lines of code, parts, projects, features). While this may have created value in the past, nowadays such management practices may actually do harm.
To better grasp this notion, one needs to understand that time is a linear resource – it cannot be expanded as other resources can be. As such, it is critical that organizational time is spent on value creating activities. Spending employee time executing on a flawed business plan and delivering a solution that nobody wants means taking that important resource away from searching for a solution that people want. This unrecognized waste does harm to the growth of the company and fails to create value for customers. We have to recognize that being busy is not the same as being productive.
When developing and launching new products how can one determine early that what they are developing will create value for customers? Not being able to predict the future in a volatile environment means that better ways of forging strategy need to be developed. If we turn back to the medical sciences, it is easy to see that continuous trial and experimentation were at the foundation of so many advancements. This practice can also be translated to the corporate environment. A process of continuous iterative experimentation has to be the bedrock of product development strategy, because only through experimentation can new and innovative growth avenues can be uncovered.
Even though most companies will get valuable insights from experimenting with product ideas before launching, a major constraint they have is that corporate environments often reward people who say ‘Look what I’ve done!’ versus people who say ‘Look what we’ve learned or prevented from happening’. However, having done something is not necessarily an indicator that you have created value. An orientation towards learning about customers and their needs might be the best way to create value and ensure that we “do no harm”.
Experimentation at T-Online.de (Deutsche Telekom AG)
In this article, we describe a process of experimentation and iteration that is being practiced at T-Online.de (Deutsche Telekom AG). Since being number one in an industry is not a guarantee for future success of newly built ventures, continuous experimenting is gradually becoming part of the modus operandi at this major telecoms company. In a conversational format, we describe why and how teams at T-Online.de (Deutsche Telekom AG) have been implementing experimentation and iteration in the development of new products and services. We also explore some the challenges they have faced in changing product development culture in such a large company.
Why is experimentation important for an established business like T-Online.de (Deutsche Telekom AG)?
The business environment has changed significantly over the last two decades. We now have increasing competition from all sides, especially from startups. We have learned that implementing our ideas without having any validated evidence from customers backing them up was not sustainable in the long term. We could not continue to rely on brand power for success.
Part of our traditional practice that we needed to set aside was basing our decisions on intuition and long term business planning. We needed to make room for a more scientific approach to new business development. The level of insight generated by formalized scientific testing helps T-Online.de (Deutsche Telekom AG) in making educated market bets. In the process, we are slowly changing our product culture from one based on our previous experiences and business planning to a more scientific approach.
On how many products have you used the method so far?
The first project we used the method on was a mobile product focused on news aggregation. Since mobile news aggregators are not really new to the market, we used our scientific method in conjunction with the Kano model. This helped the team to identify which features already offered by competitors customers perceived, as being must-be quality and which features were one-dimension and attractive quality. Our goal was to close the gap between our new product and the existing market competitors, with the smallest investment and in the shortest possible time.
Our experiments then focused on exploring two features bookmarking articles for future reading, or building custom aggregation categories. With these two examples, we saved tens of thousands of Euros in development costs through continuous iterative experimentation. At the same time, we offered the team the ability to develop value-adding components for the application – features. We made the decision that we would not further develop any feature that after several experimental iterations had traction of lower that 15% within our existing user base.
After reporting our first success with this mobile app development team, the management decided to deploy experimental methods with other teams and ultimately on a flagship product. T-Online is Germany’s biggest provider of email and news, with daily traffic of over 27,000,000 daily unique users. Being able to perform live experiments on an on going basis offered unprecedented time-to-market agility in the creation of value adding features and user-experience improvements.
On the news aggregation app we used experimentation to ensure the value gap between existing offerings and the company’s new product was closed in the most efficient way. On T-Online we used the same method to ensure that the right value is being added with the least amount of wasted resources (e.g. time and money).
Which specific tools have you used?
The principles and benefits of continuous iterative experimenting were, from the onset, fairly clear to all team members involved. However, a tool was still needed to ensure that we had the much needed transparency and engagement at all levels of the business unit. An experimentation tool also gave us the structure and physical space where the team could gather and focus on experimentation rather business plan execution.
We started with the Experiment Loop Map developed by Brant Cooper and colleagues. This was a perfect starting point for us in our quest. We were able to move through the requirement boxes on the map and benefits from its ease of use. For the first three experiments the Loop Map seemed to be up for the job, even though it was designed primarily for early stage startups.
However, as we our experiments increased in number and complexity, we realized that we needed to adapt the tool in order to cope with the requirements of large enterprises. As such, we developed the Experiment Canvas, based on the Experiment Loop Map but designed to specifically address the needs of existing products in corporate environments.
Although minor, the differences between the Experiment Loop Map and the Experiment Canvas can be easily spotted and traced back to the environment in which we used it:
- Impact on business model field. Within any businesses any hypothesis that is tested in an experiment, is related to particular blocks on the business model canvas. This in turn also impacts the other business model blocks. As such, on the Experiment Map teams are required to indicate the business model elements that are affected by each experiment. This transparency allows the team to have a clear understanding of how their business model is evolving over time.
- Time boxing. Sometime experiments can run for longer than necessary, which itself becomes a form of waste. We felt that it was not enough to just aim for our minimum success criteria; we also needed to make sure that this was all done within a set time period. Such constraints allow for rapid iteration, which is a hallmark of the lean innovation method. As such, we added a field to our canvas on which teams have to time-box the experiment they are running.
- Fail criteria. From experience we have seen that product teams, have a tendency of declaring their assumptions as validated even if they fall just short of their minimum success criteria. This is more likely to happen if the performance gap is smaller than 5%. The challenge here is that from iteration to iteration those built-in errors can add up and lead teams astray. They can be running ‘successful’ experiments but fail to create value for customers. By introducing fail criteria we wanted to force the product teams to be honest about the value they are adding and if that value is not big enough they could not declare that the assumption has been validated. As such, we created a field in which they have to state the minimum threshold for declaring the experiment a failure.
What was the cultural reaction to the introduction of continuous iterative experimentation? Any particular hurdles that needed to be overcome?
Within product teams, the reaction was relatively positive. Introducing the method has been quite straightforward, with people being excited about the idea of validating assumption before building features. For a culture that is traditionally focused on execution this was a surprisingly positive turn of event.
However, the biggest hurdle that we needed to overcome came at a later stage. The main challenge was sustaining the practice of experimentation and iteration over time; and make it part of standard operating procedure. In order to do this, we learned that we needed to engage support from senior leadership. Without this ‘air cover, it is virtually impossible to change a culture.
What long-term impact do you think adapting continuous iterative experimentation will have on the organization?
It will take time to move the entire culture towards this practice. This is the challenge of working in large organizations. Two things are clear though; first, there are now many more conversation about experimentation before execution and second, executives are starting to view not making stuff nobody wants as a key performance indicator. There is a slowly emerging view that continuous iterative experimentation can help the managers and product teams to “do no harm”.
However there is still a long way to go before a product owner can take pride in saying ‘Look what I haven’t’ done and be reimbursed for this.