Benefits of Building a Culture of Experimentation
AMA with Karl Wells, Chief Subscription Officer, Dow Jones Wall Street Journal
KARL WELLS, DOW JONES WALL STREET JOURNAL
“If people believe in the saying “are we good enough to get better”, then experimentation opens that door.”
Full Transcript
*Questions submitted by community members
Transcript and highlights from our Ask Me Anything.
Please join me in welcoming Karl Wells, Chief Subscription Officer at Dow Jones Wall Street Journal. Karl will be answering your submitted questions on the topic of the benefits of building a culture of experimentation.
Okay, lets jump right into our first question.
Question: The first step is often the hardest…how did you get started on this journey to building a culture of experimentation?
Ironically it all started with an experiment. I arrived at Dow Jones having left a culture of experimentation in my previous role with News Corp. We re-purposed a role in my team to focus on experimentation and had them start with arguably the most tangible part of the subscription journey - the checkout flow. We knew this could be improved, and it was also easy to show the financial reward of a winning experiment. From the checkout we moved up the funnel to the shop page improving the 3ps (pricing, packs, and proposition), and then to a 4th P, our paywall. Each time we showed the financial benefit of the winning variation and this led to the business investing in more talent, and the team grew as a result.
Question: We find that good ideas can come from anywhere and anyone in an org that has a great culture of experimentation... what are some ways WSJ attempts to source great ideas for testing?
We actively encourage ideas for experimentation. We know that good ideas can come from any person or any team across the business and therefore created a streamlined intake process for people to submit ideas to be considered for experimentation. Importantly we have a scoring system that takes into account potential impact and Level of Effort, which helps us to prioritize.
Question: I love the scoring system used at WSJ...can you talk a bit more about what that looks like and how other companies could create one.
Yes of course. I'll give a practical example. Let's take product improvements. The potential impact could be the reach of the piece of real estate that is being tested. A homepage would therefore win over an article template that is rarely used. The level of effort is then an estimate from the product team on the amount of work needed to create the test variants. This then provides us with an impact index which helps rank a list of ideas - this is always balanced with the art of experimentation too - something numbers will not always provide.
`Thank you, love that!
I love that you all acknowledge both the data and the experience factor -- only someone with expertise could appreciate the work that might go into an experiment build and the expected output (anticipated odds of winning!)
Question: Did you run into any early opposition, even after the first few successful tests?
The opposition was more around how things looked rather than opposition around experimentation as a whole. This taught us a valuable lesson. Involve the internal product design team early in the process.
Question: Moving to the next question ...What does Tour De France have to do with building a culture of experimentation? (softball question from Peter)
It’s funny, I think once you start embracing experimentation you start to see it being practiced in all sorts of different ways. A small nudge makes a big difference. I saw this principle being applied to the Tour De France when a team principal experimented with the pillows their riders would sleep on. Turns out they found a competitive advantage by having their riders bring their own pillows from home on tour.
Thank you for letting me share this nugget with the group. I love the story you shared with me and had to get it out there for this community.
Question: Cadence of experiments seems like a critical aspect of gaining momentum but it can work against stakeholders' comfort with costs, commitment, perceived risk and scale of result. How do you find the best balance of cadence?
Originally we would target a set number of completed monthly tests. Over time we realized that targeting a certain number of tests each month can come at the cost of tests reaching significance. I think the most important factor is to ensure you give tests long enough to reach significance and where relevant ensure tracking is in place to monitor the long-term impact of a test change (e.g. impact of a paywall on pageviews).
Love that!
I couldn't agree more!
Love this marathon mentality
Question: What groups are instrumental in having on board to be successful? How did you overcome resistance (if you encountered any)?
If you can show the direct correlation between a test winner and key financial outputs then that’s a great way to overcome internal resistance. This was a huge priority for us at the outset. Key partners that have contributed to our success have been the finance team, in partnership with the product and design teams (and where relevant news teams), and of course our partners at House of Kaizen & Daydot.
Great insight. The immeasurable outcomes can present a point of conflict
Question: Publishing editorial teams are classically protective of the content, how did you get them on board with experimentation?
Go back to the fundamentals. Experimentation leads to doing fewer things badly and more things well. If people believe in the saying “are we good enough to get better”, then experimentation opens that door. For the newsroom, it’s also a way to complement their journalistic instinct (art) with a healthy dose of science.
Love this - are we good enough to get better?
Question: Do you see the skill set of marketers and product people changing over time as experimentation becomes more popular? Moving more towards the quant than what was traditionally qual (creative and brand)?
I think they already are, as measurement and tracking become big business. Ultimately you can’t test a metric you don’t track, so it’s important that the right tracking and reporting frameworks are in place.
I love this and it reminds me of the saying..."anything you can measure, you can improve.”
I often see tons of gaps around this. Lots of important steps in the journey that have no established metric nor targets with the understanding of how they influence the full journey and ltv
Question: What platforms/tools do you rely on for experimentation?
Optimizely for MVT tools, House of Kaizen for strategy, test design, and execution, and Cxense (now part of Piano) to power a lot of the membership real estate on-site and in-app.
We are proud of the work we have done together and the sense of partnership has always added to our mutual success.
Question: We are getting closer to the last question....few more left. How do you communicate experimentation insights across your large organization?
It’s pretty simple really. The team sends around a weekly newsletter that highlights results from concluded tests, and details of upcoming tests (experiment variations and designs can be easily accessed via links in the newsletter). Acquisition tests usually require a further layer of financial analysis which depending on the significance of the change will be communicated further afield.
I would LOVE to see a copy of what makes the cut for that newsletter.
Same - could I also see a copy?
Question: We look to you for inspiration about building a culture of experimentation, where do you find inspiration for yourself?
The Times (of London, and also my previous employer) to keep track of UK news, and Strava to compete with myself and friends at running. One other inspiration has to be Richard Thaler. He is the god of Behavioral Economics. If you haven’t read it before I highly recommend reading one of his old-school classics, Nudge.
Fantastic book....agree and highly recommend everyone read Nudge.
Glad Strava can be part of your inspiration routine, and the Tour de France anecdote here is fire. Thanks for an insightful AMA, Karl!
Question: Our next AMA will be with Jason Sylva, GM and Head of Consumer Revenue at New York Magazine. What questions would you like to ask Jason at that AMA?
An easy one for you, Jason. What do you see as a key growth lever publishers should focus on over the next 12-months?
Fantastic, I will make sure that gets added to the questions for Jason!
That was a wealth of information in a short amount of time...thank you so much, Karl, for sharing your insights and taking our questions.
Thanks as well to our audience (present and future) who joined to learn from Karl and for all of your great questions.