Scientific Method

In the heat of running the campaign one needs to juggle many tasks, coordinate and sometimes cut corners in order to simply “get things done”. That’s why having a systematic experimentation process framework is so important. The speed of experimentation is one of the crucial factors to achieve growth. The rule is very simple – the faster you can experiment, analyse and improve, the quicker you’ll improve and get to your goal. This is actually nothing new. The scientific method of experimentation and testing, used to explore observations and answer questions, stems from the Empiric school of ancient Greece.

 

 

The scientific method has always been somewhat adopted by good agencies. The lean startup method applied to product development follows exactly the same protocol. In marketing we often speak of “Test & Learn”, insights and testing. But, it’s still quite rare that marketing agencies and experts rigorously follow the principles of the scientific method or strive to make the tests valid at all times. It could be because they’ve never heard of the proper way of conducting experiments, or perhaps more likely their ‘marketing method’ was good enough for what they needed.

My experiments started in 2004, first by testing some unusual channels to generate traffic (now we call them growth hacks), or A/B testing headings of a Google ad. Over the years my methods became more and more structured. I am now convinced that this type of structured approach to testing, which the scientific method offers, is the most predictable way to iteratively improve and scale any kind of marketing campaign. It’s not the only method. One could still rely on the creative minds of more traditional agencies, whose sometimes brilliance cannot be underestimated. But, just the way the early empiricists rejected the doctrines of the Dogmatic school, I treat the scientific method as the primary source of all improvements in marketing and sales activities. I don’t relinquish the creative method or my intuition to look for less evident (e.g. creative) insights, but I prefer to gather all subjective opinions in the brainstorming phase of the process, before the hypothesis is constructed.

Experimentation Frameworks in Marketing

I’ve experimented with various approaches to the management of insights gathering, tests and implementation of improvements. I’ve tried various tools that offer interactive boards, kanban boards or a simple spreadsheet. There are plenty of resources, but I was looking for something different that fits our workflow of campaigning for our clients at what.

Here’s how I defined my top requirements for our experimentation process or framework:

• • • Experiments have to be easily linked to our project management tool and time tracking
    • Monitoring tool where all experiments are gathered needs to be easily accessible
    • Avoid over engineering of the tools or just ideas, the focus should be on execution
    • Each experiment needs to start with an insight and questions
    • Avoid paid software and SASS tools
    • Flexible to fit all kinds of projects, with an ability to add fields, drop-downs, notes etc.
    • Scalable – can work with 1 or 50 experiments per month
    • Easy status tracking, an option to be pulled directly from our project management tool
    • Include or at least consider best practices from available frameworks and tools
    • Calculations
    • Available real time
    • Can integrate with APIs and data sources (e.g. Google Analytics, Facebook)
    • Ability to easily filter and search for past experiments and insights
    • Provides an overview of the current status of various experiments

Off the shelf software vs. Google Sheets

I’ve researched quite a lot of various tools and workflows designed specifically to manage experiments. This document from Ricardo Ghekiere was quite helpful, he combined Pipefy with Google Sheets to achieve a good user interface and benefit from the flexibility of Google Sheets. I’ve also looked at Experiments, but it seemed over-engineered, used too many buzzwords and subsequently none of the tools met my criteria. I’ve also learned that some of the most successful companies that run experiments rely on simple spreadsheets. That’s why I ended up combining Google Sheets, GitLab (for tracking tasks, but it could be any issue tracking software) and Google Analytics. The data analytics integration is optional if you run a small number of experiments.

Experimentation workflow

1. Set your target
It all starts with understanding of your primary metric or in growth hacking, the North Star Metric (i.e. the single metric that best captures the core value that your product delivers to customers).

2. Observe and brainstorm
Here we gather all ideas, make observations, ask questions, it’s when you can ask many people for input.

3. Prioritise
Score and research our initial ideas to have a shortlist, discarded ones go on the backlog.

4. Analyse & benchmark
Gather more data for the shortlisted ideas, including understanding the benchmark and goal. Solidify your hypothesis.

5. Plan
Set operative parameters, such as start/end date, who’s going to work on it, the task gets assigned.

6. Start your experiments

7. Project manage
Review the status of the task at least once per week

8. Track and measure
When experiments end review the status of the experiments sheet and make updates.

9. Repeat
Once you’re ready for new experiments start with the second step again.

 

The constant experimentation has always been in the center of what we do, but only recently we made it integral to our operative campaign optimisation cycle, including planning, execution and reporting. It’s a perfect fit and exciting to see how these experiments continuously get us closer to the target.

I’m keen to improve and very curious how other agencies or growth hackers manage experiments so if you have any ideas, questions or contributions please contact me or comment below.