1. Lean Analytics Workout DAAHub Phil Pearce April 2014 Fitness Consultant linkedin.com/in/philpearce
2. Harder, Better, Faster, Stronger Leaner!
3. Who have we got in the room? 1. Entrepreneur – Started company 2. Corporation – Work in big company 3. Agency – Help startups
4. Have you read any of these?
5. Quick Quiz https://www.youtube.com/watch?v=usdeiJP7xh0 and https://prezi.com/lw-arulaenh4/copy-of-lean-startup-buzz-words/
6. START START GH PvT PMF LS Cdev FINISH FINISH FINISH IyBITwC TwCIyBi P2 E4 MVP BML VM
7. Web Analytics Exchange mentor750 GA questions answered Tracking protection group About Me Phil Pearce Analytics Consultant linkedin.com/in/philpearce
8. I`m not an entrepreneur Apart from this one
9. AdWords But… I have done alot of agency consulting & I worked for some innovative startups Sold for €16m Pivoted Changed business model IPO in ~1yrs Funded by Gwyneth Paltrow Sold for €37m Crazy growth & IPO plans IPO soon Metrics Plan Massive Revenue understanding own sites digital value to understand investments Grew Taxi booking Revenue by €10m in 2yrs
10. Intrapreneur & Technical marketer 1. Build PPC reporting platform MS access 2. Enabled KW level ROI bidding in 2007. 3. Managed £600K pm Adwords account & out- performed market leader. 4. Built end-to-end affiliate tracking system. 5. Reverse engineered Adwords Algo. 6. Built mathematical ClickFraud detection tool for mobile 7. Built free version of SpeedPPC 8. Building “4clicks” SaaS for Magneto (KPIs, dataLayer, Dashboards, Remarketing -> all auto- enabled)
11. GH GH GH GH GH GH GH GH GH GH …and closet growth hacker FINISH GH
12. … I have author-ed a book on Amazon
13. Agenda • Start: 9:30am-12:30am • Introduce Lean Analytics terminology – (e.g. MVP, Iterations, Agility) • Explain why obsessing over the performance of one key metric is vital • Describe the difference between website and product innovating and testing? • Look at some examples of successful (and unsuccessful) analytics hacks • Develop a super analytics hack for your business • Define a process for testing and refining your hacks
14. If you build it … they will come.
15. They will come … if you build it
16. Because… Fast Feedback = Build what customers want Favourite Food
17. Most startups don’t know what their customers will consume (or what they are good at making) Hotmail was a database company Flickr was going to be an Video Game platform Twitter was a podcasting company Autodesk made desktop automation Paypal first built for Palmpilots Freshbooks was invoicing for a web design firm Wikipedia was to be written by experts only Mitel was a lawnmower company
18. Fast iterations/sprints using… Build > Measure > Learn BML Build • (products) Measure • (data) Learn • (ideas)
19. Fast iterations/sprints using… Build > Measure > Learn (repeat) BML Build • (products) Measure • (data) Learn • (ideas)
20. Even the book uses lean principles… 1. 5th edition in 8months (new edition every built 1.5months!) 2. “We liked to hear from you” feedback section in front & online blog comments encouraged. 3. Learnings have spawned start-up conferences
21. Build > Measure > Learn Measure
22. Problem: We lie to ourselves “We” are amazing!
23. Reality check…
24. …Analytics to the rescue
25. Analytics is the measurement of movement towards your business goals.
26. In a startup, the purpose of analytics is to iterate to product/market fit before the money runs out.
27. I have twocoins. Atleast one of them is heads.
28. What is the % probability that the other is tails?
30. Tails Tails Heads Tails Tails Heads Heads Heads
31. Heads Tails Tails Heads Heads Heads 2 of 3 (66%) are tails.
32. Some fundamentals.
33. A good metric is: Understandable If you’re busy explaining the data, you won’t be busy acting on it. Comparative Comparison is context. Aratio or rate The only way to measure change and roll up the tension between two metrics (Miles Per Hour) Behavior changing If you’re busy explaining the data, you won’t be busy acting on it.
34. simplest rule Not a good metric. If metric won’t change how you behave, it’s…
35. Metrics help you know yourself. You are just like Customers that buy >1x in 90d Your customers will buy from you Then you are in this mode Acquisition 70% of retailers Once1-15% Low acquisition cost, high checkout Hybrid 20% of retailers 2-2.5 per year 15-30% Increasing return rates, market share Focus on Loyalty 10% of retailers >2.5 per year >30% Loyalty, selection, inventory size (Thanks to Kevin Hillstrom for this.)
36. Qualitative Unstructured, anecdotal, revealing, hard to aggregate, often too positive & reassuring. Warm and fuzzy. Quantitative Numbersand stats. Hard facts, less insight, easier to analyze; often sour and disappointing. Cold and hard.
37. Exploratory Speculative. Tries to find unexpected or interesting insights. Source of unfair advantages. Cool. Reporting Predictable. Keeps you abreast of the normal, day-to-day operations. Can be managed by exception. Necessary.
38. Rumsfeld on Analytics (Or rather, Avinash Kaushik channeling Rumsfeld) Things we know don’t know we know Are facts which may be wrong and should be checked against data. we don’t know Are questions we can answer by reporting, which we should baseline & automate. we know Are intuition which we should quantify and teach to improve effectiveness, efficiency. we don’t know Are exploration which is where unfair advantage and interesting epiphanies live.
39. May A/B test: Changing one thing (i.e. color) and measuring the result (i.e. revenue.) AprMar 0 Jan Feb Segment: Cross-sectional comparison of all people divided by some attribute (age, gender, etc.) Slicing and dicing data 5,000 Active users Cohort: Comparison of similar groups along a timeline. (this is the April cohort) Multivariate analysis Changing several things at once to see which correlates with a result. ? ? ? ? ? ?
40. Which of these two companies is doing better?
41. Is this company growing or stagnating? Which of these two companies has the best Revenue/Customer? January February March April May Rev/customer $5.00 $ 4.50 $4.33 $4.25 $4.50 Cohort January February March April May Averages Cohort group5 € 5.00 € 6.00 € 7.00 € 8.00 € 9.00 € 7.00 group4 € 3.00 € 4.00 € 6.00 € 7.00 € 5.00 group3 € 2.00 € 2.00 € 5.00 € 3.00 group2 € 1.00 € 1.00 € 1.00 group1 € 0.50 € 0.50
42. Lagging Historical. Shows you how you’re doing; reports the news. Example: sales. Explaining the past. Leading Forward-looking. Number today that predicts tomorrow; reports the news. Example: pipeline. Predicting the future.
43. • AFacebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya) • If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt) • ADropbox user who puts at least one file in one folder on one device (ChenLi Wang) • Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman) • ALinkedIn user getting to X connections in Y days (Elliot Schmukler) Some examples (From the 2012 Growth Hacking conference.
44. Which means it’s time to talk about correlation.
45. Number of Analysts ChessStarTrek correlated Liked Maths causal Number of Analysts Correlated vs Causal P2 E4
46. Correlated vs Causal But it is not the cause! Strong Correlation
47. Correlated Two variables that are related (but may be dependent on something else.) Ice cream & drowning. Causal An independent variable that directly impacts a dependent one. Summertime & drowning.
48. A leading, causal metric is a superpower. h” p ://www.?ickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/
49. Growth hacking, demystified. Find correlation Test causality Optimize the causal factor Pick a metric to change
50. Why is Nigerian spam so badly written?
51. Aunshul Rege of Rutgers University, USA in 2009 Experienced scammers expect a “strike rate” of 1 or 2 replies per 1,000 messages emailed; they expect to land 2 or 3 “Mugu” (fools) each week. One scammer boasted “When you get a reply it’s 70% sure you’ll get the money” “By sending an email that repels all but the most gullible,” says [Microsoft Researcher Corman] Herley, “the scammer gets the most promising marks to self-select, and tilts the true to false positive ratio in his favor.” 1000 emails 1-2 responses 1 fool and their money, parted. Bad language (0.1% conversion) Gullible (70% conversion) 1000 emails 100 responses 1 fool and their money, parted. Good language (10% conversion) Not-gullible (.07% conversion) This would be horribly ine?cient since humans are involved.
52. Turns out the word “Nigeria” is the best way to identify promising prospects.
53. Nigerian spammers really understand their target market. They see past vanity metrics.
54. The Lean Analytics framework.
55. Eric’s three engines of growth Virality Make people invite friends. How many they tell, how fast they tell them. Price Spend money to get customers. Customers are worth more than they cost. Stickiness Keep people coming back. Approach Get customers faster than you lose them. Math that matters
56. @agatestudio Lean Analytics Stages Empathy • I’ve found a real, poorly-met need & reachable market faces Stickiness • I’ve figured out how to solve the problem, in a way they will adore and pay for! Virality • I’ve built the right product/features/functionality that keeps users around. Revenue • The users and features fuel growth organically and artificially. Scale • I’ve found a sustainable, scalable business with right margin in a healthy ecosystem.
57. 1. Ecommerce 2. Two sided marketplace 3. SaaS 4. Mobile app 5. Media/Publishing 6. User generate content Six business model types
58. Model + Stage = One Metric That Matters. One Metric That Matters. The business you’re in E-Com 2-Sided SaaS Mobile Media UCG Empathy Stickiness Virality Revenue Scale Thestageyou’reat
59. Really? Just one?
60. Yes, one!
61. Because… In a startup`s “focus” is hard to achieve.
62. Having only one metric resolves this problem.
63. www.theeastsiderla.com Prevents distraction
64. Metrics are like squeeze toys.
65. Revenue stage: CompareAndSave.com (2-sided marketplace) • Focus on one metric of CTR • Reduced CPC • Increased RPC (Effected of reverse economies of scale & tiered cpa volumes) • Marketplace: Consumers + Banks Technically a “comparison engine”
66. Empathy Stickiness Virality Revenue Scale E- commerce SaaS Media Mobile app User-gen content 2-sided market Loyalty, conversion CAC, shares, reactivation Transaction, CLV Affiliates, white-label Engagement, churn Inherent virality, CAC Upselling, CAC, CLV API, magic #, mktplace Content, spam Invites, sharing Ads, donations Analytics, user data Inventory, listings SEM, sharing Transactions, commission Other verticals (Money from transactions) Downloads, churn, virality WoM, app ratings, CAC CLV, ARPDAU Spinoffs, publishers (Money from active users) Traffic, visits, returns Content virality, SEM CPE, affiliate %, eyeballs Syndication, licenses (Money from ad clicks)
67. Workshop Task: 1. Select business type (E-Com, 2-Sided, SaaS, Mobile, Media, UCG) 2. Determine Stage (Empathy, Stickiness, Virality, Revenue, Scale) 3. Pick one metric 4. Set line in the sand (benchmark) Useful sheet bit.ly/BigLeanTable
68. Other measurement models bit.ly/kpishake
69. What other metrics do you want to know about?
70. Drawing some lines in the sand.
71. A company loses a quarter of its customers every year. Is this good or bad?
72. Baseline: 10% visitor engagement/day 30% of users/month use web or mobile app 10% of users/day use web or mobile app 1% of users/day use it concurrently
73. Baseline: 2-5% monthly churn • The best SaaS get 1.5% – 3% a month. They have multiple Ph.D’s on the job. • Get below a 5% monthly churn rate before you know you’ve got a business that’s ready to grow (Mark MacLeod) and around 2% before you really step on the gas (David Skok) • Last-ditch appeals and reactivation can have a big impact. Facebook’s “don’t leave” reduces attrition by 7%.
74. Who is worth more? Lifetime: $200 Lifetime: $200 Today A Roberto Medri, Etsy B Visits
75. The Lean Analytics cycle
76. Did we move the needle? Make changes in production Hypothesis Design a test Make changes in production Measure the results Success! Pivot or give up Pick a KPI Find a potential improvement Draw a line With data: find a commonality Without data: make a good guess Draw a new line Repeat test Did we move the needle?
77. Do AirBnB hosts get more business if their property is professionally photographed?
78. Gut instinct (hypothesis) Professional photography helps AirBnB’s business Candidate solution (MVP) 20 field photographers posing as employees Measure the results Compare photographed listings to a control group Make a decision Launch photography as a new feature for all hosts
79. 5,000 shoots per month by February 2012
80. Draw a new line Pivot or give up Find a potential improvement Try again Success! Did we move the needle? Measure the results Make changes in production Design a test Hypothesis With data: find a commonality Without data: make a good guess Draw a linePick a KPI
81. “G ee, tho se ho u se s that d o w e ll l o o k rea l l y n ic e.” Ma ybe i t ’s the ca m er a . “C o m puter : What d o a l l the h ig hl y r en te d ho u se s hav e i n co m m o n ?” C a m er a m o d el . With data: find a commonality Without data: make a good guess
82. Some non-tech examples.
83. I lied. Everyone is a tech company.
84. Cost of attention: way up. http://www.flickr.com/photos/elcapitanbsc/3936927326 Cost of experiments:down.
85. Let’s pick on restaurants for a while.
86. A line in the sand Labor costs Gross revenue 30% 20% Just right Understaffed? = 24% Too costly?
87. A leading indicator 50 reservations at 5PM 250 covers that night (Varies by restaurant. McDonalds ? Fat Duck.)
88. http://www.flickr.com/photos/southbeachcars/6892880699 Restaurant MVP
89. Is tip amount a leading indicator of long- term revenue?
90. Why does every table get the same menu?
91. Is purple ink better? http://www.tippingresearch.com/uploads/managing_tips.pdf
92. Growth hacking (is a word you should hate but will hear a lot about.)
93. Growth hacking, demystified. Find correlation Test causality Optimize the causal factor Pick a metric to change
94. Guerrilla marketing Data- driven learning Subversiveness GROWTH HACKING
95. • A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya) • If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt) • A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang) • Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman) • A LinkedIn user getting to Xconnections in Y days (Elliot Schmukler) (These are also great segments to analyze.) Leading indicators: Growth Hacks Read more examples: https://www.slideshare.net/mattangriffel/growth-hacking
96. • Growth hacking is simply what marketing should have been doing, but it fell in love with Don Draper and opinions along the way • Optimize a factor you think is correlated with growth The growth hack
97. Growth Hacking examples • Hotmail – P.S. I love you • Drobbox – Refer a friend • Facebook – Exclusive network appeal • Twitter – follow celebrities Read more examples: https://www.slideshare.net/mattangriffel/growth-hacking
98. AirBnB and Craigslist
99. What is PPC Growth? Adwords Marketing Conversion Data Sales Growth Classic model 1. Create new marketing campaign. 2. If CPA data is within target threshold = then growth achieved 3. Then Increase marketing. Repeat above steps, until threshold reached. Repeat until CPA threshold reached
100. And… needs to be Scalable, Repeatable and Sustainable. Growth Hacking for PPC SaaS Adwords Marketing Product Data Growth Hacking Note: “Product” could be value-proposition or incentives
101. SaaS hack
102. Have you Heard of “Battleships”?” Growth Hacking a PPC tool…
103. Growth Hacking a PPC tool… Download CTR battleships https://www.dropbox.com/s/f04fr0431ilwjfk/CTR%20battleships%20printable%20grid%20-%20GrowthHacking.pdf
104. And… needs to be Scalable, Repeatable and Sustainable. Growth Hacking a PPC tool… Adwords Marketing (capture/test initial lead) Product (Variation on QS grader) Adwords API Data Growth Hacking Same product but different UI/Landing page – value proposition tweaked and made fun / viral… e.g every one likes to show off 🙂
105. Growth Hacking a PPC tool… Winner must tweet winning strategy #ctrbattleships Or take a photo them holding the Prize! #ctrbattleships Download CTR battleships https://www.dropbox.com/s/f04fr0431ilwjfk/CTR%20battleships%20printable%20grid%20-%20GrowthHacking.pdf
106. Some tools and traps
107. Traction graphs Your business model The stage you’re at Yourone metric … change often if you’re doing it right. So how do you track that over time?
108. Traction graphs Jan Signup sper day Feb Mar Conversio nrate Apr Chur nrate May Jun Viral coefficient This axis changes for each metric
109. Traction graphs Jan Signups per day Feb Mar Conversion rate Apr Churn rate May Jun Viral coefficient 0%
110. Use vanity to get to meaningfulmetrics • Your goal is to produce outcomes • If the outcomes require action, and vanity motivates actors, use it! • But show how the vanity metric is a leading indicator of the real one! x Web tra?c Revenue Activation Cart Size Conversion rate VM
111. “The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them” Lloyd S. Nelson
112. Pic by Twodolla on Flickr. http://www.flickr.com/photos/twodolla/3168857844
113. ARCHIMEDES HAD TAKEN BATHS BEFORE
114. Once, a leader leader/king convinced others in the absence of data.
115. Now, a leader/king knows what questions to ask.
116. Alistair Croll & Ben Yoskovitz Special thanks to… Please buy their book if you want more info!
117. Copy of these slides bit.ly/leananalytics2
118. Appendix: Useful videos
119. 15min Lean Start-up Video https://www.youtube.com/watch?v=zOX1vC7_n6s
120. 60min Lean Analytics https://www.youtube.com/watch?v=-CB4w_OtrKw
121. 50min podcast www.twistimage.com/podcast/mp3/SPOS_351_-_Alistair_Croll.mp3