Keeping False Positives Out of Your A/B Test Results

If you already know all about correcting for familywise error rate (FWER) in suites of A/B tests, you can skip straight to our new FWER calculator web app by clicking here. Otherwise, read on!

A/B tests have become one of the most fundamental tools in a marketer’s toolbox over the past decade: from copy and ad images to landing pages and web forms, if it’s worth doing it’s worth testing. Because of this, most marketers have become very familiar with the concept of statistical significance. In simple terms, “statistically significant” means “very unlikely to happen by chance alone.” In most tests, “unlikely” means 5%: in other words, a test result is considered significant if there’s a 5% or less chance that it could have happened randomly.

This might sound boring to some folks, so lets take a second to remember why this is so important. Statistical significance matters because it lets us make better decisions with our marketing spends and creative. If we don’t check to make sure our A/B test results are statistically significant, we could spend our time and energy on something that doesn’t actually work!



Because this statistic is such a critical part of the modern marketing world, dozens of popular web tools have sprung up to help marketers make sense of it. KissMetrics’ A/B Significance Test and Evan Miller’s Sample Size Calculator are just two great examples (though the latter actually helps you figure out a related probability called power, but that’s another blog post entirely).

Sounds great, right? It’s 2015 – all marketers want more data! In reality, though, it’s not so simple. Believe it or not, testing more things can sometimes make our decisions worse rather than better. For an example of how basic statistical significance calculations can lead marketers astray, take a look at our revision of this great xkcd comic (on the left).

As you can see, if you consider any test result with a 5% or less chance to occur randomly as significant, then you’ll generate (roughly) one false positive for every 20 tests you run. If this is confusing, just think of a 20-sided die: if you were to roll it 20 times, you’d expect to roll a five at least once. Just like there’s nothing significant about the five, there’s also nothing significant about getting one “significant” result in a suite of 20 tests!

For a real life example, let’s assume that a data-driven marketer goes out and runs 20 tests on something. Just like with the die roll example, we can assume that one of the test results will be a false positive. If only two of the 20 test results come back positive, how confident can we be that a spend/creative decision we make based on the results will be the right one? Only 50% – or, in other words, about as confident as a coin flip!

Don’t panic, though; there’s a fix for this.

This phenomenon is called the “familywise error rate” (FWER), and it’s actually not that hard to stop it from ruining your results. But despite looking around for a while, no one at Greenough was able to find an online significance calculator that lets users account for FWER!

To make marketers’ lives a little bit easier, we decided to solve this problem ourselves by contributing a new app to the community. Unlike most online calculators that only give you results for one test, our app takes a list of test results and returns only the ones that are found significant vis-a-vis FWER. It even allows you to choose your method of correction, if you’re interested in going that deep. Check it out at this link – we hope it makes your life a little easier!

A big thanks to Boston’s own R Studio for making the Shiny web framework package, which we used (along with shinydashboard) to build this app. If you’re interested in using this app within your own organization, you can find the code for it on GitHub.

Zach Pearson is Manager Content and Digital at Greenough. Follow him on Twitter: @zach_p_pearson

Instagram Increases Ad Friendliness



We all saw this coming – Instagram formally announced a test run of its new advertising API two months ago – but now it’s officially active. The company turned on the switch last week allowing anyone to purchase an advertisement on the photo sharing network.

This isn’t necessarily a novel concept – Instagram Ads have been around for a few years now. But before this, marketers were forced to engage directly with Instagram’s sales team to negotiate prices for limited ad space. This meant that the price to pay was a variable and opportunities became limited to only big brands with even bigger budgets.

So what does this all mean for marketers?

The move elevates the social sharing network into a major mobile advertising business rivalling even the likes of Google and Twitter. After all, Instagram boasts over 300 million active monthly users.

Additionally, according to Hootsuite, one of the Instagram Ads API partners: “This integration will also make marketers and social-media managers more efficient as they will no longer have to switch between their phone and computer to track all aspects of a social-media campaign. For example, if a new product is launched, responses to comments and engagement with users on Instagram can be carried out from the same window that is being used to engage followers on Twitter and Facebook. Everything is in one place.”

But it also continues the natural evolution of increasing the use of visual marketing to drive engagement.

Storytelling is critical for any business and marketers need to understand the power visual storytelling has on engagement. Statistics illustrate that the human brain processes visuals 60,000 times faster than text and 90 percent of the information transmitted to the brain is visual.

During a time when users can be bombarded from any direction with loads of content, Instagram has proven we can do more with less by forcing us to focus on a single visual, rather than lengthy content to drive home their value proposition. What’s more, by adopting Twitter’s hashtag approach, advertisements can also be hyper targeted to particular groups, or insert themselves into timely newsjacking trends. The recent announcement of the “buy” button also makes Instagram the optimal choice for consumer brands looking to drive sales.

No other platform can match the combination of simplicity and niche targeting that Instagram offers and it will be only a matter of time until advertising budgets are being shifted away from the more mainstream digital platforms, such as LinkedIn and Twitter, in favor of a more visually compelling platform.

Why Twitch Is Worth Watching for Digital Marketers



Earlier this week, Amazon made the largest acquisition in the company’s history when it purchased video game streaming website Twitch for $970 million. Because video game streaming has yet to go fully mainstream, the deal has left a lot of people scratching their heads – and, in response, almost every news outlet has decided to offertheirownexplanation.

We wanted to jump in and offer our two cents on what the acquisition means from a digital marketing perspective. For the completely unfamiliar, here’s a crash course on what Twitch is, where the acquisition could take it and how this new form of high-engagement user-generated content will matter for the future of digital marketing.

Twitch 101

At its most basic, Twitch is a platform that allows people to watch other people play video games. It’s enormously popular, averaging 567,000 viewers during prime-time – that’s 7,000 above MTV’s numbers and just below Comedy Central’s.

Twitch streams fall into two broad categories: to draw a parallel with existing media, we’ll call them “Super Bowls” and “hobby shows.” Super Bowls are major gaming tournaments that attract enormous audiences – The International, one of the gaming’s largest tournaments, had a 2014 prize pool of $11 million and brought in 20 million viewers on Twitch. Hobby shows, by contrast, are more like Emeril Live or Car Talk. They attract smaller niche audiences that are both interested in learning about the hobby and also entertained by the specific personality hosting the show. Twitch has thousands of these hobby shows, each focused on a different aspect of gaming culture and hosted by a unique personality.

In addition to live video, Twitch also provides a chat room for everyone watching the game that runs alongside the show. This is what makes it stand out from traditional television – the chat function creates more engaged audiences by encouraging viewers to interact in real time with both the streamer and other viewers. This reduces viewers’ available bandwidth, making it difficult to zone out and thumb through Twitter while watching.

Where It’s Going

Amazon now owns the main distribution network for gaming Super Bowls – not bad in and of itself. But the real synergy between the two companies lies in Amazon’s ability to extend Twitch’s hobby show format across all its products. Tom Standish, the Economist’s Digital Editor, gave an example of how that might work on the newspaper’s Babbage blog this week:

“From my perspective, I go over to Twitch because there’s a new game out and I want to see what it’s like… I’m probably quite interested in buying this game… (and now) Amazon can just put a ‘buy’ button, right there, underneath the video. So one way you can think about this is that Amazon has just bought access to millions of user-generated adverts for games.”

Mr. Standish uses Twitch hobby shows – or, as he calls, them, “user-generated adverts” – as a way to vet products he’s considering purchasing. In that way, they function very much like Amazon’s user reviews, but with added benefit of offering entertainment value.

The capabilities provided by Twitch allow Amazon to bridge the divide between their core retail business and their nascent online video services, allowing both sides to direct users closer to purchases. Instead of watching cooking shows on network television, Amazon hopes that potential customers will instead visit its new streaming video service (let’s call it something like “Kindle Live”) and watch a cooking show there instead.

If the hypothetical Kindle Live can incorporate ‘buy’ buttons for the items being used on its new user-generated hobby shows, that will surely create a lot of sales – especially if users perceive the buttons not as ads but as helpful shortcuts that make it easier to emulate their favorite streamers. Imagine a customer watching their favorite home improvement Kindle Live stream, asking the host a few questions about the tool they’re using in the chat room, and then adding the tool to their Amazon cart by simply clicking on it in the stream. It’s not possible yet, but that sort of potential is the real value Twitch adds for Amazon.

Getting Ready for Live Stream Marketing

If you’re a digital marketer and had never heard of Twitch before this week, don’t worry – right now, it’s really only relevant for gaming demographics. But now that Amazon owns the company, that will likely change soon. Expect Amazon to make a big push over the next few years to expand Twitch’s trademark high-engagement user-generated content into other industries. A solid plan for this new medium will include “streamer relations” similar to existing social media influencer relations strategies, smart ideas for pre-roll stream advertisements, tactics for participating in stream chat and an understanding of the informal categories of Twitch-type streams (which, as we’ve seen, really aren’t all that different from traditional broadcast media).

Twitch is more than just a platform for game streaming – it’s a new digital media format that’s about to get a huge boost and a new monetization strategy from one of the largest companies in the world. If you haven’t studied it yet, it’s worth doing some research so you’re prepared for the arrival of Twitch-type content in your industry – even if that arrival is a few years off.

Zach Pearson is an Account Executive – Content at Greenough. Follow him on Twitter: @zach_p_pearson

How Law Firms Can Better Engage on Twitter

Photo: Salesforce

Photo: Salesforce

Although it feels like every business is tapping social media for marketing, new research released this week shows that companies are largely underutilizing social channels.  The study surveyed 300 business executives from a diverse range of industries to analyze the status of social media in the corporate world.  The findings show that law firms especially are missing the boat.

To explain why this might be the case, an article from UK marketing pub The Drum notes, “A lawyer’s training reinforces reticence, confidentiality and equivocation, all things that Twitter in particular cuts through and certainly does not encourage.”  This mentality may be why law firms are often hesitant to join the Twittersphere. However, when used thoughtfully and appropriately, social media can be a huge asset to a law firm.

I’m focusing on Twitter here because it may, for now, be the best social channel for law firms.  Twitter is the place for news and opinions, Facebook is where to find and share personal information and LinkedIn is most effective for career networking, but is still a more difficult venue on which to build visibility and establish thought leadership.  Laurel Papworth, named by Forbes as one of the 2012 Top 50 social media power influencers, agrees that Twitter is a better place for law firms. And she warns that law firms run the risk of creating “social-spam” if they over-post to Facebook.

Twitter often intimidates law firms because it is often an outlet for bold opinions or, even more troubling, risky disclosure. But there are infinitely more opportunities to post evenhanded content that doesn’t jeopardize the privacy and impartiality that are central to the legal world.  Here are a few Twitter best practices for law firms to keep in mind:

  • Tweet your firm’s news. Twitter is the ideal venue to make announcements about a new attorney you’ve brought on, certain services you provide, upcoming events you are sponsoring, or awards your practice or lawyers have earned.
  • However, don’t just tweet for your own benefit.  Offer insights or tips that are of use to your followers.  There’s nothing worse than a handle used only for self-promotion.  Share articles relevant to your practice area or broader current events that connect to cases you’ve worked on.  Posting recent news and trends shows your audience that you value staying up-to-date on what’s emerging in the legal world.
  • You can also take it one step further and directly engage with clients or the media via Twitter.  Offer advice or a link to a white paper if you notice a client posted a question.  A firm can capitalize on media opportunities by offering one of its attorneys as a future resource if the firm notices that a reporter is consistently tweeting about a subject in which the firm has expertise.
  • Lastly, make sure your firm is reaching the widest audience possible by having your attorneys link tweets from their personal handle back to the firm’s handle.  The blog The New Lawyer warns, “those firms that [make] no efforts to develop a social media presence [risk] being eclipsed by their own ‘savvy’ lawyers who exploited social media communities for their own, rather than their firms’ benefits.”  Encouraging your attorneys to include the firm in their tweets widens the reach your practice’s name has.

While Twitter can sometimes be a sounding board for the less than level-headed, law firms shouldn’t let that stop them from capitalizing on its reach.  After all, you control what you tweet and who you follow.

If that wasn’t enough to close this case (pun obviously intended), check out this list of 100 uses for social media for more ideas on how to best create and capitalize on a social presence.

Lucy Muscarella is a Consultant at Greenough. Follow her on Twitter: @lucymuscarella

Optimize for Gen Y: Do or Die

Photo: Flickr - Creative Commons 2012

Photo: Flickr – Creative Commons 2012

Every time I stumble upon an article about “Millennials” or “Generation Y,” I can’t help but read it. After all, I am part of Gen Y and curiosity gets the best of me. It’s fascinating to learn (from the media) what my daily struggles are, my life ambitions and goals, my career status, my relationship with technology (and maybe even other people) and, of course, the best ways to market products to me. Okay, I’m being a bit facetious – the reality is that sometimes these articles are on the mark, but just as often I think the articles don’t represent who I am at all.

I came across a recent article, “Generation Y: Looking for jobs – and an identity,” which I felt most definitely “hit the mark.” The article breaks down Gen Y from every angle possible: Where do we live? Did we graduate college? What did we study? What type of jobs do we have? What are our habits? When will we be able to afford a house? And, while there is some opinion and speculation, almost all of the statements are backed by hard data from various surveys and research firms.

While very comprehensive and well-written (I won’t go into all the details), the article highlights an important point: Gen Y’s world revolves around mobile and social technology.To some, this may be obvious. It’s no secret that 18-29 year olds are attached to their smartphones, obsessed with social networks and would pick “digital” over “print” any day. But, if this insight is so obvious, why are so few organizations jumping at the opportunity to optimize and reinvent themselves so they can engage more closely with Gen Y?

This “Gen Y optimization” (making mobile and social systemic) must be firmly rooted internally and externally. Organizations must enable “productive socialization” internally – embracing social networks, tools and platforms (e.g. Yammer, Facebook etc.) and encourage Gen Y workers to collaborate on those platforms. The return, in terms of productivity, employee happiness and retention, will not go unnoticed by employees and, equally important, will accrue to both the bottom and top lines of the business.

Externally, organizations must also take a leap of faith, embrace innovation and take steps to optimize in ways that demonstrably appeal to Gen Y. Today, for example, almost every bank has a mobile solution, or at least a robust online banking capability; almost every airline offers mobile ticketing (does Gen Y even own printers anymore?) and even the smallest restaurants have a grip on what Yelp and Foursquare reviewers are saying about them.

In addition to Gen Y being the most “technologically connected” generation ever, it’s also the generation that’s been hit hardest by the recession. With the election just around the corner, the possibility for an economic resurgence is on the horizon and perhaps this will give Gen Y the opportunity to truly shine. Fueled by digital, social and mobile, I believe this group will shine, and businesses surely don’t want to be left in the dust when this group finally catches its wind.

Jessica Boardman is a senior consultant at Greenough. Follow her on Twitter @J_Boardman.

IT Healthcare Vendors: Don’t Forget Social Media’s Role in Healthcare Reform

Top-20-EMR-Software-Solutions-large (1) (2)

Top-20-EMR-Software-Solutions-large (1) (2)

Hospitals and health practices today are governed by a number of standards and regulations, including the HITECH Act,  that encourage the adoption and meaningful use of electronic health systems with an investment in health information technology.

Today, IT is enabling healthcare facilities to not only meet these regulatory needs but at the same time improve all practice areas, from patient management to revenue, with solutions such as electronic medical records (EMRs). EMRs streamline the patient care workflow by providing physicians safer and quicker access to important medical information and can benefit all health systems, from large hospitals to single practitioners. Specialty software companies, such as Massachusetts-based athenahealth, focus solely on healthcare-related business services such as EMRs. These companies were quick to recognize that their tools can provide safe and cost-effective healthcare solutions, and athenahealth and some of its competitors now lead the industry in product innovation.

Recently, online enterprise software directory Capterra published an insightful view into the most popular EMR software providers (see graphic, right). The infographic’s index is compelling because it ranks companies not only by revenue but also by digital data, aggregating results for each of the vendors based on Alexa rankings, Compete traffic and Google searches. The social media component of Capterra’s infographic articulates several important takeaways about the healthcare industry; social media has huge potential in healthcare IT and, as such, the onus is on the individual software vendor to use social platforms to set themselves apart from competitors. Examination of various social platforms, such as Facebook likes, Klout score and LinkedIn company followers, also contributed to the popularity rankings and present opportunities to think critically about how these vendors should leverage social media for their content strategy.

Given that the healthcare industry is limited by government mandates that restrict creativity in product development, the key to a company effectively broadcasting its unique corporate message revolves around intelligence in storytelling, and targeted content is the golden ticket to success. As we’ve learned through our work with client Network Health, it’s important to live in the ecosystem that caters to the needs of customers. As Network Health’s partner, we create compelling storylines that reach the health insurance provider’s end users – low- and moderate-income residents in Massachusetts.

We believe that a similar approach has and will continue to achieve success for health IT companies vying for the top spot in the industry. When comparing vendors that made the leaders’ list in social media, such as GE Healthcare and Practice Fusion, it is important to recognize that these companies follow different content strategies  based on their target audience. While GE Healthcare caters predominantly to hospitals and large practices, the company’s Twitter and blog content has a global perspective. Conversely, Practice Fusion caters to medium-size practices and individual practitioners and does an excellent job at engaging with its localized audiences.

The opportunities for health IT companies to leverage social media for business development are endless, yet there is always room for improvement. One immediate fix could involve inserting the service providers into conversations around this week’s Supreme Court hearings. With so many uncertainties within healthcare reform, consumer are undoubtedly wondering what the outcome will be for them.  Moments like these are critical communication-inflection points, and social media tools are the ideal platforms for information exchange between the athenahealths and allscripts and their current and future customers. For what it’s worth, just as social media continues to be an avenue for quick and secure communication, we expect that IT innovations will continue and will weather the storm of reform. What do you think?

Sarah Hurley is a consultant at Greenough. Follow her on Twitter @Sarah_Hu