Keeping False Positives Out of Your A/B Test Results

If you already know all about correcting for familywise error rate (FWER) in suites of A/B tests, you can skip straight to our new FWER calculator web app by clicking here. Otherwise, read on!

A/B tests have become one of the most fundamental tools in a marketer’s toolbox over the past decade: from copy and ad images to landing pages and web forms, if it’s worth doing it’s worth testing. Because of this, most marketers have become very familiar with the concept of statistical significance. In simple terms, “statistically significant” means “very unlikely to happen by chance alone.” In most tests, “unlikely” means 5%: in other words, a test result is considered significant if there’s a 5% or less chance that it could have happened randomly.

This might sound boring to some folks, so lets take a second to remember why this is so important. Statistical significance matters because it lets us make better decisions with our marketing spends and creative. If we don’t check to make sure our A/B test results are statistically significant, we could spend our time and energy on something that doesn’t actually work!



Because this statistic is such a critical part of the modern marketing world, dozens of popular web tools have sprung up to help marketers make sense of it. KissMetrics’ A/B Significance Test and Evan Miller’s Sample Size Calculator are just two great examples (though the latter actually helps you figure out a related probability called power, but that’s another blog post entirely).

Sounds great, right? It’s 2015 – all marketers want more data! In reality, though, it’s not so simple. Believe it or not, testing more things can sometimes make our decisions worse rather than better. For an example of how basic statistical significance calculations can lead marketers astray, take a look at our revision of this great xkcd comic (on the left).

As you can see, if you consider any test result with a 5% or less chance to occur randomly as significant, then you’ll generate (roughly) one false positive for every 20 tests you run. If this is confusing, just think of a 20-sided die: if you were to roll it 20 times, you’d expect to roll a five at least once. Just like there’s nothing significant about the five, there’s also nothing significant about getting one “significant” result in a suite of 20 tests!

For a real life example, let’s assume that a data-driven marketer goes out and runs 20 tests on something. Just like with the die roll example, we can assume that one of the test results will be a false positive. If only two of the 20 test results come back positive, how confident can we be that a spend/creative decision we make based on the results will be the right one? Only 50% – or, in other words, about as confident as a coin flip!

Don’t panic, though; there’s a fix for this.

This phenomenon is called the “familywise error rate” (FWER), and it’s actually not that hard to stop it from ruining your results. But despite looking around for a while, no one at Greenough was able to find an online significance calculator that lets users account for FWER!

To make marketers’ lives a little bit easier, we decided to solve this problem ourselves by contributing a new app to the community. Unlike most online calculators that only give you results for one test, our app takes a list of test results and returns only the ones that are found significant vis-a-vis FWER. It even allows you to choose your method of correction, if you’re interested in going that deep. Check it out at this link – we hope it makes your life a little easier!

A big thanks to Boston’s own R Studio for making the Shiny web framework package, which we used (along with shinydashboard) to build this app. If you’re interested in using this app within your own organization, you can find the code for it on GitHub.

Zach Pearson is Manager Content and Digital at Greenough. Follow him on Twitter: @zach_p_pearson

The Re-personalization of Banking?

Photo: Jacksonville Business Journal 2013

Photo: Jacksonville Business Journal 2013

Earlier this month I read about the new video-teller ATMs Bank of America is rolling out over the next few months, starting in Boston and Atlanta. The ATMs offer live video chatting with a teller located in one of the bank’s national call centers, and will be able to accommodate a number of transactions that regular ATMs are not capable of (dispensing change, splitting a check between two accounts, etc.). BofA’s press release on their new machines announces “Bank of America Adds Human Touch to New ATMs,” but I tend to side more with New York Times Bucks blogger Ann Carrns who says it’s a touch “starship Enterprise.”

It’s not a novel concept that having a human touch and building personal relationships is valuable in banking. In the Boston Business Journal’s coverage of their “Most Admired Financial Institutions,” Boston Private Bank & Trust CEO Mark Thompson explains that developing “enduring, long-term relationships” is the key to his bank’s success. Allowing a company to manage your cash and advise your financial decisions implicitly demands a great deal of trust, and it’s far easier to trust a person who knows you and understands your background and priorities, rather than an anonymous mega-corporation.

In fact, while mega-banks are hurting in today’s economic landscape, local banks (both those technically designated as ‘community banks’ and ‘regional banks’) are on an upswing. A BBJ article published last fall reports that during the year between June 30, 2011 and June 30, 2012, Bank of America closed 13 of its 277 Massachusetts locations, while “branch closures were a rarity among the state’s local and regional players; many made no changes, while others even added a branch or two.” The article points out that most of the large bank chains in the state reported market-share decreases in the year, while a number of the local banks expanded significantly.

It’s clear why the “buy local” craze would make its way to banking. Besides building a trusting relationship with your neighborhood banker, local banks invest back into the community. When you visit the websites of Middlesex Savings Bank or the rapidly growing Berkshire Bank, ‘Community’ sections are featured prominently on the homepages, detailing the banks’ local philanthropic work and community financial support. Eastern Bank has an entire microsite –  – where customers can easily and securely donate to local nonprofits and find out about volunteer opportunities. Nothing similar appears on the front pages of national banks’ sites. And though they may participate in charity initiatives, the investment back into their customers’ neighborhood isn’t the same.

So though video chatting a live teller might make an ATM transaction more convenient, I think BofA’s new “human touch” may be missing the mark.

Lucy Muscarella is a Consultant at Greenough. Follow her on Twitter: @lucymuscarella

Cybersecurity Confusion Creates Opportunity for Security Providers to Educate

For many, the topic of cybersecurity is confusing and, depending on your business model and Internet exposure, downright scary. From last month’s New York Times and Twitter attacks, carried about by Chinese hackers, to a re-introduction in Congress of the extremely controversial Cyber Intelligence Sharing and Protection Act (CISPA), consumer and enterprises are as likely to be as afraid of showing ignorance as they are of actual attacks.
Phishing, malware and other forms of attack remind us, often too late, that companies need multiple security and data protection measures in place.  This creates a huge opportunity for the already growing online security market, but it also puts pressure on vendors to better differentiate their offerings and make a case for their place in a security stack. In other words, it’s time for the experts to be experts and demystify a confusing space.

Application security provider Veracode did just that in a recent blog post, offering several excellent tips for cybersecurity 101. In its blog post, Veracode wisely took readers back to the basics, reaffirming obvious best practices such as using the most up-to-date browser; making sure to run anti-virus software, such as Norton; and disabling all saved passwords. The post is a great reminder that an ounce of prevention is worth more than a pound of cure.

Cybersecurity threats will continue to grow in complexity and frequency, so security experts, many of whom work for vendors, must be equal to the task as educators. After all, in both our business and personal lives, we routinely download iPhone apps without carefully checking their origin and we autosave our passwords without a second thought. We must evolve and security vendors can play a pivotal role in making this happen. Those who do this will likely rise to the top of shortlists as the dust from cybersecurity confusion settles.

Gaby is a Consultant at Greenough. Follow her on Twitter: @Gabyberk

Visual Storytelling: Style versus Substance

Photo: Knight Digital Media Center 2011

Photo: Knight Digital Media Center 2011

We’ve all seen them… action-packed, special effects-driven movies with empty characters and a vague plot line.  These “blockbusters” always leave you wanting more (and not in a good way).   And as I see use of video marketing exploding among businesses, I can’t help but worry that brands will fall into the same trap.

According to a 1to1 Media post, videos are 50 times more likely to appear on Google’s first page results than non-video pages, presenting marketers with an immediate SEO advantage.  Viewers stay on sites with video five times longer than text-only sites and 80 percent of business execs watch more online content now than they did last year.

So how do companies capitalize on an effective visual storytelling plan without blowing their whole budget on one expensive, effects-driven production?

The right video crew understands the balance between style and substance.  They focus on telling a meaningful story and use animations and effects to enhance the original message.

Take Next Step Living for example: This video utilizes one of the hottest video marketing tactics right now (whiteboard video) while clearly conveying the company’s mission and commitment to making New England homes more energy efficient.

On the flip side, Geemmodity’s videos look great, but without any well-written copy to support the visuals, the “how to” productions fail to explain the product and ultimately end up confusing the viewer.

Flashy videos might be a creative way to grab a prospect’s attention, but without a substantial story to tell, even the prettiest productions will flop.

Christine Williamson is a senior consultant at Greenough. Follow her on Twitter @ChristineDBW

Showrooming: The Trend That Will Change Brick-and-Mortar Retail for Good

It’s no secret that mobile devices have made retail stores nervous for years. Ever since the widespread adoption of smartphones, it’s way too easy to find a product you like at Best Buy or Target, then scan a barcode or enter a SKU number to see if the same item is available online for cheaper. It usually is. New research from Aprimo and Forrester analyst Sucharita Mulpuru into this trend of “showrooming” found that consumers who looked at their smartphones while in stores found lower prices 55 percent of the time.
But, you may be saying, this is nothing new – stories about the impact of showrooming on retailers came out last year. This holiday season retailers are ready. Or are they? The results of the Aprimo/Forrester survey show that the trend has the potential to grow faster than retailers imagined – we’ve barely scratched the surface of showrooming. When asked why they hadn’t used a smartphone to check online prices from the store yet, about one third of consumers said, “I haven’t thought about it.” In other words, a huge segment of the brick-and-mortar customer base has continued to buy the old way simply because the idea of showrooming hasn’t occurred to them. Yet.

Maybe one-in-three isn’t scary enough for you. Drilling down further, the data are even more convincing for certain key retail audiences. Forty-three percent of 18- to 34-year olds who haven’t showroomed said they hadn’t thought about it, and the number goes up to 49 percent for women 18-to-34. That’s half of a key demographic for whom the showrooming light bulb could go on at any moment.

Photo World Travel, Flickr

Photo World Travel, Flickr

Retailers who think they have showrooming beaten have no idea what’s in store for them. Studies have shown that Cyber Monday has already surpassed Black Friday as the biggest shopping day of the year.

Think a simple price match will solve the problem? Maybe this year – after all, 57 percent of consumers say that would convince them to buy in store – but we all know that going toe-to-toe with internet retailers while brick-and-mortar store overhead weighs you down just isn’t sustainable. If retailers don’t figure out how to differentiate themselves with outstanding customer service and marketing that inspires deep brand loyalty, this could be the end of brick-and-mortar stores as we know them.

Showrooming isn’t just coming for sellers of big-ticket items either; it’s something that even supermarkets and pharmacies should be worried about. The data show that almost as many consumers showroom groceries (37 percent) as consumer electronics (39 percent).

It’s not clear how this trend will play out, but there is no doubt that any retailer who hasn’t made showrooming a top priority is in for a rude awakening. Shopping services and apps from companies such as AisleBuyer, RedLaser and Sccope are making it even easier for consumers to do mobile research from the store. I’ll still do my shopping in person this holiday season, but if retailers can’t figure out how to convince me otherwise, I’ll be checking my phone as well.

Jake Navarro is a senior consultant for Greenough. Send him an email at

IBM Aces the US Open Once Again



In my mind, the two best weeks of tennis every year happen at the US Open.  As a big tennis player and fan, there’s nothing better than watching the top players in the world sprint, lunge and sweat all over the courts at Flushing Meadow in New York City. This year, the two best weeks of tennis got bumped up from 4.5 to a 5.0, thanks to long-time United States Tennis Association (USTA) partner, IBM and the company’s innovative use of mobile, big data and predictive analytics.

For the past 22 years, IBM has been providing the Open with the technology to provide scoring and statistics. Last year, they released “Keys to the Match,” which analyzed over 39 million separate data points to come up with insights on how players needed to perform in order to win. The iPhone app with live streaming capabilities was launched in 2009, and the first iPad app followed a year later.

This year, IBM added new features to the web, iPhone and iPad apps, such as a live radio broadcast of the matches, so you can listen to how your favorites are doing even if you can’t watch them.  On the iPhone app, you can also sign up for alerts for up to five different players, so you’ll know if Roddick is about to close out his career a bit early. Plus, scores, stats and insights for each match are right at your fingertips.

The iPad app has feeds specific to all 17 courts in the Billie Jean King National Tennis Center. What’s cool about this is that it means you can keep up with every single match going on at any given time, so you won’t have to choose between watching Sharapova or Djokovic. The iPad app also has a unique social dashboard with stats and data, which stays open at all times. All of this is dubbed the “SlamTracker” technology, and incorporates SPSS technology, the predictive analytics behind “Keys to the Match.”

The concept IBM is employing is not a new one, but it’s how the company is incorporating technologies that makes it so innovative. For scoring and data, IBM is using WebSphere MQ, a foundation message-based middleware scoring system. This is also a super speedy technology (yes, it’s faster than Serena’s serve), which is why the stats can be delivered almost instantaneously. For the overall app architecture and appearance, IBM is relying on its WebSphere technologies. This is a great example of repurposing technologies in a creative and timely way.

Are you using any of the apps to enhance your US Open experience? If so, tell us how you like them!

Now if only they made an app for improving backhands…

Gaby Berkman is a Consultant at Greenough. Follow her on Twitter: @Gabyberk