–they are all lousy reflections of real life.
Look, I understand the need for testing. I’ll even admit that some of my work has improved through qualitative. But in nearly twenty-five years, I’ve never seen any piece of work escape the soul-crushing process of quantitative testing without compromising everything that made it remarkable in the first place. Worse, this kind of testing ignores the undeniable fact that people lie. Not intentionally or maliciously, but when we’re asked to reveal ourselves to another, exaggeration and overstatement rule the day. Just read the personal ads…
There may once have been a time when a product could actually have a unique selling proposition without three identical alternatives lined up beside it on the Walgreen’s shelf. But that was long before the personal computer, the cell phone, and eBay.
Still, too many otherwise smart advertisers continue to worship these false gods. Some corporate cultures even dole out media dollars based on AI scores. And so whether to earn a larger investment dollar or to simply cover the brand manager’s butt, marketers plow ridiculous resources into long processes where the only sure thing is that the advertising they develop and test will absolutely not result in sales.
Imagine if those same advertisers took the money they spend on animatics and instead made two or three rich media banners. They could then run them on-line in test markets and know immediately which one performed the best. On the fly, they could adjust the creative, tweaking the art and copy to see if it made any difference in conversion. By running this kind of real world test, you remove the all too human penchant for exaggeration and overstatement and instead, see how people actually respond in real life when there are no two way mirrors or bowls filled with M&M’s. Better still, even ads that don’t perform as well, will still drive some sales for you. It’s an indisputable win-win.
So why don’t more advertisers innovate their testing? Probably because it’s hard to tell the difference between a benchmark and a bad habit. And that’s a darned shame.