Calling one segment “A” and the other “B” doesn’t make it an A/B test.
You are not a representative sample. You have skin in the game and you know too much.
Your typical user is not a programmer. Don’t force him or her to think like one.
Designers rule at Apple. Programmers rule at Microsoft. One created the iPod. The other created the Zune. UX matters.
If you can’t find the relevant copy on the site, it doesn’t matter how cool the font is.
How and when data is captured is a vital part of decision-making in call center scripting, digital architecture, sales presentations, just about everything that touches a customer. There are things you can do to help make it better, but some issues will be always with us. So, here’s why your data is crapola, in no particular order…
1. TEXTING (yes, TEXTING)
In the data world, spelling counts. Whether data is being entered by professionals or by the customers themselves, text fields will be rife with errors. That is true no matter how clever you get with the drop-down boxes or type-ahead functionality. The proliferation of text-speak has made this worse, and spelling is rapidly becoming a matter of personal choice. That doesn’t bode well for text-mining. And no, spell-correct is not the antidote. Entire websites are devoted to how that can go horribly and hilariously wrong.
2. IT WORKLOAD
“Why can’t the digital dev folks just put another box in the online form so the prospect can enter their promo code?”
They can! They will! Just as soon as that work order rises up to the top of the priority list. Right now the project manager has it in the backlog. Where work orders go to ripen, get covered with brown spots, and die.
3.BAD DROPDOWNS
The data was shocking. Month after month, nearly a third of all disenrolling customers were dropping service due to bankruptcy. Nationwide. All segments. So I planted myself in the call center for a few days. In the CRM system, the drop-down box marked “Reason for Leaving” had 35 choices. “Bankruptcy” starts with a B, so it came up at the top of the drop-down. Too bad “Alien Invasion” wasn’t one of the choices, it would have been caught more quickly. So, yeah, that happens. In my experience, 12 choices is the limit for an effective drop-down in a call center. Any more than 12, and boom – 30% of the country mysteriously goes bankrupt.
4. POOR REQUIREMENTS
The marketer: “This email address is blank – it should be a required field.”
The analyst: “Someone input their mailing address in the email field.”
The email vendor: “Lots of these email addresses are undeliverable.”
Coding an email field with an input mask requiring a “@” character and a period will solve the analyst’s problem. Making it a required field will mean 100% of records are populated, which will satisfy the marketer (at least temporarily). Coding an email field with that input mask plus making it a required field will get you a lot of bad email addresses like “noneofyourbiz@buzzoff.com”. That can get you blacklisted with ISP’s. It can also spike your abandonment rate on the digital form page.
But marketers – it’s not the developer’s job to know that – it’s yours. He or she will code to your requirement specs. Think through the ramifications of stakeholder requirements – and make sure the requirements reflect a considered decision.
5. USER EXPERIENCE
First, let’s get one thing straight. UX and good data capture are NOT mutually exclusive. But what’s great for data capture isn’t always great for the user experience, and the two must be balanced. I once had an HR stakeholder tell me that data was vitally important to her, so it was a requirement that users submit 10 fields of personal data into a web form before they could browse jobs on the company’s career site. I told her that strategy would get her the best data and the three most committed job seekers she ever saw. The rest would bail.
6. COMMUNICATION
The promo postcards start hitting consumer mailboxes at 8am, and the TV ad blitz started airing at 9am. The campaign is a hit! Responses galore! Great! Except Marketing forgot to notify the Customer Service Manager that the promo schedule had changed, so she isn’t staffed up for the traffic onslaught. The unfortunate reps that are schededuled today can’t keep up with the volume. Calls are going unanswered. Their priority today is not great data capture. Their priority today is to take as many calls as possible and make it through their shift alive.
7. INCENTIVES
Sales reps’ income largely depends on how many sales units and revenue dollars they bring in. For customer service reps, bonus criteria often include average handle time, time-to-answer, off-hook time, etc. Sometimes data accuracy is part of the incentive formula, but it’s almost never the lion’s share. Generally, when short-term incentives are introduced for better data capture, data capture gets better. And when the incentive period is through, the gains recede, although hopefully to a little higher than baseline.
I don’t know if this is a universal trait – and I’m not entirely sure why – but my experience with Agile dev professionals is that there is a tendency to resist testing. I’m not talking about standard software Quality Assurance (incredibly, that’s not always a given either). I’m talking about A/B or multivariate testing, and true UX testing. These are things I have actually heard in my career as a marketing professional from dev groups.
“We don’t have the resources to do A/B testing – it would double development time.”
This person that said it misunderstood what A/B testing is. In most cases, you’re testing something new against your existing site – which is already developed and in production. So you’re only developing the new thing – and you were doing that anyway. Where’s the extra work?
If you’re introducing multivariate testing, developing multiple versions of something new would create extra work for a dev team that is used to allowing development of only one version at a time. But that’s the right way to do it for maximum iterative improvement.
“We are strapped for time – we don’t have time to develop things that won’t make it into production.”
You’re saying you don’t have time to find out if the change you’re working on to enhance click-thru rates will ratchet up the bounce rate instead. That’s dangerous to your website’s health. It means the whole enterprise needs to guess right 100% of the time – which just doesn’t happen. You’re also essentially saying you only have time to develop and deliver something once – and that’s it. Not very iterative, is it?
“Our dev team/marketing team/product manager would be demoralized if all their hard work never saw the light of day.”
That’s why they call it work. That’s why they have to pay you to do it. Certainly, we can make it fun. But ultimately, decisions on what does and doesn’t go into the final software release MUST be based on how that software accomplishes business goals. Team members who only feel validated by having their work seen can start a blog. Or therapy.
“Doing UX testing on only six people isn’t statistically significant.”
The same person who told me this also told me that he wanted a study on the top 1% of revenue-generators because that would be enough customers to be statistically significant (but surely you understand that the top 1% isn’t a representative sample for…aw, screw it, where’s my propeller hat?). UX testing is not the same as quantitative testing – five or six subjects are plenty. If you’re not sure why, read Jeff Sauro’s great blog post about user testing and sample sizes.
“We don’t need to test – we have a good feel for what users want.”
Robust, qualitative user testing tells you where you need to tweak to enhance your software’s usability. User testing is necessary, in part, because you may be too much of an insider, too in love with your own work, or too defensive about your work to identify the usability problems that can torpedo software adoption. It is also necessary because it helps remove our own biases and opinions (the dreaded “I think users would want”) as impediments to truly successful software. Skip it and you’ll risk shipping software that works but that users won’t use.
I had my frat buddy/luddite office mate/mother test it, and they said it was great!
User testing can’t be successful with insiders. Or insiders’ friends. Or insiders’ mothers. Real users won’t forgive you if it worked on your machine but doesn’t work on theirs. Users won’t let you explain the jargon that makes no sense to them and perfect sense to you. UX is a professional discipline – and no, you can’t do it just as well.