As smartphones and wearables collect increasingly sophisticated health data from consumers, advertisers are now considering how this can be used to serve more effective content
By Laura Entis
Earlier this month, Novak Djokovic and Roger Federer came head-to-head at the Wimbledon men’s final. While the match between the world’s No 1 and No 2 players lacked the sharp-edged suspense of their 2012 Wimbledon semi-final showdown, it was nonetheless a display of stunning, exhilarating, heart-pumping tennis.
The match’s most suspenseful moment was the second-set tiebreak, in which Federer won 10-8, tying the game at one set all. But Djokovic went on to easily cinch the next two sets, winning the match.
As Djokovic cruised toward the Wimbledon title, his fans’ heart rates remained regular, says Chris Cardew, the joint head of strategy at Mindshare, a media agency network. This wasn’t just speculation; on behalf of Jaguar, Mindshare was recording and analyzing the heart rate variability, localized audio, motion and skin temperature of 20 fans in the crowd, via sensor-equipped cuffs. The goal was to create an interactive display of the crowd’s fluctuating emotions at the match, which Jaguar could then use living ads.
While the cuff could not say if a person felt tense, surprised or excited, analyzing the data points within the context of a tennis match provided a solid window into interpreting emotion, says Cardew. “A change in heart rate variability is a good signal that there has been a shift,” he said. During tense moments, people moved less, while heart rates tended to quicken when a person’s favorite player was in trouble, like after a botched serve or before a match point. When players were in control, heart rates remained regular.
From the data, Cardew and his team of 30 analysts could infer a spectator’s emotional response to each point, and from there, infer his or her chosen player.
Mindshare launched a similar program at Cannes this year, in which it gave 100 Apple watches to attendees, collected their biometric data over the course of three days, analyzed it, and created an engagement map of the festival.
While these examples amount to little more than splashy promotional plays, it’s not a stretch to envision the underlying technology in these campaigns repurposed to collect valuable biometric data points that marketers can use to determine the most effective ads.
“If we know your sweat and pulse and additional information, we can predict: are you feeling happy, sad, excited?” says Michael Cohen, a visiting scholar at NYU Stern School of Business’s marketing department. Add location data and browsing history, and “we know where you are, and how you are feeling”.
Advertisers have long worked to determine consumer mindset, and in recent years, the available techniques have grown more sophisticated. Affectiva, an emotion measurement startup founded in 2009, uses face-reading software to analyze micro expressions, working with marketers and brands to expose consumers’ unfiltered sentiment. To date, the company’s software algorithm has analyzed more than 2.7 million faces. Clients have included Coca-Cola, Unilever, Mars, Kellogg and CBS. Its bread and butter is market research.
“Our technology records individuals’ facial expressions as they watch an ad. We can then see frame-by-frame, moment by moment, what these facial expressions reveal, measure them and provide analytics,” says Gabi Zijderveld, Affectiva’s vice president of marketing and product strategy.
Meanwhile, advertisers are able to access and mine a trove of information, compiling data points – such as consumers’ location, browsing history, purchase history, and the time of day – to build a sophisticated user profile.
Biometric data aside, the technology consumers use day-to-day already provides a window into their inner lives. Each smartphone has an ID that roots individual apps to the same device. According to Michael Lieberman, US creative director of mobile marketing agency Joule, advertisers can target consumers according to what apps they have on their smartphones. “I know you are interested in the news and I can target you that way.”
Meanwhile apps such as Facebook that require a login and are linked to consumers’ IDs allow third-party mobile ad networks to target users outside the Facebook app, using their Facebook data. (When Facebook provides user profiles to ad networks, it keeps user ID anonymous). “They know everything, these guys,” Lieberman says. “It’s why Facebook can be effective and charge a premium, because it can help you target the right people.”
On top of all this information, advertisers can access GPS data, via Google Maps or any of the multitude of apps that track location. (Many do so by default; permission is covered under the app’s terms of service.) Smartphones, in other words, reveal – down to the street corner – an individual’s location at any given moment, enabling advertisers to serve relevant location-based ads, like, say, a nearby bar offering a happy hour promotion, and determine daily routines, like “this device is in midtown Manhattan from 8am to 6pm Monday through Friday, and typically checks in at an East Village yoga studio on Tuesdays and Thursdays”. If someone looks up a Chinese restaurant on Yelp, and then visits the location two hours later, advertisers can access that data.
“Yelp, Facebook, Google Maps – they all do it. That’s what you sign up for when you say, ‘Hey I want this app’. You are opting into software and technology that you see as being valuable to your life,” says Stern’s Cohen.
All of this information is then parsed in an effort to determine a consumer’s mindset. “The strength of mobile is its ability to provide context,” says Lieberman. “It’s one thing to understand that someone is in your target audience, but it’s another to understand how receptive they are to your message right now, versus another variation of it.”
The next logical step is determining a consumer’s mood. “It’s tough, but it doesn’t mean that it’s not something we are looking at,” says Lieberman.
Emotion-based targeted advertising
As smartphones and separate wearable devices are able to collect increasingly sophisticated information, the next step feels inevitable: serving and creating ads that respond to an individual’s fluctuating biometric data. (Back in January 2014, Apple applied for a patent to use users’ moods to serve targeted content.) Indicators such as heart rate, galvanic temperature, walking speed and engagement could work together to reveal genuine emotion, a potentially more predictive indicator of a consumer’s future behavior than mindset alone, particularly when paired with browsing history, GPS data and all the other information already available to advertisers.
“Some people are emotional eaters or shoppers. Some people are the opposite,” says Cohen. “We can track and tailor a message or a shopping experience to your profile based on all the information we’ve learned from monitoring you.” Ultimately, adding biometric data to an advertiser’s arsenal sharpens its ability to deliver messages.
Right now, both Lieberman and Loren Hillberg, president and GM of Thinknear, a location technology company, say that this is all hypothetical; while Hillberg says he wouldn’t be surprised if, in the near future, clients want campaigns that incorporate health data, this hasn’t happened yet.
And there’s still the question of the data’s predictive ability. “If we understand that your heart rate is up, and you’ve been moving at a certain speed for a while, we can guess that you’ve just finished a run, and from there we can infer that you feel good,” says Lieberman. “But is it useful?” Knowing that someone’s heart rate is elevated is one thing; being able to correctly interpret and use that data point to serve an effective ad is something else entirely.
They monitor hearts, count calories … but are health apps any good for you?
Privacy is, of course, the elephant in the room. Advertisers will need to tread carefully or run the risk of spooking consumers. Until that is addressed, data collectors and advertisers alike need to be careful. Apple, for example, has banned developers from selling Healthkit data to ad networks.
“The infrastructure to deliver this advertising exists. It’s more a question of how, or when, consumers are willing to share their information,” says Hillberg. For them to do that, there has to be a value incentive, just as there is when Google Maps accesses GPS – privacy is forfeited for the sake of convenience and value.
This is exactly how Affectiva spins its desire to access consumers’ emotional data points; its mission is to “bring emotional intelligence to the digital world”. What’s currently missing from targeted ads, says Zijderveld, is emotional personalization. The question the company is continually trying to refine is “how can technology be used to create targeted content based on people’s emotional responses?”
While she won’t disclose specifics, Zijderveld says that Affectiva is currently experimenting with combining data points from its face-reading software with biometric data points in a lab in order to gain a more accurate emotional reading. There is an obvious business aspect to emotions, she says. “They influence every aspect of how we make decisions big and small, from whom we marry to what we buy.”
In Cohen’s view, it’s inevitable that consumers will eventually hand over their health data once the applications are valuable enough for them to do so. “People are already slowly giving up every aspect of their life to the information age,” he says.
Back at Wimbledon, the 20 spectators were more than willing to allow Mindshare and Jaguar to monitor their vital signs. Says Cardew: “They were superfans who wanted to be part of the experience."