Science

Eye tracking can reveal an unbelievable amount of information about you

Eye tracking can reveal an unb...
The windows of your soul can reveal an extraordinary amount about you, especially when a machine learning algorithm is watching
The windows of your soul can reveal an extraordinary amount about you, especially when a machine learning algorithm is watching
View 2 Images
Eye tracking devices can see much more than just what you're choosing to look at, and infer a huge amount of sensitive information
1/2
Eye tracking devices can see much more than just what you're choosing to look at, and infer a huge amount of sensitive information
The windows of your soul can reveal an extraordinary amount about you, especially when a machine learning algorithm is watching
2/2
The windows of your soul can reveal an extraordinary amount about you, especially when a machine learning algorithm is watching

Eye tracking technology is starting to pop up more and more, keeping track of where you're looking and how your pupils and irises are reacting for a variety of different purposes. It doesn't require particularly complex technology; a HD video camera that can watch your face is enough to collect the data.

But according to a 2020 research review, this data can divulge an extraordinary amount of information about you when it's crunched through advanced data analysis systems. "Our analysis of the literature," reads the paper's abstract, "shows that eye tracking data may implicitly contain information about a user’s biometric identity, gender, age, ethnicity, body weight, personality traits, drug consumption habits, emotional state, skills and abilities, fears, interests, and sexual preferences."

That's not all; "Certain eye tracking measures," says the review, "may even reveal specific cognitive processes and can be used to diagnose various physical and mental health conditions." According to Grandview Research, "the analyzed data is used to study a myriad of psychiatric and neurological conditions, such as autism spectrum disorder (ASD), attention deficit hyperactivity disorder (ADHD), obsessive-compulsive disorder (OCD), Parkinson‘s, Alzheimer‘s, and Schizophrenia, among others."

What exactly are they looking at? Yes, eye tracking systems can see which way your eyes are pointing to infer what you're looking at, but that's far from the end of the story. They can also track the length of fixations, rapid eye motions between fixations, smooth pursuit movements and things like the acceleration and maximum speed of your eye movements.

They can analyze your eyelids, watching how far open your eyes are, how often you're blinking and how long your eyes are staying shut when you do. They can take note of redness and see how watery or dry your eyes are through reflections. They can measure the dilation of your pupils – famously an indication of sexual interest or arousal, but also linked to drug use, fear and certain types of brain damage. They can note your eye color and iris texture.

Eye tracking devices can see much more than just what you're choosing to look at, and infer a huge amount of sensitive information
Eye tracking devices can see much more than just what you're choosing to look at, and infer a huge amount of sensitive information

They can also note your facial expressions, eyebrow movements, number and depth of wrinkles around your eyes, your eye shape and skin color. There's a reason why the eyes are spoken of as "the windows of the soul" – they can tell us a ton about the person we're interacting with, and through evolution and pattern recognition, we each learn to glean a huge amount of information about people through their eyes.

Biometric identity can be established using a combination of things. The colors and patterns in your irises, for starters, can be used almost like a fingerprint. But so can your pupil reactivity, your gaze velocity and the trajectories your eyes take when following a moving object; mechanical and brain function differences make these things unique to you.

Then there's mental workload – an area in which eye tracking sometimes performs better than an EEG. Pupil dilation can be used as a measure of task difficulty and mental effort. Your blink rate correlates with dopamine levels, signifying learning and goal-directed behavior.

Some of this stuff might be apparent to a human viewer; we can tell when somebody's thinking, or accessing memories, or getting imaginative by watching the eye movements of others. But a lot of it involves the strangely opaque world of AI and deep learning.

The way our eyes move in everyday life turns out to be a surprisingly strong predictor of our personalities when crunched through these pattern-finding machine learning algorithms. One fascinating study found it could reliably predict what range a subject would score in on four out of the "big five" personality traits (neuroticism, extraversion, agreeableness, conscientiousness), as well as curiosity. Others have been able to identify your phobias, specific mating preferences, interests and areas of expertise.

Emotional data from the eyes is very detailed, moving far beyond just positive, neutral and negative into specifics like happiness, enthusiasm, stress, worry, humorous moods, disgust, curiosity, distress, nervousness, hostility, fear, anger, sadness and surprise. Emotional intensity can be measured, and interestingly, researchers have demonstrated that they can distinguish between instinctive emotional reactions and rational ones.

The list goes on and on, but you get the picture: under the right set of circumstances, any device that can watch your eyes carefully can learn a disconcerting amount of information about you – and while you can deliberately choose where you look, to an extent, there's other things you can't help giving away.

To mitigate these concerns, the researchers point out that many ocular cues can have confusing or contradictory meanings, and lot of the public research to date has been done in a lab, where it's cheap and well controlled, rather than in the chaos of the real world.

On the other hand, this review covers only the publicly available research; "it may reasonably be assumed," write the study's authors, "that some of the companies with access to eye tracking data from consumer devices (e.g., device manufacturers, ecosystem providers) possess larger sets of training data, more technical expertise, and more financial resources than the researchers cited in this paper. Facebook, for example, a pioneer in virtual reality and eye tracking technology, is also one of the wealthiest and most profitable companies in the world with a multi-billion dollar budget for research and development and a user base of over 2.3 billion people."

The benefits of this kind of technology are clear, particularly when paired with virtual reality or augmented reality technology; devices and apps can become extremely personalized and responsive to your current state and interests. The tech offers extraordinary opportunities for marketers to target you, not only with the right products, but with the right approaches, messengers and timing to make you optimally receptive.

Since it is unlikely that companies will voluntarily refrain from using or selling personal information that can be extracted from already collected data, there should be strong regulatory incentives and controls

And of course, with so much real-time feedback coming in, the chances to surveil, manipulate and coerce you will be higher than ever. The privacy implications will be significantly worse than the world (or a prospective employer, or a prospective insurer) seeing your internet history.

In order to preserve the benefits while minimizing the risks, the researchers suggest measures like keeping raw data only for temporary use on the device itself, and only passing it on to app and device manufacturers in aggregated, pre-crunched forms or with random statistical noise added to "fuzz" the data.

But since this stuff fits into a broader picture, where we can be compromised by advanced analysis on all manner of data streams being collected on us 24/7, governments will need to get ahead of the curve on privacy. "Since it is unlikely," say the authors, "that companies will voluntarily refrain from using or selling personal information that can be extracted from already collected data, there should be strong regulatory incentives and controls."

It's certainly food for thought, particularly as we move toward the magical convenience of nerve-tracking augmented reality and Facebook pushes forward in developing hyper-realistic VR avatars that allow people to express themselves with ever-fuller ranges of facial and body language cues.

The full study – What Does Your Gaze Reveal About You? On the Privacy Implications of Eye Tracking – is available for free at Springer Link.

10 comments
guzmanchinky
Very interesting, I'm sure we'll see much more of this in the future, linked to VR/AR as well.
piperTom
Facebook?!? I'd worry more about the NSA, which has all of Facebook's data AND it's research. Good luck getting any "strong regulatory incentives" in that quarter. Put not your faith in regulations. Instead look for open source research into counter measures.
Douglas Rogers
Will my computer have auto shutdown with no camera attached?
BlueOak
Fascinating stuff if the results can be firmed up.

Will privacy protections ever catch up with technology?

Unfortunately, our legislators are ill-equipped to generalize privacy protections in anticipation of technological innovations.

Does legislation need to be so general as to say something like "all biometric information specific to an individual defaults to protected unless the collection or use of it is expressly granted by the individual"?
EJ222
One one hand, I'm super excited about eye tracking tech. On top of foveated rendering, AR and such, this data would be insightful. I want to see what my eye movements say about me.

On the other... now I'm *REALLY* never buying an Oculus product. There might be hope for Europe, but there's no way privacy legislation will keep up with this in the U.S.
wolf0579
And you know someone will write (probably already has) a piece of code to utilize the camera on your laptop or your phone to see what your looking at on the page as you browse the web.
LuckyLou
Hmmmm... suspiciously close to April 1.
Baker Steve
Presumably this data can only be interpreted properly when the subject is presented with a set 9f controlled visual stimuli, and therefore is lab conditions?
Philip Argy
"Look into my eyes" takes on more sinister connotations!
ljaques
Gee, after reading this, I might be thinking the eyes are the window to the SOUL!