University of Michigan researchers create screen protection system to fend off shoulder surfers

August 9, 2023
Eye-Shield uses an innovative pixelation scheme to obscure device screens when viewed from a distance, safeguarding against shoulder surfing attacks.
Comparison of four phone screens viewed from an angle. Two are normal/unprotected. Two are with Eye-Shield deployed - the text/images are so blurry as to be indecipherable.
Phone screens with Eye-Shield deployed to the right of unprotected phone screens. Eye-Shield reduces shoulder surfers’ ability to decipher content on your device by about 60%.

Most of us use our phones in public without worry, but accessing private information when out and about can be a risky undertaking. Increasingly, thieves are resorting to a relatively simple means of stealing data from unknowing strangers: shoulder surfing.

From public transit to waiting rooms to elevators and beyond, shoulder surfers exploit physical proximity in crowded places to view device screens and obtain potentially sensitive data, such as passwords, PINs, financial information, and more, simply by peering over your shoulder. In one recent high-profile case, a man in the UK had £22,000 stolen after a shoulder surfing incident.

In response to this threat, researchers at the University of Michigan have developed Eye-Shield, an innovative screen protection system that obscures images and text on your phone and other devices when viewed from a distance. Eye-Shield was designed by Brian Tang, doctoral candidate in computer science and engineering, and Kevin and Nancy O’Connor Professor of Computer Science Kang G. Shin.

“The key question we are trying to answer in this project,” said Tang, “is how we can improve on existing technologies to protect users’ information while browsing the web, watching videos, or doing virtually any activity on their phones, tablets, or laptops.”

Previous solutions have been ineffective, inconvenient, or limited in scope. Some involve the application of a physical privacy film to your device, which can’t be turned off or easily removed, offers only limited protection, and in many cases prevents the use of a screen protector. Others take the form of apps that target specific functions, including one that obscures numbers by overlaying low- and high-frequency images and another that replaces text with hard-to-read handwriting.

Eye-Shield, on the other hand, is designed to live in your device as a free, built-in option that you can toggle on and off depending on your needs. The program leverages our visual perception of contrast to blur text and images at a distance.

“Let’s say you have two light sources,” said Tang. “Up close, it is easy to see both light sources. But if you look at them from a distance, they merge into a single light source.”

Diagram showing the phenomenon by which two light sources merge into a single light source as a distance.
The visual perception mechanism that inspired Eye-Shield. At a distance, two light sources merge into one.

Using this phenomenon, the researchers developed a checkered grid made up of contrasting colored boxes. The use of this pattern causes pixels to merge when viewed from a certain distance or angle, making the device screen appear blurry to potential shoulder surfers but not to the person using the device.

Eye-Shield is composed of a complex set of algorithms that determine the right size of these checkered grid squares based on various parameters, such as distance from the screen, text font size, pixel density, and more. These calculations ensure appropriate protection without substantially altering user experience.

Tests with a group of over 20 participants showed that Eye-Shield was highly effective in protecting device screens from shoulder surfing. The researchers placed participants at varying distances and angles from a device to replicate a shoulder surfing situation and then asked them to identify the content, consisting of both text and images.

Through these tests, they found that Eye-Shield reduced shoulder surfers’ ability to recognize text and images by roughly 60%. On an unprotected screen, participants standing at a distance of 41 inches were able to recognize over 80% of screen content. With Eye-Shield, this number decreased to less than 25%. This number is even lower, just over 15%, for text alone. 

This demo shows how a device screen with Eye-Shield would appear to a shoulder surfer at a distance of 20 inches and a 45-degree angle. The strength of Eye-Shield’s grid effect is increased progressively across the sample screens, with the highest level at the left. The rightmost sample screen has no protection deployed.

In addition to fending off shoulder surfers, Eye-Shield is able to preserve the screen’s appearance for the intended user, an issue that existing solutions have largely failed to overcome. Eye-Shield also functions in real time with little to no effect on speed or performance.

“With Eye-Shield deployed, we found that participants were able to recognize 90% of the content on a screen when viewed up close without having seen it before,” said Tang. “Using your own phone, when you have some context for what you’re seeing, these numbers will only be higher.”

This demo shows how a device screen with Eye-Shield would appear to the intended user. The strength of Eye-Shield’s grid effect is increased progressively across the sample screens, with the highest level at the left and the rightmost screen with no protective effect.

Tang and Prof. Shin have filed a patent for Eye-Shield, and their ultimate goal is to work with device designers and manufacturers to integrate their screen protection program into commonly used operating systems. In the medium term, Tang and Prof. Shin aim to make further improvements to Eye-Shield’s system by experimenting with contrast and other parameters to boost its efficacy even further. 

“We hope that this work will lay a foundation for further advances to improve the safety of devices and protect people’s private information,” said Tang.

This work is reported in Tang and Shin’s paper titled Eye-Shield: Real-Time Protection of Mobile Device Screen Information from Shoulder Surfing, appearing at the 2023 USENIX Security Symposium.