Typed user input: not to be prepared is the greatest of crimes

                             to be prepared beforehand for any contingency is the greatest of virtues

Recovered hypothesis: not to be prepared is the greatest of crimes

                               to be prepared beforehand for any contingency is the greatest of virtues

METEOR scores: 1.00, 1.00

Typed user input: i am sitting in a chair tryinh to spell sords for a certain project

Recovered hypothesis: i a stirring in chair trying to spell words for a certain project

METEOR score: 0.80

Typed user input: i freaked out by the possibilitu of someone else reafing this

Recovered hypothesis: i created out by the possibility of committee else reading the

METEOR score: 0.65

Typed user input: if you know your enemy and you know yourself you need not fear the results of a hundred battles

                        if you know yourself but not the enemy for every victory gained you will also suffer a defeat

                        if you know neither the enemy nor yourself you will succumb in every battle

Recovered hypothesis: if you know your enemy and you know yourself you need of fear the results of hundred battles

                                      if you know yourself but not the enemy for every victory gained you i also suffer defeat

                                      if you know neither the enemy for yourself you will succumb in every battle

METEOR scores: 0.89, 0.88, 0.91

Abstract: We investigate the implications of the ubiquity of personal mobile devices and reveal new techniques for compromising the privacy of users typing on virtual keyboards. Specifically, we show that so-called compromising reflections (in, for example, a victim’s sunglasses) of a device’s screen are sufficient to enable automated reconstruction, from video, of text typed on a virtual keyboard. Despite our deliberate use of low cost commodity video cameras, we are able to compensate for variables such as arbitrary camera and device positioning and motion through the application of advanced computer vision and machine learning techniques. Using footage captured in realistic environments (e.g., on a bus), we show that we are able to reconstruct fluent translations of recorded data in almost all of the test cases, correcting users’ typing mistakes at the same time. We believe these results highlight the importance of adjusting privacy expectations in response to emerging technologies.

Automatic Reconstruction of Typed Input from Compromising Reflections

Rahul Raguram, Andrew White, Dibyendusekhar Goswami, Fabian Monrose and Jan-Michael Frahm. iSpy: Automatic Reconstruction of Typed Input from Compromising Reflections, ACM Conference on Computer and Communications Security (CCS), 2011 [PDF]

Paper

Overview Video

Results

(left: input, right: aligned frames with key detections)

Direct

Surveillance

Typed user input: the multiple human needs and desires that demand privacy among two or more people in the midst of social life must inevitably lead to cryptology wherever men thrive and wherever they write

Recovered hypothesis: the multiple human needs and desires that demand privacy among two or more people in the midst of social life must inevitably lead to cryptology wherever men thrive and wherever they write

METEOR score: 1.00

Direct

Surveillance

Direct

Surveillance

Sunglass

Reflection

Sunglass

Reflection