With so-called “gaze data” now recordable using everyday PCs, tablets, and smartphones, the research offers a new way of using technology to help people who may be trying to learn a new language, or who are seeking to improve their reading skills in some other way.
“It’s easy to imagine a program in the classroom that, instead of waiting for a teacher to intervene, could learn when to pop up suggested translations or synonyms if a student struggles with a particular word,” Ph.D. researcher Sigrid Klerke told Digital Trends.
Klerke’s work uses computer vision and machine learning technology to recognize people’s eyes as they move across a piece of text. Impressively, it can determine from this movement which words a person is likely reading on a document.
“What the eye-tracking does is to use a recording of your eyes to make a geometrical calculation about which point on the screen your eyes are fixating on at a particular moment,” Klerle said. “For reading, we can pretty accurately work out where your eyes are going to have to be in order to make sense of a certain piece of text. That’s because you can only see 1-2 degrees of the central visual field in sharp detail. As a result, people have to continuously move their eyes along a line of text in order to be able to read all the words.”
Klerke’s work builds on the observation that eyes fixate repeatedly on words or sentences that are difficult. Her research eschews building a general model to fit all readers in favor of one that is customized to each individual reader: thereby absorbing all of the personal quirks we might exhibit when we read.
Based on just a few short seconds of recorded gaze data, the model can then accurately predict whether a word is one that a person is having trouble with. As she said, “I’ve taught a machine to look at someone’s gaze and to be able to recognize stumbling behavior.”
It’s not just reading apps where this could have application, either. Klerke said that her research could also be used by services like Google Translate to suggest whether particular translations may be faulty. If a large number of French speakers struggle with a particular word or phrase when Google translates another language into French, it may suggest the translation needs to be improved.
“Our eyes can tell us a lot,” Klerke concluded. “People don’t always know if a particular word is taking more energy for them to process. There are some easy cases where it’s obvious, but others where it is much harder. In some of these cases, our eyes will reveal this without a person having to be aware of it consciously.”
