New eye tracking research helps piano students become better sight readers

A student works with eyetracker.

Cicero said our eyes are the windows to our soul, but struggling music students and their teachers may find it handy to know eyes can also pinpoint our struggles reading music.

This came to light from the curiosity of a professor of piano at UAA, Timothy Smith, looking for ways to help students pass a challenging part of their piano exam. Successful students must sight read a four-voice chorale, thick with notes. That means playing it without advance practice. Such complex music requires them to “chunk” together a lot of notes, or get more data with a single glance, while staying on tempo and playing accurately. Not easy.

Smith has been an Evelyn Woods speed-reading fan, and even taught the course one summer. Could studying eye gaze and fixation on musical scores, as the speed-reading program did for words, be of help to his struggling students? He posed that question to two UAA computer science professors skilled at programming eye tracking software.

Kenrick Mock and Bogdan Hoanca had earned a patent in 2011 for their computer security software. Essentially they programmed an eye-tracking device to allow a computer user to sign on without a password by tracking the user’s gaze on a particular set of icons. The eye gaze pattern replaced the password.

To figure out if eye tracking would help piano students, Mock and Hoanca refashioned their software over digital musical scores and Smith began recording student eye movements. A year of testing followed.

As it turns out, the answer is yes. The three professors now have a patent pending on eye tracking software capable of delivering feedback to a music student that compares to what a human teacher can deliver.

Here’s how they got there.

Typically, a music teacher assesses student performance by taking in the music as it is played.
“I would just sit back and listen,” Smith said. “How musical is it? Is it flowing? Or is it really a chore? Most of the time, it was just my appraisal of what I heard and what I saw.”

A sample of an eye tracking printout.

Of course, he noticed students’ eyes. Sometimes he even sat opposite them behind the piano, watching their eyes.
“Students who are less adept, you see them glance at their hands,” Smith said. “The idea of staying focused and also being able to generate the notes by feel or with your periphery—that’s the goal. The eye tracker can tell us when the eyes go off the page and how many times they do.”

Smith scores his students from 1-10 on their overall performance. That score is made up of about a half-dozen metrics that Smith evaluates aurally. He sent his list to the computer scientists, and they evaluated the eye tracking data to see if it could measure the same things. Turns out it could, very well.

Besides accuracy and tempo, Smith listens for pausing, regression (playing a wrong note, and playing it again), fixation, length of fixation and ability to read ahead. The eye tracking software reports exactly where the student’s eyes spend time. Advanced players tend to move forward linearly with clean up and down movements. Novice players? Their eyes are all over the place, including going back to mistakes, glancing off the page, lingering on troublesome notes.

Part of the trio just delivered a paper on their results at a national eye tracking conference. A company there presented a $100 eye tracker, which compares to the $30,000-$40,000 eye tracking equipment programmed at UAA. “There is a lot of potential out there for something like this to be inexpensive,” Hoanca said.

The ultimate goal is a low-cost device like a tablet, loaded with eye tracking software overlaid on musical scores. The team imagines having a library of these ready for students to check out. Once they achieve a score of 8 or 10, they can advance to the next one.

The student will play the music and record her eye movements. She can play the eye-tracking video back immediately, and also receive quick feedback on what the results show. Besides one overall score, it could deliver individual scores on each metric, plus some text feedback, like “You weren’t steady with the delivery of your beat pattern."

Will this replace music teachers? They don’t think so.

“It’s one thing to have music lessons once or a few times a week,” Hoanca said. “It’s something else to have a device anywhere and anytime you’re practicing that can give you this kind of feedback.”

A version of this story by Kathleen McCoy ran in the Anchorage Daily News on Sunday, April 6, 2014.