Tobii Eye Tracking Technology originally designed for Medical now being used for Video Games

My first exposure to Tobii Eye Tracking Technology occurred when I was hanging out with my cousin Buck. He has Cerebral Palsy and as a kid, I was barely able to communicate with him. Having Cerebral Palsy for him meant that he had limited motor function yet his brain was still working at full speed. He’s only 2 years younger than me but because he was unable to speak words, there was a communication barrier which made it far more difficult for me to connect with him than it normally would be for someone his age. My first conversations with him involved a smile for yes and turning his head to the left for no. This made our conversations very simple and yet, there was so much more going through Buck’s head that I couldn’t learn about.

Buck 1

However, as I grew older I got to know Buck through other methods. Buck is able to pretty precisely move his head left and right so I was able to ask him preferential questions about his favorite sports teams (Buck loves the Red Sox and the Boston Bruins). I would hold up each finger and ask him which game was on today that he was excited for. Yet, there still was a barrier to know just what Buck was thinking at a deeper level.

giphy25

When Buck acquired his first eye Tracking Technology computer, I was finally able to hear Buck speak! When setup in his wheelchair, Buck could gaze at icons on a screen and the computer would be able to track his eye movements to know where he was staring at. When he centers on an icon for long enough, the computer registers this as a click and the word or phrase can be recited by the device. Using this for the first time with Buck was unlike any other conversation we had before. He was able to ask me questions back, or make quips and interject when other people were speaking. The conversation felt much more dynamic and as a result Buck could participate more easily.

Buck 4.jpg

Pictured Below is what the Screen for Buck might look like. Customization is high and different word sets can be selected depending on the setting. For example, if he is at the Beach, it might have a bunch of Icons which are labeled Towel, Sunglasses, and Sunscreen.

Tobii Computer Speech Generator

More recently, Buck has actually gained a new feature with this device integrated with his cell phone. He is now able to make phone calls and send text messages through it. Using Eye Gaze technology, we had a quick phone call as I was telling him about this very article and the voice from his computer allowed him to speak back to me. Through Eye Gaze technology, Buck is able to participate in the texting phenomenon which previously was more difficult for him to participate in.

Tobii Technology has been around since 2001 when the company was first founded but the technology has been improving ever since. Tobii first received Venture Capital funding in 2007 ($14 Million) and have received other rounds of funding including Intel in 2012 for $21 Million. Most recently there has been a partnership with Perceptive Devices announced in August of 2017 to bring facial recognition and smile control to Tobii’s devices. Doing so would allow for people like Buck to merely smile when he wanted to click, rather than having to gaze and stare at an icon which can sometimes require more strain than a simple smile.

Unfortunately, these devices are quite expensive and they’ve been difficult to acquire in the past. However, eWeek wrote about Microsoft trying to expand the use of eye tracking technology to a much wider audience. Announced as Beta on August 3rd of 2017, Microsoft Eye Control  might be the key to unlock this treasure trove of eye tracking technology for people with disabilities. Interestingly, this technology actually has to be paired with a Tobii Eye Tracker 4C, a $149 device designed for use on a computer not produced by Tobii. Unlike the eWeek article posted up above, the promotion video from Tobii actually emphasizes the ability for everyday PC users to seamlessly and intelligently navigate windows, dim their screen and access new feature in video games (see below).

Assasin's_Creed_Origins_Thumbnail

Most people have not yet been exposed to eye tracking technology because it hasn’t been high enough quality, it’s too expensive, or there hasn’t been a use case which has been proven to justify the cost of the current technology. However, Tobii has officially done the coolest thing ever and made their product more affordable ($150) and adapted it for video games (thank you). They actually have added some sweet features highlighted here to games like Assassin’s Creed Origins, the newest release in the popular Assassin’s Creed Series. Players can turn the camera angle based upon their gaze, change the brightness of certain areas if a player is staring at the sun for example, or “tag” enemies when a player looks at them. These borderline features actually look quite cool from an immersion standpoint, but with the two analog sticks, the current camera adjustment method from first person shooter games has become second nature for many gamers anyways.

Super Mario 64.png

Super Mario 64, the first Mario Game to feature 3D Sprites and Camera Control

Games have struggled over time with camera adjustment and until the two analog stick controller arose, people constantly were fighting to be able to see where in the game they wanted to. Games on the Nintendo 64, way back in the 1996, had a single analog stick and sometimes utilized the “d-pad” for camera adjustments which when I tried it, was not a great experience (see Mario above). As mentioned above, most of the time this issue is fixed through the use of two analog sticks but “For Honor”, a new medieval sword fighting style game, has opted for a new control scheme which complements Tobii’s Eye Gaze camera control. They actually use the right analog stick for sword movements when locked in combat and as a result, changing the camera angle is impossible. Seeing more creative uses for the second analog stick or giving users a more primary focus on buttons would unlock new ways to interact with video games in the future. I’m thoroughly excited to see if this technology becomes more widely adopted and influences the Video Game Industry.

giphy26

A short clip of For Honor gameplay

7 comments

  1. markdimeglio · ·

    Awesome piece! Its very cool to see how technology has really improved Buck’s quality of life. I bet it can be so frustrating to want to connect with him but not be able to because of his physical limitations. Very glad that you have a solution that helps!

    Tobii’s product reminds me a lot of Eagle Eye which was developed by James Gips, whose a professor in the Information Systems department. I wouldn’t be surprised if you knew of him/have taken his class. The Eagle Eye program is essentially the same technology. Its cool to see how far this has come and how effective it is.

    As far as video games go, it would be cool to have a feature where if you are playing split screen with someone, the Tobii technology can ensure the person you are playing with isn’t screen watching or something like that.

    1. tuckercharette · ·

      Thanks for the reply! It does look very similar, I’m curious what some points of difference are between Eagle Eye and Tobii’s technology. Tobii has really taken off as a market leader and so they might just be more well known. Especially with regards to moving into the consumer space, I think having previous quality assurance would push people towards Tobii rather than a new brand.

  2. kikinitwithraf · ·

    Fantastic! Clearly there are so many positive uses for this technology, especially with what we’ve seen with recent injuries to athletes, and anyone else dealing with certain challenges and complexities. Before, a person’s livelihood was hindered, but with this, there’s a whole new world of possibilities.

    Further, with the explosion is eSports, I wouldn’t be surprised if investors try to add this technology to bring a whole new facet to gaming. Just off the top of my head, you have games like the classic Goldeneye or COD that could definitely add another element to these types of games. The thought of it is exciting!

  3. Jobabes121 · ·

    What a great post. Thank you for sharing Buck’s story as well. This post contained both a deep and interesting story element through Buck and gaming contents, which I found very amusing.

    Camera angle is something that is quite important unless a game enables the player to see the entire picture of the game (or the map) at once. However, as it is not the case, the integration of this eye tracking technology seems to be very creative and fit right in the sweet spot. Oh my goodness, I used to play For Honor for a while as my friend was obsessed with it, and yes the camera in that game is unable to control as soon as you enter a duel and press the control button. I just wonder how much accuracy this technology will get, as I tried staring at a particular section of the screen after reading this blog and I still stare at a significant chunk. Although the market size for video game will be a lot larger than the case for Buck, I hope this technology develops at an unprecedented rate to eventually provide a cheap, effective device for medical purpose and help people like Buck. I mean, games comes wayyy later than providing assistance with those in need.

  4. Glad to here that Tobii’s technology has facilitated the communication between you and Buck. It must mean a lot to both of you. The eye tracking technology can definitely improve game users experiences. With the increasing Virtual Reality type games released, the combination of the two will provide gamers with more lively experiences as if they have more control within the game playing and more realistic feelings.

  5. thebobbystroup · ·

    Awesome personal story in this blog post! I definitely think this Tobii is much cooler than The Office’s Toby. I know one other use of this technology has been market research for website development. As coders build web pages, they can bring in people to test the layout to see if the notifications button in the top right corner is noticed quickly, or the advertisements on the side are watched or ignored. I think the more uses we can find for this tech, the more quickly the tech will progress. I’m looking forward to seeing where it goes.

  6. Great post! (Sorry about the delay). Have you ever talked with Prof Gips in the IS department. He is actually one of the pioneers of eye tracking technology, and has done great work with kids like your cousin!

%d bloggers like this: