Video gaming system being used to see patient imaging during surgery

A team of engineers, University of Toronto residents and Sunnybrook cancer surgeons are doing much more than playing games with their ™, they are using the technology during to view critical patient imaging.

One of the challenges during cancer and other surgeries, is having to leave the sterile field around the patient to view MRI and CT scans. To overcome this, the team hooked up the popular gaming system to a computer in the operating room, which allows surgical staff to view scans of the patient by making gestures in the air, without ever having to leave the sterile field around the patient.

“We have created a software and hardware combination that works with the Microsoft Kinect™ –a hands-free control device for use with Xbox™ gaming system. Our system uses three-dimensional data from the Kinect™ to spot the surgeon and follow his or her gestures and poses. This allows the surgeon to control programs on the computer without ever touching it or breaking sterility,” says Matt Strickland, a general surgery resident at University of Toronto and engineer who co-developed this innovation with engineering colleagues Jamie Tremaine, a mechatronics engineer and Greg Brigley, a computer engineer.

In the Operating Room, clinicians work either inside a very controlled sterile field or outside of it. This environment creates challenges for interaction with technology especially when a surgeon wants to review imaging of a patient during the operation. The viewing computer is in the non-sterile area which prevents the surgeon from using the keyboard and mouse directly. Rather than choosing to break sterility the surgeon most often directs an assistant in the non-sterile field to adjust the views.

“Image-guidance is especially important in cancer surgery. We want to do our best to take out all of the tumour, but at the same time, save as much of the patient’s healthy tissue as possible for better quality of life. The imaging acts like the surgeon’s GPS, telling us exactly where everything is inside the patient. And now the surgeon can directly control all of this with a simple wave of the hand — to manipulate the view more specifically to his or her own thinking –and from within the sterile field. To me, this is pure magic for the operating room,” says Dr. Calvin Law, surgical oncologist Gastrointestinal Cancer Care Team, Sunnybrook’s Odette Cancer Centre, who specializes in complex liver, pancreatic and gastrointestinal surgeries and who helped the team enhance the technology.

Developers Strickland, Tremaine and Brigley are working with their technology across all types of surgery and plan to continue developing the system and studying its benefits in the operating room in close partnership with Sunnybrook.

“We see this as a huge opportunity to help propel and support a leading innovation with our surgical expertise at Sunnybrook,” says Dr. Andy Smith, chief, General Surgery, Sunnybrook, and Interim Chief, Sunnybrook’s Odette Cancer Centre. “This is a wonderful example of the innovations happening at Sunnybrook everyday – making technology work for our patients, especially when it matters most.”