A Medical Electronics Manufacturing Fall 1997 Feature

VIRTUAL REALITY

Providing the Power Behind Virtual Reality

Greg Freiherr

New computing platforms, imaging algorithms, and improved tracking are helping to close the gap between virtual and reality.

True virtual reality (VR)­wherein the observer is immersed in a computer-generated world indistinguishable from the world around us­is far beyond the grasp of engineers today. The display and support technologies necessary for such a creation are still too primitive, too incomplete to allow it. But these still-infant technologies are serving as stepping stones to that distant future, providing virtual solutions to real-world medical problems.

Data obtained from an MR scan are reconstructed into a three-dimensional model of the patient, showing critical brain structures and vasculature that must be avoided when using a surgical probe.

A new discipline, image-guided surgery, has formed to encompass these efforts. Some are information-based, seeking to enhance the data flowing past the operator using conventional surgical tools. Others seek to enhance the surgeon's skill, sending commands through mechanisms that transform sweeping motions into minutely precise actions while quieting the almost imperceptible tremor in the human hand as it cuts. But the ultimate expression of virtual reality as it exists today is the melding of technology to provide views of and access to the patient that escape human capability.

At the University of North Carolina (Chapel Hill), engineers, physicians, and computer scientists have crafted an augmented reality system, defined as one that combines computer graphics and virtual reality displays with images of the real world. The VR data are delivered by a diagnostic ultrasound system that generates real-time images reconstructed from sound waves bouncing off tissues inside the patient. The resulting high-resolution slices are then combined with a live video image, delivered by a camera focused on the patient. The combined data set is then presented to the operator in a head-mounted display.

By sweeping the probe over an area of the patient, the operator creates slices in a video context that define a three-dimensional volume. Generating such images requires not only the matching of data but a tracker that follows the head of the user to determine orientation in relation to the patient. The position of the ultrasound probe is tracked using a mechanical arm that provides registration information between individual ultrasound slices.

Of crucial importance is registering the data so as to present the virtual world as a part of the real one. "You have to know where the surface of the body is in order to fool the human visual system into feeling that there is sort of a hole you are looking into," says Stephen M. Pizer, director of the Medical Image Display and Analysis Group at UNC. "You have to sort of fit the pit (filled with ultrasound data) that you are drawing on the body to make it appear to be on the surface."

This data matching and fitting has been made possible by an infrared scanning system that "paints" a grid on the patient. This grid provides the basis for mapping the data sets on each other. The system is enormously computing intensive, which has strained the ability to deliver high-quality, real-time images. "In any virtual or augmented reality environment, the speed of display is one of the major limiting factors," Pizer says. "You have to do things very fast to make complicated images, and that requires a very, very capable graphics engine."

The Power of VR

The power of the VR system could expand markedly in the near future with the commercialization of a new computing platform, unveiled in August by Hewlett-Packard (Palo Alto, CA). The new product, called the HP Visualize PxFl (pixel flow), promises unlimited 3-D graphics performance, allowing visualization of the largest and most complex data sets interactively. This is accomplished by assigning the task of rendering the image to several processing modules, each one computing a full-screen image of a fraction of the image elements. A high-performance image-composition network then assembles these image segments in real time to produce a whole image. "It's a modular machine, so its speed depends on how many modules you buy, but even the basic engine is 20 times faster than the previously fastest graphic engines."

The Boston Dynamics, Inc., surgical simulator, incorporating a Phantom force-feedback interface from SensAble Technologies, Inc., allows a physician to feel as well as see virtual surgery.

This computing platform, which evolved out of a collaboration between HP and UNC, possesses the kind of functionality critical for the continued evolution of the university's medical virtual reality program. Early applications of VR using ultrasound focused on obstetric examinations­creating virtual models of the human fetus in vivo. Most recently the technology has been used to guide invasive procedures, including needle biopsies. The popularity of ultrasound-guided breast biopsy has increased in recent years, but the technique is not easy to learn or to perform. The UNC staff believe computer-augmented vision technology can enhance both the learning and performance of this technique.

But just as there are many reasons for using ultrasound, so too are there reasons for using other imaging modalities in VR. While ultrasound equipment emits no ionizing radiation, is relatively inexpensive, and provides real-time images, ultrasonic waves are blocked by air and hard tissue, which severely limits where they might be applied in the body. Magnetic resonance (MR) imaging and computed tomography (CT) are not so limited.

At the Surgical Planning Laboratory at Brigham and Women's Hospital and Harvard Medical School (Boston), surgeons are augmenting reality with MR and CT images imposed on real-time video of the patient during surgery. The combined image allows surgeons to see a 3-D model of the patient's brain reconstructed in the context of live video depicting the head, the surgeon's hands, and surgical instruments. In this way, the computer-generated model of the tissue serves as a reference for real-world intervention.

The processing is accomplished with imaging algorithms developed by engineers at the GE Research and Development Center (Schenectady, NY). These algorithms permit 3-D models of the skin and internal tissue to be constructed from conventional CT and MR slice data. A laser scanner identifies in three-dimensional space key reference points on the scalp as the patient lies on the surgical table. These references allow the computer to register the video image to the model of internal structures composed of MR or CT data. Registration is then confirmed by fading the skin model in and out. Prior to beginning the procedure, the virtual skin and skull can be "peeled back" to reveal internal structures.

Although the research team would prefer to use a head-mounted display, the images currently are visualized on a computer monitor. "The available quality of head mounts is not quite there yet," says Ron Kikinis, MD, director of the Surgical Planning Laboratory. "More importantly, the currently available technology to track head motion cannot provide enough performance."

While the patient's head is immobilized during neurosurgery, the surgeon's head is not, Kikinis points out. Head-mounted displays present data in the context of the perspective of the operator. During a procedure, the head of the neurosurgeon typically does not move very much, but head movements that do occur often happen very quickly. Commercially available trackers update positions 20 to 50 times per second. "But when someone moves their head very quickly, that is not fast enough," Kikinis says.

Tools of Tracking

The development of an adequate tracking system is more a question of cost than technological reach. "If you give me a million dollars, I could get one built today," Kikinis explains. "But for ten thousand dollars, you cannot get the speed that is needed."

The research team does make use of tracking systems to follow the surgical tool, which is represented in the virtual world as an invasive icon as it penetrates and retreats from the patient's brain. That world is color-coded by "segmentation" software that assigns different colors to different tissues, allowing tumors and critical vascular structures, for example, to stand out. This segmentation capability allows the calculation of tissue volumes in the 3-D model, thereby providing surgeons with the opportunity to quantify the amount of tumor to be removed­a handy feature when trying to estimate whether all or at least a majority of the tumor has been removed.

The 3-D model of the patient can be recomputed to subtract virtual tissues.

Ideally, the computerized model would provide that information with a moment-by-moment account of the surgical progress being made. To achieve that level of sophistication, the model would have to be updated through the acquisition and recomputation of real-time images. And that is exactly the effort in which the Boston researchers are now engaged, using a specially designed MR scanner, called the Signa SP, built by GE Medical Systems (Milwaukee). In the last year, Kikinis, along with Ferenc A. Jolesz, MD, director of the Image Guided Therapy Program, and their colleagues have completed about 200 interventional MR cases using the system.

The real-time aspect greatly complicates the computations. A preoperative model, often composed of multiple data sets from CT, MR, and nuclear medicine, is constructed in much the same way as for use in the nondynamic system. But then this model must be updated in real-time with images obtained with the Signa SP. As a tumor is removed, Kikinis explains, brain tissue typically changes position to fill the void. "The registration technology has to be able to cope with the changing anatomy, and that is an algorithmic challenge we are working on," he says. "We have the necessary environment here to do this, being a high-end research lab. But for this to diffuse to clinical sites will take a few years."

GE has been carefully building a case for the clinical utility of this technology. Even though FDA granted 510(k) clearance for the Signa SP last November, the company wants plenty of evidence of its value before coming to market with a commercial product, if indeed it ever does. The estimated price for the system is $3 million, a hefty bill to pay for a system that would be dedicated to a still uncertain future in intervention.

At least for the time being, minimally invasive surgery will continue to be dominated by the real-time video that guides physicians as they snip, grasp, and suture tissues at the end of a laparoscope or endoscope. The popularity of these surgeries has soared as medical institutions have struggled to restrain the cost of health care. But by their very nature, such minimally invasive procedures test the skills of surgeons. They are akin to operating through a keyhole, where the reduced field of view and access decrease the margin for error. The very limitations of this approach, however, make it fertile ground for the development of new VR technologies.

Improving Surgical Training

Researchers in the Human Interface Technology Laboratory at the University of Washington (Seattle) are developing a simulator that promises to improve the training of surgeons for endoscopic procedures. In collaboration with Lockheed Martin (Akron, OH), the Ohio Supercomputer Center (Columbus, OH), and Immersion Corp. (San Jose), a HITLab research team has developed a simulator for endoscopic sinus surgery. The system uses force feedback to create the kind of resistance experienced when plumbing the sinus cavity of a real patient, while depicting the video images that would accompany those sensations. The difference, of course, is that neither animals nor real patients are put at risk.

"VR simulation has broad appeal for medical training, but its impact may be especially significant for surgeons," says Suzanne Weghorst, HITLab assistant director for research. "Too often in current practice a resident's first hands-on experience is with real patients, and opportunities to practice skills and procedures are rare and usually come at the expense of lab animals."

This feeling of touch, provided by force feedback, is created when the computer commands a device to resist motions made by a person in the same way that a real object would resist. An extension of that concept is the establishment of haptic interfaces that use sensors built into the device to provide this sense of touch at the skin level, as well as force on the operator's muscles and joints.

The VR simulator at the University of Washington provides these tactile and video stimuli. It has the added attraction of scaling the tasks from the "novice" level, which is aimed at training basic endoscope navigation and instrument skill, through advanced levels involving complex anatomical models of the sinus, culminating in the expert or unassisted simulated procedure level.

Along the way, the student encounters training aids embedded in the simulation, such as clearly defined targets and structured clinical protocols to be completed. The simulator also provides the ability to play back the student's performance and even contrast it to the performance of an expert surgeon.

As might be expected, the greatest technological change encountered in developing this simulator was incorporation of what Weghorst terms "interactive realism." The simulator had to accurately reflect the dynamics of the tissues when being probed endoscopically in real time. That meant limiting the scope of the simulator somewhat. "We opted to focus this simulator on tasks that are not so dependent on precise tissue deformation, so as to maintain a frame rate as close to 30 per second as possible," Weghorst says.

Currently the HITLab team is assessing the importance of the force feedback for various aspects of training. The challenge is to determine what is really needed in sinus surgery training and to infuse the simulator with the appropriate mechanisms for that training. To do that, the researchers are hoping to correlate performance on the simulator with performance in the operating room, a goal being addressed in six tests being run with the simulator at the Madigan Army Medical Center (Tacoma, WA). The tests began last spring and will be complete before year's end.

Commercial Horizons

While academicians are on the cutting edge of clinical research, the tools of virtual reality will never be accepted until commercial systems are available. And that is beginning to happen.

SensAble Technologies (Cambridge, MA) is currently marketing a product that generates those feelings. This product, called the PHANToM/GHOST, is a "haptics platform" that allows users to feel the physical properties of virtual objects. "It even goes beyond that," says Thomas H. Massie, who invented the founding technology of SensAble Technologies. "Not only can you feel objects but you can change them­you can change their shape and their form."

Selective subtraction of the image allows surgeons to better plan actual tumor removal.

The device is deceptively simple looking, appearing as a pair of articulated arms attached to swivel bases. It provides three degrees of freedom (x, y, and z planes) for force feedback and optionally three additional degrees of freedom for measurement. These computations can be accomplished with Pentium-driven PCs, although more powerful workstations of the kind populating the Silicon Graphics family are recommended for running more sophisticated applications.

The PHANToM combines robotics and the principles of force feedback to mimic the sensations that may be as simple as a scalpel cutting through tissue or as complex as the varied resistance that accompanies the journey of a biopsy probe through the skull, into the brain, to the periphery of a tumor. The complexity of the simulation is determined by the software that guides the device. Boston Dynamics, Inc. (BDI; Boston), has integrated the PHANToM into a VR simulator that creates "tangible reality"­a three-dimensional image of the surgical field, as well as tactile senses of cutting, clamping, and sewing. The engineers have programmed into their simulator the physics of anastomosis­the surgical task of suturing tube-like tissues, such as blood vessels, ureters, tracheas, colons, and ducts.

"We have simulated vessels that are flexible and compliant," says Robert Playter, technical manager for force feedback training at BDI. "You can touch them; grasp them; deform them."

These virtual objects move and change shape when grasped. The surface flexes the moment before a needle pierces the vessel wall, then pops back as the needle penetrates.

The illusion of three dimensionality is created using glasses composed of rotating shutters timed to coincide with the frame rate of a monitor displaying the images of tubes such as ducts or blood vessels. An optical system projects the image into the field view of the surgeon, who peers downward as though the patient were lying on a table in an operating room. The sense of touch is created using force feedback and haptic software that simulates the feel of surgical tools cutting through tissue. That feeling is transmitted through the actual surgical tools held by the operator. "The force feedback device conveys those contact forces so you can feel them," Playter explains.

All the data­tactile and visual­are routed through the PHANToM. A control computer reads data from the PHANToM, simulates virtual objects, determines contact between the PHANToM and virtual objects, calculates contact forces, and applies those forces to the user through the PHANToM. The control computer also sends real-time data to a graphics computer to update the three-dimensional display. The software that guides the interactions between operator and device simulate the touching and grasping of object in virtual space, as well as contact between objects. Algorithms that define basic physical laws guide the movement of objects during interactions between the virtual environment and operator.

A virtual reality by combining data from the MR scan and live video of the patient.

The potential, says Playter, is to integrate this simulator into surgical curricula, providing not only the means to practice surgical technique but also to evaluate those efforts quantitatively. "We can measure how accurately you can place a needle; when you puncture the tissue with the needle and the kind of forces applied; whether you follow the curve of the needle as you push it through the tissue; how long you take," says Playter.

Currently the surgical skills of students are graded subjectively. BDI's tangible reality device could change that. But before that can happen, the Boston engineers must identify the specific skills that expert surgeons possess, so the simulated evaluation might be focused on just those critical few points. "If you are good at virtual surgery, does that mean you have the skills to do real surgery?" Playter asks. "Answering that question is the direction in which we are headed now."

The next step in VR is to enhance human skills using electromechanical interlopers that refine crude human strokes into microprecise actions. That is the goal of Zeus, a microsurgical aid developed by Computer Motion (Santa Barbara, CA) that allows surgeons to work on smaller and more delicate structures than would otherwise be possible. "We believe Zeus will do for surgeons' hands what the microscope did for their eyes," says Yulan Wang, chief technical officer, executive vice president, and cofounder of Computer Motion.

Zeus resembles a VR simulator, providing a control arm for each hand of the surgeon and using force feedback systems to communicate with the operator. But rather than indicating resistance from the tissue of a computer-generated patient, these mechanical arms relay sensations encountered during the actual procedure and then translate the actions of the operator to fit the microsurgical theater. Sweeping motions become submillimeter actions of the tiny effectors at the end of the control arms. Algorithms running tiny motors damp out the tremors that beset even the steadiest hand. "When working with extremely small blood vessels, every surgeon's hand shakes somewhat," says Michel Gagner, MD, a surgeon in the Minimally Invasive Surgery Center of the Cleveland Clinic, one of three sites in the United States where Zeus is now being clinically tested.

Ultimately, Zeus may improve dexterity so much that cardiac surgeons will be able to complete multiple-vessel coronary artery bypass grafting through an incision less than an inch long, potentially reducing morbidity and decreasing the patient recovery time while cutting the cost of medical care. "As we work with smaller incisions and smaller vessels, we test the limits of human physical capabilities and require the assistance of new technology," says Joseph F. Hahn, MD, chairman of the Cleveland Clinic Division of Surgery.

This symbiotic relationship between the real and virtual world is continuing to evolve as new turf is trod on. Technologies common to both rely on the same kinds of sensors, and the same kinds of software to sense and relay forces and commands. And, in a way, their parallel evolution may be both the greatest challenge and the most significant result of VR in surgery. The development of VR tools promises to help prepare the next generation of practitioners while simultaneously leading to a better understanding of surgical practice itself. In doing so, these tools will both reflect the nature and correct the foibles of their makers.