By: Cynthia Adams | Photos by: Nancy Evelyn

I stopped a human heart from beating with a flick of my wrist. Then, effortlessly, I restarted it again. Yes, I did. Then, I poked around inside the heart’s cavity for a look-see.

It was a David Copperfield kind of moment.

Holding a small cereal-sized box of “Lion Flakes” in my right hand, I aimed a smart phone at the box and scanned it. Instead of the box, a throbbing human heart appeared on  the screen.

With a twist of the box, the heart rotated. It pumped. It was fully dimensional, anatomically correct and reactive to each movement of my hand. It felt as if I could reach into the screen and massage that heart if it suddenly arrested.

The technology used to create this spectacular illusion— partially computer-generated, partially technology driven, partially painstaking human illustration—is called augmented reality, or simply, AR. AR removes the dividing membrane of perceived reality and can potentially render any anatomical subject transparent, interactive and multidimensional.

“The most common reactions are amazement and wondering what’s magical about the box!” says Jim Moore, who is possibly the only full-time working retiree on campus. “Augmented reality has the potential for engaging students,” adds his colleague, Scott Brown. “It wows people.”

But Moore stresses that augmented reality is only one amazing endeavor at the Educational Resources Center, which he co-directs with Brown. There is more inside, they say.

Above left to right: Co-directors of the Educational Resource Center, or ERC, Jim Moore and Scott Brown. Bottom left, ERC medical illustrator Joe Samson and right, UGA doctoral student Rafael Silva. An interdisciplinary project team includes graduate students and staff from engineering, virtual reality, medical illustration, scientific illustration, 3-D animation, photography, videography, graphic design and dramatic media production. photography/videographer Chris Herron.

Left to right: Row 1: Assistant professor Kyle Johnsen; graduate student Aryabrata Basu; medical illustrator Neil McMillan. Row 2: Medical illustrator Kip Carter; graduate student Will McAbee; graduate student Dominique Edwards. Edwards received a graduate assistantship for the past two years, funded by the Graduate School. Row 3: Scientific illustrator Stephanie Pfeiffer; medical illustrator Brad Gilleland; graphics artist Harsh Jain. Row 4: 3-D graphics artist Thel Melton; graduate student Tasha Obrin; photographer/videographer Chris Herron.

Creative Building Blocks in an Unassuming Place: Imagining a Different Teaching Dynamic

The obscure Educational Resources Center, or ERC, building sits within the campus of the College of Veterinary Medicine. A smallish, squat white building is tucked near a paddock and a small hospital. It teems with an ensemble of technical, medical, scientific and creative thinkers, who are also tinkers, and illusionists.

“They—the team involved—are the ones who do the work,”

Brown insists as Moore listens. “If you gave Jim and me a year, and all the computing equipment you want, and a cereal box—you would still just have a cereal box.” The two men howl with laughter.

“The people from ERC are incredibly talented artists,” Moore agrees. Developing projects like AR, interactive text books and dynamic experiences requires art and finesse.

To create future-forward tools like AR and virtual reality, it really takes a cast of developers, including medical illustrators, scientists, researchers, 3-D graphic artists, engineering faculty, and engineering/computer science students.

Using NIH grant proceeds, Moore and Brown hired Joe Samson, a medical illustrator from Johns Hopkins. Sampson  created the 3-D heart and ERC collaborated with engineering assistant professor Kyle Johnsen to render it interactive.

“Then Rafael Silva came into the picture with an interest in augmented reality,” says Moore.

Augmented Reality Lies at the Vector of Science and Art

It has significant differences from virtual reality. The experience is directly viewed—while virtual reality requires headgear or head-mounted displays. Augmented reality requires nothing more than a tablet or smart phone camera to read code embedded upon an object. One source calls this method “rotation representation with exponential map.”

The code that looks like a graphic on the small box contains an algorithm written by Silva, and employs imagery from a library created by Vuforia.

The notion behind AR is another seminal change—one radically augmenting the way we educate, says Moore. No more dusty chalk lectures. Thus the beating, interactive heart, which requires neither headset nor helmet, is a product of augmented reality.

“Brown believes that AR is most useful for teaching complex concepts that involve imagery,” says Moore.

“Like body functions, or molecular interactions, seen in real—almost real—three-dimension interactions. It’s very powerful where anatomy is important in studying a body part,” Brown agrees. “The heart is a complex organ to see come to life.”

Indeed.

Rafael Silva -close-up of augmented reality heart on tablet and phone

A Different Set of Tools in the Box

The unassuming Center has quietly gone about its business for four decades—Moore and Brown say few such entities even exist on other campuses. The ERC used to solely produce videos, still images, print projects and posters—all manner of technical support to faculty and graduate students. They still do, of course, and some of the staffers who do this work have logged as many as three decades there. But today, they do work that is attention-getting and elicits the “Wow!” reaction.

But the directors emphasize this group was doing amazing work then, given the technology that was available. The ERC, Moore says, “has always tried to be the sticky side of the envelope.” He is emphatic that the difference is technology has given them greater tools to expand their creative skills.

“Our push now is to make the best use of technological advances coming out of computer engineering,” says Moore. “That translates into working with Johnsen and two of his graduate students (Silva and Aryabrata Basu) and pursuing extramural funding for graduate assistantships.”

But think of this modest-looking compound as an innovations incubator. Soon, the ERC will be ready for a public roll-out of another brain child.

Playing with a New/Old Idea

This is not their first rodeo when it comes to key players developing interactive educational projects on campus. Moore, Brown and an extended group within and beyond the ERC have long been interested in interactive teaching tools. Over a decade earlier, Steve Oliver, a professor in the College of Education, was principal investigator on an NIH project which resulted in dynamic teaching models.

In 2001, a group animation project called the Glass Horse came to fruition. This was followed by the Glass Dog. The interdisciplinary project included two from ERC: Thel Melton and Brad Gilleland. Melton worked on both the animations, and Gilleland played a key role on the Glass Dog.

Oliver worked with faculty (including, among others, Brown, Moore, Tom Robertson and Mike Hussey) and students on an educational project producing animated 3-D games. Those games, intended to engage and delight students on difficult to learn physiologic processes such as osmosis, were intended for high schools as well as college classrooms. Their idea, then as now, was to articulate scientific processes in a more realistic way. (See our Winter, 2011 GSM feature.)

UGA researchers have been working with 3-D models and animations. A current project for the National Institute of Diabetes and Digestive and Kidney Diseases is headed by Brown. It is creating materials for use in undergraduate physiology, in an effort to interest students in research careers related to diabetes and kidney disease. According to Moore, the project’s development is heavily dependent upon staff “which includes Joe Samson and Neil McMillan. Neil is handling the pancreas/diabetes side of that project.”

Faculty member Johnsen teaches virtual reality courses and works with cross-disciplinary research. He is also working on the NIDDKD project as a co-investigator. Silva is the graduate assistant.

Virtual reality and multimodal 3-D user interfaces—a hybrid of computer science, computer engineering, and research—are Johnsen’s specialties.

Theater and engineering have long been team players in the progression of virtual reality technologies. As futuristic as it may be, virtual reality has inched forward for more than a century. In the 1950s, it became the focus of cinematographer Morton Heilig’s “Experience Theatre,” but was moved forward in the next decade with the development of Air Force flight simulators. By the late 1970s, MIT researchers created a 3-D model and program called the Aspen Movie Map.

VPL Research (the company that developed virtual reality) created a system known as ‘goggles and gloves’ in the 1980s. By the 1990s, a NASA scientist used virtual reality to create the means to control a Mars rover from Earth—but also in real time. This year, the company Oculus VR, who makes virtual reality headsets, was acquired by Facebook. (It may take more than virtual reality to unfriend a nosy Grandma, unfortunately.)

Augmented reality, however, is an innovation that seeks to augment reality with computer-generated images, sounds, or data. AR is attributed to Harvard professor Ivan Sutherland. But the earlier history of AR was one drawn from fiction and cinematography, segueing ultimately with technologies used in the gaming industries and for the military.

As was the case with virtual reality, augmented technology found application and advancements by the U.S. Air Force. It has since leaped into the public sector. In 2013, Google rolled out the Google Glass, which employs augmented reality. BMW announced in April, 2014 it had incorporated AR into its windshield display on its luxury sedans, and project speed limits and road signs onto the windshield.

“The American Veterinary Medical Association accreditation team came in last year, and we had 15 minutes to show what we were doing here in the ERC,” says Moore. It was a very full 15 minutes. The ERC group walked through the various things happening in the unit, ending with an electronic book under development. They talked about other possibilities that could involve additional interactivity for the student.

Some of the AVMA accreditation team asked to return for a chance to have additional time, wishing to see the Glass Horse and Glass Dog products in action.

“It’s important for people to see what’s happening here that isn’t happening anywhere else,” says Moore with visible pride. But there was more—the cherry on the top of the cake. What the AVMA team did not yet see was the work done in developing the augmented reality model of the human heart.

“The Education Resource Center existed 40 years ago,” says Moore, as students and faculty busily move through the workplace. Graphics artists Kip Carter and Harsh Jain have been there much of that time. Carter, reportedly a gifted medical illustrator, has served as president of the Association of Medical Illustrators.

Today, the ERC is an of-the-moment place where a diversity of talent brings vastly different tool boxes to work.

In addition to the list of projects, the ERC group is working on seven interactive books, on topics such as neurology. These interactive books also include self assessment tools, and UGA students are offered the electronic books free of charge for downloading. “Kyle Johnsen wants to know how AR is going to improve students learning,” says Brown. “Part of our experience as veterinary faculty is realizing that learning anatomy is tedious. It’s normal anatomy they are learning, and they want to learn to treat animals that are sick. We can also use this approach to show them anatomical anomalies, which are sometimes the best way for them to learn.”

“Can We Make it Beat?” Medical Illustration and High Tech Advancements

Scientific illustration is taught at the University of Georgia. However, only four universities in North America offer postgraduate programs in medical illustration. On June 1, graduate students Tasha Obrin and Will McAbee entered a new one- year certificate program in Comparative Medical Illustration.

Remember that the medical illustrations were required to build 3-D models; both of those steps preceded the development of an AR model and many of the interactive projects underway at ERC. “It speaks to the transformative way this is helping teaching. This has always been a point of pride that we support medical illustration at the Center,” says Moore.

It is an integral piece of the interactive, animated, and now augmented realities that will keep a student on the edge of their seat. These tools will grab them, just as the creative principals involved imagineered.

Silva, an engineering graduate student working with Johnsen, requested the heart model, which required 167 pages of medical illustration to create. Silva then used Sampson’s model to create an application that would allow the heart to jump off the page, by devising the algorithm and code that is embedded on the small paper box.

He insists his role was minor, and points out the preceding work that made it possible.

“We have weekly meetings at the ERC,” explains Moore, who remembers the meeting when it coalesced very clearly.

“Rafael was attending those meetings, and without any directions from us, he saw the heart model. Having an interest in augmented reality he conceived of the idea,” Moore adds. “Rafael asked Joe for the heart model. At the end of a March meeting, Rafael said quietly, ‘There’s just something I wanted to show you.’”

Silva pulled out a tiny box. The team members scanned it. “Initially it was just the heart that you rotated around,” says Moore. “All of us were momentarily silenced. Wowed. We said, ‘Could you make it beat?’ Then, we asked Rafael, ‘Could we look on the inside?’” Moore laughs. “We kept asking him to make the heart do more things, and he did!”

Silva had been working with an Atlanta tech firm before leaving industry to return for a graduate degree. (“Very boring,” he says, pulling a face, as he describes life before UGA) “Dr. Johnsen is a specialist in virtual reality. Everything  I know I learned from him,” he stresses.

He first showed Moore the application and the box, which featured subtle geometric patterns the computer could read. His next step “is to learn how to replicate the experience of extracting the spinal fluid from a horse,” using a joy stick to exert the same virtual pressure required in the intricate procedure.

“I’m now making less than half what I made at a corporate job,” Silva says. “But it’s worth it. It’s a wonderful quality of life. I want to go to work with a smile on my face.” Silva’s wife is a UGA graduate, and the reason he chose UGA. He plans to earn a doctorate in engineering.

“It’s unfair that I take credit for it,” says Silva, who finished his master’s in May and returned to begin his doctoral program this fall. As he notes, each of the players in the project stand upon the shoulders of the last—as seamlessly as a Cirque de Soleil ensemble performance.

The Glass Horse and Glass Dog Project

In 1996, an interactive teaching project began in the University of Georgia’s College of Veterinary Medicine. Their idea was to create teaching tools which could help veterinary students visualize the complex anatomy of the horse’s gastrointestinal tract. With guidance from the University of Georgia Research Foundation, an interdisciplinary team of inventors developed an interactive interface in which their 3-D models and animations could be viewed. This interactive animation was named The Glass Horse CD and The Glass Horse Project was off and running.

In 2010, The Glass Horse Project grew to become a new company known as Science In 3D, Inc. The Glass Dog was 3D’s initial product, extending beyond equine medicine into canine anatomy. The new program included interactive models and animated movies, with more than 250 detailed and narrated images. Illustrations from The Glass Dog went on to win a 2011 Award of Excellence from the Association of Medical Illustrators.