I use a tablet application to generate an augmented reality “microscope view” of the hemoglobin over the tangible cells. This tangible representation is reinforced by the augmented reality view that visually shows both types of hemoglobin in its respective structures. On top of each cell is a tag for the augmented reality software to recognize the three dimensional orientation for image projection.
The microscope mode uses the Vuforia augmented reality software development kit (SDK) and its extension to the Unity game engine, allowing easy porting to both the Android and iOS operating systems. The Vuforia extensions are used to track the physical red blood cell model in 3D space once it is in view of the tablet’s camera. This information is then used to render a semi-transparent virtual model of the cell (see Figures \ref{fig:xray} & \ref{fig:xray-usage}). The virtual model shows the interior composition on top of the camera’s image, making the physical cell appear transparent when viewed on the tablet. This is akin to a microscope effect: when a user looks at the physical cell he only sees it’s exterior, but when viewed through the tablet, additional details of the interior of the cell are now visible. This application runs in real-time – as the user pans and tilts the camera, or moves it closer or farther, the virtual model follows and allows the user to explore the cell from multiple angles and zoom levels.
To assist in tracking, Vuforia requires a unique tag to be placed on each physical object. In the case of our cells, we created labels for “Normal” and “Sickle” cells, placed in the center of the object for both tracking purposes as well as cell identification for the user. This provides the app with an estimate of the space in the image that the physical object occupies. Next, a model of the cell was created to render a transparent version of the cell. The 3D representation for these cells are modeled using Blender, an open source 3D animation software package. Within Blender I defined models for both a normal red blood cell and a sickle cell. This included specifying other properties such as the textures, transparencies, colors, and interior contents. These models were then imported into the Unity engine for rendering. In reality, the hemoglobins in normal red blood cells have some freedom of movement. To create a more realistic image, I added C# code to define the rules for movement (rotation, velocity, and containment) of the hemoglobin in the blood cells.