The Concord Consortium

In his Critique of Pure Reason, the Enlightenment philosopher Immanuel Kant asserted that “conception without perception is empty, perception without conception is blind. The understanding can intuit nothing, the senses can think nothing. Only through their unison can knowledge arise.” More than 200 years later, his wisdom is still enlightening our NSF-funded Mixed-Reality Labs project.

Mixed reality (more commonly known as augmented reality) refers to the blending of real and virtual worlds to create new environments where physical and digital objects co-exist and interact in real time to provide user experiences that are impossible in only real or virtual world. Mixed reality provides a perfect technology to promote the unison of perception and conception. Perception happens in the real world, whereas conception can be enhanced by the virtual world. Knitting the real and virtual worlds together, we can build a pathway that leads perceptual experiences to conceptual development.

We have developed and perfected a prototype of mixed reality for teaching the Kinetic Molecular Theory and the gas laws using our Frame technology. This Gas Frame uses three different types of sensors to translate user inputs into changes of variables in a molecular simulation on the computer: A temperature sensor is used to detect thermal changes in the real world and then change the temperature of the gas molecules in the virtual world; a gas pressure sensor is used to detect gas compression or decompression in the real world and then change the density of the gas molecules in the virtual world; a force sensor is used to detect force changes in the real world and then change the force on a piston in the virtual world. Because of this underlying linkage with the real world through the sensors, the simulation appears to be "smart" enough to detect user actions and react in meaningful ways accordingly.

Each sensor is attached to a physical object installed along the edge of the computer screen (see the illustration above). The temperature sensor is attached to a thermal contact area made of highly conductive material, the gas pressure sensor is attached to a syringe, and the force sensor is attached to a spring that provides some kind of force feedback. These three physical objects provide the real-world contextualization of the interactions. In this way, the Gas Frame not only produces an illusion as if students could directly manipulate tiny gas molecules, but also creates a natural association between microscopic concepts and macroscopic perception. Uniting the actions of students in the real world and the reactions of the molecules in the virtual world, the Gas Frame provides an unprecedented way of learning a set of important concepts in physical science.

Pilot tests of the Gas Frame will begin at Concord-Carlisle High School this week and, collaborating with our project partners Drs. Jennie Chiu and Jie Chao at the University of Virginia, unfold at several middle schools in Virginia shortly. Through the planned sequence of studies, we hope to understand the cognitive aspects of mixed reality, especially on whether perceptual changes can lead to conceptual changes in this particular kind of setup.

Acknowledgements: My colleague Ed Hazzard made a beautiful wood prototype of the Frame (in which we can hide the messy wires and sensor parts). The current version of the Gas Frame uses Vernier's sensors and a Java API to their sensors developed primarily by Scott Cytacki. This work is made possible by the National Science Foundation.

Natural user interfaces (NUIs) are the third generation of user interface for computers, after command line interfaces and graphical user interfaces. A NUI uses natural elements or natural interactions (such as voice or gestures) to control a computer program. Being natural means that the user interface is built upon something that most people are already familiar with. Thus, the learning curve can be significantly shortened. This ease of use allows computer scientists to build more complicated but richer user interfaces that simulate the existing ways people interact with the real world.

Research on NUIs is currently one of the most active areas in computer science and engineering. It is one of the most important directions of Microsoft Research. In line with this future, our NSF-funded Mixed-Reality Labs (MRL) project has proposed a novel concept called the Natural Learning Interfaces (NLIs), which represents our latest ambition to realize the educational promise of cutting-edge technology. In the context of science education, an NLI provides a natural user interface to interact with a scientific simulation on the computer. It maps a natural user action to the change of a variable in the simulation. For example, the user uses a hot or cold source to control a temperature variable in a thermal simulation. The user exerts a force to control the pressure of a gas simulation. NLIs use sensors to acquire real-time data that are then used to drive the simulation in real time. In most cases, it involves a combination of multiple sensors (or multiple types of sensors) to feed more comprehensive data to a simulation and to enrich the user interface.

I have recently invented a technology called the Frame, which may provide a rough idea of what NLIs may look like as an emerging learning technology for science education. The Frame technology is based on the fact that the frame of a computer screen is the natural boundary between the virtual world and the physical world and is, therefore, an intuitive user interface for certain human-computer interactions. Compared with other interfaces such as touch screens or motion trackers, the Frame allows users to interact with the computer from the edges of the screen.

Collaborating with Jennie Chiu's group at the University of Virginia (UVA), we have been working on a few Frame prototypes that will be field tested with several hundred Virginia students in the fall of 2012. These Frame prototypes will be manufactured using UVA's 3D printers. One of the prototypes shown in this blog post is a mixed-reality gas lab, which was designed for eighth graders to learn the particulate nature of temperature and pressure of a gas. With this prototype, students can push or pull a spring to exert a force on a virtual piston, or use a cup of hot water or ice water to adjust the temperature of the virtual molecules. The responsive simulation will immediately show the effect of those natural actions on the state of the virtual system. Besides the conventional gas law behavior, students may discover something interesting. For example, when they exert a large force, the gas molecules can be liquified, simulating gas liquifying under high pressure. When they apply a force rapidly, a high-density layer will be created, simulating the initiation of a sound wave. I can imagine that science centers and museums may be very interested in using this Frame lab as a kiosk for visitors to explore gas molecules in a quick and fun way.

A mixed-reality gas lab (a Frame prototype)

As these actions can happen concurrently, two students can control the simulation using two different mechanisms: changing temperature or changing pressure. This makes it possible for us to design a student competition in which two students use these two different mechanisms to push the piston into each other's side as far as possible. To the best of our knowledge, this is the first collaborative learning of this kind mediated by a scientific simulation.

NLIs are not just the results of some programming fun. NLIs are deeply rooted in cognitive science. Constructivism views learning as a process in which the learner actively constructs or builds new ideas or concepts based upon current and past knowledge or experience. In other words, learning involves constructing one's own knowledge from one's own experiences. NLIs are learning systems built on what learners already know or what they feel natural. The key of a NLI is that it engineers natural interactions that connect prior experiences to what students are supposed to learn, thus building a bridge for stronger mental association and deeper conceptual understanding.

Watch a demo video

We have started to develop a high quality three-dimensional motion tracking system for science education based on the Microsoft Kinect controller, which was released about 18 months ago. This development is part of the Mixed-Reality Labs project funded by the National Science Foundation.

KTracker will provide a versatile interface between the Kinect and many physics experiments commonly conducted in the classroom. It will also provide natural user interfaces for students to control the software for data collection, analysis, and task management. For example, the data collector will automatically pause while the Kinect detects that the experimenter is adjusting the apparatus to create a new experimental condition (during which the data collection should be suspected). Or the user can "wave" to the Kinect to instruct the software to invoke a procedure. In this way, the user will not need to switch hands between the apparatus and the keyboard or mouse of the computer (this "hand-switching" scene seems familiar to the experimentalists reading this post, huh?). The Kinect sensor has the capacity to recognize both gestures of the experimenter and motions of the subject, making it an ideal device for carrying out performance assessment based on motor skill analysis.

KTracker is not a post-processing tool. It is not based on video analysis. Thanks to the high performance infrared-based depth camera built in the Kinect, KTracker is capable of doing motion tracking and kinematic analysis in real time. This is very important as it helps to accelerate the data analysis process and contributes to enhancing the interactivity of laboratory experiments.

KTracker will also integrate a popular physics engine, Box2D, to support simulation fitting. For example, the user can design a computer model of the pendulum shown in the above video and adjust the parameters so that its motion will fit what the camera is showing--all in real time. Like the graph demonstrated in the above video, the entire Box2D will be placed in a translucent pane on top of the camera view, making it easy for the user to align the simulation view and the experiment view.

KTracker will soon be available for download on our websites. We will keep you posted.

Click here to watch a video.

Microsoft's Kinect controller offers the first affordable 3D camera that can be used to detect complex three-dimensional motions such as body language, gestures, and so on. It provides a compelling solution to motion tracking, which--up to this point--is often based on analyzing the conventional RGB data from one or more video cameras.

The conventional wisdom of motion tracking based on RGB data requires complicated algorithms to process a large amount of video data, making it harder to implement a real-time application. The Kinect adds a depth camera that detects the distances between the subjects and the sensor based on the difference of the infrared beams it emits and the reflection it receives. This gives us a way to dynamically construct a 3D model of what is in front of the Kinect with a rate of about 10-30 frames per second, fast enough to build interactive applications (see the video linked under the above image). For as low as $100, we now have a revolutionary tool for tracking 3D motions of almost anything.

The demo video in this post shows an example of using the Kinect sensor to track and analyze the motion of a pendulum. The left part of the above image shows the overlay of trajectory and velocity vector to the RGB image of the pendulum, whereas the right part shows the slice of the depth data that is relevant to analyzing the pendulum.

The National Science Foundation provides funding to this work.

IR: Watch the YouTube video

Augmented reality (AR) presents a live view of the real world whose elements are augmented by computer-generated data such as sound or graphics. The technology promises to enhance the user's current perception of reality. AR is considered as an extension of virtual reality (VR). But unlike VR that replaces the real world with a simulated one, AR bridges and takes advantage of the real world and the simulated world.

Augmentation is conventionally in real-time and in semantic context with environmental elements. With the help of AR technology, the information about the surrounding real world of the user becomes digitally manipulable. Artificial information about the environment and its objects can be overlaid on the real world to achieve seamless effects and user experiences.

Our NSF-funded Mixed-Reality (MR) Labs Project has set out to explore how AR/MR technologies can support "augmented inquiry" to help students learn abstract concepts that cannot be directly seen or felt in purely hands-on lab activities.

AR: Watch the YouTube video

One of the first classes of prototype we have built is what we call "Augmented Reality Thermal Imaging." The concepts related to heat and temperature are somehow difficult to some students because thermal energy is invisible to the naked eye. Thermal energy can now be visualized using infrared (IR) imaging. But we have developed AR technology that provides another means of "seeing" thermal energy and its flow.

The first image in this post shows an IR image of a poster board heated by a hair dryer. The second image shows a demo of AR thermal imaging: When a hair dryer blows hot air to a liquid crystal display (LCD), the AR system reacts as if hot air could flow into the screen and leave a trace of heat on the screen, just like what we see in the IR image above. You may click the links below the images to watch the recorded videos.

The tricky part of MR Labs is that, in order to justify the augmentation of a computer simulation to a physical activity, the simulation should be a good approximation of what happens in the real world. We used our computational fluid dynamics (CFD) program, Energy2D, to accomplish this. There are many more demos of MR Labs using Energy2D, which can be viewed at this website.