Monthly Archives: March 2012

Augmented reality thermal imaging

IR: Watch the YouTube video
Augmented reality (AR) presents a live view of the real world whose elements are augmented by computer-generated data such as sound or graphics. The technology promises to enhance the user's current perception of reality. AR is considered as an extension of virtual reality (VR). But unlike VR that replaces the real world with a simulated one, AR bridges and takes advantage of the real world and the simulated world.

Augmentation is conventionally in real-time and in semantic context with environmental elements. With the help of AR technology, the information about the surrounding real world of the user becomes digitally manipulable. Artificial information about the environment and its objects can be overlaid on the real world to achieve seamless effects and user experiences.

Our NSF-funded Mixed-Reality (MR) Labs Project has set out to explore how AR/MR technologies can support "augmented inquiry" to help students learn abstract concepts that cannot be directly seen or felt in purely hands-on lab activities.

AR: Watch the YouTube video
One of the first classes of prototype we have built is what we call "Augmented Reality Thermal Imaging." The concepts related to heat and temperature are somehow difficult to some students because thermal energy is invisible to the naked eye. Thermal energy can now be visualized using infrared (IR) imaging. But we have developed AR technology that provides another means of "seeing" thermal energy and its flow.

The first image in this post shows an IR image of a poster board heated by a hair dryer. The second image shows a demo of AR thermal imaging: When a hair dryer blows hot air to a liquid crystal display (LCD), the AR system reacts as if hot air could flow into the screen and leave a trace of heat on the screen, just like what we see in the IR image above. You may click the links below the images to watch the recorded videos.

The tricky part of MR Labs is that, in order to justify the augmentation of a computer simulation to a physical activity, the simulation should be a good approximation of what happens in the real world. We used our computational fluid dynamics (CFD) program, Energy2D, to accomplish this. There are many more demos of MR Labs using Energy2D, which can be viewed at this website.

Streaming Arduino Data to a Browser without Flash or Java

This blog post describes an older method for connecting sensors to a web browser. You can learn about a newer, more robust method using Web Bluetooth in this Under the Hood article from our fall 2017 @Concord newsletter.

What if you were reading a blog or working through an online lesson and you could just plug in your Arduino and start taking data or interacting with models right in your browser?

Here at the Concord Consortium we are very interested in making sensors that are easy to use in the classroom or embedded directly into rich online curriculum. We’ve done some work in the past using applets as an intermediary to read data from commercial sensors and displaying them in lightweight graphs in the browser. When we think of fun, hackable, multi-probe sensors, though, we naturally think of Arduinos — we are open-source geeks after all.

In thinking of ways to display Arduino data in a browser with the minimum amount of fuss, we considered both our existing applet technique and using the new HID capabilities of the Arduino Unos. But while we will probably still find uses for both strategies, it occurred to Scott Cytacki, our Senior Developer, that we could simply use the common Ethernet Shields (or the new Arduino Ethernets) to send the data directly to the browser.

With this idea, it was quick work to hack the Arduino Server example to send JSON values of the analog pins and create a webpage that would rapidly poll the Arduino for data. So here is the first example that I wrote in about 70 lines of code (including the Arduino sketch) usable on any Ethernet-capable Arduino on any browser:

  1. Upload the tiny server sketch to your Arduino
  2. Plug in your ethernet shield, connect the Arduino to your computer with an ethernet cable and wait about 30 seconds for the Arduino server to boot up
  3. Optionally connect a sensor to pin A0. (The demo below is scaled for a L35 temperature sensor, but you don’t need it — you might need to rescale the graph by dragging on the axis to see the plot, though)
  4. Click the “Start Reading” button below

You should see your Arduino data filling up the graph. If not, wait another 20 seconds to ensure the server is fully booted and click the “play” button at the top right to start it again.

Wow, that was actually pretty easy!

I created the slightly more complicated example below reads data from all six analog pins, applies an optional conversion, and graphs any one of the data streams. If you were already reading data above, you don’t need to do anything new, just hit the button:

Direct link to stand-alone version

We think this is really cool, and we can’t wait to come up with new ways to integrate this into online content. Why not feed the temperature data into the HTML5 version of Molecular Workbench we’re developing under our new grant from, for instance, and see the atoms speed up as the temperature increases? Or set up an online classroom where students across the globe can take environmental readings and easily upload and pool their data?

Even by itself, the example above (perhaps expanded further by an interested reader) makes a great workbench for developing on an Arduino — much better than watching the raw Serial Out panel. And of course all the programming can happen in your friendly JavaScript environment, instead of needing to keep recompiling code and uploading it to your Arduino as you iterate.

Technical details:

  • This Arduino Sketch creates a server running on, which is a private local IP that will automatically try to not conflict with other servers, allowing for easier connection without a DHCP server. The sketch then returns JSON data using the JSON-P technique of calling back a function, which allows us to make cross-domain requests.
  • Click on the tabs at the tops of the embedded jsFiddle examples to see the source code for streaming data to the webpage, or fork and edit any of the examples yourself.

  • The graphs are creating using D3.js, and make use of the simple-graph library created by Stephen Bannasch.

Energy2D: Computational fluid dynamics at your fingertips

Online Energy2D simulations
Energy2D is our signature software for simulating heat transfer and fluid dynamics. Fifty online simulations are now available to the world through Energy2D's website. These simulations run speedily on most computers, bringing a vivid, colorful world of science to your computer screen and allowing you to experiment with them.

All these simulations can be downloaded for editing, provided that you have also installed the standalone Energy2D software on your computer (you don't need it to run the online simulations--only when you need to edit or create a simulation will you need to install it). The editing interface still has limited functionalities, but we are hoping to make it ten times better in the future.

One of our next steps is to make a version that runs on Android. This will allow the simulations you have created to run on tablets and smartphones as well. Work is also underway to include other energy flows and transformations to enrich the natural phenomena it can simulate, and to integrate data from sensors to enable richer user interfaces.

The National Science Foundation provides the funding to make this possible.