Seeing student learning with visual analytics

Technology allows us to record almost everything happening in the classroom. The fact that students' interactions with learning environments can be logged in every detail raises the interesting question about whether or not there is any significant meaning and value in those data and how we can make use of them to help students and teachers, as pointed out in a report sponsored by the U.S. Department of Education:
New technologies thus bring the potential of transforming education from a data-poor to a data-rich enterprise. Yet while an abundance of data is an advantage, it is not a solution. Data do not interpret themselves and are often confusing — but data can provide evidence for making sound decisions when thoughtfully analyzed.” — Expanding Evidence Approaches for Learning in a Digital World, Office of Educational Technology, U.S. Department of Education, 2013
A digraph of action transition
A histogram of action intensity
Here we are not talking about just analyzing students' answers to some multiple-choice questions, or their scores in quizzes and tests, or their frequencies of logging into a learning management system. We are talking about something much more fundamental, something that runs deep in cognition and learning, such as how students conduct a scientific experiment, solve a problem, or design a product. As learning goes deeper in those directions, data produced by students becomes bigger. It is by no means an easy task to analyze large volumes of learner data, which contain a lot of noisy elements that cast uncertainty to assessment. The validity of an assessment inference rests on  the strength of evidence. Evidence construction often relies on the search for relations, patterns, and trends in student data.With a lot of data, this mandates some sophisticated computation similar to cognitive computing.

Data gathered from highly open-ended inquiry and design activities, key to authentic science and engineering practices that we want students to learn, are often intensive and “messy.” Without analytic tools that can discern systematic learning from random walk, what is provided to researchers and teachers is nothing but a DRIP (“data rich, information poor”) problem.

A polar chart of problem space exploration
Recognizing the difficulty in analyzing the sheer volume of messy student data, we turned to visual analytics, a whole category of techniques extensively used in cutting-edge business intelligence systems such as software developed by SAS and others. We see interactive, visual process analytics key to accelerating the analysis procedures so that researchers can adjust mining rules easily, view results rapidly, and identify patterns clearly. This kind of visual analytics optimally combines the computational power of the computer, the graphical user interface of the software, and the pattern recognition power of the brain to support complex data analyses in data-intensive educational research.

A scatter plot of action timeline
Within a week, I was able to write four interactive graphs and charts that can be used to study four different aspects of the design action data that we collected from our Energy3D CAD software. Recording several weeks of student work on complex engineering design challenges, these datasets are high-dimensional, meaning that it is improper to treat them from a single point of view. For each question we are interested in getting answers from student data, we usually need a different representation to capture the outstanding features specific to the question.

In the long run, our overall objective is to add as many graphic representations as possible as we move along in answering more and more research questions based on our datasets. Given time, this growing library of visual analytics would develop sufficient power to the point that it may also become useful for teachers to monitor their students' work and thereby conduct formative assessment. To guarantee that our visual analytics runs on all devices, this library is written in JavaScript/HTML/CSS. A number of touch gestures are also supported for users to use the library on a multi-touch screen. A neat feature of this library is that multiple graphs and charts can be grouped together so that when you are interacting with one of them, the linked ones also change at the same time. As the datasets are temporal in nature, you can also animate these graphs to reconstruct and track exactly what students do throughout.

The National Science Foundation funds SmartCAD—an intelligent learning system for engineering design

We are pleased to announce that the National Science Foundation has awarded the Concord Consortium, Purdue University, and the University of Virginia a $3 million, four-year collaborative project to conduct research and development on SmartCAD, an intelligent learning system that informs engineering design of students with automatic feedback generated using computational analysis of their work.

Engineering design is one of the most complex learning processes because it builds on top of multiple layers of inquiry, involves creating products that meet multiple criteria and constraints, and requires the orchestration of mathematical thinking, scientific reasoning, systems thinking, and sometimes, computational thinking. Teaching and learning engineering design becomes important as it is now officially part of the Next Generation Science Standards in the United States. These new standards mandate every student to learn and practice engineering design in every science subject at every level of K-12 education.
Figure 1

In typical engineering projects, students are challenged to construct an artifact that performs specified functions under constraints. What makes engineering design different from other design practices such as art design is that engineering design must be guided by scientific principles and the end products must operate predictably based on science. A common problem observed in students' engineering design activities is that their design work is insufficiently informed by science, resulting in the reduction of engineering design to drawing or crafting. To circumvent this problem, engineering design curricula often encourage students to learn or review the related science concepts and practices before they try to put the design elements together to construct a product. After students create a prototype, they then test and evaluate it using the governing scientific principles, which, in turn, gives them a chance to deepen their understanding of the scientific principles. This common approach of learning is illustrated in the upper image of Figure 1.

There is a problem in the common approach, however. Exploring the form-function relationship is a critical inquiry step to understanding the underlying science. To determine whether a change of form can result in a desired function, students have to build and test a physical prototype or rely on the opinions of an instructor. This creates a delay in getting feedback at the most critical stage of the learning process, slowing down the iterative cycle of design and cutting short the exploration in the design space. As a result of this delay, experimenting and evaluating "micro ideas"--very small stepwise ideas such as those that investigate a design parameter at a time--through building, revising, and testing physical prototypes becomes impractical in many cases. From the perspective of learning, however, it is often at this level of granularity that foundational science and engineering design ultimately meet.

Figure 2
All these problems can be addressed by supporting engineering design with a computer-aided design (CAD) platform that embeds powerful science simulations to provide formative feedback to students in a timely manner. Simulations based on solving fundamental equations in science such as Newton’s Laws model the real world accurately and connect many science concepts coherently. Such simulations can computationally generate objective feedback about a design, allowing students to rapidly test a design idea on a scientific basis. Such simulations also allow the connections between design elements and science concepts to be explicitly established through fine-grained feedback, supporting students to make informed design decisions for each design element one at a time, as illustrated by the lower image of Figure 1. These scientific simulations give the CAD software tremendous disciplinary intelligence and instructional power, transforming it into a SmartCAD system that is capable of guiding student design towards a more scientific end.

Despite these advantages, there are very few developmentally appropriate CAD software available to K-12 students—most CAD software used in industry not only are science “black boxes” to students, but also require a cumbersome tool chaining of pre-processors, solvers, and post-processors, making them extremely challenging to use in secondary education. The SmartCAD project will fill in this gap with key educational features centered on guiding student design with feedback composed from simulations. For example, science simulations can be used to analyze student design artifacts and compute their distances to specific goals to detect whether students are zeroing in towards those goals or going astray. The development of these features will also draw upon decades of research on formative assessments of complex learning.

Book review: "Simulation and Learning: A Model-Centered Approach" by Franco Landriscina

Interactive science (Image credit: Franco Landriscina)
If future historians were to write a book about the most important contributions of technology to improving science education, it would be hard for them to skip computer modeling and simulation.

Much of our intelligence as humans originates from our ability to run mental simulations or thought experiments in our mind to decide whether it would be a good idea to do something or not to do something. We are able to do this because we have already acquired some basic ideas or mental models that can be applied to new situations. But how do we get those ideas in the first place? Sometimes we learn from our experiences. Sometimes we learn from listening to someone. Now, we can learn from computer simulation, which was carefully programmed by someone who knows the subject matter well and is typically expressed by a computer through interactive visualization based on some sort of calculation. In the cases when the subject matter is entirely alien to students such as atoms and molecules, computer simulation is perhaps the most effective form of instruction. Given the importance of mental simulation in scientific reasoning, there is no doubt that computer simulation, bearing some similarity with mental simulation, should have great potential in fostering learning.

Constructive science (Image credit: Franco Landriscina)
Although enough ink has been spilled on this topic and many thoughts have existed in various forms for decades, I found the book "Simulation and Learning: A Model-Centered Approach" by Dr. Franco Landriscina, an experimental psychologist in Italy, is a masterpiece that I must have on my desk and chew over from time to time. What Dr. Landriscina has accomplished in a book less than 250 pages is amazingly deep and wide. He starts with fundamental questions in cognition and learning that are related to simulation-based instruction. He then gradually builds a solid theoretical foundation for understanding why computer simulation can help people learn and think by grounding cognition in the interplay between mental simulation (internal) and computer simulation (external). This intimate coupling of internalization and externalization leads to some insights as for how the effectiveness of computer simulation as an instructional tool can be maximized in various cases. For example, Landriscina's two illustrations, embedded in this blog post, represent how two ways of using simulations in learning, which I coined as "Interactive Science" and "Constructive Science," differ in terms of the relationships among the foundational components in cognition and simulation.

This book is not only useful to researchers. Developers should benefit from reading it, too. Developers tend to create educational tools and materials based on the learning goals set by some education standards, with less consideration on how complex learning actually happens through interaction and cognition in reality. This succinct book should provide a comprehensive, insightful, and intriguing guide for those developers who would like to understand more deeply about simulation-based learning in order to create more effective educational simulations.

SimBuilding on iPad

SimBuilding (alpha version) is a 3D simulation game that we are developing to provide a more accessible and fun way to teach building science. A good reason that we are working on this game is because we want to teach building science concepts and practices to home energy professionals without having to invade someone's house or risk ruining it (well, we have to create or maintain some awful cases for teaching purposes, but what sane property owner would allow us to do so?). We also believe that computer graphics can be used to create some cool effects that demonstrate the ideas more clearly, providing complementary experiences to hands-on learning. The project is funded by the National Science Foundation to support technical education and workforce development.

SimBuilding is based on three.js, a powerful JavaScript-based graphics library that renders 3D scenes within the browser using WebGL. This allows it to run on a variety of devices, including the iPad (but not on a smartphone that has less horsepower, however). The photos in this blog post show how it looks on an iPad Mini, with multi-touch support for navigation and interaction.

In its current version, SimBuilding only supports virtual infrared thermography. The player walks around in a virtual house, challenged to correctly identify home energy problems in a house using a virtual IR camera. The virtual IR camera will show false-color IR images of a large number of sites when the player inspects them, from which the player must diagnose the causes of problems if he believes the house has been compromised by problems such as missing insulation, thermal bridge, air leakage, or water damage. In addition to the IR camera, a set of diagnostics tools is also provided, such as a blower-door system that is used to depressurize a house for identifying infiltration. We will also provide links to our Energy2D simulations should the player become interested in deepening their understanding about heat transfer concepts such as conduction, convection, and radiation.

SimBuilding is a collaborative project with New Mexico EnergySmart Academy at Santa Fe. A number of industry partners such as FLIR Systems and Building Science Corporation are also involved in this project. Our special thanks go to Jay Bowen of FLIR, who generously provided most of the IR images used to create the IR game scenes free of charge.

Modeling solar thermal power using heliostats in Energy2D

An array of heliostats in Energy2D (online simulation)
A new class of objects was added in Energy2D to model what is called a heliostat, a device that can automatically turn a mirror to reflect sunlight to a target no matter where the sun is in the sky. Heliostats are often used in solar thermal power plants or solar furnaces that use mirrors. With an array of computer-controlled heliostats and mirrors, the energy from the sun can be concentrated on the target to heat it up to a very high temperature, enough to vaporize water to create steam that drives a turbine to generate electricity.

Image credit: Wikipedia
The Ivanpah Solar Power Facility in California's Mojave Desert, which went online on February 13, 2014, is currently the world's largest solar thermal power plant. With a gross capacity of 392 megawatts, it is enough to power 140,000 homes. It deploys 173,500 heliostats, each controlling two mirrors.

A heliostat in Energy2D contains a planar mirror mounted on a pillar. You can drop one in at any location. Once you specify its target, it will automatically reflect any sunlight beam hitting on it to the target.

Strictly speaking, heliostats are different from solar trackers that automatically face the sun like sunflowers. But in Energy2D, if no target is specified, as is the default case, a heliostat becomes a solar tracker. Unlike heliostats, solar trackers are often used with photovoltaic (PV) panels that absorb, instead of reflecting, sunlight that shine on them. A future version of Energy2D will include the capacity of modeling PV power plants as well.

Complete undo/redo support in Energy2D

In Version 2.3 of Energy2D, I have added full support of undo/redo for most actions. With this feature, you can undo all the way back to your starting point and redo all the way forward to your latest state. This is not only a must-have feature for a design tool with a reasonable degree of complexity, but also a simple -- yet powerful -- mechanism for reliably collecting very fine-grained data for understanding how a user interacts with the software.

Why are we interested in collecting these action data?

From the perspective of software engineering, these action data provide first-hand information for quality assurance (QA). QA engineers can analyze these data to measure the usability of the software, to identify behavior patterns of users, and to track results from version to version to gauge if an adjustment has led to better user experience.

From the perspective of education and training, these action data encode users' cognitive processes. Any interaction with the software, especially with a piece of highly visual and responsive software like Energy2D, is automatically a process of cognition. A fundamental thesis in learning science is to understand how we can design interactive materials that maximize learning for all students. These precious fine-grained action data may hold an important key to that understanding.

This idea of using the stack of actions stored in the undo manager of a piece of software to record and replay the entire process of interaction is a unique feature that has been implemented in our Energy2D and Energy3D software and proven a non-obtrusive, high-fidelity, and low-bandwidth technique for data collection.

A stock-and-flow model for building thermal analysis

Figure 1. A stock-and-flow model of building energy.
Our Energy3D CAD software has two built-in simulation engines for performing solar energy analysis and building thermal analysis. I have extensively blogged about solar energy analysis using Energy3D. This article introduces building thermal analysis with Energy3D.

Figure 2. A colonial house.
The current version of the building energy simulation engine is based on a simple stock-and-flow model of building energy. Viewed from the perspective of system dynamics—a subject that studies the behavior of complex systems, the total thermal energy of a building is a stock and the energy gains or losses through its various components are flows. These gains or losses usually happen via the energy exchange between the building and the environment through the components. For instance, the solar radiation that shines into a building through its windows are inputs; the heat transfer through its walls may be inputs or outputs depending on the temperature difference between the inside and the outside.

Figure 3. The annual energy graph.
Figure1 illustrates how energy flows into and out of a building in the winter and summer, respectively. In order to maintain the temperature inside a building, the thermal energy it contains must remain constant—any shortage of thermal energy must be compensated and any excessive thermal energy must be removed. These are done through heating and air conditioning systems, which, together with ventilation systems, are commonly known as HVAC systems. Based on the stock-and-flow model, we can predict the energy cost of heating and air conditioning by summing up the energy flows in various processes of heat transfer, solar radiation, and energy generation over all the components of the building such as walls, windows, or roofs and over a certain period of time such as a day, a month, or a year.

Figure 2 shows the solar radiation heat map of a house and the distribution of the heat flux density over its building envelope. Figure 3 shows the results of the annual energy analysis for the house shown in Figure 2.

More information can be found in Chapter 3 of Energy3D's User Guide.

Energy2D video tutorials in English and Spanish

Many users asked if there is any good tutorial of Energy2D. I apologize for the lack of a User Manual and other tutorial materials (I am just too busy to set aside time for writing up some good documentations).


So Carmen Trudell, an architect who currently teaches at the School of Architecture of the University of Virginia, decided to make a video tutorial of Energy2D for her students. It turned out to be an excellent overview of what the software is capable of doing in terms of illustrating some basic concepts related to heat transfer in architectural engineering. She also kindly granted permission for us to publish her video on Energy2D's website so that other users can benefit from her work.

If you happen to come from the Spanish-speaking part of the world, there is also a Spanish video tutorial made by Gabriel Concha based on an earlier version of Energy2D.

Beautiful Chemisty won the Vizzies Award

The National Science Foundation and the Popular Science Magazine have announced that “Beautiful Chemistry” won the Expert's Choice Award for Video at the 2015 Visualization Challenge, known as Vizzies. According to the Popular Science Magazine,
For many, the phrase “chemical reactions” conjures memories of tedious laboratory work and equations scribbled on exams. But Yan Liang, a professor at the University of Science and Technology of China in Hefei, sees art in the basic science. Last September, Liang and colleagues launched beautiful​chemistry.net to highlight aesthetically pleasing chemistry. Their video showcases crystallization, fluorescence, and other reactions or structures shot in glorious detail. Liang says finding experiments that meet their visual standards has been a challenge. “Many reactions are very interesting, but not beautiful,” he says. “But sometimes, when shot at close distance without the distraction of beakers or test tubes, ordinary reactions such as precipitation can be very beautiful.”
Beautiful Chemistry is the first of the Beautiful Science Series that Prof. Liang has been planning. The series will include two new titles, Beautiful Simulations and Beautiful Infrared, which we will co-produce with Prof. Liang this summer while he visits Boston.

Congratulations to Prof. Liang for this amazing work!

Comparing two smartphone-based infrared cameras

Figure 1
With the releases of two competitively priced IR cameras for smartphones, the year 2014 has become a milestone for IR imaging. Early in 2014, FLIR unveiled the $349 FLIR ONE, the first IR camera that can be attached to an iPhone. Months later, a startup company Seek Thermal released a $199 IR camera that has an even higher resolution and is attachable to most smartphones. In addition, another company Therm-App released an Android mobile thermal camera that specializes in long-range night vision and high-resolution thermography, priced at $1,600. The race is on... Into 2015, FLIR announced a new version of FLIR ONE that supports both Android and iOS and will probably be even more aggressively priced.

Figure 2
All these game changers can take impressive IR images just like taking conventional photos and record IR videos just like recording conventional videos, and then share them online through an app. The companies also provide a software developers kit (SDK) for a third party to create apps linked to their cameras. Excited by these new developments, researchers at several Swedish universities and I have embarked an international collaboration towards the vision that IR cameras will one day become as necessary as microscopes in science labs.

Figure3
To test these new IR cameras, I did an easy-to-do experiment (Figure 1) that shows a paradoxical warming effect on a piece of paper placed on top of a cup of (slightly cooler than) room-temperature water. This seemingly simple experiment actually leads to very deep science at the molecular level, as blogged before.

I took images using FLIR ONE (Figure 2) and SEEK (Figure 3), respectively. These images are shown to the right for comparison. As you can see, both cameras are sensitive enough to capture the small temperature rise caused by water absorption and condensation underside the paper.

The FLIR ONE has a nice feature that contextualizes the false-color IR image by overlaying it on top of the edges (where brightness changes sharply) of the true-color image taken at the same time by the conventional camera of the smartphone. With this feature, you can see the sharp edges of the paper in Figure 2.