Visual Process Analytics (VPA) launched


Visual Process Analytics (VPA) is a Web-based program that we are developing for visualizing and analyzing student learning from complex, fine-grained process data collected by interactive learning software such as computer-aided design tools. We envision a future in which every classroom would be powered by informatics and infographics such as VPA to support day-to-day learning and teaching at a highly responsive level. In a future when every business person relies on visual analytics every day to stay in business, it would be a shame that teachers still have to read through tons of paper-based work from students to make instructional decisions. The research we are conducting with the support of the National Science Foundation is paving the road to a future that would provide the fair support for our educational systems that is somehow equivalent to business analytics and intelligence.

This is the mission of VPA. Today we are announcing the launch of this cyberinfrastructure. We decided that its first version number should be 0.1. This is just a way to indicate that the research and development on this software system will continue as a very long-term effort and what we have done is a very small step towards a very ambitious goal.

VPA is written in plain JavaScript/HTML/CSS. It runs within any browser but it looks and works like a typical desktop app. This means that while you are in the middle of mining the data, you can save what we call "the perspective" as a file onto your disk (or in the cloud) so that you can keep track of what you have done. Later, you can load the perspective back into VPA. Each perspective opens the datasets that you have worked on, with your latest settings and results. So if you are half way through your data mining, your work can be saved for further analyses.

So far Version 0.1 has seven analysis and visualization tools, each of which shows a unique aspect of the learning process with a unique type of interactive visualization. We admit that, compared with the daunting high dimension of complex learning, this is a tiny collection. But we will be adding more and more tools as we go. At this point, only one repository -- our own Energy3D process data -- is connected to VPA. But we expect to add more repositories in the future. Meanwhile, more computational tools will be added to support in-depth analyses of the data. This will require a tremendous effort in designing a smart user interface to support various computational tasks that researchers may be interested in defining.

Eventually, we hope that VPA will grow into a versatile platform of data analytics for cutting-edge educational research. As such, VPA represents a critically important step towards marrying learning science with data science and computational science.

The National Science Foundation funds large-scale applications of infrared cameras in schools


We are pleased to announce that the National Science Foundation has awarded the Concord Consortium, Next Step Living, and Virtual High School a grant of $1.2M to put innovative technologies such as infrared cameras into the hands of thousands of secondary students. This education-industry collaborative will create a technology-enhanced learning pathway from school to home and then to cognate careers, establishing thereby a data-rich testbed for developing and evaluating strategies for translating innovative technology experiences into consistent science learning and career awareness in different settings. While there have been studies on connecting science to everyday life or situating learning in professional scenarios to increase the relevance or authenticity of learning, the strategies of using industry-grade technologies to strengthen these connections have rarely been explored. In many cases, often due to the lack of experiences, resources, and curricular supports, industry technologies are simply used as showcases or demonstrations to give students a glimpse of how professionals use them to solve problems in the workplace.


Over the last few years, however, quite a number of industry technologies have become widely accessible to schools. For example, Autodesk has announced that their software products will be freely available to all students and teachers around the world. Another example is infrared cameras that I have been experimenting and blogging since 2010. Due to the continuous development of electronics and optics, what used to be a very expensive scientific instrument is now only a few hundred dollars, with the most affordable infrared camera falling below $200.

The funded project, called Next Step Learning, will be the largest-scale application of infrared camera in secondary schools -- in terms of the number of students that will be involved in the three-year project. We estimate that dozens of schools and thousands of students in Massachusetts will participate in this project. These students will use infrared cameras provided by the project to thermally inspect their own homes. The images in this blog post are some of the curious images I took in my own house using the FLIR ONE camera that is attached to an iPhone.

In the broader context, the Next Generation Science Standards (NGSS) envisions “three-dimensional learning” in which the learning of disciplinary core ideas and crosscutting concepts is integrated with science and engineering practices. A goal of the NGSS is to make science education more closely resemble the way scientists and engineers actually think and work. To accomplish this goal, an abundance of opportunities for students to practice science and engineering through solving authentic real-world problems will need to be created and researched. If these learning opportunities are meaningfully connected to current industry practices using industry-grade technologies, they can also increase students’ awareness of cognate careers, help them construct professional identities, and prepare them with knowledge and skills needed by employers, attaining thereby the goals of both science education and workforce development simultaneously. The Next Step Learning project will explore, test, and evaluate this strategy.

Twelve Energy3D designs by Cormac Paterson

Cormac Paterson, a 17-years old student from Arlington High School in Massachusetts, has created yet another set of beautiful architectural designs using our Energy3D CAD software. The variety of his designs can be used to gauge the versatility of the software. His work is helping us push the boundary of the software and imagine what may be possible with the system.

This is the second year Cormac has worked with us as a summer intern. We are constantly impressed by his perseverance in working with the limitations of the software and around problems, as well as his ingenuity in coming up with new solutions and ideas. Working with Cormac has inspired us on how to improve our software so that it can support more students to do this kind of creative design. Our objective in the long run is to develop our software into a CAD system that is appropriate for children and yet capable of supporting authentic engineering design. Cormac's work might be an encouraging sign that we may actually be very close to realizing this goal.

Cormac also designed a building surrounded by solar trees. Solar tree is a concept that blends art and solar energy technology in a sculptural expression. An image of this post shows the result of the solar energy gains of these "trees" using the improved computational engine for solar simulation in Energy3D. 

Seeing student learning with visual analytics

Technology allows us to record almost everything happening in the classroom. The fact that students' interactions with learning environments can be logged in every detail raises the interesting question about whether or not there is any significant meaning and value in those data and how we can make use of them to help students and teachers, as pointed out in a report sponsored by the U.S. Department of Education:
New technologies thus bring the potential of transforming education from a data-poor to a data-rich enterprise. Yet while an abundance of data is an advantage, it is not a solution. Data do not interpret themselves and are often confusing — but data can provide evidence for making sound decisions when thoughtfully analyzed.” — Expanding Evidence Approaches for Learning in a Digital World, Office of Educational Technology, U.S. Department of Education, 2013
A radar chart of design space exploration.
A histogram of action intensity.
Here we are not talking about just analyzing students' answers to some multiple-choice questions, or their scores in quizzes and tests, or their frequencies of logging into a learning management system. We are talking about something much more fundamental, something that runs deep in cognition and learning, such as how students conduct a scientific experiment, solve a problem, or design a product. As learning goes deeper in those directions, data produced by students grows bigger. It is by no means an easy task to analyze large volumes of learner data, which contain a lot of noisy elements that cast uncertainty to assessment. The validity of an assessment inference rests on  the strength of evidence. Evidence construction often relies on the search for relations, patterns, and trends in student data.With a lot of data, this mandates some sophisticated computation similar to cognitive computing.

Data gathered from highly open-ended inquiry and design activities, key to authentic science and engineering practices that we want students to learn, are often intensive and “messy.” Without analytic tools that can discern systematic learning from random walk, what is provided to researchers and teachers is nothing but a DRIP (“data rich, information poor”) problem.

A scatter plot of action timeline.
Recognizing the difficulty in analyzing the sheer volume of messy student data, we turned to visual analytics, a whole category of techniques extensively used in cutting-edge business intelligence systems such as software developed by SAS, IBM, and others. We see interactive, visual process analytics key to accelerating the analysis procedures so that researchers can adjust mining rules easily, view results rapidly, and identify patterns clearly. This kind of visual analytics optimally combines the computational power of the computer, the graphical user interface of the software, and the pattern recognition power of the brain to support complex data analyses in data-intensive educational research.

A digraph of action transition.
So far, I have written four interactive graphs and charts that can be used to study four different aspects of the design action data that we collected from our Energy3D CAD software. Recording several weeks of student work on complex engineering design challenges, these datasets are high-dimensional, meaning that it is improper to treat them from a single point of view. For each question we are interested in getting answers from student data, we usually need a different representation to capture the outstanding features specific to the question. In many cases, multiple representations are needed to address a question.

In the long run, our objective is to add as many graphic representations as possible as we move along in answering more and more research questions based on our datasets. Given time, this growing library of visual analytics would develop sufficient power to the point that it may also become useful for teachers to monitor their students' work and thereby conduct formative assessment. To guarantee that our visual analytics runs on all devices, this library is written in JavaScript/HTML/CSS. A number of touch gestures are also supported for users to use the library on a multi-touch screen. A neat feature of this library is that multiple graphs and charts can be grouped together so that when you are interacting with one of them, the linked ones also change at the same time. As the datasets are temporal in nature, you can also animate these graphs to reconstruct and track exactly what students do throughout.

The National Science Foundation funds SmartCAD—an intelligent learning system for engineering design

We are pleased to announce that the National Science Foundation has awarded the Concord Consortium, Purdue University, and the University of Virginia a $3 million, four-year collaborative project to conduct research and development on SmartCAD, an intelligent learning system that informs engineering design of students with automatic feedback generated using computational analysis of their work.

Engineering design is one of the most complex learning processes because it builds on top of multiple layers of inquiry, involves creating products that meet multiple criteria and constraints, and requires the orchestration of mathematical thinking, scientific reasoning, systems thinking, and sometimes, computational thinking. Teaching and learning engineering design becomes important as it is now officially part of the Next Generation Science Standards in the United States. These new standards mandate every student to learn and practice engineering design in every science subject at every level of K-12 education.
Figure 1

In typical engineering projects, students are challenged to construct an artifact that performs specified functions under constraints. What makes engineering design different from other design practices such as art design is that engineering design must be guided by scientific principles and the end products must operate predictably based on science. A common problem observed in students' engineering design activities is that their design work is insufficiently informed by science, resulting in the reduction of engineering design to drawing or crafting. To circumvent this problem, engineering design curricula often encourage students to learn or review the related science concepts and practices before they try to put the design elements together to construct a product. After students create a prototype, they then test and evaluate it using the governing scientific principles, which, in turn, gives them a chance to deepen their understanding of the scientific principles. This common approach of learning is illustrated in the upper image of Figure 1.

There is a problem in the common approach, however. Exploring the form-function relationship is a critical inquiry step to understanding the underlying science. To determine whether a change of form can result in a desired function, students have to build and test a physical prototype or rely on the opinions of an instructor. This creates a delay in getting feedback at the most critical stage of the learning process, slowing down the iterative cycle of design and cutting short the exploration in the design space. As a result of this delay, experimenting and evaluating "micro ideas"--very small stepwise ideas such as those that investigate a design parameter at a time--through building, revising, and testing physical prototypes becomes impractical in many cases. From the perspective of learning, however, it is often at this level of granularity that foundational science and engineering design ultimately meet.

Figure 2
All these problems can be addressed by supporting engineering design with a computer-aided design (CAD) platform that embeds powerful science simulations to provide formative feedback to students in a timely manner. Simulations based on solving fundamental equations in science such as Newton’s Laws model the real world accurately and connect many science concepts coherently. Such simulations can computationally generate objective feedback about a design, allowing students to rapidly test a design idea on a scientific basis. Such simulations also allow the connections between design elements and science concepts to be explicitly established through fine-grained feedback, supporting students to make informed design decisions for each design element one at a time, as illustrated by the lower image of Figure 1. These scientific simulations give the CAD software tremendous disciplinary intelligence and instructional power, transforming it into a SmartCAD system that is capable of guiding student design towards a more scientific end.

Despite these advantages, there are very few developmentally appropriate CAD software available to K-12 students—most CAD software used in industry not only are science “black boxes” to students, but also require a cumbersome tool chaining of pre-processors, solvers, and post-processors, making them extremely challenging to use in secondary education. The SmartCAD project will fill in this gap with key educational features centered on guiding student design with feedback composed from simulations. For example, science simulations can be used to analyze student design artifacts and compute their distances to specific goals to detect whether students are zeroing in towards those goals or going astray. The development of these features will also draw upon decades of research on formative assessments of complex learning.

Book review: "Simulation and Learning: A Model-Centered Approach" by Franco Landriscina

Interactive science (Image credit: Franco Landriscina)
If future historians were to write a book about the most important contributions of technology to improving science education, it would be hard for them to skip computer modeling and simulation.

Much of our intelligence as humans originates from our ability to run mental simulations or thought experiments in our mind to decide whether it would be a good idea to do something or not to do something. We are able to do this because we have already acquired some basic ideas or mental models that can be applied to new situations. But how do we get those ideas in the first place? Sometimes we learn from our experiences. Sometimes we learn from listening to someone. Now, we can learn from computer simulation, which was carefully programmed by someone who knows the subject matter well and is typically expressed by a computer through interactive visualization based on some sort of calculation. In the cases when the subject matter is entirely alien to students such as atoms and molecules, computer simulation is perhaps the most effective form of instruction. Given the importance of mental simulation in scientific reasoning, there is no doubt that computer simulation, bearing some similarity with mental simulation, should have great potential in fostering learning.

Constructive science (Image credit: Franco Landriscina)
Although enough ink has been spilled on this topic and many thoughts have existed in various forms for decades, I found the book "Simulation and Learning: A Model-Centered Approach" by Dr. Franco Landriscina, an experimental psychologist in Italy, is a masterpiece that I must have on my desk and chew over from time to time. What Dr. Landriscina has accomplished in a book less than 250 pages is amazingly deep and wide. He starts with fundamental questions in cognition and learning that are related to simulation-based instruction. He then gradually builds a solid theoretical foundation for understanding why computer simulation can help people learn and think by grounding cognition in the interplay between mental simulation (internal) and computer simulation (external). This intimate coupling of internalization and externalization leads to some insights as for how the effectiveness of computer simulation as an instructional tool can be maximized in various cases. For example, Landriscina's two illustrations, embedded in this blog post, represent how two ways of using simulations in learning, which I coined as "Interactive Science" and "Constructive Science," differ in terms of the relationships among the foundational components in cognition and simulation.

This book is not only useful to researchers. Developers should benefit from reading it, too. Developers tend to create educational tools and materials based on the learning goals set by some education standards, with less consideration on how complex learning actually happens through interaction and cognition in reality. This succinct book should provide a comprehensive, insightful, and intriguing guide for those developers who would like to understand more deeply about simulation-based learning in order to create more effective educational simulations.

SimBuilding on iPad

SimBuilding (alpha version) is a 3D simulation game that we are developing to provide a more accessible and fun way to teach building science. A good reason that we are working on this game is because we want to teach building science concepts and practices to home energy professionals without having to invade someone's house or risk ruining it (well, we have to create or maintain some awful cases for teaching purposes, but what sane property owner would allow us to do so?). We also believe that computer graphics can be used to create some cool effects that demonstrate the ideas more clearly, providing complementary experiences to hands-on learning. The project is funded by the National Science Foundation to support technical education and workforce development.

SimBuilding is based on three.js, a powerful JavaScript-based graphics library that renders 3D scenes within the browser using WebGL. This allows it to run on a variety of devices, including the iPad (but not on a smartphone that has less horsepower, however). The photos in this blog post show how it looks on an iPad Mini, with multi-touch support for navigation and interaction.

In its current version, SimBuilding only supports virtual infrared thermography. The player walks around in a virtual house, challenged to correctly identify home energy problems in a house using a virtual IR camera. The virtual IR camera will show false-color IR images of a large number of sites when the player inspects them, from which the player must diagnose the causes of problems if he believes the house has been compromised by problems such as missing insulation, thermal bridge, air leakage, or water damage. In addition to the IR camera, a set of diagnostics tools is also provided, such as a blower-door system that is used to depressurize a house for identifying infiltration. We will also provide links to our Energy2D simulations should the player become interested in deepening their understanding about heat transfer concepts such as conduction, convection, and radiation.

SimBuilding is a collaborative project with New Mexico EnergySmart Academy at Santa Fe. A number of industry partners such as FLIR Systems and Building Science Corporation are also involved in this project. Our special thanks go to Jay Bowen of FLIR, who generously provided most of the IR images used to create the IR game scenes free of charge.

Modeling solar thermal power using heliostats in Energy2D

An array of heliostats in Energy2D (online simulation)
A new class of objects was added in Energy2D to model what is called a heliostat, a device that can automatically turn a mirror to reflect sunlight to a target no matter where the sun is in the sky. Heliostats are often used in solar thermal power plants or solar furnaces that use mirrors. With an array of computer-controlled heliostats and mirrors, the energy from the sun can be concentrated on the target to heat it up to a very high temperature, enough to vaporize water to create steam that drives a turbine to generate electricity.

Image credit: Wikipedia
The Ivanpah Solar Power Facility in California's Mojave Desert, which went online on February 13, 2014, is currently the world's largest solar thermal power plant. With a gross capacity of 392 megawatts, it is enough to power 140,000 homes. It deploys 173,500 heliostats, each controlling two mirrors.

A heliostat in Energy2D contains a planar mirror mounted on a pillar. You can drop one in at any location. Once you specify its target, it will automatically reflect any sunlight beam hitting on it to the target.

Strictly speaking, heliostats are different from solar trackers that automatically face the sun like sunflowers. But in Energy2D, if no target is specified, as is the default case, a heliostat becomes a solar tracker. Unlike heliostats, solar trackers are often used with photovoltaic (PV) panels that absorb, instead of reflecting, sunlight that shine on them. A future version of Energy2D will include the capacity of modeling PV power plants as well.

Complete undo/redo support in Energy2D

In Version 2.3 of Energy2D, I have added full support of undo/redo for most actions. With this feature, you can undo all the way back to your starting point and redo all the way forward to your latest state. This is not only a must-have feature for a design tool with a reasonable degree of complexity, but also a simple -- yet powerful -- mechanism for reliably collecting very fine-grained data for understanding how a user interacts with the software.

Why are we interested in collecting these action data?

From the perspective of software engineering, these action data provide first-hand information for quality assurance (QA). QA engineers can analyze these data to measure the usability of the software, to identify behavior patterns of users, and to track results from version to version to gauge if an adjustment has led to better user experience.

From the perspective of education and training, these action data encode users' cognitive processes. Any interaction with the software, especially with a piece of highly visual and responsive software like Energy2D, is automatically a process of cognition. A fundamental thesis in learning science is to understand how we can design interactive materials that maximize learning for all students. These precious fine-grained action data may hold an important key to that understanding.

This idea of using the stack of actions stored in the undo manager of a piece of software to record and replay the entire process of interaction is a unique feature that has been implemented in our Energy2D and Energy3D software and proven a non-obtrusive, high-fidelity, and low-bandwidth technique for data collection.

A stock-and-flow model for building thermal analysis

Figure 1. A stock-and-flow model of building energy.
Our Energy3D CAD software has two built-in simulation engines for performing solar energy analysis and building thermal analysis. I have extensively blogged about solar energy analysis using Energy3D. This article introduces building thermal analysis with Energy3D.

Figure 2. A colonial house.
The current version of the building energy simulation engine is based on a simple stock-and-flow model of building energy. Viewed from the perspective of system dynamics—a subject that studies the behavior of complex systems, the total thermal energy of a building is a stock and the energy gains or losses through its various components are flows. These gains or losses usually happen via the energy exchange between the building and the environment through the components. For instance, the solar radiation that shines into a building through its windows are inputs; the heat transfer through its walls may be inputs or outputs depending on the temperature difference between the inside and the outside.

Figure 3. The annual energy graph.
Figure1 illustrates how energy flows into and out of a building in the winter and summer, respectively. In order to maintain the temperature inside a building, the thermal energy it contains must remain constant—any shortage of thermal energy must be compensated and any excessive thermal energy must be removed. These are done through heating and air conditioning systems, which, together with ventilation systems, are commonly known as HVAC systems. Based on the stock-and-flow model, we can predict the energy cost of heating and air conditioning by summing up the energy flows in various processes of heat transfer, solar radiation, and energy generation over all the components of the building such as walls, windows, or roofs and over a certain period of time such as a day, a month, or a year.

Figure 2 shows the solar radiation heat map of a house and the distribution of the heat flux density over its building envelope. Figure 3 shows the results of the annual energy analysis for the house shown in Figure 2.

More information can be found in Chapter 3 of Energy3D's User Guide.