Posts Tagged ‘Time series analysis’

On the instructional sensitivity of computer-aided design logs

July 20th, 2014 by Charles Xie
Figure 1: Hypothetical student responses to an intervention.
In the fourth issue this year, the International Journal of Engineering Education published our 19-page-long paper on the instructional sensitivity of computer-aided design (CAD) logs. This study was based on our Energy3D software, which supports students to learn science and engineering concepts and skills through creating sustainable buildings using a variety of built-in design and analysis tools related to Earth science, heat transfer, and solar energy. This paper proposed an innovative approach of using response functions -- a concept borrowed from electrical engineering -- to measure instructional sensitivity from data logs (Figure 1).

Many researchers are interested in studying what students learn through complex engineering design projects. CAD logs provide fine-grained empirical data of student activities for assessing learning in engineering design projects. However, the instructional sensitivity of CAD logs, which describes how students respond to interventions with CAD actions, has never been examined, to the best of our knowledge.
Figure 2. An indicator of statistical reliability.

For the logs to be used as reliable data sources for assessments, they must be instructionally sensitive. Our paper reports the results of our systematic research on this important topic. To guide the research, we first propose a theoretical framework for computer-based assessments based on signal processing. This framework views assessments as detecting signals from the noisy background often present in large temporal learner datasets due to many uncontrollable factors and events in learning processes. To measure instructional sensitivity, we analyzed nearly 900 megabytes of process data logged by Energy3D as collections of time series. These time-varying data were gathered from 65 high school students who solved a solar urban design challenge using Energy3D in seven class periods, with an intervention occurred in the middle of their design projects.

Our analyses of these data show that the occurrence of the design actions unrelated to the intervention were not affected by it, whereas the occurrence of the design actions that the intervention targeted reveals a continuum of reactions ranging from no response to strong response (Figure 2). From the temporal patterns of these student responses, persistent effect and temporary effect (with different decay rates) were identified. Students’ electronic notes taken during the design processes were used to validate their learning trajectories. These results show that an intervention occurring outside a CAD tool can leave a detectable trace in the CAD logs, suggesting that the logs can be used to quantitatively determine how effective an intervention has been for each individual student during an engineering design project.

The first paper on learning analytics for assessing engineering design?

January 30th, 2014 by Charles Xie
Figure 1
The International Journal of Engineering Education published our paper ("A Time Series Analysis Method for Assessing Engineering Design Processes Using a CAD Tool") on learning analytics and educational data mining for assessing student performance in complex engineering design projects. I believe this is the first time learning analytics was applied to the study of engineering design -- an extremely complicated process that is very difficult to assess using traditional methodologies because of its open-ended and practical nature.

Figure 2
This paper proposes a novel computational approach based on time series analysis to assess engineering design processes using our Energy3D CAD tool. To collect research data without disrupting a design learning process, design actions and artifacts are continuously logged as time series by the CAD tool behind the scenes, while students are working on an engineering design project such as a solar urban design challenge. These "atomically" fine-grained data can be used to reconstruct, visualize, and analyze the entire design process of a student with extremely high resolution. Results of a pilot study in a high school engineering class suggest that these data can be used to measure the level of student engagement, reveal the gender differences in design behaviors, and distinguish the iterative (Figure 1) and non-iterative (Figure 2) cycles in a design process.

From the perspective of engineering education, this paper contributes to the emerging fields of educational data mining and learning analytics that aim to expand evidence approaches for learning in a digital world. We are working on a series of papers to advance this research direction and expect to help with the "landscaping" of  those fields.

Measuring the effects of an intervention using computational process analytics

September 15th, 2013 by Charles Xie
"At its core, scientific inquiry is the same in all fields. Scientific research, whether in education, physics, anthropology, molecular biology, or economics, is a continual process of rigorous reasoning supported by a dynamic interplay among methods, theories, and findings. It builds understanding in the form of models or theories that can be tested."  —— Scientific Research in Education, National Research Council, 2002
Actions caused by the intervention
Computational process analytics (CPA) is a research method that we are developing in the spirit of the above quote from the National Research Council report. It is a whole class of data mining methods for quantitatively studying the learning dynamics in complex scientific inquiry or engineering design projects that are digitally implemented. CPA views performance assessment as detecting signals from the noisy background often present in large learner datasets due to many uncontrollable and unpredictable factors in classrooms. It borrows many computational techniques from engineering fields such as signal processing and pattern recognition. Some of these analytics can be considered as the computational counterparts of traditional assessment methods based on student articulation, classroom observation, or video analysis.

Actions unaffected by the intervention
Computational process analytics has wide applications in education assessments. High-quality assessments of deep learning hold a critical key to improving learning and teaching. Their strategic importance has been highlighted in President Obama’s remarks in March 2009: “I am calling on our nation’s Governors and state education chiefs to develop standards and assessments that don’t simply measure whether students can fill in a bubble on a test, but whether they possess 21st century skills like problem-solving and critical thinking, entrepreneurship, and creativity.” However, the kinds of assessments the President wished for often require careful human scoring that is far more expensive to administer than multiple-choice tests. Computer-based assessments, which rely on the learning software to automatically collect and sift learner data through unobtrusive logging, are viewed as a promising solution to assessing increasingly prevalent digital learning.

While there have been a lot of work on computer-based assessments for STEM education, one foundational question has rarely been explored: How sensitive can the logged learner data be to instructions?

Actions caused by the intervention.
According to the assessment guru Popham, there are two main categories of evidence for determining the instructional sensitivity of an assessment tool: judgmental evidence and empirical evidence. Computer logs provide empirical evidence based on user data recording—the logs themselves provide empirical data for assessment and their differentials before and after instructions provide empirical data for evaluating the instructional sensitivity. Like any other assessment tools, computer logs must be instructionally sensitive if they are to provide reliable data sources for gauging student learning under intervention. 


Actions unaffected by the intervention.
Earlier studies have used CAD logs to capture the designer’s operational knowledge and reasoning processes. Those studies were not designed to understand the learning dynamics occurring within a CAD system and, therefore, did not need to assess students’ acquisition and application of knowledge and skills through CAD activities. Different from them, we are studying the instructional sensitivity of CAD logs, which describes how students react to interventions with CAD actions. Although interventions can be either carried out by human (such as teacher instruction or group discussion) or generated by the computer (such as adaptive feedback or intelligent tutoring), we have focused on human interventions in this phase of our research. Studying the instructional sensitivity to human interventions will enlighten the development of effective computer-generated interventions for teaching engineering design in the future (which is another reason, besides cost effectiveness, why research on automatic assessment using learning software logs is so promising).

The study of instructional effects on design behavior and performance is particularly important, viewing from the perspective of teaching science through engineering design, a practice now mandated by the newly established Next Generation Science Standards of the United States. A problem commonly observed in K-12 engineering projects, however, is that students often reduce engineering design challenges to construction or craft activities that may not truly involve the application of science. This suggests that other driving forces acting
Distribution of intervention effect across 65 students.
on learners, such as hunches and desires for how the design artifacts should look, may overwhelm the effects of instructions on how to use science in design work. Hence, the research on the sensitivity of design behavior to science instruction requires careful analyses using innovative data analytics such as CPA to detect the changes, however slight they might be. The insights obtained from studying this instructional sensitivity may result in the actionable knowledge for developing effective instructions that can reproduce or amplify those changes.

Our preliminary CPA results have shown that CAD logs created using our Energy3D CAD tool are instructionally sensitive. The first four figures embedded in this post show two pairs of opposite cases with one type of action sensitive to an instruction that occurred outside the CAD tool and the other not. This is because the instruction was related to one type of action and had nothing to do with the other type. The last figure shows that the distribution of instructional sensitivity across 65 students. In this figure, the largest number means higher instructional sensitivity. A number close to one means that the instruction has no effect. From the graph, you can see that the three types of actions that are not related to the instruction fluctuate around one whereas the fourth type of action is strongly sensitive to the instruction.

These results demonstrate that software logs can not only record what students do with the software but also capture the effects of what happen outside the software.

Significant gender differences found (confirmed?) in CAD research

March 13th, 2013 by Charles Xie
A student design
In a pilot study conducted in December 2012, high school students in an engineering class used our Energy3D CAD tool to do an urban solar design project -- they must consider the sun path in four seasons and the existing buildings in the neighborhood as the design constraints to optimize solar penetration to the new buildings and minimize obstruction of sunlight to the existing buildings.

Energy3D can log any student actions and intermediate steps, which provide extremely detailed information about student design processes. With such a high-resolution lens, we could characterize student patterns and analyze how they solve the design challenge closely. For example, the CAD log allows us to reconstruct the entire design process of each student and show it in an unprecedentedly fine-grained timeline graph. A timeline graph may show how students went through different iterative steps while shaping their designs. For instance, did they consider the interactions among the buildings they designed? Did they go back to revise a previously erected building that may be affected by a newly added one? The timeline data we have collected show that the students' designs demonstrated more iterative features as they moved on to explore and design alternatives following the initial attempts (perhaps encouraged by the gained familiarity with and confidence in the CAD tool).

A design timeline (click to enlarge)
Our analyses also suggest that there appears to be a significant gender difference in both design products and processes. The main differences are: 1) The boys tended to push the limit of the software and produced unconventional designs that looked "cool" but did not necessarily meet the design specifications; and 2) The girls spent more time carefully revising their designs than building new structures. While these findings may not be surprising to some seasoned educators, the significance is that this may be the first time this kind of gender difference was revealed or confirmed by empirical data from CAD logs. Using CAD logs may provide a fairer basis of assessing student performance based on the entire learning process rather than just looking at their final products or self reports.

Summary of the results
The implication of this study is that if we can identify patterns in student design learning and understand their cognitive meanings, we could devise a software system that can provide real-time feedback to help students learn in the future. For example, could the software prompt students to consider the design criteria more when it detects that students are ignoring them? Could the software stimulate students to think out of the box more when it detects that students are underexploring the design space?

For more information about this research project, visit: http://energy.concord.org/research.html.

Engineering engineering research: Understanding the fabric of engineering design

December 27th, 2012 by Charles Xie
A house designed using our Energy3D CAD software.
Perhaps the most important change in the Next Generation Science Standards to be released in March 2013 is the elevation of engineering design to the same level of importance as of scientific inquiry (which was enshrined as a doctrine of science education in the 1996 science standards). But how much do we know about teaching engineering design in K-12 classrooms?

A house made using our Energy3D CAD software.
Surprisingly, our knowledge about students’ learning and ideation in engineering design is dismal. The Committee on Standards for K-12 Engineering Education assembled by the National Research Council in 2010 found “very little research by cognitive scientists that could inform the development of standards for engineering education in K–12.” Most educational engineering projects lacked data collection and analysis to provide reliable evidence of learning. Many simply replicated the “engineering science” model from higher education, which focuses on learning basic science for engineering rather than learning engineering design. Little was learned from these projects about students’ acquisition of design skills and development of design thinking. In the absence of in-depth knowledge about students’ design learning, it would be difficult to teach and assess engineering design.

In response to these problems, we have proposed a research initiative that will hopefully start to fill the gap. As in any scientific research, our approach is to first establish a theory of cognitive development for engineering design and then invent a variety of experimental techniques to verify research hypotheses based on the theory. This blog post introduces these ideas.

In order to study engineering design on a rigorous basis, we need a system that can automatically monitor student workflows to provide us all the fine-grain data we need to understand how they think and learn when they become expert designers from novice designers. This means we have no choice but to move the entire engineering design process onto the computer -- to be more exact, into computer-aided design (CAD) systems -- so that we can keep track of students’ workflows and extract information for inferring their learning. While some educators may be uncomfortable with the virtualization of engineering design, this actually complies with contemporary engineering practices that ubiquitously rely on CAD tools. If we have a CAD system, we can add some data mining mechanisms to turn it into a powerful experimental system for investigating student learning. Fortunately, we have created our own CAD software, Energy3D, from scratch (see the above images about it). So we can do anything we want with it. Since all the CAD tools are similar, the research results should be generalizable.

A cognitive theory of engineering design.
Next we need a cognitive theory of engineering design. Engineering design is interdisciplinary, dynamic, and complicated. It requires students to apply STEM knowledge to solve open-ended problems with a given set of criteria and constraints. It is such a complex process that I am almost certain that any cognitive theory will not be perfect. But without a cognitive theory our research would be aimless. So we must invent one.

Our cognitive theory assumes that engineering design is a process of “knitting” science and engineering. Inquiry and design are at the hearts of science and engineering practices. In an engineering project, both types of practices are needed. All engineering systems are tested during the development phase. A substantial part of engineering is to find problems through tests in order to build robust products. The diagnosis of a problem is, as a matter of fact, a process of scientific inquiry into an engineered system. The results of this inquiry process provide explanations of the problem, as well as feedback to revise the design and improve the system. The modified system with new designs is then put through further tests. Testing a new design can lead to more questions worth investigating, starting a new cycle of inquiry. This process of interwoven inquiry and design repeats itself until the system is determined to be a mature product. 

These elements in our cognitive theory all sound logical and necessary. Now the question is: If we agree on this theory, how are we going to make it happen in the classroom and how are we going to measure its degree of success? Formative assessment seems to be the key. So the next thing we need to invent is a method of formative assessment. But what should we assess in order not to miss the entire picture of learning? This requires us to develop a deep understanding of the fabric of engineering design.

A time series model of design assessment.
Engineering design is a complex process that involves multiple types of science and engineering tasks and subprocesses that occur iteratively. Along with the properties and attributes of the designed artifacts that can be calculated, the order, frequency, and duration learners handle the tasks provide invaluable insights into the fabric of engineering design. These data can be monitored and collected as time series. Formative assessment can then be viewed as the analysis of a set of time series, each representing an aspect of learning or performance. In other words, each time series logs a “fiber” of engineering design.

At first glance, the time series data may look stochastic, just like the Dow Jones index. But buried under the noisy data are students’ behavioral and cognitive patterns. Time series analysis, which has been widely used in signal processing and pattern recognition, will provide us the analytic power to detect learner behaviors from the seemingly random data and then generate adaptive feedback to steer learning to less arbitrary, more productive paths. For example, spectral or wavelet analysis can be used to calculate the frequency of using a design or test tool. Auto-correlation analysis can be used to find repeating patterns in a subprocess. Cross-correlation analysis can be used to examine if an activity or intervention in one subprocess has resulted in changes in another. Cross-correlation provides a potentially useful tool for tracking a designer’s activity with regard to knowledge integration and system thinking.

In the next six months, we will undertake this ambitious research project and post our findings in this blog as we move forward. Stay tuned!