Search NASA⌕ Search

Engineering topics

Anderson-Cook, Christine Michaela

Publications and source records attributed to Anderson-Cook, Christine Michaela.

At least 19 records

Microbial Drivers of Plant Performance during Drought Depend upon Community Composition and the Greater Soil Environment

The increasing occurrence of drought is a global challenge that threatens food security through direct impacts to both plants and their interacting soil microorganisms. Plant growth promoting microbes are increasingly being harnessed to improve plant performance under stress. However, the magnitude of microbiome impacts on both structural and physiological plant traits under water limited and water replete conditions are not well-characterized. Using two microbiomes sourced from a ponderosa pine forest and an agricultural field, we performed a greenhouse experiment that used a crossed design to test the individual and combined effects of the water availability and the soil microbiome composition on plant performance. Specifically, we studied the structural and leaf functional traits of maize that are relevant to drought tolerance. We further examined how microbial relationships with plant phenotypes varied under different combinations of microbial composition and water availability. We found that water availability and microbial composition affected plant structural traits. Surprisingly, they did not alter leaf function. Maize grown in the forest-soil microbiome produced larger plants under well-watered and water-limited conditions, compared to an agricultural soil community. Although leaf functional traits were not significantly different between the watering and microbiome treatments, the bacterial composition and abundance explained significant variability in both plant structure and leaf function within individual treatments, especially water-limited plants. Our results suggest that bacteria-plant interactions that promote plant performance under stress depend upon the greater community composition and the abiotic environment.

59 BASIC BIOLOGICAL SCIENCES↗

Gauge R & R studies for angular measurements

Angular measurements lie on the circumference of a circle and have different characteristics than standard scalar measurements. For applications involving angular data, treating the measured values as scalars can lead to misinterpretation of results if its wrap-around nature is not taken into account. In this article, we propose a variance components wrapped normal model for angular measurements that is analogous to the standard normal model for continuous measurements. This model allows decomposition of contributions to the overall variance to be separated and compared to understand the drivers of the spread of the data. In this work, we analyze gauge R & R study data using Bayesian methods and illustrate the use of this wrapped normal model with simulated and real data. We also performed a small simulation study in considering the design of gauge R & R studies with angular measurements.

42 ENGINEERING↗

Radiation Detection Data Competition Report

In FY2018 through FY2020, NA-22, the Defense Nuclear Nonproliferation Research and Development Program, funded a Data Science project to develop and implement statistical methodology to effectively host data competitions with the goal of leveraging the opportunity provided by crowdsourcing. By accessing and engaging expertise from a broader research community, there is an opportunity to attract innovative solutions from a variety of different research disciplines to advance the ability to solve important non-proliferation problems. This report summarizes the key results of this project after hosting two data competitions focused on urban radiation detection. The first competition was focused on attracting participants from the U.S. national laboratories, while the second, hosted by TopCoder, was open to the broader international community and awarded prize money to the top 10 competitors. At the start of the project, there was strong interest from NA-22 to explore and develop the capability to host data competitions as a means of leveraging the broader community to solve important nuclear nonproliferation problems. Having a standard data set on which to compare different approaches based on clearly defined criteria was desirable to be able to evaluate the state of solutions for important problems. Initially, it was not clear that it would even be possible logistically and bureaucratically to host a competition with an international field of competitors and to award the prize money needed to attract solutions from top competitors. Happily, a path to host the competitions was ultimately found that allowed this powerful accelerator of improvements to be leveraged.

61 RADIATION PROTECTION AND DOSIMETRY↗

Practical choices for space-filling designs

Space-filling designs are now commonly used as a flexible model-free strategy for providing good coverage throughout an input space of interest for a variety of computer and physical experiment design scenarios. Some of the preliminary choices about how to frame the problem and which type of design to use can have a substantial impact on the success or failure of the experiment, and yet how to make these critical choices is often under-emphasized in the literature. In this paper, we explore several of the practical choices required by the experimenter and describe a sequence of steps to help create an ideal design that matches the goals and constraints of the experiment. These choices include the specification of the input space, the scaling of the variables, the degree of uniformity of the design points across the input space, and the space-filling characteristics. In addition, some new tools for defining the weights to implement non-uniform space-filling designs are provided. Finally, the methods are demonstrated with several illustrative examples and a real-world chemical engineering experiment for carbon capture.

42 ENGINEERING↗

Incorporating uncertainty for enhanced leaderboard scoring and ranking in data competitions

Data competitions have become a popular and cost-effective approach for crowdsourcing versatile solutions from diverse expertise. Current practice relies on the simple leaderboard scoring based on a given set of competition data for ranking competitors and distributing the prize. However, a disadvantage of this practice in many competitions is that a slight difference in the scores due to the natural variability of the observed data could result in a much larger difference in the prize amounts. In this article, we propose a new strategy to quantify the uncertainty in the rankings and scores from using different data sets that share common characteristics with the provided competition data. By using a bootstrap approach to generate many comparable data sets, the new method has four advantages over current practice. Furthermore, during the competition, it provides a mechanism for competitors to get feedback about the uncertainty in their relative ranking. After the competition, it allows the host to gain a deeper understanding of the algorithm performance and their robustness across representative data sets. It also offers a transparent mechanism for prize distribution to reward the competitors more fairly with superior and robust performance. Finally, it has the additional advantage of being able to explore what results might have looked like if competition goals evolved from their original choices. The implementation of the strategy is illustrated with a real data competition hosted by Topcoder on urban radiation search.

42 ENGINEERING↗

Improved learning from data competitions through strategic design of training and test data sets

Leveraging the depth and breadth of solutions generated through crowdsourcing can be a powerful accelerator to method development for high consequence problems. While data science competitions have become quite popular and prevalent, particularly in supervised learning formats, their implementations by the host are highly variable. Without careful planning, a supervised learning competition is vulnerable to overfitting, where the winning solutions are so closely tuned to the particular set of provided data that they cannot generalize to the general underlying problem of interest to the host. This paper outlines important considerations for strategically designing relevant and informative data sets to maximize the learning outcome from hosting a competition. These include: precisely defining the scope of the problem, encouraging participation by competitors from diverse technical backgrounds, specifying the most interesting solution space in order to encourage improvement and distinguish between competitors, strategically generating data sets that enable testing for interpolation and extrapolation to new scenarios of interest, leveraging design of experiment principles for strategic data design while preventing unintentional artifacts in the competition data set that an algorithm could exploit without addressing the real problem of interest, and carefully designing the leaderboard scoring metric to select top solutions that exactly match the overall competition goals. The methods are illustrated with a recently completed competition in the context of urban radiological search to evaluate algorithms capable of detecting, identifying, and locating radioactive materials. Simulated data were used in the urban search competition. Ideas for using measured real data in competitions are also suggested

97 MATHEMATICS AND COMPUTING↗

QP Statistics Spotlight: Practical versus Statistical Priorities

Randomization during data collection in a designed experiment is an effective strategy for reducing system bias and ensuring independent observations1, but at what cost should it be fully implemented? We recently encountered a scenario that highlights an important aspect of collaboration2 between statisticians, scientists and engineers – the idea of compromise for the sake of achieving wins for all team members. In a multi-million-dollar experiment with a fixed amount of time on the test equipment, each experimental run involved resetting the input values and then waiting for the continuous process to reach a new equilibrium. For example, the temperature of the furnace could be reset, but it takes some time for the test chamber to reach the new temperature and rest at that temperature for a while until all components in the chamber settle at that new setting. Similarly, a critical gas flow component also required a settling period before data could be collected that would be representative of that input setting value. In all, the experiment involved 5 factors, where 4 of the 5 were time-consuming and costly to adjust, with the duration of time needed to reach new equilibrium being dependent on how far the previous setting was from the new one for the combination of these 4 factors. For example, changing from 150° to 200° would reach equilibrium faster than changing from 100° to 200°.

97 MATHEMATICS AND COMPUTING↗

How to Host An Effective Data Competition: Statistical Advice for Competition Design and Analysis

Data competitions rely on real-time leaderboards to rank competitor entries and stimulate algorithm improvement. While such competitions have become quite popular and prevalent, particularly in supervised learning formats, their implementations by the host are highly variable. Without careful planning, a supervised learning competition is vulnerable to overfitting, where the winning solutions are so closely tuned to the particular set of provided data that they cannot generalize to the underlying problem of interest to the host. This paper outlines some important considerations for strategically designing relevant and informative data sets to maximize the learning outcome from hosting a competition based on our experience. It also describes a postcompetition analysis that enables robust and efficient assessment of the strengths and weaknesses of solutions from different competitors, as well as greater understanding of the regions of the input space that are well-solved. The postcompetition analysis, which complements the leaderboard, uses exploratory data analysis and generalized linear models (GLMs). The GLMs not only expand the range of results we can explore, they also provide more detailed analysis of individual subquestions including similarities and differences between algorithms across different types of scenarios, universally easy or hard regions of the input space, and different learning objectives. When coupled with a strategically planned data generation approach, the methods provide richer and more informative summaries to enhance the interpretation of results beyond just the rankings on the leaderboard. The methods are illustrated with a recently completed competition to evaluate algorithms capable of detecting, identifying, and locating radioactive materials in an urban environment.

97 MATHEMATICS AND COMPUTING↗

Discussion of “Is reliability a new science?”

I would like to thank the participants of the panel at the 10th International Conference on Mathematical Methods in Reliability (MMR 2017) for their thoughtful and thorough discussion on this topic. In considering the fundamentals of a discipline, it is often possible to gain additional understanding about its strengths, weaknesses, and potential gaps. I confess that my perspective on the topic most closely matches thoughts expressed by William Meeker about the crux of the matter being tied to the choice of definitions. There are a multitude of candidate definitions, and each of the panelists chose to focus on different aspects from different sources. I opted for the Oxford English Dictionary definition of “The intellectual and practical activity encompassing the systematic study of the structure and behaviour of the physical and natural world through observation and experiment.” Based on this definition, I would conclude that reliability can qualify as a science because (1) it shares elements of intellectual and practical activity through the development of methodologies to characterize observed patterns, (2) it considers both the structure and behavior of the systems that it seeks to characterize, and (3) it involves notions of obtaining data (observation) and prioritizes the collection of high-quality data with known pedigree through experimentation. More broadly, however, I think that it has elements that expand beyond just a science and contain elements of engineering. To illustrate this, I present some of the thinking that has recently been occurring related to the distinction between statistical science and statistical engineering.

97 MATHEMATICS AND COMPUTING↗

Rejoinder

We would like to thank all of the discussants for their thoughtful and insightful comments. We heartily agree with Willis Jensen’s remark that the collection of views from multiple different perspectives possible with a discussion paper has added greatly to the overall contribution. Roger Hoerl compactly and eloquently summarized our thoughts about the role of statisticians on teams: “to objectively lead a team of subject matter experts through the minefield of problem resolution.” We agree with Benjamin Slade that this leadership role can be achieved either through formally leading the team or by being an active, engaged member of the team with an eye on the overall goals of the team and impacting the structure of the decision-making process. Hoerl also highlights what solutions interdisciplinary teams should be seeking – namely those that are “the best business … or scientific solution[s], rather than the best statistical solutions.” If this is how we define the objective of the team’s problem-solving or decision-making efforts, then establishing ourselves as fully-included team members or even working towards leading the team efforts should become easier.

97 MATHEMATICS AND COMPUTING↗

Effective interdisciplinary collaboration between statisticians and other subject matter experts

Progress and innovative solutions to challenging problems often come at the intersection of multiple disciplines. Statisticians frequently are presented with opportunities to participate on or lead interdisciplinary teams, where how well their contributions are received is a function of their effectiveness as collaborators. In this article, we outline six fundamentals for effective collaboration: respect, shared common goals, trust, commitment, intercommunication, and execution. We focus on how these core aspects of a successful collaboration can be encouraged by statisticians. Through an example, we illustrate how problems can arise when some of the key components are missing and what strategies can be used to mitigate problems. Lastly, we describe how early career statisticians can work to improve their collaboration skills to improve their impact on teams with diverse backgrounds.

42 ENGINEERING↗

Straight Line or Not?

Until this spring, I had never heard of the book Factfulness. Then Bill Gates made a big splash in the news by giving each U.S. graduate from college a copy. In reading the book, I found it to be a remarkable combination of hopefulness and commentary about some of our blindspots regarding numeracy, and how we interpret and internalize information in a media-saturated world. If you have not read the book, I recommend it highly—not only for how it can help us think about the world in a more informed and balanced way, but also how it analyzes classic mistakes that we are all prone to. The book describes our instincts to erroneously divide things into groups when distributions are more accurate (the gap instinct), misallocate attention to fear-triggering but low probability events (the fear instinct), plus eight more. But beyond that, Factfulness also presents helpful strategies to tame these debilitating instincts and prevent them from leading us toward false conclusions and perceptions. In this column, I would like to focus on one of these 10 instincts: the straight line instinct, which suggests that we often are tempted to extrapolate patterns with straight lines, when these may not be appropriate. Factfulness gives several examples in which incongruous results are obtained, including one that projects the growth rate of newborns unaltered through adulthood and reaches outlandish heights.

97 MATHEMATICS AND COMPUTING↗

Bayesian Design of Experiments for Logistic Regression to Evaluate Multiple Forensic Algorithms

When evaluating the performance of several forensic classification algorithms, it is desirable to construct a design that considers a variety of performance levels for each of the algorithms. We describe a strategy to use Bayesian design of experiments with multiple prior estimates to capture anticipated performance. Our goal is to characterize results from the different classification algorithms as a function of multiple explanatory variables and use this to choose a design about which units to test. Bayesian design of experiments has been successful for generalized linear models, including logistic regression models. Here, we develop methodology for the case where there are several potentially nonoverlapping priors for anticipated performance under consideration. The weighted priors method performs well for a broad range of true underlying model parameter choices and is more robust when compared to other candidate design choices. Additionally, we show how this can be applied in the multivariate input case and provide some useful summary measures. The shared information plot is used to evaluate design point allocation, and the D-value difference plot allows for the comparison of design performance across multiple potential parameter values in higher dimensions. Here, we illustrate the methods with several examples.

97 MATHEMATICS AND COMPUTING↗