Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Challenge”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,027 records · Page 57

On-the-fly data assessment for high-throughput x-ray diffraction measurements

Investment in brighter sources and larger and faster detectors has accelerated the speed of data acquisition at national user facilities. The accelerated data acquisition offers many opportunities for the discovery of new materials, but it also presents a daunting challenge. The rate of data acquisition far exceeds the current speed of data quality assessment, resulting in less than optimal data and data coverage, which in extreme cases forces recollection of data. Herein, we show how this challenge can be addressed through the development of an approach that makes routine data assessment automatic and instantaneous. By extracting and visualizing customized attributes in real time, data quality and coverage, as well as other scientifically relevant information contained in large data sets, is highlighted. Deployment of such an approach not only improves the quality of data but also helps optimize the usage of expensive characterization resources by prioritizing measurements of the highest scientific impact. We anticipate our approach will become a starting point for a sophisticated decision-tree that optimizes data quality and maximizes scientific content in real time through automation. Finally, with these efforts to integrate more automation in data collection and analysis, we can truly take advantage of the accelerating speed of data acquisition.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

PDF Entity Annotation Tool (PEAT)

While different text mining approaches – including the use of Artificial Intelligence (AI) and other machine based methods - continue to expand at a rapid pace, the tools used by researchers to create the labeled datasets required for training, modeling, and evaluation remain rudimentary. Labeled datasets contain the target attributes the machine is going to learn; for example, training an algorithm to delineate between images of a car or truck would generally require a set of images with a quantitative description of the underlying features of each vehicle type. Development of labeled textual data that can be used to build natural language machine learning models for scientific literature is not currently integrated into existing manual workflows used by domain experts. Published literature is rich with important information, such as different types of embedded text, plots, and tables that can all be used as inputs to train ML/natural language processing (NLP) models, when extracted and prepared in machine readable formats. Currently, both normalized data extraction of use to domain experts and extraction to support development of ML/NLP models are labor intensive and cumbersome manual processes. Automatic extraction of data and information from formats such as PDFs that are optimized for layout and human readability, not machine readability. The PDF (Portable Document Format) Entity Annotation Tool (PEAT) was developed with the goal of allowing users to annotate publications within their current print format, while also allowing those annotations to be captured in a machine-readable format. One of the main issues with traditional annotation tools is that they require transforming the PDF into plain text to facilitate the annotation process. While doing so lessens the technical challenges of annotating data, the user loses all structure and provenance that was inherent in the underlying PDF. Also, textual data extraction from PDFs can be an error prone process. Challenges include identifying sequential blocks of text and a multitude of document formats (multiple columns, font encodings, etc.). As a result of these challenges, using existing tools for development of NLP/ML models directly from PDFs is difficult because the generated outputs are not interoperable. We created a system that allows annotations to be completed on the original PDF document structure, with no plain text extraction. The result is an application that allows for easier and more accurate annotations. In addition, by including a feature that grants the user the ability to easily create a schema, we have developed a system that can be used to annotate text for different domain-centric schemas of relevance to subject matter experts. Different knowledge domains require distinct schemas and annotation tags to support machine learning.

97 MATHEMATICS AND COMPUTING↗

Trends and Challenges in Smart Healthcare Research: A Journey from Data to Wisdom

Smart Healthcare is a relatively new context-aware healthcare paradigm influenced by several fields of knowledge, namely medical informatics, communications and electronics, bioengineering, ethics and so on. Thus, many challenging problems are related to smart healthcare but in many cases they are explored individually in their respective fields and, as a result, they are not always known by the smart healthcare research community working in more specific domains. The aim of this article is to identify some of the most relevant trends and research lines that are going to affect the smart healthcare field in the years to come. To do so, the article considers a systematic approach that classifies the identified research trends and problems according to their appearance within the data life cycle, this is, from the data gathering in the physical layer (lowest level) until their final use in the application layer (highest level). By identifying and classifying those research trends and challenges, we help to pose questions that the smart healthcare community will need to address. Consequently, we set a common ground to explore important problems in the field, which will have significant impact in the years to come.

behavior analysis↗

Using Science Data and Models for Space Weather Forecasting - Challenges and Opportunities

Space research, and, consequently, space weather forecasting are immature disciplines. Scientific knowledge is accumulated frequently, which changes our understanding or how solar eruptions occur, and of how they impact targets near or on the Earth, or targets throughout the heliosphere. Along with continuous progress in understanding, space research and forecasting models are advancing rapidly in capability, often providing substantially increases in space weather value over time scales of less than a year. Furthermore, the majority of space environment information available today is, particularly in the solar and heliospheric domains, derived from research missions. An optimal forecasting environment needs to be flexible enough to benefit from this rapid development, and flexible enough to adapt to evolving data sources, many of which may also stem from non-US entities. This presentation will analyze the experiences obtained by developing and operating both a forecasting service for NASA, and an experimental forecasting system for Geomagnetically Induced Currents.

Hesse, Michael↗

The General Circulation of the Venus Atmosphere: An Assessment

The overall spin or "superrotation" of the Venus atmosphere is a striking phenomenon. In the 15 years since the NASA Pioneer Venus mission, a first-order understanding has been reached of the dynamics of the atmospheric region near and just above the Venus cloud tops. Tidal motions induced by solar heating produce a traveling disturbance whose vertical momentum transports are balanced by mean flow advection. The balance explains the strength of the mean flow above the clouds, and partially explains the strength of the mean flow at the cloud level where the strongest superrotation of the atmosphere occurs. But the fundamental cause of the global superrotation remains a mystery in spite of data from Earth-based observatories, from Pioneer Venus, from several Russian probes, from a Russian/French balloon experiment, and from the NASA Galileo flyby. The key missing knowledge is of momentum transfer processing the deep atmosphere, between the surface and the cloud deck. Neither the forcing nor the drag and dissipation mechanisms are known. The existing data are reviewed here and theoretical suggestions are listed. It is concluded that further measurements, in conjunction with numerical modeling, will be required to resolve this puzzling and challenging question. New data must improve by an order of magnitude on the accuracies achieved by the Pioneer Venus probes. Velocities in the deep atmosphere must be measured to better than 0.1 m/s and relative temperatures to beter than 0.1 K near the surface.

Gierasch, P. J.↗

Improve Data Mining and Knowledge Discovery Through the Use of MatLab

Data mining is widely used to mine business, engineering, and scientific data. Data mining uses pattern based queries, searches, or other analyses of one or more electronic databases/datasets in order to discover or locate a predictive pattern or anomaly indicative of system failure, criminal or terrorist activity, etc. There are various algorithms, techniques and methods used to mine data; including neural networks, genetic algorithms, decision trees, nearest neighbor method, rule induction association analysis, slice and dice, segmentation, and clustering. These algorithms, techniques and methods used to detect patterns in a dataset, have been used in the development of numerous open source and commercially available products and technology for data mining. Data mining is best realized when latent information in a large quantity of data stored is discovered. No one technique solves all data mining problems; challenges are to select algorithms or methods appropriate to strengthen data/text mining and trending within given datasets. In recent years, throughout industry, academia and government agencies, thousands of data systems have been designed and tailored to serve specific engineering and business needs. Many of these systems use databases with relational algebra and structured query language to categorize and retrieve data. In these systems, data analyses are limited and require prior explicit knowledge of metadata and database relations; lacking exploratory data mining and discoveries of latent information. This presentation introduces MatLab(R) (MATrix LABoratory), an engineering and scientific data analyses tool to perform data mining. MatLab was originally intended to perform purely numerical calculations (a glorified calculator). Now, in addition to having hundreds of mathematical functions, it is a programming language with hundreds built in standard functions and numerous available toolboxes. MatLab's ease of data processing, visualization and its enormous availability of built in functionalities and toolboxes make it suitable to perform numerical computations and simulations as well as a data mining tool. Engineers and scientists can take advantage of the readily available functions/toolboxes to gain wider insight in their perspective data mining experiments.

Shaykhian, Gholam Ali↗

Improve Data Mining and Knowledge Discovery through the use of MatLab

Data mining is widely used to mine business, engineering, and scientific data. Data mining uses pattern based queries, searches, or other analyses of one or more electronic databases/datasets in order to discover or locate a predictive pattern or anomaly indicative of system failure, criminal or terrorist activity, etc. There are various algorithms, techniques and methods used to mine data; including neural networks, genetic algorithms, decision trees, nearest neighbor method, rule induction association analysis, slice and dice, segmentation, and clustering. These algorithms, techniques and methods used to detect patterns in a dataset, have been used in the development of numerous open source and commercially available products and technology for data mining. Data mining is best realized when latent information in a large quantity of data stored is discovered. No one technique solves all data mining problems; challenges are to select algorithms or methods appropriate to strengthen data/text mining and trending within given datasets. In recent years, throughout industry, academia and government agencies, thousands of data systems have been designed and tailored to serve specific engineering and business needs. Many of these systems use databases with relational algebra and structured query language to categorize and retrieve data. In these systems, data analyses are limited and require prior explicit knowledge of metadata and database relations; lacking exploratory data mining and discoveries of latent information. This presentation introduces MatLab(TradeMark)(MATrix LABoratory), an engineering and scientific data analyses tool to perform data mining. MatLab was originally intended to perform purely numerical calculations (a glorified calculator). Now, in addition to having hundreds of mathematical functions, it is a programming language with hundreds built in standard functions and numerous available toolboxes. MatLab's ease of data processing, visualization and its enormous availability of built in functionalities and toolboxes make it suitable to perform numerical computations and simulations as well as a data mining tool. Engineers and scientists can take advantage of the readily available functions/toolboxes to gain wider insight in their perspective data mining experiments.

Shaykahian, Gholan Ali↗

Bringing Fenton Hill into the Digital Age: Data Conversion in Support of the Geothermal Technologies Office Code Comparison Study Challenge Problems

The Geothermal Technologies Office Code Comparison Study (GTO-CCS) was established by the U.S. Department of Energy to facilitate collaboration among members of the geothermal modeling community and to evaluate and improve upon the ability of existing codes to simulate thermal, hydrological, mechanical, and chemical processes associated with complex enhanced geothermal systems (EGS). The first stage of the project, which has been completed, involved comparing simulations for seven benchmark problems that were primarily designed using well-prescribed, simplified data sets. In the second stage, the participating teams are tackling two challenge problems based on the EGS research conducted in hot dry rock (HDR) at Fenton Hill, near Los Alamos, New Mexico. The Fenton Hill project, conducted by Los Alamos National Laboratory (LANL) from 1970 to 1995, was the world’s first HDR demonstration project. One of the criteria for selecting this experiment as the basis for the challenge problems was the amount and availability of data for generating model inputs. The Fenton Hill HDR system consisted of two reservoirs – an earlier Phase I reservoir tested from 1974 to 1981 and a deeper Phase II reservoir tested from 1980 to 1995. Detailed accounts of both phases of the HDR project have been presented in a number of books and reports, including a recently published summary of the lessons learned and a final report with a chronological description of the Fenton Hill project, prepared by LANL. Project documents and records have been archived and made public through the National Geothermal Data System (NGDS). Some of the data acquired from Phase II are available in electronic format readable on modern computers. These include the microseismic data from some of the important experiments (e.g. the massive hydraulic fracturing test conducted in 1983) and the injection/production wellhead data from the circulation tests conducted between 1992-1995. However, much of the data collected during the project, while publicly available, currently only exist in the form of tables or graphs within scanned documents. Therefore, in support of the GTO-CCS, the data needed for developing simulation inputs are being compiled and converted to platform independent, open readable formats so that all participating teams will have access to the same electronic data set. In some cases this requires conversion using optical character recognition, digitizing existing images, and generating the appropriate metadata from project documents. The GTO-Velo knowledge management framework, developed by Pacific Northwest National Laboratory (PNNL), was used for the benchmark problem stage of the comparison study and will also be used as the data repository for the challenge problem data sets. It is staggering and impractical to convert all published data for the Fenton Hill site, so the focus is on data that supports simulations for the three topical areas defined by the study for the challenge problems: 1) reservoir creation/stimulation, 2) reactive and passive transport, and 3) thermal recovery. Conversion of these data provide value not only to GTO-CCS participants, but also to members of the geothermal community at large who may be interested in revisiting the Fenton Hill experiment in the future.

Code Comparison Study↗

Build Less Code, Deliver More Science: An Experience Report on Composing Scientific Environments using Component-based and Commodity Software Platforms

Modern scientific software is daunting in its diversity and complexity. From massively parallel simulations running on the world’s largest supercomputers, to visualizations and user support environments that manage ever growing complex data collections, the challenges for software engineers are plentiful. While high performance simulators are necessarily specialized codes to maximize performance on specific supercomputer architectures, we argue the vast majority of supporting infrastructure, data management and analysis tools can leverage commodity open source and component-based technologies. This approach can significantly drive down the effort and costs of building complex, collaborative scientific user environments, as well as increase their reliability and extensibility. In this paper we describe our experiences in creating an initial user environment for scientists involved in modeling the detailed effects of climate change on the environment of selected geographical regions. Our approach composes the user environment using the Velo scientific knowledge management platform and the MeDICi Integration Framework for scientific workflows. These established platforms leverage component-based technologies and extend commodity open source platforms with abstractions and capabilities that make them amenable for broad use in science. Using this approach we were able to deliver an operational user environment capable of running thousands of simulations in a 7 month period, and achieve significant software reuse.

Gorton, Ian↗

Machine Learning and Social Media to Mine and Disseminate Big Scientific Data

One of the challenges in supplying the communities with wider access to scientific databases is the need for knowledge of database languages like Structured Query Language (SQL). Although the SQL language has been published in many forms, not everybody is able to write SQL queries. Another challenge is that it might not be practical to make the public aware of the structure of databases. There is a need for novice users to query relational databases using their natural language. To solve this problem, many natural language interfaces to structured databases have been developed. The goal is to provide a more intuitive method for generating database queries and delivering responses. Through social media, which makes it possible to interact with a wide section of the population, and with the help of natural language processing, researchers at the Atmospheric Radiation Measurement (ARM) Data Center at Oak Ridge National Laboratory (ORNL) have developed a concept to enable easy search and retrieval of data from several environmental data centers for the scientific community through social media.Using a machine learning framework that maps natural language text to thousands of datasets, instruments, variables, and data streams, the prototype system would allow users to request data through Twitter and receive a link (via tweet) to applicable data results on the project's search catalog tailored to their key words. This automated identification of relevant data from various petascale archives at ORNL could increase convenience, access, and use of the project's data by the broader community. In this paper we discuss how some data-intensive projects at ORNL are using innovative ways to help in data discovery.

Devarakonda, Ranjeet↗

WE-H-BRB-03: Learning Health Systems for Radiation Oncology: Needs and Challenges for Future Success

Big Data in Radiation Oncology: (1) Overview of the NIH 2015 Big Data Workshop, (2) Where do we stand in the applications of big data in radiation oncology?, and (3) Learning Health Systems for Radiation Oncology: Needs and Challenges for Future Success The overriding goal of this trio panel of presentations is to improve awareness of the wide ranging opportunities for big data impact on patient quality care and enhancing potential for research and collaboration opportunities with NIH and a host of new big data initiatives. This presentation will also summarize the Big Data workshop that was held at the NIH Campus on August 13–14, 2015 and sponsored by AAPM, ASTRO, and NIH. The workshop included discussion of current Big Data cancer registry initiatives, safety and incident reporting systems, and other strategies that will have the greatest impact on radiation oncology research, quality assurance, safety, and outcomes analysis. Learning Objectives: To discuss current and future sources of big data for use in radiation oncology research To optimize our current data collection by adopting new strategies from outside radiation oncology To determine what new knowledge big data can provide for clinical decision support for personalized medicine L. Xing, NIH/NCI Google Inc.

60 APPLIED LIFE SCIENCES↗

Data Generation for Machine Learning Interatomic Potentials and Beyond

The field of data-driven chemistry is undergoing an evolution, driven by innovations in machine learning models for predicting molecular properties and behavior. Recent strides in ML-based interatomic potentials have paved the way for accurate modeling of diverse chemical and structural properties at the atomic level. The key determinant defining MLIP reliability remains the quality of the training data. A paramount challenge lies in constructing training sets that capture specific domains in the vast chemical and structural space. This Review navigates the intricate landscape of essential components and integrity of training data that ensure the extensibility and transferability of the resulting models. We delve into the details of active learning, discussing its various facets and implementations. We outline different types of uncertainty quantification applied to atomistic data acquisition and the correlations between estimated uncertainty and true error. The role of atomistic data samplers in generating diverse and informative structures is highlighted. Furthermore, we discuss data acquisition via modified and surrogate potential energy surfaces as an innovative approach to diversify training data. The Review also provides a list of publicly available data sets that cover essential domains of chemical space.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON

For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organisations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analysed by researchers, or archived for safe long-term storage. To fulfil these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS at LHC to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown, many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities that arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments Belle II and CMS, the neutrino experiment DUNE, the scattering radar experiment EISCAT3D, the gravitational wave observatories LIGO and VIRGO, the SKA radio telescope, and the dark matter search experiment XENON.

Lassnig, Mario↗

Exploring Future Storage Options for ATLAS at the BNL/SDCC facility

The ATLAS experiment is expected to deliver an unprecedented amount of scientific data in the High Luminosity(HL-LHC) era. As the demand for disk storage capacity in ATLAS continues to rise steadily, the BNL Scientific Data and Computing Center (SDCC) faces challenges in terms of cost implications for maintaining multiple disk copies and adapting to the coming ATLAS storage requirements. To address these challenges, the SDCC Storage team has undertaken a thorough analysis of the ATLAS experiment's requirements, matching them to suitable storage options and strategies, and has explored alternatives to enhance or replace the current storage solution. This paper aims to present the main challenges encountered while supporting big data experiments such as ATLAS. We describe the experiment's specific requirements and priorities, particularly focusing on the critical storage system characteristics of the high-luminosity run and how the key storage components provided by the Storage team work together: the dCache disk storage system; its archival back-end, HPSS; and its OS-level backend Storage. Specifically, we investigate a novel approach to integrate Lustre and XRootD. In this setup, Lustre serves as backend storage and XRootD acts as an access layer frontend, supporting various grid access protocols. Additionally, we also describe the validation and commissioning tests, including the performance comparison between dCache and XRootd. Furthermore, we provide a performance and cost analysis comparing OpenZFS and LINUX MD RAID, evaluate different storage software stacks, and showcase stress tests conducted to validate Third Party Copy (TPC) functionality.

99 GENERAL AND MISCELLANEOUS↗

Resistive switching in ultra-thin La{sub 0.7}Sr{sub 0.3}MnO{sub 3}/SrRuO{sub 3} superlattices

Superlattices may play an important role in next generation electronic and spintronic devices if the key-challenge of the reading and writing data can be solved. This challenge emerges from the coupling of low dimensional individual layers with macroscopic world. Here, we report the study of the resistive switching characteristics of a hybrid structure made out of a superlattice with ultrathin layers of two ferromagnetic metallic oxides, La{sub 0.7}Sr{sub 0.3}MnO{sub 3} (LSMO) and SrRuO{sub 3} (SRO). Bipolar resistive switching memory effects are measured on these LSMO/SRO superlattices, and the observed switching is explainable by ohmic and space charge-limited conduction laws. It is evident from the endurance characteristics that the on/off memory window of the cell is greater than 14, which indicates that this cell can reliably distinguish the stored information between high and low resistance states. The findings may pave a way to the construction of devices based on nonvolatile resistive memory effects.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

Energy Efficiency Finance Programs: Use Case Analysis to Define Data Needs and Guidelines

There are over 200 energy efficiency loan programs—across 49 U.S. states—administered by utilities, state/local government agencies, or private lenders.1 This distributed model has led to significant variation in program design and implementation practices including how data is collected and used. The challenge of consolidating and aggregating data across independently administered programs has been illustrated by a recent pilot of an open source database for energy efficiency financing program data. This project was led by the Environmental Defense Fund (EDF), the Investor Confidence Project, the Clean Energy Finance Center (CEFC), and the University of Chicago. This partnership discussed data collection practices with a number of existing energy efficiency loan programs and identified four programs that were suitable and willing to participate in the pilot database (Diamond 2014).2 The partnership collected information related to ~12,000 loans with an aggregate value of ~$100M across the four programs. Of the 95 data fields collected across the four programs, 30 fields were common between two or more programs and only seven data fields were common across all programs. The results of that pilot study illustrate the inconsistencies in current data definition and collection practices among energy efficiency finance programs and may contribute to certain barriers.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Intelligent Systems: Terrestrial Observation and Prediction Using Remote Sensing Data

NASA has made science and technology investments to better utilize its large space-borne remote sensing data holdings of the Earth. With the launch of Terra, NASA created a data-rich environment where the challenge is to fully utilize the data collected from EOS however, despite unprecedented amounts of observed data, there is a need for increasing the frequency, resolution, and diversity of observations. Current terrestrial models that use remote sensing data were constructed in a relatively data and compute limited era and do not take full advantage of on-line learning methods and assimilation techniques that can exploit these data. NASA has invested in visualization, data mining and knowledge discovery methods which have facilitated data exploitation, but these methods are insufficient for improving Earth science models that have extensive background knowledge nor do these methods refine understanding of complex processes. Investing in interdisciplinary teams that include computational scientists can lead to new models and systems for online operation and analysis of data that can autonomously improve in prediction skill over time.

Coughlan, Joseph C.↗

Transforming Science Data for GIS: How to Find and Use NASA Earth Observation Data Without Being a Rocket Scientist

NASAs Earth Observing System Data Information System (EOSDIS) manages Earth Observation satellites and the Distributed Active Archive Centers (DAACs), where the data is stored and processed. The challenge is that Earth Observation data is complicated. There is plenty of data available, however, the science teams have had a top-down approach: define what it is you are trying to study -select a set of satellite(s) and sensor(s), and drill down for the data.Our alternative is to take a bottom-up approach using eight environmental fields of interest as defined by the Group on Earth Observations (GEO) called Societal Benefit Areas (SBAs): Disaster Resilience (DR) Public Health Surveillance (PHS) Energy and Mineral Resource Management (EMRM) Water Resources Management (WRM) Infrastructure and Transport Management (ITM) Sustainable Urban Development (SUD) Food Security and Sustainable Agriculture (FSSA) Biodiversity and Ecosystems Sustainability (BES).

DAAC↗