Search NASA⌕ Search

SEARCH · Search NASA

Results for “data ingest”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

battery-data-toolkit

This software package contains utilities for ingesting battery cycling data from various equipment and converting it to a common file format including metadata. Further development of the package will include utilities for further processing the data for a variety of machine learning tasks.

PAULSON, NOAHHARRIS↗

Data and scripts from: “Denoising autoencoder for reconstructing sensor observation data and predicting evapotranspiration: noisy and missing values repair and uncertainty quantification”

This data package includes data and scripts from the manuscript “Denoising autoencoder for reconstructing sensor observation data and predicting evapotranspiration: noisy and missing values repair and uncertainty quantification”.The study addressed common challenges faced in environmental sensing and modeling, including uncertain input data, missing sensor observations, and high-dimensional datasets with interrelated but redundant variables. Point-scaled meteorological and soil sensor observations were perturbed with noises and missing values, and denoising autoencoder (DAE) neural networks were developed to reconstruct the perturbed data and further predict evapotranspiration. This study concluded that (1) the reconstruction quality of each variable depends on its cross-correlation and alignment to the underlying data structure, (2) uncertainties from the models were overall stronger than those from the data corruption, and (3) there was a tradeoff between reducing bias and reducing variance when evaluating the uncertainty of the machine learning models.This package includes:(1) Four ipython scripts (.ipynb): “DAE_train.ipynb” trains and evaluates DAE neural networks, “DAE_predict.ipynb” makes predictions from the trained DAE models, “ET_train.ipynb” trains and evaluates ET prediction neural networks, and “ET_predict.ipynb” makes predictions from trained ET models.(2) One python file (.py): “methods.py” includes all user-defined functions and python codes used in the ipython scripts.(3) A “sub_models” folder that includes five trained DAE neural networks (in pytorch format, .pt), which could be used to ingest input data before being fed to the downstream ET models in ‘ET_train.ipynb” or ‘ET_predict.ipynb’.(4) Two data files (.csv). Daily meteorological, vegetation, and soil data is in “df_data.csv”, where “df_meta.csv” contains the location and time information of “df_data.csv”. Each row (index) in “df_meta.csv” corresponds to each row in “df_data.csv”. These data files are formatted to follow the data structure requirements and be directly used in the ipython scripts, and they have been shuffled chronologically to train machine learning models. The meteorological and soil data was collected using point sensors between 2019-2023 at(4.a) Three shrub-dominated field sites in East River, Colorado (named “ph1”, “ph2” and “sg5” in “df_meta.csv”, where “ph1” and “ph2” were located at PumpHouse Hillslopes, and “sg5” was at Snodgrass Mountain meadow) and(4.b) One outdoor, mesoscale, and herbaceous-dominated experiment in Berkeley, California (named “tb” in “df_meta.csv”, short for Smartsoils Testbed at Lawrence Berkeley National Lab).- See "df_data_dd.csv" and "df_meta_dd.csv" for variable descriptions and the Methods section for additional data processing steps. See "flmd.csv" and "README.txt" for brief file descriptions.- All ipython scripts and python files are written in and require PYTHON language software.

54 ENVIRONMENTAL SCIENCES↗

Collaborative Platforms Aid Emergency Decision Making

Terra. Aqua. Cloudsat. Landsat. NASA runs and partners in many missions dedicated to monitoring the Earth, and the tools used in these missions continuously return data on everything from shifts in temperature to cloud formation to pollution levels over highways. The data are of great scientific value, but they also provide information that can play a critical role in decision making during times of crisis. Real-time developments in weather, wind, ocean currents, and numerous other conditions can have a significant impact on the way disasters, both natural and human-caused, unfold. "NASA has long recognized the need to make its data from real-time sources compatible and accessible for the purposes of decision making," says Michael Goodman, who was Disasters Program manager at NASA Headquarters from 2009-2012. "There are practical applications of NASA Earth science data, and we d like to accelerate the use of those applications." One of the main obstacles standing in the way of eminently practical data is the fact that the data from different missions are collected, formatted, and stored in different ways. Combining data sets in a way that makes them useful for decision makers has proven to be a difficult task. And while the need for a collaborative platform is widely recognized, very few have successfully made it work. Dave Jones, founder and CEO of StormCenter Communications Inc., which consults with decision makers to prepare for emergencies, says that "when I talk to public authorities, they say, If I had a nickel for every time someone told me they had a common operating platform, I d be rich. But one thing we ve seen over the years is that no one has been able to give end users the ability to ingest NASA data sets and merge them with their own."

Source record↗

Simulation INsight and Analysis

Sina is a tool set for modern scientific data management that provides flexible, light-weight support of non-bulk data capture for retention in and queries against SQL and noSQL data stores. HPC simulations traditionally maintain their data in files. Extracting data of interest for subsequent analysis then requires the time-consuming process of traversing directories and scraping data from files in a variety of formats. Sina facilitates capturing relevant data during execution or post-processing of simulation runs for retention in and queries from a modern data store. The tools are sufficiently general to allow for the inclusion of new fields as scientists learn more about their data. Libraries, currently in C++ and Python, and a command line interface (CLI) are provided. Sina's flexibility starts with a general schema, in JSON, for the collection of non-bulk simulation data. JSON provides a flexible, human-readable representation of the data that of interest. Sina currently has a C++ library for simulations to write data to and read from a schema-compliant file for subsequent ingestion into one of the supported data stores. However, applications are free to write their data directly into a schema-compliant file. Python packages provide data ingestion, management, query, and export capabilities. A command line interface (CLI) provides simplified access to these features. A common application programming interface (API) is used to maintain and query data in any of the supported data stores, which are currently limited to SQL and Apache Cassandra (a column store). Tutorials, demonstrations, and examples illustrate aspects of the process using scripts and Jupyter notebooks.

Pauli, EstebanT↗

Industrial Scale Demonstration of Smart Manufacturing, Achieving Transformational Energy Productivity Gains

While many U.S. manufacturing operations utilize optimization for individual unit processes, smart manufacturing (SM) systems that integrate manufacturing intelligence in real time across an entire production operation are rare in large companies and virtually nonexistent in smaller organizations. One example of an area where SM systems can be applied is in the management of waste heat. A smart system that not only seeks to recover waste heat, but also to use energy more efficiently, is a more cost-effective approach because energy use, waste energy loss and product output are optimized together. The overall objective of this project was to develop an industry-accepted SM Platform that enables smart systems by accelerating and lowering the cost of development and implementation scales for use in diverse sets of manufacturing industry sectors, manufacturing operations and company sizes. Specifically, the project (1) designed and demonstrated the application of a prototype SM Platform for two diverse commercial test beds, (2) demonstrated at one and estimated for the other test bed the potential reduction in waste heat generation, and (3) worked with leading automation vendors to catalyze low-cost commercialization of the technology developed. The Smart Manufacturing Application and Data Platform (SM Platform) developed during this project is an innovative approach that marries IT virtualization technologies with operational data and modeling system requirements to construct a ready-to-go platform for enabling sensor – data – modeling – actuation systems, driven by real-time plant data and performance metrics. This SM Platform allows manufacturing organizations, regardless of their industry or size, to assemble new operational systems at a much lower cost, optimizing process knowledge and improving energy productivity by integrating with existing process control and automation systems. Also developed during this project is a methodology for developing energy productivity metrics that use real-time process information and provide comparisons for potential, practical, and actual performance, as well as examples and the methodology for provisioning of an Energy Dashboard that displays alternatives to optimize energy use within specific business contexts aimed at helping improve energy productivity and reduce emissions in U.S. manufacturing. The SM Platform designed utilizes current manufacturing, operational and IT industry standards. The platform can be used to expedite sensor – data – model – actuation system development and deployment in a cloud-based environment using a novel end-to-end workflow as a service capability that is already integrated with data connection, ingestion, and contextualization services. In addition, the methodology for development of energy productivity metrics and reusable application templates called Toolkits have been defined so that they can be tailored to a specific business situation. The project’s two test beds, one in a hydrogen production plant and the other in a forging, heat treatment, and machining operation, utilize sensor-driven modeling, measurement, and simulation systems. In these two test beds, the implementation of these as a comprehensive system has made it possible for energy productivity to be managed in real-time throughout the plant and enterprise using an energy dashboard. The development pathway to direct machine actuation was articulated. During the first year, the project team developed the most highly instrumented steam-methane reformer (SMR) in the U.S. using infrared cameras and thermocouples. Real-time data are streamed to high fidelity and reduced-order models for analysis. A high fidelity CFD model, parallelized in the SM Platform cloud environment, was demonstrated for operational use. This is the first real-time operational productivity system of a steam methane reformer using both high fidelity and data-driven models together in a real time situation to balance natural gas flow to the reformer tubes. An infrared camera system was developed and shown to outperform thermocouples in coverage, accuracy, reliability, and cost effectiveness for use in a high temperature, harsh environment. A forging, heat-treating, machining line operation test bed includes new instrumentation for data capture analysis, modeling, and simulation and will be integrated into SM Platform workflows so that business performance can be managed in real-time. Composable metrics were developed using the SM Platform to drive improvements in energy productivity, environmental performance, safety, asset management, costs, and overall operations. A major focus of the SM systems developed was optimization of energy productivity at the enterprise level, which resulted in reductions in waste heat generation for the two project test beds. Reductions in the waste heat generated will result in greater energy savings than will waste heat recovery due to inefficiencies in the heat transfer process. For the SMR test bed an efficiency improvement of 1 – 2% with a corresponding waste heat reduction of 5% were demonstrated. It should be noted that the test bed SMR unit had an efficiency of more than 90% as compared to 70 to 80% efficiencies of most SMR units. So implementation of similar SM systems at typical SMR units could yield more than 15 to 25% waste heat reductions. Similarly the forging, heat-treating, machining line operation test bed is a DOE AMO Platinum Level Superior Energy Performance plant with a much higher than typical overall operating efficiency. For this test bed heat integration using recuperators is estimated to result in fuel savings of 15.9% due to waste heat recovery compared to the base case without recuperators with negligible changes in part exit conditions. This results in an 18.9% energy efficiency improvement. The project team estimates that broad multi-sector adoption of the developed technology could reduce waste heat generation by an amount equivalent to 1.3% of total U.S. energy use. This waste heat reduction is associated with 69 million tons reduction in annual carbon dioxide emissions. Other potential benefits include reducing process times, solid and liquid wastes, environmental impacts, and water use. It is estimated that this investment has an anticipated payback period of less than one year in energy-intensive applications. The SM Platform could also reduce the costs of deploying SM systems by 50% in manufacturing operations. Commercialization of the SM Platform will require significant cross-industry collaboration and technology provider market changes. Steps taken toward commercialization included 1) analyzing savings from deployment of the SM Platform and 2) holding SM Platform workshops, webinars, and conferences with targeted individual industry segments, with a focus on small and medium-sized organizations. An upgraded community website has been established as a portal for accessing and distributing information. Most importantly, the SM platform enabling technology developed during this project and its commercialization will continue, evolve and expand under the Clean Energy Smart Manufacturing Innovation Institute (CESMII).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Influence of Local Water Vapor Analysis Uncertainty on Ensemble Forecasts of Tropical Cyclogenesis Using Hurricane Irma (2017) as a Testbed

Abstract Tropical cyclone formation is known to require abundant water vapor in the lower to middle troposphere within the incipient disturbance. In this study, we assess the impacts of local water vapor analysis uncertainty on the predictability of the formation of Hurricane Irma (2017). To this end, we reduce the magnitude of the incipient disturbance’s water vapor perturbations obtained from an ensemble-based data assimilation system that constrained moisture by assimilating all-sky infrared and microwave radiances. Five-day ensemble forecasts are initialized two days before genesis using each set of modified analysis perturbations. Growth of convective differences and intensity uncertainty are evaluated for each ensemble forecast. We observe that when initializing an ensemble forecast with only moisture uncertainty within the incipient disturbance, the resulting intensity uncertainty at every lead time exceeds half that of an ensemble containing initial perturbations to all variables throughout the domain. Although ensembles with different initial moisture uncertainty amplitudes reveal a similar pathway to genesis, uncertainty in genesis timing varies substantially across ensembles since moister members exhibit earlier spinup of the low-level vortex. These differences in genesis timing are traced back to the first 6–12 h of integration, when differences in the position and intensity of mesoscale convective systems across ensemble members develop more quickly with greater initial moisture uncertainty. In addition, the rapid growth of intensity uncertainty may be greatly modulated by the diurnal cycle. Ultimately, this study underscores the importance of targeting the incipient disturbance with high spatiotemporal water vapor observations for ingestion into data assimilation systems. Significance Statement Hurricanes form from clusters of thunderstorms that organize into a coherent system. One of the key ingredients for the formation process is an abundance of moisture. In this study, we test the sensitivity of hurricane formation to the initial moisture content in the vicinity of the cluster of thunderstorms that would become Hurricane Irma (2017). To do so, we initialize sets of forecasts each having a different variability of initial moisture content within the embryonic disturbance. Our results show that the predictability of hurricane formation is highly dependent on the uncertainty of the moisture content within the initial disturbance. Consequently, more high-quality observations of the moisture within the precursor disturbances to hurricanes are expected to improve forecasts of their formation.

Hartman, Christopher M.↗

Informing Wildfire Needs: The Expanded User Interface of NASA's Fire Information for Resource Management System (Firms)

As the global community continues to experience, and respond to, living in a changing environment, access to tools, technologies, and timely data utilized by an increasingly diverse set of stakeholders is increasing. This year, 2023, has thus far seen an unprecedented number of extreme events, increasingly driven by changes in the climate and a strong 2023 ENSO pattern. In Canada, a record number of wildfires, and associated weather events, evacuations, and infrastructure and habitat destruction has taken place, and is ongoing. Large swaths of Greece have experienced similar wildfire destruction. Most recently, Maui has experienced destructive wildfires, and early 2023 saw massive wildfires in Chile. NASA's Fire Information for Resource Management System, or FIRMS, has a fifteen-year history of providing timely and comprehensive data and information on wildfires to stakeholders. FIRMS was initially developed in 2007 by the University of Maryland, with funds from NASA's Applied Sciences Program and the United Nations Food and Agriculture Organization (UN FAO), to provide near real-time active fire Locations to natural resource managers that faced challenges obtaining timely satellite-derived fire information. FIRMS has consistently evolved to address the needs of stakeholders Living in a changing environment; in 2012 it transitioned to NASA LANCE and in 2021 through a partnership between NASA and the US Forest Service, an updated version of FIRMS was released for the US and Canada. As NASA and other federal agencies continue to accelerate Open Science through integrated efforts such as the Year of Open Science, the provision of readily discoverable, findable, accessible, interoperable, reusable data represents a major focus to facilitate equitable outcomes. FIRMS supports this acceleration in Open Science by continuing to provision data and information for its traditional user base, while addressing the novel user needs of an increasingly diverse set of stakeholders seeking robust, reliable, transparently generated data and information. Increasingly, FIRMS is utilized by citizen scientists and individuals directly affected by wildfires - through evacuations, risks to structures/homes, poor air quality. etc. FIRMS has also been Leveraged to detect and assess the impacts resulting from ongoing conflicts. This further highlights the multi-faceted impacts of wildfires and other events. In the Fall of 2023, FIRMS will release an expanded User Interface (UI). This interface captures and reflects the needs of, and input from, a multitude of users. These users range from federal agency representatives to non-government organizations to the private sector to citizen science entities. To respond to this expansive and diverse user need base, the updated FIRMS UI will capture a range of features to support those beginning to explore the range of data and tools available to inform wildfire awareness and knowledge. These users are supported through a Basic Mode interface, furnishing access to a light set of functionalities that provision straight-forward, readily usable information and data, and ingestible knowledge. The Advanced Mode interface supports those stakeholder groups already proficient in navigating FIRMS. These stakeholders, representing fire managers and others, perform active fire management and tactical wildfire response activities. For these stakeholders, additional datasets have been included which require in-depth knowledge of both the utility as well as the caveats of such datasets. Additional functionalities have also been embedded to aid specific user queries. The expanded UI will introduce a new Experimental Mode. The focus of this UI will be to support the provision of emerging and innovative datasets that are in development for review and comment by the user community Examples include post-fire products generated by NASA's Earth Information System (EIS) Fire. This presentation will provide an overview of the expanded FIRMS UI. We will discuss how this UI is designed to be scalable and support the unique needs of an expanding and diverse user base. We will highlight key features, elements, and datasets, and describe how user needs have informed and guided the design of the UI. We will also share recent use cases to convey, and increase awareness, among conference participants. As the global community faces more extreme wildfires, due to climate variability and change, there is an increased need for reliable data to inform, manage, and mitigate the impacts of these events. Through this work, NASA FIRMS is striving to level the playing field, by making information accessible to all; from policy makers to the private sector to historically marginalized communities. In doing so, NASA is promoting the all-hands-on-deck response needed to minimize the impacts of wildfires and harness the strengths of open science to address the greatest environmental challenge faced.

Jenny Hewson↗

A network of soil moisture, soil temperature, air temperature, net radiation, ground heat flux and ground water for Chicago, Illinois

This dataset contains environmental monitoring data collected using solar-powered Multi-Function Research (MFR) Long Range Wide Area (LoRaWAN)-enabled nodes at 11 sites in Chicago, Illinois, as part of the DOE Urban Integrated Field Lab CROCUS project. The MFR node system consists of an Input/Output Digital Input Module (IB8) interface box (ICT International) providing wired connections for environmental sensors and an MFR-Node-L data logger that manages power, data processing, and LoRaWAN communication. The wireless data are ingested via Sage network (https://sagecontinuum.org/) nodes that contain LoRaWAN antennae. Measurements were collected from 11 MFR nodes deployed across Chicago State University (CSU), Northeastern Illinois University (NEIU), Northwestern University (NU), University of Illinois Chicago (UIC), West Woodlawn "Blacks in Green" (BIG), and Indian Boundary Prairies (IBP). Each MFR node supports a consistent suite of sensors measuring atmospheric, soil, and hydrological variables. Atmospheric measurements include 2m air temperature (°C), 2m vapor pressure deficit (kPa), and 2m shortwave/longwave radiation (incoming and outgoing, W/m²) measured using ATH-VPD and Apogee SN500 sensors. Soil measurements include volumetric water content (VWC, %) and temperature (°C) at four depths (15, 30, 45, and 60 cm below surface) using Meter Teros54 sensors, and heat flux (W/m²) at 10 cm depth using Huske HFP01-05 sensors. At selected locations, Meter Hydros21 sensors measure groundwater depth (mm), specific conductivity (dS/m), and temperature (°C). The dataset includes timestamps, site identifiers with location names, device IDs, Global Positioning System (GPS) coordinates, variable names with units, measurement depths, values, sensor names, and Sage node identifiers. All timestamps are in local Chicago time (CDT/CST). Quality control flags are provided using a 6-bit binary system indicating physical range violations, step spikes, 24-hour flat-line conditions, 6-hour jitter, 7-day ultra-low variance, and persistent high offset. Data is provided in CSV and CF-compliant NetCDF formats. This dataset is part of a larger collection of CROCUS environmental monitoring data, including linked datasets from Air Quality Transmitter (AQT) sensors, Weather Transmitter (WXT) sensors, and Sap Flow Meter (SFM1x) sensors.

Chicago↗

Deep learning to estimate permeability using geophysical data

Time-lapse electrical resistivity tomography (ERT) is a popular geophysical method to estimate three-dimensional (3D) permeability fields from electrical potential difference measurements. Traditional inversion and data assimilation methods are used to ingest this ERT data into hydrogeophysical models to estimate permeability. Due to ill-posedness and the curse of dimensionality, existing inversion strategies provide poor estimates and low resolution of the 3D permeability field. Recent advances in deep learning provide us with powerful algorithms to overcome this challenge. This paper presents a deep learning (DL) framework to estimate the 3D subsurface permeability from time-lapse ERT data. To test the feasibility of the proposed framework, we train DL-enabled inverse models on simulation data. Each measurement in both synthetic and field data is standardized by removing the mean and scaling the time-series to unit variance. This pre-processing step is necessary to bring simulation data closer to field observations. Subsurface process models based on hydrogeophysics are used to generate this synthetic data. Training performed on limited simulation data resulted in the DL model over-fitting. An advanced data augmentation based on mixup is implemented to generate additional training samples to overcome this issue. This mixup technique creates weakly labeled (low-fidelity) samples from strongly labeled (high-fidelity) data. The weakly labeled training data is then used to develop DL-enabled inverse models and reduce over-fitting. As both time-lapse ERT (1133048 features/realization) and 3D permeability (585453 features/realization) data samples are from a high-dimensional space, principal component analysis (PCA) is employed to reduce dimensionality. Encoded ERT and encoded permeability are generated using the trained PCA estimators. A deep neural network is then trained to map the encoded ERT to encoded permeability. This mixup training and unsupervised learning allowed us to build a fast and reasonably accurate DL-based inverse model under limited simulation data. Results show that proposed weak supervised learning can capture salient spatial features in the 3D permeability field. Quantitatively, the average mean squared error (in terms of the natural log) on the strongly labeled training, validation, and test datasets is less than 0.5. The R 2 -score (global metric) is greater than 0.75, and the percent error in each cell (local metric) is less than 10%. Finally, an added benefit in terms of computational cost is that the proposed DL-based inverse model is at least O(10 4 ) times faster than running a forward model once it is trained. Data generation, DL model training, and hyperparameter tuning to identify optimal neural network architectures utilized high-performance computing resources while the DL inference is performed on a standard laptop. Approximately, O(10 5 ) processor hours are used for generating data and DL tuning and training. We acknowledge that the data generation and DL model development are expensive. But once a DL model is trained, it can be re-used for inversion rapidly for the given system, with set physics and domain. Note that traditional inversion may require multiple forward model simulations (e.g., in the order of 10 to 1000), which are very expensive. This computational savings ≈ O(10 5 ) – O(10 7 )) makes the proposed DL-based inverse model attractive for subsurface imaging and real-time ERT monitoring applications due to fast and yet reasonably accurate estimations of permeability field.

58 GEOSCIENCES↗

Making Common Fund data more findable: catalyzing a data ecosystem

The Common Fund Data Ecosystem (CFDE) has created a flexible system of data federation that enables researchers to discover datasets from across the US National Institutes of Health Common Fund without requiring that data owners move, reformat, or rehost those data. This system is centered on a catalog that integrates detailed descriptions of biomedical datasets from individual Common Fund Programs’ Data Coordination Centers (DCCs) into a uniform metadata model that can then be indexed and searched from a centralized portal. This Crosscut Metadata Model (C2M2) supports the wide variety of data types and metadata terms used by individual DCCs and can readily describe nearly all forms of biomedical research data. We detail its use to ingest and index data from 11 DCCs.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Traffic Control via Connected and Automated Vehicles (CAVs): An Open-Road Field Experiment with 100 CAVs

The CIRCLES project aims to reduce instabilities in traffic flow, which are naturally occurring phenomena due to human driving behavior. Also called “phantom jams” or “stop-and-go waves,” these instabilities are a significant source of wasted energy. Toward this goal, the CIRCLES project designed a control system, referred to as the MegaController by the CIRCLES team, that could be deployed in real traffic. Our field experiment, the MegaVanderTest (MVT), leveraged a heterogeneous fleet of 100 longitudinally controlled vehicles as Lagrangian traffic actuators, each of which ran a controller with the architecture described in this article. The MegaController is a hierarchical control architecture that consists of two main layers. The upper layer is called the Speed Planner and is a centralized optimal control algorithm. It assigns speed targets to the vehicles, conveyed through the LTE cellular network. The lower layer is a control layer, running on each vehicle. It performs local actuation by overriding the stock adaptive cruise controller, using the stock onboard sensors. The Speed Planner ingests live data feeds provided by third parties as well as data from our own control vehicles and uses both to perform the speed assignment. The architecture of the Speed Planner allows for the modular use of standard control techniques, such as optimal control, model predictive control (MPC), kernel methods, and others. The architecture of the local controller allows for the flexible implementation of local controllers. Corresponding techniques include deep reinforcement learning (RL), MPC, and explicit controllers. Depending on the vehicle architecture, all onboard sensing data can be accessed by the local controllers or only some. Likewise, control inputs vary across different automakers, with inputs ranging from torque or acceleration requests for some cars to electronic selection of adaptive cruise control (ACC) setpoints in others. The proposed architecture technically allows for the combination of all possible settings proposed previously, that is {Speed Planner algorithms} × {local Vehicle Controller algorithms} × {full or partial sensing} × {torque or speed control}. As a result, most configurations were tested throughout the ramp up to the MegaVandertest (MVT).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Splash-ML v0.1

The Splash database and portal, also under development (https://lbl.wellspringsoftware.net/kms/invention/detail/3980/), ingests experimental data and references to datasets from a variety of sources, primarily lightsource beamlines. After deployment, it will contain a great deal of metadata and references to data from which many insights can be extracted.

McReynolds, Dylan↗

The catalog-to-cosmology framework for weak lensing and galaxy clustering for LSST

We present TXPipe, a modular, automated and reproducible pipeline for ingesting catalog data and performing all the calculations required to obtain quality-assured two-point measurements of lensing and clustering, and their covariances, with the metadata necessary for parameter estimation. The pipeline is developed within the Rubin Observatory Legacy Survey of Space and Time (LSST) Dark Energy Science Collaboration (DESC), and designed for cosmology analyses using LSST data. In this paper, we present the pipeline for the so-called 3x2pt analysis -- a combination of three two-point functions that measure the auto- and cross-correlation between galaxy density and shapes. We perform the analysis both in real and harmonic space using TXPipe and other LSST-DESC tools. We validate the pipeline using Gaussian simulations and show that it accurately measures data vectors and recovers the input cosmology to the accuracy level required for the first year of LSST data under this simplified scenario. We also apply the pipeline to a realistic mock galaxy sample extracted from the CosmoDC2 simulation suite (Korytov et al. 2019). TXPipe establishes a baseline framework that can be built upon as the LSST survey proceeds. Furthermore, the pipeline is designed to be easily extended to science probes beyond the 3x2pt analysis.

79 ASTRONOMY AND ASTROPHYSICS↗

Final LDRD Report: Using Linkography of Cyber Attack Patterns to Inform Honeytoken Placement

The war to establish cyber supremacy continues, and the literature is crowded with strictly technical cyber security measures. We present the results of a three year LDRD project using Linkography, a methodology new to the field of cyber security, we establish the foundation necessary to track and profile the microbehavior of humans attacking cyber systems. We also propose ways to leverage this understanding to influence and deceive these attackers. We studied the science of linkography, applied it to the cyber security domain, implemented a software package to manage linkographs, generated the preprocessing blocks necessary to ingest raw data, produced machine learning models, created ontology refinement algorithms and prototyped a web application for researchers and practitioners to apply linkography. Machine learning produced some of our key results: We trained and validated multinomial classifiers with a real world data set and predicted the attacker's next category of action with 86 to 98% accuracy; dimension reduction techniques indicated that the linkography-based features were among the most powerful. We also discovered ontology refinement algorithms that advanced the state of the art in linkography in general and cyber security in particular. We conclude that linkography is a viable tool for cyber security; we look forward to expanding our work to other data sources and using our prediction results to enable adversary deception techniques.

97 MATHEMATICS AND COMPUTING↗

Titan on the Red (advanced AI / ML system) [Slides]

Born out of a need to make LANL Weapons Program archival material available to scientists and engineers. Material is the end result of decades of consolidation of mini-libraries and mini archives at LANL. Latest consolidation brought together LANL’s digital archives and physical archives. This houses over 75 years of nuclear weapons research, designs, procedures, videos, photos, and other reports. For the Titan on the Red machine learning project, the system must be able to automatically extract metadata from digitized documents, perform natural-language search, enforce security classification and NTK protocols, be expandable to ingest various data stores (Online Vault, PDMLink, shared drives, SharePoint, etc.), and utilize commercial, public domain, and LANL ontologies.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

Evaluation of Data Lake Design for the Accelerator Control System

In this modern world, the user expects faster processing and real-time response to operate accelerator control devices. The existing framework with its infrastructure does not have the ability to satisfy these future needs. Therefore, modernization is required to reach industry standards and develop a modular framework that can provide flexibility and dynamic scalability. The data lake architecture, comprising three layers, ingestion, processing, and data consumption, provides flexibility and scalability to meet current and future demands for the accelerator control system.

Jaikar, Amol [Fermilab]↗

A Semi-Automated Workflow Solution for Data Set Publication

In order to address the need for published data, considerable effort has gone into formalizing the process of data publication. From funding agencies to publishers, data publication has rapidly become a requirement. Digital Object Identifiers (DOI) and data citations have enhanced the integration and availability of data. The challenge facing data publishers now is to deal with the increased number of publishable data products and most importantly the difficulties of publishing diverse data products into an online archive. The Oak Ridge National Laboratory Distributed Active Archive Center (ORNL DAAC), a NASA-funded data center, faces these challenges as it deals with data products created by individual investigators. This paper summarizes the challenges of curating data and provides a summary of a workflow solution that ORNL DAAC researcher and technical staffs have created to deal with publication of the diverse data products. Finally, the workflow solution presented here is generic and can be applied to data from any scientific domain and data located at any data center.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Flood and Landslide Applications of High Time Resolution Satellite Rain Products

Experimental, potentially real-time systems to detect floods and landslides related to heavy rain events are described. A key basis for these applications is high time resolution satellite rainfall analyses. Rainfall is the primary cause for devastating floods across the world. However, in many countries, satellite-based precipitation estimation may be the best source of rainfall data due to insufficient ground networks and absence of data sharing along many trans-boundary river basins. Remotely sensed precipitation from the NASA's TRMM Multi-satellite Precipitation Analysis (TMPA) operational system (near real-time precipitation at a spatial-temporal resolution of 3 hours and 0.25deg x 0.25deg) is used to monitor extreme precipitation events. Then these data are ingested into a macro-scale hydrological model which is parameterized using spatially distributed elevation, soil and land cover datasets available globally from satellite remote sensing. Preliminary flood results appear reasonable in terms of location and frequency of events, with implementation on a quasi-global basis underway. With the availability of satellite rainfall analyses at fine time resolution, it has also become possible to assess landslide risk on a near-global basis. Early results show that landslide occurrence is closely associated with the spatial patterns and temporal distribution of TRMM rainfall characteristics. Particularly, the number of landslides triggered by rainfall is related to rainfall climatology, antecedent rainfall accumulation, and intensity-duration of rainstorms. For the purpose of prediction, an empirical TMPA-based rainfall intensity-duration threshold is developed and shown to have skill in determining potential areas of landslides. These experimental findings, in combination with landslide surface susceptibility information based on satellite-based land surface information, form a starting point towards a potential operational landslide monitoring/warning system around the globe.

Adler, Robert F.↗