Search NASA⌕ Search

SEARCH · Search NASA

Results for “data ingest”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15

New and Improved GLDAS Data Sets and Data Services at NASA GES DISC

The goal of a Land Data Assimilation System (LDAS) is to ingest satellite- and ground-based observational data products, using advanced land surface modeling and data assimilation techniques, in order to generate optimal fields of land surface states and fluxes data and, thereby, facilitate hydrology and climate modeling, research, and forecast. With the motivation of creating more climatologically consistent data sets, NASA GSFC's Hydrological Sciences Laboratory has generated more than 60 years (Jan. 1948-- Dec. 2008) of Global LDAS Version 2 (GLDAS-2) data, by using the Princeton Forcing Data Set and upgraded versions of Land Surface Models (LSMs). GLDAS data and data services are provided at NASA GES DISC Hydrology Data and Information Services Center (HDISC), in collaboration with HSL and LDAS.

Rui, Hualan↗

Moving Away from Ones and Zeros, Designing a Ground Data System Based on Higher Levels of Abstraction

Previous JPL ground systems have been designed with the Ground Data System (GDS) engineer in mind. The focus on these systems has been on packaging and delivery of low level information (frames, packets, telemetry values) to the end user. It was not that long ago when project teams would be huddled over a workstation, examining crude displays of telemetry bits organized in various ways, trying to determine the status of a spacecraft. Understanding the data often required additional levels of GDS expertise, or worse, transformation of the raw data into alternative formats followed by ingestion into other tools so that the data became meaningful. The primary focus was often to answer these types of questions: "Why did this particular frame fail Reed-Solomon decode? Why did this packet get marked as invalid? Why am I missing a block of telemetry from my query?" -- which are completely valid questions to ask from a GDS Engineer's point of view, and large families of tools have been designed to help answer these questions. But these are not the questions that most users care about - which are more like: "Why is the battery state of charge trending down? Show me a summary image report for the last traverse to the target. Show me a data accountability summary for the last DSN pass." Answers to these questions, which are what users are looking for, requires a higher level of abstraction and supporting tools than mining through ones and zeros. JPL has created a next generation capability called the Mission Data Processing and Control System (MPCS) which is designed to support this higher level of abstraction by providing customizable views of the ground system combining collections of lower level information into more meaningful ways. Instead of examining frames, packets, and individual telemetry data points -- MPCS is capable of providing comprehensive summary reports, product status, overall flight/ground event status, as well as payload health summaries. Based on these higher level views, end users can make tactical or strategic decisions, or drop into detailed analysis as needed. System designers need to continue building systems that support low level GDS troubleshooting - but the basic design of a GDS should be geared towards what end users actually need to see. This paper will describe the capabilities of MPCS that directly support these higher levels of abstraction, and which are being used today in missions such as the Mars Science Laboratory and other NASA missions.

MPCS↗

Hourly GPP Estimation In Australia Using Himawari-8 AHI Products

We extended the MODIS 17 algorithm to hourly Gross Primary Product (GPP) estimation for Australia in 2018. AHI 1-km data are available at NASA Earth Exchange (NEX) facility. The high-frequent observation of Advanced Himawari Imager (AHI) enabled us to calculate hourly GPP, which is corresponding to ground observation of carbon flux measured by TERN OzFlux. We interpolated the weather observation data to 1-km grid over Australia ingesting AHI reflectance and radiometric temperature data (NEX-Gridded Hourly Meteorology (NEX-GHM)). Then, we applied MODIS 17 algorithm to each grid with optimized climate regulation functions. The estimation of climate and GPP were well correlated with OzFlux ground observation data.

Himawari-8↗

Earth Science Enterprise Scientific Data Purchase Project: Verification and Validation

This paper presents viewgraphs on the Earth Science Enterprise Scientific Data Purchase Project's verification,and validation process. The topics include: 1) What is Verification and Validation? 2) Why Verification and Validation? 3) Background; 4) ESE Data Purchas Validation Process; 5) Data Validation System and Ingest Queue; 6) Shipment Verification; 7) Tracking and Metrics; 8) Validation of Contract Specifications; 9) Earth Watch Data Validation; 10) Validation of Vertical Accuracy; and 11) Results of Vertical Accuracy Assessment.

Jenner, Jeff↗

krowkee

krowkee is a toolkit for scalably and efficiently summarizing many data streams in distributed memory. krowkee is intended for applications where one needs to summarize huge loosely structured data, such as matrices or graphs, where individual components such rows/columns or vertex adjacency information are impractical to store and directly inspect. krowkee ingests these objects as data streams - unstructured, arbitrarily ordered lists of updates - and accumulates summaries thereof in the form of data sketches.

Dunton, AlecM.↗

The fraction of total hand surface area involved in young children's outdoor hand-to-object contacts

Information on the fraction of total hand surface area touching a contaminated object is necessary in accurately estimating contaminant (e.g., pesticides, pathogens) loadings onto the hands during hand-to-object contacts. While several existing physical-stochastic human exposure models require such surface area data to estimate dermal and non-dietary ingestion exposure, there are very limited data sets. This paper provides statistical distributions of fractional surface areas (FSAs) for children's outdoor hand contacts. These distributions were constructed by combining information collected from two distinct studies exploring children's activity patterns and quantifying hand contact surface area. Results show that for outdoor contacts with 'All Objects', a range of 0.13-0.27 captured median FSAs, while a range of 0.12-0.24 captured time-weighted FSAs. Overall, an FSA of 0.31 captured 80-100% of FSAs involved in each child's outdoor hand contacts, depending upon the object of interest. These values are much lower than the often conservative assumptions of up to 1 (i.e., the entire hand) that researchers currently make regarding FSAs involved in indoor and outdoor contacts [USEPA, 1997. Standard operating procedures (SOPs) for residential exposure assessments. Contract no. 68-W6-0030. < (http://www.epa.gov/pesticides/trac/science/trac6a05.pdf)>].

60 APPLIED LIFE SCIENCES↗

Development and Validation of a Polarimetric-MCScene 3D Atmospheric Radiation Model

Polarimetric measurements can substantially enhance the ability of both spectrally resolved and single band imagery to detect the proliferation of weapons of mass destruction, providing data for locating and identifying facilities, materials, and processes of undeclared and proliferant nuclear weapons programs worldwide. Unfortunately, models do not exist that efficiently and accurately predict spectral polarized signatures for the materials of interest embedded in complex 3D environments. Having such a model would enable one to test hypotheses and optimize both the enhancement of scene contrast and the signal processing for spectral signature extraction. The Phase I set the groundwork for development of fully validated polarimetric spectral signature and scene simulation models. This has been accomplished 1. by (a) identifying and downloading state-of-the-art surface and atmospheric polarimetric data sources, (b) implementing tools for generating custom polarimetric data, and (c) identifying and requesting US Government funded field measurement data for use in validation; 2. by formulating an approach for upgrading the radiometric spectral signature model MODTRAN to generate polarimetric intensities through (a) ingestion of the polarimetric data, (b) polarimetric vectorization of existing MODTRAN modules, and (c) integration of a newly developed algorithm for computing polarimetric multiple scattering contributions; 3. by generating an initial polarimetric model that demonstrates calculation of polarimetric solar and lunar single scatter intensities arising from the interaction of incoming irradiances with molecules and aerosols; 4. by developing a design and implementation plan to (a) automate polarimetric scene construction and (b) efficiently sample polarimetric scattering and reflection events, for use in a to be developed polarimetric version of the existing first-principles synthetic scene simulation model, MCScene; and 5. by planning a validation field measurement program in collaboration with the Remote Sensing and Exploitation group at Sandia National Laboratories (SNL) in which data from their ongoing polarimetric field and laboratory measurement program will be shared and, to the extent allowed, tailored for model validation in exchange for model predictions under conditions and for geometries outside of their measurement domain.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

New and Improved GLDAS and NLDAS Data Sets and Data Services at HDISC/NASA

Terrestrial hydrological variables are important in global hydrology, climate, and carbon cycle studies. Generating global fields of these variables, however, is still a challenge. The goal of a land data assimilation system (LDAS)is to ingest satellite-and ground-based observational data products, using advanced land surface modeling and data assimilation techniques, in order to generate optimal fields of land surface states and fluxes data and, thereby, facilitate hydrology and climate modeling, research, and forecast.

Rui, Hualan↗

The NASA Open Science Data Repository: Biomedical Fair Data, Analysis Tools, User Communities, Publications, and Discoveries for Deep Space Missions

Increased biomedical risks and challenges associated with deep space missions require new knowledge discovery, new health countermeasures, and development of novel ecosystems, life support, crop production, and biomedical support capabilities. To meet NASA’s Moon to Mars strategic program goals for Human and Biological Sciences, findable, accessible, interoperable, reusable (FAIR), and maximally open-access data is going to be required to enable humanity to thrive in deep space. Indeed, this cornerstone perspective on FAIR and maximally open access data was also recommended in the recent 2023-2032 Decadal Survey from the National Academies of Sciences, Engineering, and Medicine. The NASA Open Science Data Repository (OSDR) is a maximally open access and FAIR database, and meets various scientific, technical, and operational spaceflight needs. It offers public users and submitters the ability to upload, download, search, share, analyze, and visualize data across ‘omics, physiological, phenotypic, behavioral, bioimaging, video, and environmental monitoring telemetry datasets. OSDR includes NASA GeneLab, NASA Ames Life Sciences Data Archive, and the NASA Biological Institutional Scientific Collection. OSDR has >455 studies with datasets from model organisms and non-NASA human astronauts. There are ~12 datasets from the Inspiration 4 (I4) mission, spanning metagenomics, comprehensive metabolic panels, clonal hematopoiesis, spatial transcriptomics, proteomics, and cytokine panels. In the interest of data privacy, two I4 datasets have raw FASTQ and FASTA files relating to the epitranscriptome, and a new request feature is live in OSDR (with a backend review process established) which was developed based on industry norms. OSDR also recently began a collaboration with the European Space Agency (ESA) to scientifically curate and make available >200 terabytes of human and model organism space-relevant data. The OSDR submission portal is designed to ingest and curate ~25 ‘omics assay data types, and ~50 physiological-phenotypic-imaging assay data types, spanning ultrasonography, micro-computed tomography, histology, morphometric photography, rebound tonometry, gait analysis, optical coherence tomography, novel object recognition, flow cytometry, and immunohistochemistry. A suite of analysis tools are available for OSDR users including: 1) an Environmental Data Application to compare radiation, CO2, relative humidity, temperature, and other telemetry across missions and subjects, 2) the RadLab database, a collaboration between NASA, ESA, the German and Italian Space Agencies, and the Bulgarian Academy of Sciences, which compiles radiation measurements relevant to human spaceflight and provides tools for accessing and manipulating the data, and 3) a Multi-study visualization tool which enables users to look across and combine GeneLab’s omics datasets across different experiments and missions. There are ~600 volunteer OSDR Analysis Working Group (AWG) members who: 1) provide feedback on scientific standards for reuse (subject and assay metadata; processing pipelines; dataset formats and uniformed structures for machine-readability), and 2) collaborate to mine-reuse OSDR data conducting scientific analysis. OSDR has enabled 60 publications as of September 2023, many directly from AWG collaborations most notably the Cell Press package in 2020. Lastly, there are at least 15 articles which mine OSDR data part of a package of ~50 articles across Nature Portfolio with research stemming from I4, the Japan Aerospace Exploration Agency, NASA Space Biology, and the NASA Human Research Program.

space biology↗

NASA Open Science Data Repository: Biomedical FAIR Data, Analysis Tools, User Communities, and Discoveries for Deep Space Missions

Increased biomedical risks and challenges associated with deep space missions require new knowledge discovery, new health countermeasures, and development of novel ecosystems, life support, crop production, and biomedical support capabilities. To meet NASA’s Moon to Mars strategic program goals for Human and Biological Sciences, findable, accessible, interoperable, reusable (FAIR), and maximally open-access data is going to be required to enable humanity to thrive in deep space. Indeed, this cornerstone perspective on FAIR and maximally open access data was also recommended in the recent 2023-2032 Decadal Survey from the National Academies of Sciences, Engineering, and Medicine. The NASA Open Science Data Repository (OSDR) is a maximally open access and FAIR database, and meets various scientific, technical, and operational spaceflight needs. It offers public users and submitters the ability to upload, download, search, share, analyze, and visualize data across ‘omics, physiological, phenotypic, behavioral, bioimaging, video, and environmental monitoring telemetry datasets. OSDR includes NASA GeneLab, NASA Ames Life Sciences Data Archive, and the NASA Biological Institutional Scientific Collection. OSDR has >455 studies with datasets from model organisms and non-NASA human astronauts. There are ~12 datasets from the Inspiration 4 (I4) mission, spanning metagenomics, comprehensive metabolic panels, clonal hematopoiesis, spatial transcriptomics, proteomics, and cytokine panels. In the interest of data privacy, two I4 datasets have raw FASTQ and FASTA files relating to the epitranscriptome, and a new request feature is live in OSDR (with a backend review process established) which was developed based on industry norms. OSDR also recently began a collaboration with the European Space Agency (ESA) to scientifically curate and make available >200 terabytes of human and model organism space-relevant data. The OSDR submission portal is designed to ingest and curate ~25 ‘omics assay data types, and ~50 physiological-phenotypic-imaging assay data types, spanning ultrasonography, micro-computed tomography, histology, morphometric photography, rebound tonometry, gait analysis, optical coherence tomography, novel object recognition, flow cytometry, and immunohistochemistry. A suite of analysis tools are available for OSDR users including: 1) an Environmental Data Application to compare radiation, CO2, relative humidity, temperature, and other telemetry across missions and subjects, 2) the RadLab database, a collaboration between NASA, ESA, the German and Italian Space Agencies, and the Bulgarian Academy of Sciences, which compiles radiation measurements relevant to human spaceflight and provides tools for accessing and manipulating the data, and 3) a Multi-study visualization tool which enables users to look across and combine GeneLab’s omics datasets across different experiments and missions. There are ~600 volunteer OSDR Analysis Working Group (AWG) members who: 1) provide feedback on scientific standards for reuse (subject and assay metadata; processing pipelines; dataset formats and uniformed structures for machine-readability), and 2) collaborate to mine-reuse OSDR data conducting scientific analysis. OSDR has enabled 60 publications as of September 2023, many directly from AWG collaborations most notably the Cell Press package in 2020. Lastly, there are at least 15 articles which mine OSDR data part of a package of ~50 articles across Nature Portfolio with research stemming from I4, the Japan Aerospace Exploration Agency, NASA Space Biology, and the NASA Human Research Program.

open access↗

Low cost real time interactive analysis system

Efforts continue to develop a low cost real time interactive analysis system for the reception of satellite data. A multi-purpose ingest hardware software frame formatter was demonstrated for GOES and TIROS data and work is proceeding on extending the capability to receive GMS data. A similar system was proposed as an archival and analysis system for use with INSAT data and studies are underway to modify the system to receive the planned SeaWiFS (ocean color) data. This system was proposed as the core of a number of international programs in support of U.S. AID activities. Systems delivered or nearing final testing are listed.

Stetina, F.↗

Assimilation of GRACE Terrestrial Water Storage Data into a Land Surface Model

The NASA Gravity Recovery and Climate Experiment (GRACE) system of satellites provides observations of large-scale, monthly terrestrial water storage (TWS) changes. In. this presentation we describe a land data assimilation system that ingests GRACE observations and show that the assimilation improves estimates of water storage and fluxes, as evaluated against independent measurements. The ensemble-based land data assimilation system uses a Kalman smoother approach along with the NASA Catchment Land Surface Model (CLSM). We assimilated GRACE-derived TWS anomalies for each of the four major sub-basins of the Mississippi into the Catchment Land Surface Model (CLSM). Compared with the open-loop (no assimilation) CLSM simulation, assimilation estimates of groundwater variability exhibited enhanced skill with respect to measured groundwater. Assimilation also significantly increased the correlation between simulated TWS and gauged river flow for all four sub-basins and for the Mississippi River basin itself. In addition, model performance was evaluated for watersheds smaller than the scale of GRACE observations, in the majority of cases, GRACE assimilation led to increased correlation between TWS estimates and gauged river flow, indicating that data assimilation has considerable potential to downscale GRACE data for hydrological applications. We will also describe how the output from the GRACE land data assimilation system is now being prepared for use in the North American Drought Monitor.

Reichle, Rolf H.↗

International Space Station Data Collection for Disaster Response

Natural disasters - including such events as tropical storms, earthquakes, floods, volcanic eruptions, and wildfires -effect hundreds of millions of people worldwide, and also cause billions of dollars (USD) in damage to the global economy. Remotely sensed data acquired by orbital sensor systems has emerged as a vital tool to identify the extent of damage resulting from a natural disaster, as well as providing near-real time mapping support to response efforts on the ground and humanitarian aid efforts. The International Space Station (ISS) is a unique terrestrial remote sensing platform for acquiring disaster response imagery. Unlike automated remote-sensing platforms it has a human crew; is equipped with both internal and externally-mounted remote sensing instruments; and has an inclined, low-Earth orbit that provides variable views and lighting (day and night) over 95 percent of the inhabited surface of the Earth. As such, it provides a useful complement to free-flyer based, sun-synchronous sensor systems in higher altitude polar orbits. While several nations have well-developed terrestrial remote sensing programs and assets for data collection, many developing nations do not have ready access to such resources. The International Charter, Space and Major Disasters (also known as the "International Disaster Charter", or IDC; http://www.disasterscharter.org/home) addresses this disparity. It is an agreement between agencies of several countries to provide - on a best-effort basis - remotely sensed data of natural disasters to requesting countries in support of disaster response. The lead US agency for interaction with the IDC is the United States Geological Survey (USGS); when an IDC request or "activation" is received, the USGS notifies the science teams for NASA instruments with targeting information for data collection. In the case of the ISS, the Earth Sciences and Remote Sensing (ESRS) Unit, part of the Astromaterials Research and Exploration Science Directorate and supporting the ISS Program Science Office at NASA's Johnson Space Center, receives notification from the USGS and coordinates targeting and data collection with the NASA ISS sensor teams. If data is collected, it is passed back to the USGS for posting on their Hazards Data Distribution System and made available for download. The ISS International Partners (CSA, ESA, JAXA, Roscosmos/Energia) have their own procedures for independently supporting IDC activations using their assets on ISS, and there is currently no joint coordination with NASA ISS sensor teams. Following completion of ISS assembly, NASA remote sensing assets began collecting IDC response data in May 2012. The initial NASA ISS sensor systems available to respond to IDC activations included the ISS Agricultural Camera (ISSAC), an internal multispectral visible-near infrared wavelength system mounted in the Window Observational Research Facility, or WORF; the Crew Earth Observations (CEO) Facility, where the crew collects imagery through Station windows using off-the-shelf handheld digital visible-wavelength cameras; and the Hyperspectral Imager for the Coastal Oceans (HICO), a visible to near-infrared system mounted externally on the Japan Experiment Module Exposed Facility. The ISSAC completed its primary mission and was removed from the WORF in January 2013. It was replaced by the very high resolution ISS SERVIR Environmental Research and Visualization System (ISERV) Pathfinder, a visible-wavelength digital camera, telescope, and pointing system. Since the start of IDC response by NASA sensors on the ISS in May 2012 and as of this report, there have been eighty IDC activations; NASA sensor systems have collected data for twenty-three of these events. Of the twenty-three successful data collections, five involved 2 or more ISS sensor systems responding to the same event. Data has also been collected by International Partners in response to natural disasters, most notably JAXA and Roscosmos/Energia through the Urugan program. Data collected in response to IDC activations is delivered by the ISS sensor teams to the ESRS for quality review and transfer to the USGS, where it is ingested into the Hazards Data Distribution System, or HDDS (https://hdds.usgs.gov/hdds2/; figure 1). This system allows the local agencies that issued the IDC activation request to review and download data. The data is then used to develop secondary products useful for humanitarian response such as flood maps. As of this report, approximately 1000 images collected by NASA ISS sensor systems have been downloaded from the HDDS, indicating that the ISS has assumed a valuable role in disaster response efforts. The ISS is also a unique platform in that it will have multiple users over its lifetime, and that no single remote sensing system has a permanent internal or external berth. This scheduled turnover provides for development of new remote sensing capabilities relevant to disaster response -as well as both research and applied science-and represents a significant contribution to continuance and enhancement of the NASA mission to investigate changes on our home planet.

Stefanov, William L.↗

Phasor-Measurement-Unit-Based Data Analytics Using Digital Twin and PhasorAnalytics Software

A major objective of this project was to apply GE’s commercial machine learning and data analytics toolsets to large-scale, real-world, anonymized Phasor Measurement Unit (PMU) datasets in order to extract signatures, correlated and/or causal factors, and precursor patterns associated with significant power system phenomena. The project had a particular emphasis on extraction of insights relevant to asset health monitoring, real-time load modeling and cybersecurity monitoring. Additionally, the team was directed to undertake a comprehensive data quality analysis for the provided datasets and encouraged to estimate the ‘machine-learning readiness’ of the datasets by documenting any major obstacles to the application of commercial machine learning algorithms. To accomplish the aforementioned objectives, the project team’s work centered around the identification of key event signatures and application of the identified event signatures for event detection and event classification. The industry-validated, semi-supervised machine learning strategy employed for event signature identification involved several major tasks, including data-preprocessing, generation of an overabundance of features, normal data identification, normality modeling, and event signature identification through a methodical, quantitative ranking of features in order of relevance to each studied event type. Throughout the project, data quality issues and mitigation techniques were investigated. In this report, insights are provided regarding the readiness of the provided synchrophasor datasets for application of machine learning and data analytics. The methodologies employed for this technical strategy are summarized in this report. With regards to data preprocessing and feature generation, the provided Training and Test Datasets were ingested into GE’s big data environment. Subsequently, the team applied bad data cleansing and data imputation scripts, event detection scripts, and application programming interfaces (APIs) to the datasets for convenient data access. The project team completed development and validation of dozens of physics-based, statistics-based and transformation-based feature functions used for the extraction of over 60 synchrophasor features. Using a new parallel feature generation technology developed on this project, over 60 features have been rapidly generated for the full two years’ worth of Training and Test Dataset data associated with both the Eastern and Western interconnects. Even accommodating for temporal down-sampling inherent to the feature extraction procedure, this parallel feature generation activity resulted in a massive feature set with a storage requirement approximately equal to that of the raw training dataset itself. With regards to normal data identification and normality modeling, a normality model was built using the feature data extracted from the Training Dataset and iteratively refined subsequent to incremental adjustments and expansions of the Training Dataset feature data. With respect to event characterization and signature identification, an event signature identification pipeline was developed and used in conjunction with the normality model to identify over 15 event signatures for key event categories within the Training Dataset. The identified event signatures were used to characterize hundreds of key events in terms of relative severity, duration, and location of the event. An investigation was undertaken to identify correlated and causal factors involved in transformer events. A separate investigation into temporal trends in ring-down analysis results was undertaken to determine possible associations between system dynamics and various other factors such as loading, season or year. To validate the identified event signatures, additional work was undertaken to develop signature-based anomaly detection and classification tools suitable for convenient application to the synchrophasor datasets. The anomaly detection and classification tools, suitable for online application, were then applied to the entirety of the Eastern Interconnect Training and Test Datasets. Performance of the event detection and classification tools was evaluated upon receipt of the Test Dataset event logs (i.e., the labels for events contained in the Test Dataset), and promising results were obtained despite several challenges (documented herein) associated with application of supervised or semi-supervised machine learning methods to large-scale, anonymized datasets. Finally, the detection and classification tools were used to detect, classify, and characterize thousands of new events not included in the original event logs provided by the DOE within both the Training and Test Datasets.

24 POWER TRANSMISSION AND DISTRIBUTION↗

A scalable messaging system for accelerating discovery from large scale scientific simulations

Emerging scientific and engineering simulations running at scale on leadership-class High End Computing (HEC) environments are producing large volumes of data, which has to be transported and analyzed before any insights can result from these simulations. The complexity and cost (in terms of time and energy) associated with managing and analyzing this data have become significant challenges, and are limiting the impact of these simulations. Recently, data-staging approaches along with in-situ and in-transit analytics have been proposed to address these challenges by offloading I/O and/or moving data processing closer to the data. However, scientists continue to be overwhelmed by the large data volumes and data rates. In this paper we address this latter challenge. Specifically, we propose a highly scalable and low-overhead associative messaging framework that runs on the data staging resources within the HEC platform, and builds on the staging-based online in-situ/in- transit analytics to provide publish/subscribe/notification-type messaging patterns to the scientist. Rather than having to ingest and inspect the data volumes, this messaging system allows scientists to (1) dynamically subscribe to data events of interest, e.g., simple data values or a complex function or simple reduction (max()/min()/avg()) of the data values in a certain region of the application domain is greater/less than a threshold value, or certain spatial/temporal data features or data patterns are detected; (2) define customized in-situ/in-transit actions that are triggered based on the events, such as data visualization or transformation; and (3) get notified when these events occur. The key contribution of this paper is a design and implementation that can support such a messaging abstraction at scale on high- end computing (HEC) systems with minimal overheads. We have implemented and deployed the messaging system on the Jaguar Cray XK6 machines at Oak Ridge National Laboratory and the Lonestar system at the Texas Advanced Computing Center (TACC), and we present the experimental performance evaluation using these HEC platforms in the paper.

Jin, Tong↗

Assessing the Needs of NASA's Near Real-Time Earth Observation Products

"The 2017-2027 Decadal Survey for Earth Science and Applications from Space stated that NASA's Earth Science with planned implementation of applications provides sustained earth observations for societal benefits [1]. The Decadal Survey indicated that data latency is invaluable for time-sensitive applications including disaster risk reduction, wildland fire carbon emissions quantification, real-time measurements of the state of the hydrologic systems and many more. Data latency refers to the time between earth observation and data products available to users. During the past 13 years, NASA's Land, Atmosphere Near Real-Time Capability for Earth Observing Systems (LANCE) continues to provide free access to earth observation products that are made available much quicker than routine processing allows. The latency of most LANCE data products is Near Real-time (NRT) which is defined as less than three hours from satellite observations [2]. LANCE is managed by the Earth Science Data and Information System (ESDIS) Project at NASA Goddard Space Flight Center [3], and a User Working Group (UWG) is responsible for providing guidance to LANCE. LANCE data are used by direct users and brokers who add value to the data [4]. NASA Earth Applied Sciences Program (ASP) is one of the primary users of LANCE, which collaborates with partner organizations and provides support to scientists to solve problems in applications of earth observations. ASP promotes the use of LANCE NRT data products to demonstrate applications in decision making, facilitates end-user feedback to the science team to improve data products, and provides information on future demands for research. LANCE supports applications that need a rapid response including detecting wildland fires and volcanic eruptions, tracking smoke, ash and dust plumes, monitoring air quality and tracking extreme weather events such as hurricanes, landslides, and floods. To gather feedback regarding the availability, accessibility and actionability of NASA's NRT data products for societal benefit, three surveys and a few discussions with experts involved in the topic within ASP were conducted from the perspective of users. Feedback has been collected from users who are interested in using low latency NASA data within application communities of agriculture, disasters, water resources, health and air quality, ecological conservation, wildland fires and capacity building. Analysis-ready NRT data products in a variety of formats have been mentioned many times in the collected feedback, especially for applied users with little to no experience using research-grade earth observation products. Users prefer to have products that can be easily integrated into their existing workflows and take their analysis to the data. HDF5 is a commonly used data format for research, but typically requires some conversion to a more friendly format for applications and regular use in decision-making. Users prefer the GeoTIFF data format that can be directly ingested into a GIS mapping software and platform for data analysis and visualization. For example, LANCE’s fire, flood, SO2 and Black Marble Nighttime Blue/Yellow Composite data products have been integrated into NASA Disasters Mapping Portal, which is an GIS-based open data portal, for users in the disaster management community. There are 291 LANCE NRT layers available through GIBS and Worldview, where users can download a snapshot in GeoTIFF format. Operational users expect data to be processed as close to the user as possible. The collected feedback indicates that LANCE fire products within 3 hours latency would meet the needs of the wildland fire community. The ideal latency for volcanic application is 10-15 minutes. Users in Volcanic Ash Advisory Centers (VAAC) reported that the first forecast volcanic product should be issued within 75 minutes from the volcano eruption [5]. Overall, for disaster applications, data latency within 3 hours is useful while latency greater than 12 hours is not timely enough for operational use. Capacity building and training are critical for users to be able to access, interpret and use data products and tools for their decision making, especially for applied users with limited experience using earth observation products. LANCE data products have been used in a number of capacity building projects domestically and internationally [6]. As LANCE continues to bring new products into the system, users request training to utilize LANCE new and upcoming data products and capabilities in their applications. Due to the limitation of bandwidth and downstream flow paths, users in some developing countries need tools to select and download data for a specific area of interest instead of bulk downloads. The collected feedback also shows the lack of available SAR satellite low latency data products. The advantages of SAR to monitor conditions and changes on the ground through darkness, clouds, volcanic ash, and other atmospheric conditions, are appealing to low latency users. For example, terabytes of low latency but cloudy optical images are not helpful in rapidly identifying the extent of flood or fire impacts. LANCE could be complemented with low latency measurements via the upcoming NASA-ISRO Synthetic Aperture Radar (NISAR) mission [7]. Requests for higher spatial resolution products are expressed. A user from the wildland fire management community reported that products with 30-m spatial resolution could be used to detect small fires. The 30-m Landsat OLI fire data is now part of NASA’s Fire Information for Resource Management System (FIRMS) US/Canada [8]. Within the open and free NASA resources, LANCE disseminates NRT data products in a manner that allows them to be accessible and understandable to both scientific and applied users. In many application areas, latency plays an important or even decisive role where low latency earth observations help people to observe areas of interest, detect and track changes in the environment and make timely decisions. NASA’s Earth Applied Sciences Program promotes the use of LANCE NRT products and builds a bridge between application users and research teams. The collected feedback indicates data latency within 3 hours is useful for most of the applications, and shows the needs of user-friendly, analysis-ready products, and requests training on LANCE’s new and upcoming data products. User feedback has been provided to LANCE UWG for guidance and recommendations, and for translating findings into something actionable.

Tian Yao↗

Big Data Framework with Machine Learning for D and D Applications - 19108

The nuclear industry is experiencing a steady increase in maintenance costs even though plants are maintained under high levels of safety, capability and reliability. Nuclear power plants are expected to run every unit at maximum capacity at all times, efficiently utilizing assets with minimal downtime. Surveillance and maintenance of nuclear decommissioning infrastructure provides lot challenges with respect to maintenance or decommissioning of the buildings. There is a need for a framework to analyze the huge amount of data generated by the sensors on the nuclear reactor components as well as structures, to monitor the conditions of these building over a period of time. Emerging technologies such as big data analytics have become a requirement in the nuclear industry to improve structural health monitoring and diagnostics. FIU will make use of existing mature technologies in the areas of imaging, robotics, big data, and machine learning/deep learning to assess the structural integrity of aging facilities at DOE sites. As these facilities await decommissioning, there is a need to understand the structural health of these structures. Many of these facilities were built over 50 years ago and in some cases these facilities have gone beyond the operational life expectancy. In other cases, the facilities have been placed in a state of 'cold and dark' and they are sitting unused, awaiting decommissioning. Finally, some older facilities are one-of-a-kind operational/production facilities supporting critical DOE missions and cannot be shut down. In any of these scenarios, the structural integrity of these facilities may be compromised, so it is imperative that adequate inspections and data collection/analysis be performed on a continuous and ongoing basis. The primary goals of the research include collecting various formats of data such as structured, unstructured and streaming data from the various sensors deployed in buildings, collect videos/pictures from various imaging sources, ingest them using a Hadoop distributed file system and process the data using Spark to perform batch processing and real time analytics. Research and development on various machine learning/deep learning algorithms will also be performed to analyze the heterogeneous data collected from nuclear decommission infrastructure. FIU will design the big data framework to ingest, store and process huge amounts of heterogeneous data collected from many sources and optimize the algorithms to provide insights into the data and predict anomalies observed when compared against baseline conditions. Various modules of this framework will include heterogeneous data sources, message broker, Hadoop distributed file system, Spark for stream and batch processing, machine learning/ deep learning, Cassandra - persistent data store and visualization. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

PC-SEAPAK user's guide, version 4.0

PC-SEAPAK is designed to provide a complete and affordable capability for processing and analysis of NOAA Advanced Very High Resolution Radiometer (AVHRR) and Nimbus-7 Coastal Zone Color Scanner (CZCS) data. Since the release of version 3.0 over a year ago, significant revisions were made to the AVHRR and CZCS programs and to the statistical data analysis module, and a number of new programs were added. This new version has 114 procedures listed in its menus. The package continues to emphasize user-friendliness and interactive data analysis. Additionally, because the scientific goals of the ocean color research being conducted have shifted to larger space and time scales, batch processing capabilities were enhanced, allowing large quantities of data to be easily ingested and analyzed. The development of PC-SEAPAK was paralled by two other activities that were influential and assistive: the global CZCS processing effort at GSFC and the continued development of VAX-SEAPAK. SEAPAK incorporates the instrument calibration and support all levels of data available from the CZCS archive.

Mcclain, Charles R.↗