Search NASA⌕ Search

SEARCH · Search NASA

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24
All sourcesNASA NTRSDOE OSTI
NASA NTRS2007

Wireless Data-Acquisition System for Testing Rocket Engines

A prototype wireless data-acquisition system has been developed as a potential replacement for a wired data-acquisition system heretofore used in testing rocket engines. The traditional use of wires to connect sensors, signal-conditioning circuits, and data acquisition circuitry is time-consuming and prone to error, especially when, as is often the case, many sensors are used in a test. The system includes one master and multiple slave nodes. The master node communicates with a computer via an Ethernet connection. The slave nodes are powered by rechargeable batteries and are packaged in weatherproof enclosures. The master unit and each of the slave units are equipped with a time-modulated ultra-wide-band (TMUWB) radio transceiver, which spreads its RF energy over several gigahertz by transmitting extremely low-power and super-narrow pulses. In this prototype system, each slave node can be connected to as many as six sensors: two sensors can be connected directly to analog-to-digital converters (ADCs) in the slave node and four sensors can be connected indirectly to the ADCs via signal conditioners. The maximum sampling rate for streaming data from any given sensor is about 5 kHz. The bandwidth of one channel of the TM-UWB radio communication system is sufficient to accommodate streaming of data from five slave nodes when they are fully loaded with data collected through all possible sensor connections. TM-UWB radios have a much higher spatial capacity than traditional sinusoidal wave-based radios. Hence, this TM-UWB wireless data-acquisition can be scaled to cover denser sensor setups for rocket engine test stands. Another advantage of TM-UWB radios is that it will not interfere with existing wireless transmission. The maximum radio-communication range between the master node and a slave node for this prototype system is about 50 ft (15 m) when the master and slave transceivers are equipped with small dipole antennas. The range can be increased by changing to larger antennas and/or greater transmission power. The battery life of a slave node ranges from about six hours during operation at full capacity to as long as three days when the system is in a "sleep" mode used to conserve battery charge during times between setup and rocket-engine testing. Batteries can be added to prolong operational lifetimes. The radio transceiver dominates the power consumption.

Lin, Chujen↗
DOE OSTI2022

FTICR-MS, Sensor, and Environmental Data from 5 Streams Impacted by the 2020 Holiday Farm Fire Associated with: "Spatiotemporal controls on the delivery of dissolved organic matter to streams following a wildfire"

This data package is associated with the publication "Spatiotemporal Controls on the Delivery of Dissolved Organic Matter to Streams Following a Wildfire" submitted to Geophysical Research Letters (Roebuck et al., 2022). The study aims to understand storm induced transport of pyrogenic materials to streams impacted by varying degrees of burn severity. Time series samples (24 samples in 1-hour intervals) were collected at 5 sites within the McKenzie River Watershed (Oregon, USA) whose catchment were each completely engulfed by the 2020 Holiday Farm Fire. The samples were collected in November 2020 during the first major storm pulse following the conclusion of the wildfire. Samples were characterized for dissolved organic carbon, total dissolved nitrogen, and by ultra-high resolution mass spectrometry. In situ turbidity data also collected.This data package contains 4 primary folders that include the following: 1) Metadata, 2) EnvData (Environmental Data), 3) SensorData, and 4) FTICR_SupportingData. The package contains a single file-level metadata (flmd) file. Each primary folder also contains individual data dictionaries (dd) to define and provide descriptors of column/row headers and data flags. The FTICR_SupportingData, folder 4, contains raw, unprocessed FTICR-MS Data files in addition to a csv containing processed FTICR-MS data. This package contains the following file types: csv, xml, pdf.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2013

Discrete Data Qualification System and Method Comprising Noise Series Fault Detection

A Sensor Data Qualification (SDQ) function has been developed that allows the onboard flight computers on NASA s launch vehicles to determine the validity of sensor data to ensure that critical safety and operational decisions are not based on faulty sensor data. This SDQ function includes a novel noise series fault detection algorithm for qualification of the output data from LO2 and LH2 low-level liquid sensors. These sensors are positioned in a launch vehicle s propellant tanks in order to detect propellant depletion during a rocket engine s boost operating phase. This detection capability can prevent the catastrophic situation where the engine operates without propellant. The output from each LO2 and LH2 low-level liquid sensor is a discrete valued signal that is expected to be in either of two states, depending on whether the sensor is immersed (wet) or exposed (dry). Conventional methods for sensor data qualification, such as threshold limit checking, are not effective for this type of signal due to its discrete binary-state nature. To address this data qualification challenge, a noise computation and evaluation method, also known as a noise fault detector, was developed to detect unreasonable statistical characteristics in the discrete data stream. The method operates on a time series of discrete data observations over a moving window of data points and performs a continuous examination of the resulting observation stream to identify the presence of anomalous characteristics. If the method determines the existence of anomalous results, the data from the sensor is disqualified for use by other monitoring or control functions.

Fulton, Christopher↗
DOE OSTI2025

Real-time data processing for serial crystallography experiments

We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames ('hits'). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data, with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the `data deluge' problem from recent and future photon-science experiments, in particular on calibration requirements, computing access patterns and the need for the preservation of raw data.

47 OTHER INSTRUMENTATION↗
DOE OSTI2026

Evaluating lightweight unsupervised online IDS for masquerade attacks in CAN

Vehicular controller area networks (CANs) are susceptible to masquerade attacks by malicious adversaries. In masquerade attacks, adversaries silence a targeted ID and then send malicious frames with forged content at the expected timing of benign frames. As masquerade attacks could seriously harm vehicle functionality and are the stealthiest attacks to detect in CAN, recent work has devoted attention to compare frameworks for detecting masquerade attacks in CAN. However, most existing works report offline evaluations using CAN logs already collected using simulations that do not comply with the domain’s real-time constraints. Here we contribute to advance the state of the art by presenting a comparative evaluation of four different non-deep learning (DL)-based unsupervised online intrusion detection systems (IDS) for masquerade attacks in CAN. Our approach differs from existing comparative evaluations in that we analyze the effect of controlling streaming data conditions in a sliding window setting. In doing so, we use realistic masquerade attacks being replayed from the ROAD dataset. We show that although evaluated IDS are not effective at detecting every attack type, the method that relies on detecting changes in the hierarchical structure of clusters of time series produces the best results at the expense of higher computational overhead. We discuss limitations, open challenges, and how the evaluated methods can be used for practical unsupervised online CAN IDS for masquerade attacks.

Anomaly detection↗
NASA NTRSDate not supplied

Anomaly Detection in Flight Operational Data Using Deep Learning

In this session, we demonstrate two recently developed deep learning models for anomaly detection in flight operational data by the Data Sciences Group at NASA Ames Research Center. The first model is Convolutional Variational Auto-Encoder (CVAE) [1], which is an unsupervised deep encoder-decoder model, designed specifically for finding anomalies in heterogeneous multivariate time series data. We will demonstrate its application to finding anomalies in streaming data from NASA’s Digital Information Platform’s Fuser source. CVAE identifies data instances that are not representative of expected nominal behavior as anomalous. Since it is an unsupervised approach, the flagged anomalies will need to be reviewed by the subject matter experts (SMEs) for validation and labeling and is designed to assist with vulnerability discovery within Safety Monitoring System programs. The second model is Robust and Explainable Semi-supervised Anomaly Detection (RESAD) model [2], which builds on CVAE to allow learning from both minimally labeled data (previously reviewed by the SMEs) as well as majority unlabeled data. RESAD takes advantage of graph theoretic techniques to propagate the labels from the labeled data to the unlabeled data based on a pre-defined similarity metric and structures the learned feature space from flight time-series so that data of the same class would cluster tightly together. This model characteristic is enabled by training with an augmented loss function and allows learning of a more informative feature space for down-stream tasks such as search and active learning. We demonstrate RESAD using data from the NASA DASHlink project [3].

anomaly detection↗
DOE OSTI2019

Efficient Modeling of HIL Multi-Grid System for Scalability & Concurrency in CPS Security Testbed

Cyber-event-triggered power grid blackout compels utility operators to intensify cyber-aware and physics-constrained recovery and restoration process. Recently, coordinated cyber attacks on the Ukrainian grid witnessed such a cyber-event-triggered power system blackout. Various cyber-physical system (CPS) testbeds have attempted with multitude designs to analyze such interdependent events and evaluate remedy measures. However, resource constraints and modular integration designs have been significant barriers while modeling large-scale grid models (scalability) and multi-grid isolated models (concurrency) under a single real-time execution environment for the hardware-in-the-loop (HIL) CPS security testbeds. This paper proposes a meticulous design and effective modeling for simulating large-scale grid models and multi-grid isolated models in a HIL realtime digital simulator environment integrated with industry-grade hardware and software systems. We have used our existing HIL CPS security testbed to demonstrate scalability by the realtime performance of a Texas-2000 bus US synthetic grid model and concurrency by the real-time performance of simultaneous ten IEEE-39 bus grid models and an IEEE-118 bus grid model. The experiments demonstrated significant results by 100% realtime performance with zero overruns, low latency while receiving and executing control signals from SEL Relays via IEC-61850 protocol and low latency while computing and transmitting grid data streams including stability measures via IEEE C37.118 synchrophasor data protocol to SEL Phasor Data Concentrators.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2013

AGR-2 Data Qualification Report for ATR Cycles 151B-2, 152A, 152B, 153A, 153B and 154A

This report documents the data qualification status of AGR-2 fuel irradiation experimental data from Advanced Test Reactor (ATR) Cycles 152A, 152B, 153A, 153B, and 154A, as recorded in the Nuclear Data Management and Analysis System (NDMAS). The AGR-2 data streams addressed include thermocouple (TC) temperatures, sweep gas data (flow rate, pressure, and moisture content), and fission product monitoring system (FPMS) data for each of the six capsules in the experiment. A total of 13,400,520 every minute instantaneous TC and sweep gas data records were received and processed by NDMAS for this period. Of these data, 8,911,791 records (66.5% of the total) were determined to be Qualified based on NDMAS accuracy testing and data validity assessment. For temperature, there were 4,266,081 records (74% of the total TC data) that were Failed due to TC instrument failures. For sweep gas flows, there were 222,648 gas flow records (2.91% of the flow data) that were Failed. The inlet gas flow failures due to gas flow cross-talk and leakage problems that occurred after Cycle 150A were corrected by using the same gas mixture in all six capsules and the Leadout. For FPMS data, NDMAS received and processed preliminary release rate and release-to-birth rate ratio (R/B) data for three reactor cycles (Cycles 149B, 150B, and 151A) . This data consists of 45,983 release rate records and 45,235 R/B records for the 12 radionuclides reported. The qualification status of these FPMS data has been set to In Process until receipt of Quality Assurance-approved data generator reports. All of the above data have been processed and tested using a SAS®-based enterprise application software system, stored in a secure Structured Query Language database, made available on the NDMAS Web portal (http://ndmas.inl.gov), and approved by the INL STIM for release to both internal and appropriate external Very High Temperature Reactor Program participants.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2021

HAIMOS Ensemble Forecasts for Intra-day and Day- Ahead GHI, DNI and Ramps

The objective of this research is to develop a hybrid physics-based/data-driven forecast model to improve direct normal and global horizontal irradiance (DNI and GHI) prediction for horizons ranging from 1 to 72 hours. Project objectives also address key gaps in state-of-the-art solar forecasting: accurate probabilistic solar forecasts and the forecasting of large irradiance ramps (ramp onset and magnitude). The proposed model ensembles Numerical Weather Prediction (NWP) forecasts, determinist physics-based algorithms, and new-generation cloud cover products (high-resolution rapid refresh satellite images and Large Eddy Simulations). The result is the Hybrid Adaptive Input Model Objective Selection (HAIMOS) ensemble model. HAIMOS blends state of the art machine learning methodologies with physics-based models for cloud cover and cloud optical depth forecasts. The technical activities followed a two-pronged strategy. First, the preprocessing of data, the selection of inputs to the nonlinear approximators, the type of approximator and objective functions, and post-processing ensembling techniques included in HAIMOS were all optimized adaptively to find the best model for a specific goal (reduce DNI/GHI forecast error, improve the prediction of ramp onset, etc.). Second, a large effort was put in improving cloud identification and the forecast of cloud cover and cloud optical depth. To this end, new-generation cloud parametrization products were developed in this work. These include improved algorithms to assist in cloud identification, cloud classification and cloud parametrization from satellite images – three key factors in the accuracy of 1 to 6-hours irradiance forecasts and prediction of ramp onset. Furthermore, we also included cloud information extracted from high resolution rapid refresh satellite images (GOES-16) and Large Eddy Simulations (LES). LES was used to model the atmosphere in detail over locations of interest and produce cloud optical depth forecasts. Once these data streams were validated, they were used as input data to the HAIMOS forecast. The model was developed using data from several climatologically distinct locations with potential for high solar penetration. In the last year of the project, we conducted a validation campaign according to the guidelines stipulated by the Topic Area 1 project as described in the FOA. This effort brings, for the first time, proven machine-learning methodologies for generating state-of-the-art solar forecasts interweaved with detailed physics-based models for cloud detection, and cloud optical depth forecasts. HAIMOS will generate accurate irradiance probabilistic forecast to assist in reducing solar generation prediction error. Globally optimized solar forecast models are more likely to impact solar energy stakeholders. The goal of this project was to increase the state-of-the-art forecast skill from their present values of 10 to 35%. At the end of the project, we achieved between 30% and 50% forecast skill across a wide range of horizons for both GHI and DNI.

14 SOLAR ENERGY↗
DOE OSTI2019

A Parallel Graph Environment for Real-World Data Analytics Workflows

Economic competitiveness and national security depend increasingly on the insightful analysis of large data sets. The diversity of real-world data sources and analytic workflows impose challenging hardware and software requirements for parallel graph platforms. The irregular nature of graph methods is not supported well by the deep memory hierarchies of conventional distributed systems, requiring new processor and runtime system designs to tolerate memory and synchronization latencies. Moreover, the efficiency of relational table operations and matrix computations are not attainable when data is stored in common graph data structures. In this paper, we present HAGGLE, a high-performance, scalable data analytics platform. The platform’s hybrid data model supports a variety of distributed, thread-safe data structures, parallel programming constructs, and persistent and streaming data. An abstract runtime layer enables us to map the stack to conventional, distributed computer systems with accelerators. The runtime uses multithreading, active messages, and data aggregation to hide memory and synchronization latencies on large-scale systems.

Graph Analytics↗
NASA NTRS1978

Tracking performance of unbalanced QPSK demodulators. I - Biphase Costas loop with passive arm filters

Unbalanced quadriphase-shift-keying (QPSK) is an attractive means for transmitting two digital data streams which in general have different average powers, data rates, and data formats. Previous analyses of the tracking performance of Costas loop demodulators of unbalanced QPSK have accounted only for the filtering effect produced by the loop's two arm filters on the equivalent additive noise perturbing the loop. When the bandwidth of these filters is selected on the basis of the order of the data rate, as is typical of optimum Costas loop design, the filtering degradations of the data modulations themselves and the cross-modulation noise produced by their multiplication in the loop often cannot be neglected. The purpose of this paper is to incorporate these additional filtering effects into the analysis. Many of the results obtained herein are in the form of closed-form expressions which can easily be evaluated numerically for design and performance prediction purposes.

Simon, M. K.↗
NASA NTRS1995

Data management and scientific integration within the Atmospheric Radiation Measurement Program

The Atmospheric Radiation Measurement (ARM) Program has been developed by the U.S. Department of Energy with the goal to improve the predictive capabilities of General Circulation Models (GCM's) in their treatment of clouds and radiative transfer effects. To achieve this goal, three experimental testbeds were designed for the deployment of instruments that will collect atmospheric data used to drive the GCM's. Each site, known as a Cloud and Radiation Testbed (CART), consists of a highly available, redundant data system for the collection of data from a variety of instrumentation. The first CART site was deployed in April 1992 in the Southern Great Plains (SGP), Lamont, Oklahoma, with the other two sites to follow in September 1995 in the Tropical Western Pacific and in 1997 on the North Slope of Alaska. Approximately 400 MB of data are transferred per day via the Internet from the SGP site to the ARM Experiment Center at Pacific Northwest Laboratory in Richland, Washington. The Experiment Center is central to the ARM data path and provides for the collection, processing, analysis, and delivery of ARM data. Data are received from the CART sites from a variety of instrumentation, observational systems, amd external data sources. The Experiment Center processes these data streams on a continuous basis to provide derived data products to the ARM Science Team in near real-time while providing a three-month running archive of data. A primary requirement of the ARM Program is to preserve and protect all data produced or acquired. This function is performed at Oak Ridge National Laboratory where leading edge technology is employed for the long-term storage of ARM data. The ARM Archive provides access to data for participation outside of the ARM Program. The ARM Program involves a collaborative effort by teams from various DOE National Laboratories, providing multi-disciplinary areas of expertise. This paper will discuss the collaborative methods in which the ARM teams translate the scientific goals of the Program into data products. By combining atmospheric scientists, systems engineers, and software engineers, the ARM Program has successfully designed and developed an environment where advances in understanding the parameterizations of GCM's can be made.

Gracio, Deborah K.↗
NASA NTRS1989

Fiber optic voice/data network

An asynchronous, high-speed, fiber optic local area network originally developed for tactical environments with additional benefits for other environments such as spacecraft, and the like. The network supports ordinary data packet traffic simultaneously with synchronous T1 voice traffic over a common token ring channel; however, the techniques and apparatus of this invention can be applied to any deterministic class of packet data networks, including multitier backbones, that must transport stream data (e.g., video, SAR, sensors) as well as data. A voice interface module parses, buffers, and resynchronizes the voice data to the packet network employing elastic buffers on both the sending and receiving ends. Voice call setup and switching functions are performed external to the network with ordinary PABX equipment. Clock information is passed across network boundaries in a token passing ring by preceeding the token with an idle period of non-transmission which allows the token to be used to re-establish a clock synchronized to the data. Provision is made to monitor and compensate the elastic receiving buffers so as to prevent them from overflowing or going empty.

Bergman, Larry A.↗
NASA NTRSDate not supplied

Global 3D Data Visualization and Analysis Platform With Advanced Machine Learning Capabilities in Support of Lunar Exploration

Introduction: The science goals for NASA’s Artemis program include: a) Understanding the character and origin of lunar polar volatiles, b) Conducting experimental science in the lunar environment and c) Investigating and mitigating exploration risks [1]. The permanently shadowed regions (PSRs) on the Lunar south pole are expected to host large quantities of water-ice and volatiles that are important for sustainable Lunar exploration [2]. There are several missions such as onboard Korea Pathfinder Lunar Orbiter (KPLO: Korean name Danuri) with onboard ShadowCam camera [3], Astrobotic Peregrine Mission One [4], and other efforts underway to obtain high resolution topographic, minerals, volatiles and other information on the moon. We envision a need in immediate future for platforms to integrate these data sets, provide rendering and visualization capabilities in the context of a 3D Lunar globe for easier information access and analysis. NASA's Celestial Mapping System (CMS) [5] is developed to address the need for 3D tools for planetary science investigations, mission planning, in-situ operations, in a 3D-first design constructed around a unified view of a planetary globe. At present CMS provides many critical functionalities that include: 1) equipment planning and optimized placement on Lunar surface 2) line-of-sight (LOS) analysis 3) powerful measurement tools based on 3D terrain with realistic 3D models to represent rovers, astronauts and equipment 4) visualization of derived mapping products (e.g. resource maps), and 5) a data engine for hosting new observations that are not available in other contemporary lunar data tools [5, 7]. Planetary Data Ingestion: CMS can consume and analyze data from locally hosted and external third party sources. It is compatible with Open Geospatial Consortium (OGC) data and file standards and currently integrates datasets from the Astrogeology Science Center of USGS. This includes global and local data acquired from NASA (LRO, Clementine, Lunar Orbiter) and JAXA (SELENE/Kaguya), with capability of integrating more datasets. In addition, users can specify other WMS-hosted data endpoints, which CMS can then query and stream data from automatically. To set-up an automated process for ingestion and accurate rendering, visualization and analysis of external 3rd party planetary datasets within CMS, we initiated the process of ingesting unique dataset of super-enhanced images of the permanently shadowed regions (PSRs) at the lunar poles which were produced by the Hyper-effective nOise Removal U-net Software (HORUS) tool [8]. This tool was developed to enhance the extremely low-light images of the interior of PSRs and provide the ability to see within these regions at and discern surface features (i.e. boulders and craters) down to 3 meters in size. We focused on the Nobile region on the Lunar south pole, selected site for VIPER mission and stitched several images to create a high-resolution map within one of the PSR of Nobile crater. Figure 1 shows the dark PSR zone form the original NAC layer of LRO as the base layer (left image) and the illuminated areas within that crater (center) which was created by ingesting and merging several of HORUS generated images. At present we employ a semi-automated process to ensure spatial accuracy and merger of several overlapping zones. However, we are in the process of completely automating this process by employing AI based techniques that would rank, sort, and stack the images based on their information density. The georectification of the images would employ selected features. Analysis on Ingested Planetary Datasets: Once an external planetary data-set is successfully ingested, georectified and merged seamlessly as a data-layer; CMS’ numerous analysis tools can be used on this data. A Line of Sight (LOS) tool has been developed for CMS which analyzes terrain profiles and obstructions to determine visibility for remote observers [5,6]. Figure 1 (right image) shows the viewshed analysis on the same PSR in the Nobile region. The yellow pin shows the observer location outside the PSR. The yellow area shows the visible part of PSR. The obstructed area with no visibility for the observer is shown in red. The Measurements tool allows the user to take area and distance measurements of features on the terrain using various shapes. Measurement type can be specified in a number of ways: Line, Path, Polygon, Circle, Ellipse, Square, Rectangle or Freehand. Once the shape is specified, elevation information can then be extracted along each of these shapes. Figure 2 (left) shows the measurements performed on a crater n illuminated PSR in Nobile region. The equipment placement tool allows the user to place a 3D equipment model at a desired location and analyze its coverage area. The equipment placement tool is coupled with LOS to determine the coverage. Figure 2 (right) shows an equipment placed on the Lunar terrain and it’s coverage area. The red rays are blocked sight lines and the green rays are non-obstructed sight lines with the cyan lines showing the point of intersection with the terrain. More details are provided in the video demonstrations in Reference 5. Overcoming Polar Distortions: 3D geospatial applications exhibit significant distortions in polar imagery due to several reasons: 1) distortions in the source imagery, 2) incompatible tessellation algorithms at the poles, and 3) map projections. We are leveraging new tessellation algorithms and reprojecting data using projections that are better suited for Lunar poles. The goal is to seamlessly switch to polar projections while maintaining 3D view and navigation.

Maps↗
NASA NTRSDate not supplied

Global 3D Data Visualization and Analysis Platform with Advanced Machine Learning Capabilities in Support of Lunar Exploration

Introduction: The science goals for NASA’s Artemis program include: a) Understanding the character and origin of lunar polar volatiles, b) Conducting experimental science in the lunar environment and c) Investigating and mitigating exploration risks. The permanently shadowed regions (PSRs) on the Lunar south pole are expected to host large quantities of water-ice and volatiles that are important for sustainable Lunar exploration. There are several missions such as onboard Korea Pathfinder Lunar Orbiter (KPLO: Korean name Danuri) with onboard ShadowCam camera, Astrobotic Peregrine Mission One [4], and other efforts underway to obtain high resolution topographic, minerals, volatiles and other information on the moon. We envision a need in immediate future for platforms to integrate these data sets, provide rendering and visualization capabilities in the context of a 3D Lunar globe for easier information access and analysis. NASA's Celestial Mapping System (CMS) is developed to address the need for 3D tools for planetary science investigations, mission planning, in-situ operations, in a 3D-first design constructed around a unified view of a planetary globe. At present CMS provides many critical functionalities that include: 1) equipment planning and optimized placement on Lunar surface 2) line-of-sight (LOS) analysis 3) powerful measurement tools based on 3D terrain with realistic 3D models to represent rovers, astronauts and equipment 4) visualization of derived mapping products (e.g. resource maps), and 5) a data engine for hosting new observations that are not available in other contemporary lunar data tools. Planetary Data Ingestion: CMS can consume and analyze data from locally hosted and external third party sources. It is compatible with Open Geospatial Consortium (OGC) data and file standards and currently integrates datasets from the Astrogeology Science Center of USGS. This includes global and local data acquired from NASA (LRO, Clementine, Lunar Orbiter) and JAXA (SELENE/Kaguya), with capability of integrating more datasets. In addition, users can specify other WMS-hosted data endpoints, which CMS can then query and stream data from automatically. To set-up an automated process for ingestion and accurate rendering, visualization and analysis of external 3rd party planetary datasets within CMS, we initiated the process of ingesting unique dataset of super-enhanced images of the permanently shadowed regions (PSRs) at the lunar poles which were produced by the Hyper-effective nOise Removal U-net Software (HORUS) tool. This tool was developed to enhance the extremely low-light images of the interior of PSRs and provide the ability to see within these regions at and discern surface features (i.e. boulders and craters) down to 3 meters in size. We focused on the Nobile region on the Lunar south pole, selected site for VIPER mission and stitched several images to create a high-resolution map within one of the PSR of Nobile crater. Figure 1 shows the dark PSR zone form the original NAC layer of LRO as the base layer (left image) and the illuminated areas within that crater (center) which was created by ingesting and merging several of HORUS generated images. At present we employ a semi-automated process to ensure spatial accuracy and merger of several overlapping zones. However, we are in the process of completely automating this process by employing AI based techniques that would rank, sort, and stack the images based on their information density. The georectification of the images would employ selected features. Analysis on Ingested Planetary Datasets: Once an external planetary data-set is successfully ingested, georectified and merged seamlessly as a data-layer; CMS’ numerous analysis tools can be used on this data. A Line of Sight (LOS) tool has been developed for CMS which analyzes terrain profiles and obstructions to determine visibility for remote observers. Figure 1 (right image) shows the viewshed analysis on the same PSR in the Nobile region. The yellow pin shows the observer location outside the PSR. The yellow area shows the visible part of PSR. The obstructed area with no visibility for the observer is shown in red. The Measurements tool allows the user to take area and distance measurements of features on the terrain using various shapes. Measurement type can be specified in a number of ways: Line, Path, Polygon, Circle, Ellipse, Square, Rectangle or Freehand. Once the shape is specified, elevation information can then be extracted along each of these shapes. Figure 2 (left) shows the measurements performed on a crater n illuminated PSR in Nobile region. The equipment placement tool allows the user to place a 3D equipment model at a desired location and analyze its coverage area. The equipment placement tool is coupled with LOS to determine the coverage. Figure 2 (right) shows an equipment placed on the Lunar terrain and it’s coverage area. The red rays are blocked sight lines and the green rays are non-obstructed sight lines with the cyan lines showing the point of intersection with the terrain. More details are provided in the video demonstrations in Reference 5. Overcoming Polar Distortions: 3D geospatial applications exhibit significant distortions in polar imagery due to several reasons: 1) distortions in the source imagery, 2) incompatible tessellation algorithms at the poles, and 3) map projections. We are leveraging new tessellation algorithms and reprojecting data using projections that are better suited for Lunar poles. The goal is to seamlessly switch to polar projections while maintaining 3D view and navigation.

Maps↗
DOE OSTI2025

Streaming Compression of Scientific Data via Weak-SINDy

Here, in this paper, a streaming weak-SINDy algorithm is developed specifically for compressing streaming scientific data. The production of scientific data, either via simulation or experiments, is undergoing a stage of exponential growth, which makes data compression important and often necessary for storing and utilizing large scientific data sets. As opposed to classical “offline” compression algorithms that perform compression on a readily available data set, streaming compression algorithms compress data “online” while the data generated from simulation or experiments is still flowing through the system. This feature makes streaming compression algorithms well suited for scientific data compression, where storing the full data set offline is often infeasible. This work proposes a new streaming compression algorithm, streaming weak-SINDy, which takes advantage of the underlying data characteristics during compression. The streaming weak-SINDy algorithm constructs feature matrices and target vectors in the online stage via a streaming integration method in a memory efficient manner. The feature matrices and target vectors are then used in the offline stage to build a model through a regression process that aims to recover equations that govern the evolution of the data. For compressing high-dimensional streaming data, we adopt a streaming proper orthogonal decomposition (POD) process to reduce the data dimension and then use the streaming weak-SINDy algorithm to compress the temporal data of the POD expansion. We propose modifications to the streaming weak-SINDy algorithm to accommodate the dynamically updated POD basis. By combining the built model from the streaming weak-SINDy algorithm and a small amount of data samples, the full data flow could be reconstructed accurately at a low memory cost, as shown in the numerical tests.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Machine Learning and Social Media to Mine and Disseminate Big Scientific Data

One of the challenges in supplying the communities with wider access to scientific databases is the need for knowledge of database languages like Structured Query Language (SQL). Although the SQL language has been published in many forms, not everybody is able to write SQL queries. Another challenge is that it might not be practical to make the public aware of the structure of databases. There is a need for novice users to query relational databases using their natural language. To solve this problem, many natural language interfaces to structured databases have been developed. The goal is to provide a more intuitive method for generating database queries and delivering responses. Through social media, which makes it possible to interact with a wide section of the population, and with the help of natural language processing, researchers at the Atmospheric Radiation Measurement (ARM) Data Center at Oak Ridge National Laboratory (ORNL) have developed a concept to enable easy search and retrieval of data from several environmental data centers for the scientific community through social media.Using a machine learning framework that maps natural language text to thousands of datasets, instruments, variables, and data streams, the prototype system would allow users to request data through Twitter and receive a link (via tweet) to applicable data results on the project's search catalog tailored to their key words. This automated identification of relevant data from various petascale archives at ORNL could increase convenience, access, and use of the project's data by the broader community. In this paper we discuss how some data-intensive projects at ORNL are using innovative ways to help in data discovery.

Devarakonda, Ranjeet↗
DOE OSTI2021

In Situ Transmission Electron Microscopy: Signal processing challenges and examples

Transmission electron microscopy (TEM) is a powerful tool for imaging material structure and characterizing material chemistry. Recent advances in data collection technology for TEM have enabled high-volume and high-resolution data collection at a microsecond frame rate. Here, taking advantage of these advances in data collection rates requires the development and application of data processing tools, including image analysis, feature extraction, and streaming data processing techniques. In this article, we highlight a few areas in materials science that have benefited from combining signal processing and statistical analysis with data collection capabilities in TEM and present a future outlook on opportunities of integrating signal processing with automated TEM data analysis.

36 MATERIALS SCIENCE↗