Search NASASearch

SEARCH · Search NASA

Results for “Streaming Algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Entropy encoding applied to a video DM bit stream

This paper investigates bandwidth compression of delta modulated video signals using a variety of novel techniques. The video signal is encoded at 16 MHz with an adaptive delta modulator (ADM) using the Song video mode algorithm. One method of achieving bandwidth compression is through the detection and reduction of steady state patterns in the DM bit stream. Further reduction is obtained by deleting the horizontal and vertical drive information. Finally, a modified ADM algorithm is introduced to increase the frequency of steady state patterns. When the modified ADM is combined with entropy coding a 30-50% data compression can be realized.

Barba, J.

Lightning Jump Algorithm Development for the GOES·R Geostationary Lightning Mapper

Current work on the lightning jump algorithm to be used in GOES‐R Geostationary Lightning Mapper (GLM)'s data stream is multifaceted due to the intricate interplay between the storm tracking, GLM proxy data, and the performance of the lightning jump itself. This work outlines the progress of the last year, where analysis and performance of the lightning jump algorithm with automated storm tracking and GLM proxy data were assessed using over 700 storms from North Alabama. The cases analyzed coincide with previous semi‐objective work performed using total lightning mapping array (LMA) measurements in Schultz et al. (2011). Analysis shows that key components of the algorithm (flash rate and sigma thresholds) have the greatest influence on the performance of the algorithm when validating using severe storm reports. Automated objective analysis using the GLM proxy data has shown probability of detection (POD) values around 60% with false alarm rates (FAR) around 73% using similar methodology to Schultz et al. (2011). However, when applying verification methods similar to those employed by the National Weather Service, POD values increase slightly (69%) and FAR values decrease (63%). The relationship between storm tracking and lightning jump has also been tested in a real‐time framework at NSSL. This system includes fully automated tracking by radar alone, real‐time LMA and radar observations and the lightning jump. Results indicate that the POD is strong at 65%. However, the FAR is significantly higher than in Schultz et al. (2011) (50‐80% depending on various tracking/lightning jump parameters) when using storm reports for verification. Given known issues with Storm Data, the performance of the real‐time jump algorithm is also being tested with high density radar and surface observations from the NSSL Severe Hazards Analysis & Verification Experiment (SHAVE).

Schultz. E.

Simulation of an inductively coupled plasma with a two-dimensional Darwin particle-in-cell code

A two-dimensional particle-in-cell code for the simulation of low-frequency electromagnetic processes in laboratory plasmas has been developed. The code uses the Darwin method omitting the electromagnetic wave propagation. The Darwin method separates the electric field into solenoidal and irrotational parts. The solenoidal electric field is calculated with a new algorithm based on the equation for the electric field vorticity. The system of linear equations in the new algorithm is readily solved using a standard iterative method. The irrotational electric field is the electrostatic field calculated with the direct implicit algorithm. The code is verified by reproducing the two-stream instability, electron electromagnetic waves, and shear Alfvén waves. The code is applied to simulate an inductively coupled plasma with the driving current flowing around the plasma region. In this simulation, a ring of dense plasma forms at the initial stage but then the density becomes maximal in the center and decays monotonically toward the walls. The skin effect is in the transitional mode between local and non-local, and the electron velocity distribution function is non-Maxwellian.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY

Basic data requirements for microwave radiometer systems

Microwave radiometry has emerged over the last two decades to become an integral part of the field of environmental remote sensing. Numerous investigations were conducted to evaluate the use of microwave radiometry for atmospheric, oceanographic, hydrological, and geological applications. Remote sensing of the earth using microwave radiometry began in 1968 by the Soviet satellite Cosmos 243, which included four microwave radiometers (Ulably, 1981). Since then, microwave radiometers were included onboard many spacecraft, and were used to infer many physical parameters. Some of the basic concepts of radiometric emission and measurement will be discussed. Several radiometer systems are presented and an overview of their operation is discussed. From the description of the radiometer operation the data stream required from the radiometer and the general type of algorithm required for the measurement is discussed.

Lawrence, R. W.

Data Mining at NASA: From Theory to Applications

This slide presentation demonstrates the data mining/machine learning capabilities of NASA Ames and Intelligent Data Understanding (IDU) group. This will encompass the work done recently in the group by various group members. The IDU group develops novel algorithms to detect, classify, and predict events in large data streams for scientific and engineering systems. This presentation for Knowledge Discovery and Data Mining 2009 is to demonstrate the data mining/machine learning capabilities of NASA Ames and IDU group. This will encompass the work done re cently in the group by various group members.

Srivastava, Ashok N.

Computation of Domain-Averaged Shortwave Irradiance by a One-Dimensional Algorithm Incorporating Correlations between Optical Thickness and Direct Incident Radiation

A one-dimensional radiative transfer algorithm that accounts for correlations between the optical thickness and the incident direct solar radiation is developed to compute the domain-averaged shortwave irradiance profile. It divides the direct irradiance into four components and treats the direct irradiance in two separate, clear and cloudy columns to account for the fact that clouds attenuate the direct irradiance more than clear-sky. The horizontal inhomogeneity of clouds in the cloudy column is treated by the gamma weighted two-stream approximation, which assumes that the optical thickness of clouds follows a gamma distribution. The algorithm inputs the cloud fraction, cumulative cloud fraction as a function of height, and a parameter expressing the shape of the probability density function of the cloud optical thickness distribution in addition to inputs required for a two-stream radiative transfer model. These cloud property inputs can be obtained using ground- and satellite-based instruments. Therefore, the algorithm can treat realistic cloud overlap features and horizontal inhomogeneity of clouds in a framework of one- dimensional radiative transfer. Heating rates computed by the algorithm using cloud fields generated by cloud resolving models agree with those computed with a Monte Carlo model. If optical properties in computational layers that divide a vertically extensive cloud are correlated, the irradiance profile computed by the algorithm further improves.

Kato, S.

Support for the Harvard University Water Vapor and Total Water Instruments for the 2004 NASA WB57 Middle Latitude Cirrus Experiment

In order to improve our understanding of the role clouds play in the climate system, NASA is investing considerable effort in characterizing clouds with instruments ranging from passive remote sensors on board the EOS platforms, to the forthcoming active remote sensors on Cloudsat and Calipso. These missions, when taken together, have the capacity to advance our understanding of the coupling between various components of the hydrologic cycle and the atmospheric circulation, and hold the additional potential of leading to significant improvements in the characterization of cloud feedbacks in global models. This is especially true considering that several of these platforms will be flown in an identical orbit within several minutes of one another-a constellation of satellites known as the A-Train. The algorithms that are being implemented and developed to convert these new data streams from radiance and reflectivity measurements into geophysical parameters invariably rely on some set of simplifymg assumptions and empirical constants. Uncertainties in these relationships lead to poorly understood random and systematic errors in the retrieved properties. This lack of understanding introduces ambiguity in interpreting the data and in using the global data sets for their intended purposes. In light of this, a series of flights with the W57F was proposed to address certain specific issues related to the basic properties of mid latitude cirrus clouds: the NASA WE357 Middle Latitude Cirrus Experiment ("MidCiX"). The science questions addressed are: 1) Can cloud property retrieval algorithms developed for A-Train active and passive remote sensing measurements accurately characterize the microphysical properties of synoptic and convectively generated cirrus cloud systems? 2) What are the relationships between the cirrus particle mass, projected area, and particle size spectrum in various genre of cirrus clouds? 3) Does the present compliment of state of the art in situ cloud probes provide the level of precision and accuracy needed to develop and validate algorithms and to contribute to our understanding of the characteristics and microphysical processes operating in cirrus clouds?

Anderson, James G.

PvaPy streaming framework for real-time data processing

User facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynamically and modify them in real time. In turn, this requires a computing framework for real-time processing capable of moving data quickly from the detector to local or remote computing resources, processing data, and returning results to users. In this paper, we discuss the streaming framework built on top of PvaPy, a Python API for the EPICS pvAccess protocol. We describe the framework architecture and capabilities, and discuss scientific use cases and applications that benefit from streaming workflows implemented on top of this framework. We also illustrate the framework's performance in terms of achievable data-processing rates for various detector image sizes.

EPICS pvAccess

Numerical studies of porous airfoils in transonic flow

A numerical tool is constructed to examine the effects of a porous surface on transonic airfoil performance and to help understand the flow structure of passive shockwave/boundary layer interactions. The porous region is located near the shock with a cavity underneath it. This study is composed of two parts. Solved in the first part, with an inviscid-flow approach, is the transonic full-potential equation associated with transpiration boundary conditions which are obtained from porosity modeling. The numerical results indicate that a porous airfoil has a wave drag lower than that of a solid airfoil. The observed lambda-shock structure in the wind-tunnel testing can be predicted. Furthermore, the lift could be increased with an appropriate porosity distribution. In the second part of this work, the modified version of either an interactive boundary layer (IBL) algorithm or a thin-layer Navier-Stokes (TLNS) algorithm is used to study the outer flow, while a stream-function formulation is used to model the inner flow in the shallow cavity. The coupling procedure at the porous surface is based on Darcy's law and the assumption of a constant total pressure in the cavity. In addition, a modified Baldwin-Lomax turbulence model is used to describe the transpired turbulent boundary layer in the TLNS approach, while the Cebeci turbulence model is used in the IBL approach. According to the present analysis, a porous surface can reduce the wave drag appreciably, but can also increase the viscous losses. As has been observed experimentally, the numerical results indicate that the total drag is reduced at higher Mach numbers and increased at lower Mach numbers when the angles of attack are small. Furthermore, the streamline pattern of passive shock/boundary layer interaction are revealed.

Chow, C. Y.

Laser Remote Sensing from ISS: CATS Cloud and Aerosol Level 2 Data Products (Heritage Edition)

With the recent launch of the Cloud-Aerosol Transport System (CATS) we have the opportunity to acquire a continuous record of space based lidar measurements spanning from the Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) era to the start of the EarthCARE mission. Utilizing existing well-validated science algorithms from the CALIPSO mission, we will ingest the CATS data stream and deliver high-quality lidar data sets to the user community at the earliest possible opportunity. In this paper we present an overview of procedures necessary to generate CALIPSO-like lidar level 2 data products from the CATS level 1 data products.

cloud-aerosol lidar

Data Assimilation for Robust UQ Within Agent-Based Simulation on HPC Systems

Agent-based simulation provides a powerful tool for in silico system modeling. However, these simulations do not provide built-in methods for uncertainty quantification (UQ). Within these types of models a typical approach to UQ is to run multiple realizations of the model then compute aggregate statistics. This approach is limited due to the compute time required for a solution. When faced with an emerging biothreat, public health decisions need to be made quickly and solutions for integrating near real-time data with analytic tools are needed. We propose an integrated Bayesian UQ framework for agent-based models based on sequential Monte Carlo sampling. Given streaming or static data about the evolution of an emerging pathogen this Bayesian framework provides a distribution over the parameters governing the spread of a disease through a population. These estimates of the spread of a disease may be provided to public health agencies seeking to abate the spread. By coupling agent-based simulations with Bayesian modeling in a data assimilation, our proposed framework provides a powerful tool for modeling dynamical systems in silico. We propose a method which reduces model error and provides a range of realistic possible outcomes. Moreover, our method addresses two primary limitations of ABMs: the lack of UQ and an inability to assimilate data. Our proposed framework combines the flexibility of an agent-based model with UQ provided by the Bayesian paradigm in a workflow which scales well to HPC systems. We provide algorithmic details and results on a simulated outbreak with both static and streaming data.

Spannaus, Adam [ORNL] (ORCID:0000000225213657)

Generating Accurate and Consistent Top-Of-Atmosphere Reflectance Products from the New Generation Geostationary Satellite Sensors

GeoNEX is a collaborative project by scientists from NASA, NOAA, JAXA, and other organizations around the world with the purpose of generating a suite of Earth-monitoring products using data streams from the latest geostationary (GEO) sensors including the GOES-16/17 ABI and the Himawari-8/9 AHI. An accurate and consistent top-of-atmosphere (TOA) reflectance product, in particular the bidirectional reflectance factor (BRF), is the starting point in the scientific processing chain. We describe the main considerations and corresponding algorithms in generating the GeoNEX TOA BRF product. First, a special advantage of geostationary data streams is their high temporal resolution (~10 minutes per full-disk scan), providing a key source of information for many downstream products. To fully utilize this high temporal frequency demands a high georegistration accuracy for every acquired image. Our analysis shows that there can be substantial georegistration uncertainties in both GOES and Himawari L1b data which we addressed by implementing a phase-based correction algorithm to remove residual errors. Second, geostationary sensors have distinct illumination-view geometry features in that the solar angle changes for every pixel. Therefore, to accurately derive a BRF requires a solar position algorithm and the estimation of the pixel-wise acquisition time within an uncertainty of 10 seconds. Third, we discuss the measures we adopted to check and correct residual radiometric calibration issues of individual sensors to enable time-series analysis as well as the cross calibration between different satellite sensors (including those from low-Earth orbit). Finally, we also explain the rationale for the choice of the global grid/tile system of the GeoNEX TOA BRF product.

Wang, Weile

The Operational MODIS Cloud Optical and Microphysical Property Product: Overview of the Collection 6 Algorithm and Preliminary Results

Operational Moderate Resolution Imaging Spectroradiometer (MODIS) retrievals of cloud optical and microphysical properties (part of the archived products MOD06 and MYD06, for MODIS Terra and Aqua, respectively) are currently being reprocessed along with other MODIS Atmosphere Team products. The latest "Collection 6" processing stream, which is expected to begin production by summer 2012, includes updates to the previous cloud retrieval algorithm along with new capabilities. The 1 km retrievals, based on well-known solar reflectance techniques, include cloud optical thickness, effective particle radius, and water path, as well as thermodynamic phase derived from a combination of solar and infrared tests. Being both global and of high spatial resolution requires an algorithm that is computationally efficient and can perform over all surface types. Collection 6 additions and enhancements include: (i) absolute effective particle radius retrievals derived separately from the 1.6 and 3.7 !-lm bands (instead of differences relative to the standard 2.1 !-lm retrieval), (ii) comprehensive look-up tables for cloud reflectance and emissivity (no asymptotic theory) with a wind-speed interpolated Cox-Munk BRDF for ocean surfaces, (iii) retrievals for both liquid water and ice phases for each pixel, and a subsequent determination of the phase based, in part, on effective radius retrieval outcomes for the two phases, (iv) new ice cloud radiative models using roughened particles with a specified habit, (v) updated spatially-complete global spectral surface albedo maps derived from MODIS Collection 5, (vi) enhanced pixel-level uncertainty calculations incorporating additional radiative error sources including the MODIS L1 B uncertainty index for assessing band and scene-dependent radiometric uncertainties, (v) and use of a new 1 km cloud top pressure/temperature algorithm (also part of MOD06) for atmospheric corrections and low cloud non-unity emissivity temperature adjustments.

Platnick, Steven

A packet data compressor

We are in the preliminary stages of creating an operational system for losslessly compressing packet data streams. The end goal is to reduce costs. Real world constraints include transmission in the presence of error, tradeoffs between the costs of compression and the costs of transmission and storage, and imperfect knowledge of the data streams to be transmitted. The overall method is to bring together packets of similar type, split the data into bit fields, and test a large number of compression algorithms. Preliminary results are very encouraging, typically offering compression factors substantially higher than those obtained with simpler generic byte stream compressors, such as Unix Compress and HA 0.98.

Grunes, Mitchell R.

Adaptive System Modeling for Spacecraft Simulation

This invention introduces a methodology and associated software tools for automatically learning spacecraft system models without any assumptions regarding system behavior. Data stream mining techniques were used to learn models for critical portions of the International Space Station (ISS) Electrical Power System (EPS). Evaluation on historical ISS telemetry data shows that adaptive system modeling reduces simulation error anywhere from 50 to 90 percent over existing approaches. The purpose of the methodology is to outline how someone can create accurate system models from sensor (telemetry) data. The purpose of the software is to support the methodology. The software provides analysis tools to design the adaptive models. The software also provides the algorithms to initially build system models and continuously update them from the latest streaming sensor data. The main strengths are as follows: Creates accurate spacecraft system models without in-depth system knowledge or any assumptions about system behavior. Automatically updates/calibrates system models using the latest streaming sensor data. Creates device specific models that capture the exact behavior of devices of the same type. Adapts to evolving systems. Can reduce computational complexity (faster simulations).

Thomas, Justin

rustpix

rustpix is a high-performance, open-source Rust library with first-class Python bindings (via PyO3) for processing pixel-detector data in neutron imaging. It targets time-stamping detectors such as Timepix3 (TPX3) at ORNL's Spallation Neutron Source (VENUS beamline), where each detected neutron deposits charge across a cluster of pixels within a very high-rate event stream (96M+ hits/sec). rustpix parses TPX3 event data in parallel using memory-mapped I/O, offers four interchangeable clustering algorithms (ABS adjacency-based search, DBSCAN, graph/union-find connected components, and a parallel grid method), and extracts weighted, super-resolved centroids to produce neutron-event lists. A streaming architecture lets it process files larger than available memory. rustpix is distributed as a pip-installable Python package (with NumPy integration), Rust crates, a command-line tool, and an interactive GUI; it writes HDF5, Apache Arrow, and CSV; and it is designed to extend to TPX4 and other detector types. Released as open-source under the MIT License.

Zhang, Chen [Oak Ridge National Laboratory (ORNL),

Earth Independent Medical Operations (EIMO) Datascope: Challenges and Potential Solutions

Data flows and storage/retrieval capacity are severely constrained during missions in space and challenges will become even greater during exploration class missions. There is a need for an artificial intelligence (AI)-based clinical decision support system (CDSS) to monitor and analyze data to provide real-time consultative support for crew medical officer (CMO) decision-making. EIMO is defined as the gradual transition of medical care and decision making from terrestrial to space-based assets, enabling support of astronaut health and performance and reducing overall mission risk. While a hallmark of this paradigm shift from low-earth orbit is that on-board care will increasingly become the responsibility of the astronauts for primary management and decision making, terrestrial assets will continue to be paramount in pre-mission screening and planning, as well as prevention, health maintenance and long-term care contingencies. New capabilities and systems that enable progressively more robust and resilient systems and crews will be necessary to reduce risk and increase probability of deep space exploration mission success. An aspiration for EIMO is to develop AI-enhanced solutions for analysis of crew health & performance data and to facilitate clinical decision support for autonomous medical operations. A “system of systems” approach is envisioned whereby EIMO will deploy AI-supported natural language processing and machine learning (ML) techniques to utilize embedded reference databases and real-time data streams [input vectors] from multiple data sources. Constituent input vectors may include environmental controls, countermeasures data, behavioral data, physiologic wearables, point-of-care laboratory tests, personalized medical records, inventory trade space risk assessments, COTS medical databases, and ground support inputs. An ideal AI capability would possess trained fusion algorithms to cross reference input vectors with medical ‘knowledge’ [cultivated database] to stratify relevant data streams for predictive and actionable capabilities. In addition, EIMO will feature mobility, in that it can be accessed and can push/pull data within and between multiple vehicles/habitats. Large amounts and variable sources of data can be leveraged to diagnose, inform treatment strategies, and potentially predict medical events and performance decrements. Inclusion of advanced training tools using extended reality will enable increasingly autonomous medical care to aid a CMO when ground support is unavailable or time-delayed beyond required action window, e.g., emergent medical situations. EIMO CDSS would require very large datasets to train pre-flight and significant amounts of data are needed to support ML via in-flight CDSS operations. An additional challenge will be to find sufficient data to train a model relevant to astronaut demographics. The rapid, accelerating evolution of this field creates a propitious solution space to leverage multi-modal AI through public-private partnership(s). The status of multi-modal AI systems today would preclude their use for long duration missions as they remain unreliable and are subject to “digital hallucinations” and other errors that could pose operational risk. A federated labs structure is being considered to test and optimize data flow from the multiple input vectors leading to field testing in suitable ground/flight analogs. Critical to the success of an EIMO CDSS will be integration and interoperability and success will be defined by a system that can serve as an in-flight medical consult for the CMO providing critical support during medical contingencies. Benefits to terrestrial medicine may be significant as an outflow of the EIMO medical system, particularly for remote areas and communities lacking significant infrastructure, personnel and resources.

J Lemery

Earth Independent Medical Operations (EIMO) Datascope: Challenges and Potential Solutions

Data flows and storage/retrieval capacity are severely constrained during missions in space and challenges will become even greater during exploration class missions. There is a need for an artificial intelligence (AI)-based clinical decision support system (CDSS) to monitor and analyze data to provide real-time consultative support for crew medical officer (CMO) decision-making. EIMO is defined as the gradual transition of medical care and decision making from terrestrial to space-based assets, enabling support of astronaut health and performance and reducing overall mission risk. While a hallmark of this paradigm shift from low-earth orbit is that on-board care will increasingly become the responsibility of the astronauts for primary management and decision making, terrestrial assets will continue to be paramount in pre-mission screening and planning, as well as prevention, health maintenance and long-term care contingencies. New capabilities and systems that enable progressively more robust and resilient systems and crews will be necessary to reduce risk and increase probability of deep space exploration mission success. An aspiration for EIMO is to develop AI-enhanced solutions for analysis of crew health & performance data and to facilitate clinical decision support for autonomous medical operations. A “system of systems” approach is envisioned whereby EIMO will deploy AI-supported natural language processing and machine learning (ML) techniques to utilize embedded reference databases and real-time data streams [input vectors] from multiple data sources. Constituent input vectors may include environmental controls, countermeasures data, behavioral data, physiologic wearables, point-of-care laboratory tests, personalized medical records, inventory trade space risk assessments, COTS medical databases, and ground support inputs. An ideal AI capability would possess trained fusion algorithms to cross reference input vectors with medical ‘knowledge’ [cultivated database] to stratify relevant data streams for predictive and actionable capabilities. In addition, EIMO will feature mobility, in that it can be accessed and can push/pull data within and between multiple vehicles/habitats. Large amounts and variable sources of data can be leveraged to diagnose, inform treatment strategies, and potentially predict medical events and performance decrements. Inclusion of advanced training tools using extended reality will enable increasingly autonomous medical care to aid a CMO when ground support is unavailable or time-delayed beyond required action window, e.g., emergent medical situations. EIMO CDSS would require very large datasets to train pre-flight and significant amounts of data are needed to support ML via in-flight CDSS operations. An additional challenge will be to find sufficient data to train a model relevant to astronaut demographics. The rapid, accelerating evolution of this field creates a propitious solution space to leverage multi-modal AI through public-private partnership(s). The status of multi-modal AI systems today would preclude their use for long duration missions as they remain unreliable and are subject to “digital hallucinations” and other errors that could pose operational risk. A federated labs structure is being considered to test and optimize data flow from the multiple input vectors leading to field testing in suitable ground/flight analogs. Critical to the success of an EIMO CDSS will be integration and interoperability and success will be defined by a system that can serve as an in-flight medical consult for the CMO providing critical support during medical contingencies. Benefits to terrestrial medicine may be significant as an outflow of the EIMO medical system, particularly for remote areas and communities lacking significant infrastructure, personnel and resources.

Medical Operations