Search NASA⌕ Search

SEARCH · Search NASA

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,207 records · Page 67

Predictive Interfaces for Long-Distance Tele-Operations

We address the development of predictive tele-operator interfaces for humanoid robots with respect to two basic challenges. Firstly, we address automating the transition from fully tele-operated systems towards degrees of autonomy. Secondly, we develop compensation for the time-delay that exists when sending telemetry data from a remote operation point to robots located at low earth orbit and beyond. Humanoid robots have a great advantage over other robotic platforms for use in space-based construction and maintenance because they can use the same tools as astronauts do. The major disadvantage is that they are difficult to control due to the large number of degrees of freedom, which makes it difficult to synthesize autonomous behaviors using conventional means. We are working with the NASA Johnson Space Center's Robonaut which is an anthropomorphic robot with fully articulated hands, arms, and neck. We have trained hidden Markov models that make use of the command data, sensory streams, and other relevant data sources to predict a tele-operator's intent. This allows us to achieve subgoal level commanding without the use of predefined command dictionaries, and to create sub-goal autonomy via sequence generation from generative models. Our method works as a means to incrementally transition from manual tele-operation to semi-autonomous, supervised operation. The multi-agent laboratory experiments conducted by Ambrose et. al. have shown that it is feasible to directly tele-operate multiple Robonauts with humans to perform complex tasks such as truss assembly. However, once a time-delay is introduced into the system, the rate of tele\ioperation slows down to mimic a bump and wait type of activity. We would like to maintain the same interface to the operator despite time-delays. To this end, we are developing an interface which will allow for us to predict the intentions of the operator while interacting with a 3D virtual representation of the expected state of the robot. The predictive interface anticipates the intention of the operator, and then uses this prediction to initiate appropriate sub-goal autonomy tasks.

Wheeler, Kevin R.↗

Integrated inertial stellar attitude sensor

An integrated inertial stellar attitude sensor for an aerospace vehicle includes a star camera system, a gyroscope system, a controller system for synchronously integrating an output of said star camera system and an output of said gyroscope system into a stream of data, and a flight computer responsive to said stream of data for determining from the star camera system output and the gyroscope system output the attitude of the aerospace vehicle.

Brady, Tye M.↗

Closing the Gap between FAIR Data Repositories and Hierarchical Data Formats

Many in the scientific community, particularly in publicly funded research, are pushing to adhere to more accessible data standards to maximize the findability, accessibility, interoperability, and reusability (FAIR) of scientific data, especially with the growing prevalence of machine learning augmented research. Online FAIR data repositories, such as the Open Science Framework (OSF), help facilitate the adoption of these standards by providing frameworks for storage, access, search, APIs, and other features that create organized hubs of scientific data. However, the wider acceptance of such repositories is hindered by the lack of support of hierarchical data formats, such as Technical Data Management Streaming (TDMS) and Hierarchical Data Format 5 (HDF5), that many researchers rely on to organize their datasets. Various tools and strategies should be used to allow hierarchical data formats, FAIR data repositories, and scientific organizations to work more seamlessly together. A pilot project at Los Alamos National Laboratory (LANL) addresses the disconnect between them by integrating the OSF FAIR data repository with hierarchical data renderers, extending support for additional file types in their framework. The multifaceted interactive renderer displays a tree of metadata alongside a table and plot of the data channels in the file. This allows users to quickly and efficiently load large and complex data files directly in the OSF webapp. Users who are browsing files can quickly and intuitively see the files in the way they or their colleagues structured the hierarchical form and immediately grasp their contents. This solution helps bridge the gap between hierarchical data storage techniques and FAIR data repositories, making both of them more viable options for scientific institutions like LANL which have been put off by the lack of integration between them.

97 MATHEMATICS AND COMPUTING↗

Fast Vector-Quantizing Data Compressor

Proposed encoding digital electronic data-processing system compresses 7-Mb/s input stream of data into 1-Mb/s output stream. Intended to reduce flow of data from orbiting synthetic-aperture-radar (SAR) system to rate low enough for transmission to ground. Features include reliability, light weight, and low power. Proves advantageous in such terrestrial applications as compression of speech and of video signals.

Chang, Chi-Yung↗

Seismology in the Cloud: A New Streaming Workflow

Data-intensive research in seismology is experiencing a recent boom, driven in part by large volumes of available data and advances in the growing field of data science. However, there are significant barriers to processing large data volumes, such as long retrieval times from data repositories, complex data management, and limited computational resources. New tools and platforms have reduced the barriers to entry for scientific cluster computing, including the maturation of the commercial cloud as an accessible instrument for research. Here, we build a customized research cluster in the cloud to test a new workflow for large-scale seismic analysis, in which data are processed as a stream (retrieved on-the-fly and acted upon without storing), with data from the Incorporated Research Institutions for Seismology Data Management Center. We use this workflow to deploy a spectral peak detection algorithm over 5.6 TB of compressed continuous seismic data from 2074 stations of the USArray Transportable Array EarthScope network. Using a 50-node cluster in the cloud, we completed the noise survey in 80 hr, with an average data throughput of 1.7 GB per minute. By varying cluster sizes, we find the scaling of our analysis to be sublinear, due to a combination of algorithmic limitations and data center response times. The cloud-based streaming workflow represents an order-of-magnitude increase in acquisition and processing speed compared to a traditional download-store-process workflow, and offers the additional benefits of employing a flexible, accessible, and widely used computing architecture. It is limited, however, due to its reliance on Internet transfer speeds and data center service capacity, and may not work well for repeated analyses or those for which even higher data throughputs are needed. These research applications will require a new class of cloud-native approaches in which both data and analysis are in the cloud.

58 GEOSCIENCES↗

Walker Branch Watershed: Weekly Stream Water Chemistry

This data set reports weekly stream water chemistry in the West and East Forks of Walker Branch Watershed (WBW). The long-term Walker Branch stream chemistry monitoring is intended to provide data on watershed output of chemicals via streamflow and long-term changes in stream chemical composition. This data set contains two data files of stream water chemistry collected at weekly time intervals from 1989 through 2013. Weekly sampling of stream water started in 1989 in the West Fork and in 1995 in the East Fork. Measured parameters include water level at the weir and stream discharge, water temperature, specific conductivity, pH, alkalinity, and dissolved organic carbon (DOC), soluble reactive phosphorus (SRP), total dissolved phosphorus (TDP), ammonium-N, nitrate+nitrite-N, total dissolved nitrogen (TDN), anion (chloride and sulfate), and cation and trace metal (calcium, magnesium, sodium, potassium, iron, manganese, silicon, aluminum, barium, cadmium, nickel, lead, strontium, zinc, copper, molybdenum) concentrations. The Walker Branch Watershed is located on the U.S. Department of Energy's Oak Ridge Reservation in eastern Tennessee.

54 ENVIRONMENTAL SCIENCES↗

Preliminary estimates of Gulf Stream characteristics from TOPEX data and a precise gravimetric geoid

TOPEX sea surface height data has been used, with a gravimetric geoid, to calculate sea surface topography across the Gulf Stream. This topography was initially computed for nine tracks on cycles 21 to 29. Due to inaccurate geoid undulations on one track, results for eight tracks are reported. The sea surface topography estimates were used to calculate parameters that describe Gulf Stream characteristics from two models of the Gulf Stream. One model was based on a Gaussian representation of the velocity while the other was a hyperbolic representation of velocity or the sea surface topography. The parameters of the Gaussian velocity model fit were a width parameter, a maximum velocity value, and the location of the maximum velocity. The parameters of the hyperbolic sea surface topography model were the width, the height jump, position, and sea surface topography at the center of the stream. Both models were used for the eight tracks and nine cycles studied. Comparisons were made between the width parameters, the maximum velocities, and the height jumps. Some of the parameter estimates were found to be highly (0.9) correlated when the hyperbolic sea surface topography fit was carried out, but such correlations were reduced for either the Gaussian velocity fits or the hyperbolic velocity model fit. A comparison of the parameters derived from 1-year TOPEX data showed good agreement with values derived by Kelly (1991) using 2.5 years of Geosat data near 38 deg N, 66 deg W longitude. Accuracy of the geoid undulations used in the calculations was of order of +/- 16 cm with the accuracy of a geoid undulation difference equal to +/- 15 cm over a 100-km line in areas with good terrestrial data coverage. This paper demonstrates that our knowledge or geoid undulations and undulation differences, in a portion of the Gulf Stream region, is sufficiently accurate to determine characteristics of the jet when used with TOPEX altimeter data. The method used here has not been shown to be more accurate than methods that average altimeter data to form a reference surface used in analysis to obtain the Gulf Stream characteristics. However, the results show the geoid approach may be used in areas where lack of current meandering reduces the accuracy of the average surface procedure.

Rapp, Richard H.↗

Spatial Study 2022: Surface Water Samples, Cotton Strip Degradation, and Hydrologic Sensor Data across the Yakima River Basin, Washington, USA (v3)

This dataset supports a broader study examining the drivers of spatial variability in sediment respiration rates in the Yakima River Basin. The dataset provides data and photos generated from sample collection during the same one-week period at 48 sites within multiple rivers throughout the Yakima River Basin in Washington, USA. The contents include surface water geochemistry data; river substrate grain size photos; stream depth data; manual chamber open channel respiration data; and field metadata (including qualitative information on instream and river corridor characteristics). Grain size photos can be used to improve estimates of channel substrate D50 data. The dataset also includes tensile strength and photos from cotton strip field degradation experiments; five-week sensor time series temperature, dissolved oxygen, pressure, pH, specific conductance, chlorophyll A, and turbidity data; plots of the sensor data; and R scripts used to generate the plots. Samples collected during this study were labeled as “Second Spatial Study” or “SSS.” A subset of data from the SSS samples were published in the contiguous United States (CONUS)-Scale Model-Sample (CM) study data package available at https://data.ess-dive.lbl.gov/view/doi:10.15485/1923689 that presents data from across the CONUS. SSS data published in the CM data package were not included in this data package. They include dissolved organic carbon (DOC, measured as non-purgeable organic carbon, NPOC), total nitrogen (TN), grain size, aerobic sediment respiration, dissolved oxygen (DO), and temperature. Parent IDs and Site IDs are consistent between the SSS and CM data packages, and they can be mapped directly so data across packages can be used together. Additionally, sensor data from a similar 2021 spatial study can be found at https://data.ess-dive.lbl.gov/view/doi:10.15485/1892052 and 2021 sample data can be found at https://data.ess-dive.lbl.gov/view/doi:10.15485/1898914. The 2021 spatial study had some sites in common with this 2022 spatial study. This dataset is comprised of three photo folders and one main data folder with six subfolders. The photo folders contain photographs and videos of cotton strip retrieval and sediment quadrats. The main data folder consists of (1) file-level metadata; (2) data dictionary; (3) field metadata; (4) total suspended solids (TSS) data and cotton strip tensile strength data and averages; (5) field protocol; (6) readme; (7) methods codes; (8) international generic sample number (IGSN) mapping file; (9) sensor installation methods summary; (10) stream depth and averages; and (11) Ultrameter data and averages. The Sonar subfolder consists of Sonar time-series depth data and a processing script. The BarotrollAtm, DepthHOBO, MantaRiver, miniDOT, and miniDOTManualChamber subfolders contain time-series data, plots, and summary files. All files are .csv, .pdf, .txt, .R, .Rmd, .jpg, .jpeg, .AVI, .mp4, or .mov. The data package was originally published in April 2023. It was updated in August 2023 (v2; modified files) and September 2024 (v3; modified files). See the change history section in the readme for details. We acknowledge the Yakama Nation as owners and caretakers of the lands where we collected these data. We thank the Confederated Tribes and Bands of the Yakama Nation Tribal Council and Yakama Nation Fisheries for working with us to facilitate sample collection and optimization of data usage according to their values and worldview.

54 ENVIRONMENTAL SCIENCES↗

The Sagittarius stream in Gaia Early Data Release 3 and the origin of the bifurcations

The Sagittarius dwarf spheroidal (Sgr) is a dissolving galaxy being tidally disrupted by the Milky Way (MW). Its stellar stream still poses serious modelling challenges, which hinders our ability to use it effectively as a prospective probe of the MW gravitational potential at large radii. Our goal is to construct the largest and most stringent sample of stars in the stream with which we can advance our understanding of the Sgr-MW interaction, focusing on the characterization of the bifurcations. We improved on previous methods based on the use of the wavelet transform to systematically search for the kinematic signature of the Sgr stream throughout the whole sky in the Gaia data. We then refined our selection via the use of a clustering algorithm on the statistical properties of the colour-magnitude diagrams. Our final sample contains more than 700 000 candidate stars and is three times larger than previous Gaia samples. With it, we have been able to detect the bifurcation of the stream in both the northern and southern hemispheres, which requires four branches (two bright and two faint) to fully describe the system. We present the detailed proper motion distribution of the trailing arm as a function of the angular coordinate along the stream, showing, for the first time, the presence of a sharp edge (on the side of the small proper motions) beyond which there are no Sgr stars. We also characterize the correlation between kinematics and distance. Finally, the chemical analysis of our sample shows that the faint branch of the bifurcation is more metal poor than the bright. We provide analytical descriptions for the proper motion trends as well as for the sky distribution of the four branches of the stream. Based on our analysis, we interpret the bifurcations as a misaligned overlap of the material stripped at the antepenultimate pericentre (faint branches) with the stars ejected at the penultimate pericentre (bright branch), given that Sgr just went through its perigalacticon. The source of this misalignment is still unknown, but we argue that models with some internal rotation in the progenitor – at least during the time of stripping of the stars that are now in the faint branches – are worth exploring.

79 ASTRONOMY AND ASTROPHYSICS↗

Active workflow system for near real-time extreme-scale science. In: PPAA '14 Proceedings of the first workshop on Parallel programming for analytics applications

In recent years, streaming-based data processing has been gaining substantial traction for dealing with overwhelming data generated by real-time applications, from both enterprise sources and scientific computing. In this work, however, we look at an emerging class of scientific data with Near Real-Time (NRT) requirement, in which data is typically generated in a bursty fashion with the near real-time constraints being applied primarily between bursts, rather than within a stream. A key challenge for this types of data sources is that the processing time per data element is not uniform, and not always feasible to predict. Given the observations on the increasing unpredictability of compute load and system dynamics, this work looks to adapt streaming-based approach to the context of this new class of large experiments and simulations that have complex run-time control and analysis issues. In particular, we deploy a novel two-tier scheme for handling the increasing unpredictability of runtime behaviors: Instead of relying on determining what and where to run the scientific workflows beforehand or partial dynamically, the decision will also be adaptively enhanced online according to system runtime status. This is enabled by embedding workflow along with data streams. Specifically, we break data outputs generated from experiments or simulations into multiple self-describing "chunks", which we call active data objects. As such, if there is a transient hotspot observed, a data object with unfinished workflow pipeline can break its previous schedule and search for a least loaded location to continue the execution. Our preliminary experiment results based on synthetic workloads demonstrate the proposed active workflow system as a very promising solution by outperforming the state-of-the-art semi-dynamic workflow schedulers with an improved workflow completion time, as well as a good scalability.

Zhang, Yanwei↗

Jet stream winds - Enhanced aircraft data acquisition and analysis over Southwest Asia

A project is described for providing the accurate initial and verification analyses for the jet stream in regions where general circulation models are known to have large systematic errors, either due to the extreme sparsity of data or to incorrect physical parameterizations. For this purpose, finely spaced aircraft-based meteorological data for the Southwest Asian regions collected for three 10-day periods in the winter of 1988-1989 will be used, together with corresponding data for the North American regions used as a control, to rerun the assimilation cycles and forecast models of the NMC and the NASA Goddard Laboratory for Atmospheres. Data for Southeast Asia will be collected by three carriers with extensive wide-body routes crossing the total region, while data for the North American region will be obtained from the archives of ACARS and GTS.

Tenenbaum, J.↗

DynPaC: Coarse-Grained, Dynamic, and Partially Reconfigurable Array for Streaming Applications

Coarse-grained reconfigurable arrays (CGRAs) provide higher flexibility than application-specific integrated circuits (ASICs) and higher efficiency than fine-grained reconfigurable devices such as Field Programmable Gate Arrays (FPGAs). However, CGRAs are generally designed to support offloading of a single kernel. While their design, based on communicating functional units, appears to naturally suit streaming applications composed of multiple cooperating kernels, current approaches only statically partition the resources across kernels. However, streaming applications often are data-dependent, leading to variable kernel execution times depending on the input data and impacting the throughput of the entire pipeline if resources are statically allocated. Therefore, in this paper, we discuss the design of DynPaC — a coarse-grained, dynamically, and partially reconfigurable array for data-dependent streaming applications. We discuss the required software and hardware components to manage partial dynamic reconfiguration. We demonstrate that by supporting partial dynamic reconfiguration, we can obtain an average speedup of 1.44X for a representative set of applications w.r.t. static partitioning, with a limited area overhead (6.4% of the entire chip).

Tan, Cheng↗

Continuous Spacecraft Communications via Make-Before-Break Antenna Array Beam Steering

Active phased array (APA) antennas can electronically form multiple beams to track several targets simultaneously. Spacecraft equipped with these antennas can achieve continuous communications through a constellation of relay satellites by forming a second beam to an upcoming relay satellite before the spacecraft moves beyond the current relay’s line of sight. These make-before-break operations have utility for streaming critical data without gaps. Using characterization data from a commercially-available APA-based satellite communications terminal, we simulate performance from low-Earth orbit to a representative geosynchronous relay satellite system at Ka-band. Average data rates of 3 Mbps are achievable with a minimum rate of 0.8 Mbps during worst-case handovers. We prototype the beam-splitting algorithm in the terminal hardware and evaluate its performance in an antenna range. In an over-the-air test emulating a handover between two relays we observe error-free data in the combined telemetry stream.

active phased array antennas↗

An Active Learning-Based Streaming Pipeline for Reduced Data Training of Structure Finding Models in Neutron Diffractometry

Structure determination workloads in neutron diffractometry are computationally expensive and routinely require several hours to many days to determine the structure of a material from its neutron diffraction patterns. The potential for machine learning models trained on simulated neutron scattering patterns to significantly speed up these tasks have been reported recently. However, the amount of simulated data needed to train these models grows exponentially with the number of structural parameters to be predicted and poses a significant computational challenge. To overcome this challenge, we introduce a novel batch-mode active learning (AL) policy that uses uncertainty sampling to simulate training data drawn from a probability distribution that prefers labelled examples about which the model is least certain. We confirm its efficacy in training the same models with ∼ 75% less training data while improving the accuracy. We then discuss the design of an efficient stream-based training workflow that uses this AL policy and present a performance study on two heterogeneous platforms to demonstrate that, compared with a conventional training workflow, the streaming workflow delivers ∼ 20% shorter training time without any loss of accuracy.

Wang, Tianle [Brookhaven National Laboratory (BNL)↗

Leveraging In-Network Computing and Programmable Switches for Streaming Analysis of Scientific Data

With the emergence of programmable network devices that match the performance of fixed function devices, several recent projects have explored in-network computing, where the processing that is traditionally done outside the network is offloaded to the network devices. In-network computing has typically been applied to network functions (e.g., load balancing, NAT, and DNS), caching, data reduction/aggregation, and coordination/consensus functions. In some cases it has been used to accelerate stream-processing tasks that involve small payloads and simple operations. In this work we focus on leveraging in-network computing for stream processing of scientific datasets with large payloads that require complex operations such as floating-point computations and logarithmic functions. We demonstrate in-network computing for a real-world scientific application performing streaming normalization of a 2-D image from a light source experiment. We discuss the challenges we encountered and potential approaches to address them.

Sankaran, Ganesh↗

Virtual Log-Structured Storage for High-Performance Streaming

Over the past decade, given the higher number of data sources (e.g., Cloud applications, Internet of things) and critical business demands, Big Data transitioned from batch-oriented to real-time analytics. Stream storage systems, such as Apache Kafka, are well known for their increasing role in real-time Big Data analytics. For scalable stream data ingestion and processing, they logically split a data stream topic into multiple partitions. Stream storage systems keep multiple data stream copies to protect against data loss while implementing a stream partition as a replicated log. This architectural choice enables simplified development while trading cluster size with performance and the number of streams optimally managed. This paper introduces a shared virtual log-structured storage approach for improving the cluster throughput when multiple producers and consumers write and consume in parallel data streams. Stream partitions are associated with shared replicated virtual logs transparently to the user, effectively separating the implementation of stream partitioning (and data ordering) from data replication (and durability). We implement the virtual log technique in the KerA stream storage system. When comparing with Apache Kafka, KerA improves the cluster ingestion throughput by up to 4x when multiple producers write over hundreds of data streams.

consistent stream ordering↗

Gila Water Resources II: Using Earth Observations to Identify Wildfire Impacts on Hydrologic Functions and Recovery in the Gila National Forest

Wildfires have the potential to cause devastating and long-lasting impacts on ecological systems. In the Gila National Forest (Gila NF), wildfire events have occurred with increasing frequency and severity over recent years. These disturbances, such as the historic Whitewater Baldy Complex Fire (2012) and Silver Fire (2013), have raised concerns over post-fire flooding, debris flows, and vegetation recovery. Understanding connections between burn events and ecological functions is crucial for developing effective land management practices within the Gila NF that ensure conservation of the watershed. The Gila Water Resources II team worked in partnership with the US Department of Agriculture (USDA) US Forest Service’s (USFS) Gila National Forest and Region 3. This project provided insight into the influence of wildfires on increased flooding events and determined if restoration efforts in the Gila NF are having a beneficial impact on vegetation regeneration. To understand recovery trends and hydrologic impact in the Gila NF between 2000-2019, this project used Landsat 5 Thematic Mapper, Landsat 7 Enhanced Thematic Mapper Plus, Landsat 8 Operational Land Imager, Global Precipitation Measurement Integrated Multi-satellite Retrievals for GPM precipitation, along with ancillary data from USGS stream gauges and data provided by USDA USFS’s Gila National Forest and Region 3. Based on these data, the team identified burn areas that received restorative treatments and compared Normalized Burn Ratio for different land cover types to better inform land management decisions. Additionally, the team analyzed the relationship between precipitation and streamflow from stream gauges to investigate the impact wildfires have on hydrology within the watershed.

Water Resources↗

Surface Water Quality Data from Beaver-Impacted Streams; Trail Creek and East River, Colorado 2025

This data package contains surface water chemistry measurements collected in 2025 to evaluate how beaver damming and low-tech process-based stream restoration influence water quality and metal mobility in mountainous headwater systems of the Upper Colorado River Basin. Sampling was conducted at Trail Creek (Taylor Park watershed, Colorado), a tributary undergoing restoration through installation of low-tech process-based structures (i.e., beaver dam analogs), and at off-channel beaver ponds within the East River floodplain (East River watershed, Colorado). Samples were collected along longitudinal transects spanning upstream control reaches, beaver-influenced ponded reaches, and downstream segments. Additional samples were collected from near-surface pore waters within a beaver dam seepage face. The dataset includes concentrations of major and trace elements measured by inductively coupled plasma–mass spectrometry (ICP-MS) and inductively coupled plasma–optical emission spectrometry (ICP-OES), major anions measured by ion chromatography (IC), and dissolved organic carbon (DOC; reported as non-purgeable organic carbon, NPOC). Samples were size-fractionated at 0.45 micrometers (µm), 0.22 µm, and 0.02 µm to distinguish particulate (>0.45 µm), colloidal (0.22–0.02 µm), and dissolved (<0.02 µm) fractions. The data package consists of comma-separated value (.csv) files containing tabulated chemical concentration data, sample metadata (site identifiers, geographic coordinates, sampling dates, fraction type), and quality control flags. All files are provided in open, non-proprietary formats that can be accessed using standard data analysis software such as Microsoft Excel, R, Python, MATLAB, or other programs capable of reading .csv files. Units, detection limits, and analytical methods are documented in accompanying metadata files. The dataset is designed to support analyses of (1) how beaver impoundment and restoration structures alter elemental partitioning and transport, (2) the role of iron and organic carbon in mediating trace metal mobility, and (3) reach-scale changes in water quality across restoration gradients. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231. Part of this work was performed at SLAC Accelerator Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-76SF00515.

Anions↗