Search NASA⌕ Search

SEARCH · Search NASA

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Understanding the Impact of Data Staging for Coupled Scientific Workflows

We report the rate of data generated by cutting-edge experimental science facilities and large-scale simulations enabled by current high-performance computing (HPC) systems has continued to grow at a far greater pace than the development of the network and storage capabilities on which these systems rely. To cope with this challenge, scientist are moving toward the creation of autonomous experiments and HPC simulations using machine learning. However, efficiently moving, storing, and processing large amounts of data away from the point of origin presents an incredible challenge. In-memory computing, in situ analysis, data staging, and data streaming are recognized viable alternatives to traditional file-based methods for transferring data between coupled workflows. However, the performance trade-offs and limitations for these methods are not fully understood when used in HPC applications. This article presents a comprehensive performance assessment of the current solutions for data staging when applied to applications that are not necessary I/O intensive which makes them not ideal candidates for these methods. Our study is based on experiments running at scale on Oak Ridge National Laboratory's Summit supercomputer using applications and simulations that cover typical computational motifs and patterns. We investigated the usability and cost/benefit trade-offs of staging algorithms for HPC applications under different scenarios and highlight opportunities for optimizing the dataflow between coupled simulation workflows.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

L-PBF High-Throughput Data Pipeline Approach for Multi-modal Integration

Abstract Metal-based additive manufacturing requires active monitoring solutions for assessing part quality. Multiple sensors and data streams, however, generate large heterogeneous data sets that are impractical for manual assessment and characterization. In this work, an automated pipeline is developed that enables feature extraction from high-speed camera video and multi-modal data analysis. The framework removes the need for manual assessment through the utilization of deep learning techniques and training models in a weakly supervised paradigm. We demonstrate this pipeline’s capability over 700,000 high-speed camera frames. The pipeline successfully extracts melt pool and spatter geometries and links them to corresponding pyrometry, radiography, and processparameter information. 715 individual prints are examined to reveal melt pool areas that exceeds 0.07 mm 2 and pyrometry signal over a threshold (375 pyrometry units) were more likely to have defects. These automated processes enable massive throughput of characterization techniques.

36 MATERIALS SCIENCE↗
DOE OSTI2011

Real-Time Dynamic Brake Assessment Proof of Concept Final Report

This proof-of-concept research was performed to explore the feasibility of using real-world braking data from commercial motor vehicles to make a diagnosis of brake condition similar to that of the performance-based brake tester (PBBT). This was done by determining the relationship between pressure and brake force (P-BF), compensating for the gross vehicle weight (GVW). The nature of this P-BF relationship (e.g., low braking force for a given brake application pressure) may indicate brake system problems. In order to determine the relationship between brake force and brake application pressure, a few key parameters of duty cycle information were collected. Because braking events are often brief, spanning only a few seconds, a sample rate of 10 Hz was needed. The algorithm under development required brake application pressure and speed (from which deceleration was calculated). Accurate weight estimation was also needed to properly derive the braking force from the deceleration. In order to ensure that braking force was the predominant factor in deceleration for the segments of data used in analysis, the data was screened for grade as well. Also, the analysis needed to be based on pressures above the crack pressure. The crack pressure is the pressure below which the individual brakes are not applied due the nature of the mechanical system. This value, which may vary somewhat from one wheel end to another, is approximately 10 psi. Therefore, only pressures 15 psi and above were used in the analysis. The Department of Energy s Medium Truck Duty Cycle research has indicated that under the real-world circumstances of the test vehicle brake pressures of up to approximately 30 psi can be expected. Several different types of data were collected during the testing task of this project. Constant-pressure stopping tests were conducted at several combinations of brake application pressure (15, 20, 25, and 30 psi), load conditions (moderately and fully laden), and speeds (20 and 30 mph). Data was collected at 10 Hz. Standard and stepped-pressure performance-based brake tests with brake pressure transducers were performed for each loading condition. The stepped-pressure test included the constant-pressure intervals of brake application at 15, 20, 25, and 30 psi. The PBBT data files included 10 Hz streaming data collected during the testing of each axle. Two weeks of real-world duty cycle (driving and braking) data was also collected at 10 Hz. Initial analysis of the data revealed that the data collected in the field (i.e., day-to-day operations) provided the same information as that obtained from the controlled tests. Analysis of the data collected revealed a strong linear relationship between brake application pressure and deceleration for given GVWs. As anticipated, initial speed was not found to be a significant factor in the deceleration-pressure relationship, unlike GVW. The positive results obtained from this proof of concept test point to the need for further research to expand this concept. A second phase should include testing over a wider range of speeds and include medium brake application pressures in addition to the low pressures tested in this research. Testing on multiple vehicles would also be of value. This future phase should involve testing to determine how degradation of braking performance affects the pressure-deceleration relationship.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2014

Hybrid methods for cybersecurity analysis :

Early 2010 saw a signi cant change in adversarial techniques aimed at network intrusion: a shift from malware delivered via email attachments toward the use of hidden, embedded hyperlinks to initiate sequences of downloads and interactions with web sites and network servers containing malicious software. Enterprise security groups were well poised and experienced in defending the former attacks, but the new types of attacks were larger in number, more challenging to detect, dynamic in nature, and required the development of new technologies and analytic capabilities. The Hybrid LDRD project was aimed at delivering new capabilities in large-scale data modeling and analysis to enterprise security operators and analysts and understanding the challenges of detection and prevention of emerging cybersecurity threats. Leveraging previous LDRD research e orts and capabilities in large-scale relational data analysis, large-scale discrete data analysis and visualization, and streaming data analysis, new modeling and analysis capabilities were quickly brought to bear on the problems in email phishing and spear phishing attacks in the Sandia enterprise security operational groups at the onset of the Hybrid project. As part of this project, a software development and deployment framework was created within the security analyst work ow tool sets to facilitate the delivery and testing of new capabilities as they became available, and machine learning algorithms were developed to address the challenge of dynamic threats. Furthermore, researchers from the Hybrid project were embedded in the security analyst groups for almost a full year, engaged in daily operational activities and routines, creating an atmosphere of trust and collaboration between the researchers and security personnel. The Hybrid project has altered the way that research ideas can be incorporated into the production environments of Sandias enterprise security groups, reducing time to deployment from months and years to hours and days for the application of new modeling and analysis capabilities to emerging threats. The development and deployment framework has been generalized into the Hybrid Framework and incor- porated into several LDRD, WFO, and DOE/CSL projects and proposals. And most importantly, the Hybrid project has provided Sandia security analysts with new, scalable, extensible analytic capabilities that have resulted in alerts not detectable using their previous work ow tool sets.

Dunlavy, Daniel M.↗
DOE OSTI2025

Tactical Analysis for Calculating Contextual Risk at Boundaries: Summary of Laboratory Directed Research & Development Effort

The Tactical Analysis for Calculating Contextual Risk at Boundaries (TACCRAB) tool is an innovative digital twin (DT) platform and automated risk algorithm designed to transform operational decision-making in structured screening environments, with an initial focus on Southern Border Land Ports of Entry (POEs). The invention provides integration points for advanced artificial intelligence, predictive modeling, and real-time data analysis to produce a comprehensive risk management tool that enables proactive, data-informed security strategies. The core inventive features of TACCRAB center on its unique risk algorithm, which dynamically calculates contextual risk by synthesizing historical data, near real-time streaming data from the checkpoints themselves, and AI-generated predictions. Unlike traditional risk assessment methods, TACCRAB utilizes a DT to provide comprehensive operational insights, allowing stakeholders to visualize, simulate, and optimize checkpoint configurations with unprecedented speed and contextual awareness. TACCRAB's key innovation lies in its ability to combine multiple complex inputs - including technology detection probabilities, resource availability, screening pathway characteristics, and threat actor behavioral patterns - into a unified risk calculation and update these inputs based on changing operational and environmental conditions. By leveraging a DT that continuously updates and learns from linked data, TACCRAB can suggest adaptive mitigation strategies that minimize risk while maintaining operational efficiency. Particularly novel is the platform's approach to decision support, which goes beyond static risk assessment. The DT provides dynamic metrics such as wait times, resource allocation effectiveness, and potential emerging threat scenarios, enabling users to view sophisticated, relevant what-if simulations and optimize checkpoint operations in near real-time. The system's architecture allows for generalized application across different screening environments, such as secure facilities, ports of entry, and soft targets, making it a versatile tool for security and operational management. The invention distinguishes itself through its comprehensive integration of predictive modeling, AI-driven pattern discovery, and user-friendly interface design. By combining these elements, TACCRAB transforms complex risk data into actionable insights, supporting decision-makers at various organizational levels - from booth agents making split-second screening decisions to checkpoint managers optimizing the day's resource allocation to strategic planners managing long-term investments.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2015

What’s in Your Pocket? Trends in Mobile Apps for Biosurveillance and Decision-Making

Mobile applications offer tremendous potential for collecting data for biosurveillance through participatory surveillance and the integration of new and traditional data streams. This panel features epidemiologists, data scientists, and technology partners who will address the full application lifecycle -- from the assessment of the specific demand, resource, and technology specifications, to implementation and evaluation-- of several innovative apps. Audience engagement using a new web app will gather input on other uses of mobile apps for data collection, visualization, and analyses. The panel and audience discussion will be compiled into guidance for future development of mobile apps for biosurveillance purposes to advance population health.

60 APPLIED LIFE SCIENCES↗
NASA NTRS1977

Telemetry and data relay for manned space programs

Previously and currently planned methods of transferring or relaying telemetered information from a remote receiving station to the Mission Control Center, at the National Aeronautics and Space Administration Lyndon B. Johnson Space Center, are discussed. The data flow techniques of the Apollo Program (fixed formats), the Skylab Program (data compression), and the Space Shuttle Program (data throughput) are defined and analyzed according to their advantages and deficiencies from an operations viewpoint. Excluded from the data flow technique discussions are command data, tracking data, television data, and Teletype and Datafax messages. Voice transmissions are addressed in the Space Shuttle Program discussion because they form a part of the telemetry data stream. The tradeoffs between actual data flow requirements and existing communications limitations are addressed for each of these programs. Finally, a short discussion of current problem areas in the planned ground data transmission system for the Space Shuttle Program is presented.

Anson, K. W.↗
NASA NTRS1982

LANDSAT-D accelerated payload correction subsystem output computer compatible tape format

The NASA GSFC LANDSAT-D Ground Segment (GS) is developing an Accelerated Payload Correction Subsystem (APCS) to provide Thematic Mapper (TM) image correction data to be used outside the GS. This correction data is computed from a subset of the TM Payload Correction Data (PCD), which is downlinked from the spacecraft in a 32 Kbps data stream, and mirror scan correction data (MSCD), which is extracted from the wideband video data. This correction data is generated in the GS Thematic Mapper Mission Management Facility (MMF-T), and is recorded on a 9-track 1600 bit per inch computer compatible tape (CCT). This CCT is known as a APCS Output CCT (AOT). The AOT follows standardized corrections with respect to data formats, record construction and record identification. Applicable documents are delineated; common conventions which are used in further defining the structure, format and content of the AOT are defined; and the structure and content of the AOT are described.

Source record↗
NASA NTRS1991

The MAMS Quick View System-2 (QVS2) - A workstation for NASA aircraft scanner data evaluation

This paper describes a ground-based data-evaluation workstation named Quick View System-2 (QVS2) developed to support postflight evaluation of data supplied by the Multispectral Atmospheric Mapping Sensor (MAMS), one of the four spectrometers that can be used with the Daedalus scanner flown on the ER-2 aircraft. The QVS2 provides advanced analysis capabilities and can be applied to other airborne scanners used throughout NASA for earth-system-science investigations, because of the commonality in the data stream and in the generalized data structure.

Jedlovec, Gary J.↗
NASA NTRS1997

Sustainable Biosphere Initiative Project

The goal of the Advanced Technology in Ecological Sciences project is to gain broad participation within the environmental scientific community in developing a research agenda addressing the development and refinement of technologies instrumental to research that responds to these challenges (e.g. global climate change, unsustainable resource use, and threats to biological diversity). The following activities have been completed: (1) A listserve 'eco-tech was set up to serve as a clearinghouse of information about activities and events relating to advanced technologies; (2) A series of conference calls were organized on specific topics including data visualization and spatial analysis, and remote sensing; and (3) Two meetings were organized at the 19% ESA Annual Meeting in Providence, Rhode Island. Topics covered included concerns about tool and data sharing; interest in expanded development of ground-based remote sensing technologies for monitoring; issues involved in training for using new technologies and increasing data streams, and- associated implications of data processing capabilities; questions about how to develop appropriate standards (i.e. surface morphology classification standards) that facilitate the exchange and comparison of analytical results; and some thoughts about remote sensing platforms and vehicles.

Source record↗
DOE OSTI2023

Conquering Data Chaos: Research Data Management with Kubernetes

Managing massive volumes of data and effectively making it accessible to researchers poses significant challenges and is a barrier to scientific discovery. In many cases, critical data is locked up in unwieldy file formats or one-off databases and is too large to effectively process on a single machine. This talk explores the role of Kubernetes, an open-source container orchestration platform, in addressing research data management challenges. I will discuss how we are using a set of publicly available open-source and home-grown tools in the National Renewable Energy Lab (NREL) Data, Analysis, and Visualization (DAV) group to help researchers overcome data-related bottlenecks. The talk will begin by providing an overview of the data challenges faced in research data management, including data storage, processing, and analysis. I will highlight Kubernetes' ability to handle large-scale data by leveraging containerization and distributed computing, including distributed storage. Kubernetes allows researchers to encapsulate data processing infrastructure and workflows into portable containers, enabling reproducibility and ease of deployment. Kubernetes can then schedule and manage the resource allocation of these containers to enable efficient utilization of limited computing resources, leading to more efficient data processing and analysis. I will discuss some limitations of traditional, siloed approaches to dealing with data and emphasize the need for solutions which foster collaboration. I will highlight how we are using Kubernetes at NREL to facilitate data sharing and cooperation among research teams. Kubernetes' flexible architecture enables the deployment of shared computing environments, such as Apache Superset, where researchers can seamlessly access and analyze shared datasets. Providing the ability to have one research team easily consume data generated by another, utilizing Kubernetes' as a central data platform, is one of the major wins we've encountered by adopting the platform. Finally, I will showcase real-world use cases from NREL where we have used Kubernetes to solve some persistent data challenges involving large volumes of sensor and monitoring data. I will discuss the challenges we encountered when creating our cluster and making it available as a production-ready resource. I will also discuss the specific suite of tools, including Postgres and Apache Druid for columnar and timeseries data, and Redpanda Kafka for streaming data we have deployed in our infrastructure, and the process that went into the selection of these tools.

collaborative environment↗
DOE OSTI2020

1.2 Mfps standalone X-ray detector for Time-Resolved Experiments

We present a standalone and autonomous X-ray detector capable of operation with the speed of up to1.2Mfps. The detector utilizes UFXC32k hybrid pixel detectors for sensing X-rays, Spartan-6 LX45 FPGA placed in commercially available sbRIO 9628 controller for data acquisition and processing including a compression with zero-suppression algorithm. A Linux-RT system working on the 400 MHz Dual-Core CPU is used for FPGA control and data streaming to the higher-level system over 1 Gbps Ethernet connection. 1.2 M frames per second is achieved in so-called burst mode of operation while in zerodead-time mode 70 kfps is possible. Due to efficient data compression in FPGA there’s no need of using high-speed transceivers and Frame-Grabber cards on the data server side and the detector can stream the data infinitely over standard 1 Gbps network connection. Operation modes were tested at Advanced Photon Source Synchrotron at Argonne National Laboratory.

47 OTHER INSTRUMENTATION↗
DOE OSTI2019

Event Detection for Undersampled Electron Microscopy Experiments: A Control Chart Case Study

Real-time systems often consist of high-volume, high-velocity data streams in which making efficient and timely inferences about the data are challenging. In particular, when a user interacts with a real-time system to gain insights, detect events, and make decisions about the system, the rate and amount of information the user is required to process is generally overwhelming. In addition, analytically processing large volumes of data can be computationally expensive and, in real-time, renders traditional inferential methods effectively useless. One approach to mitigate these challenges is to reduce both the amount of information presented to the user and the volume of data placed in the stream. In this manuscript we will describe a method constructed under this approach in addition to the development and deployment of an online tool, Real time Event Detector for Subsampled Images (REDSI), designed to provide feedback on a real-time system by characterizing and detecting events of interest. We will discuss REDSI in the context of scanning transmission electron microscopy (STEM), which is a powerful real-time system that provides high spatial and temporal resolution on nanoscale structures and processes. The data produced by STEM experiments are a stream of images relaying structural, compositional, and dynamic interphase information to scientists in fields ranging from microbiology and neuroscience to materials science and energetics. While REDSI was designed to processes STEM images in real time, we will also discuss other real-time system applications for this methodology.

multivariate statistics, event detection↗
DOE OSTI2023

Driving Next-Generation Workflows from the Data Plane

We observe the emergence of a new generation of scientific workflows that process data produced at a sustained rate by scientific instruments and large scale numerical simulations. This data is consumed by multiple analysis, visualization, or Machine Learning components not only to enable inference and justify the scientific program, but also to monitor and steer the evolution of these experiments. In such workflows, moving intermediate data efficiently is key to performance, more than efficiently scheduling computational tasks. However, most traditional workflow management systems focus on optimizing task scheduling and then deal with data management, assuming a “move little, compute for long” model, which makes them unfit to the efficient management of this new generation of workflows. Therefore, we advocate for a new way to manage scientific workflows. We propose to consider an efficiently and independently managed data plane that can store and stream data. Workflows compute components, in the application plane can then interact with the data plane, abstracted from complexities of data management. Then, the role of a workflow management system would become that of a control plane that allows users to connect services together to execute the workflow and manages connections between the application and data planes. In this position paper, we characterize several next-generation workflow motifs and describe how their interaction with the data plane is a challenge to traditional workflow management systems. Then, we express a set of requirements that a workflow management system should meet to efficiently manage next-generation workflows at different scales. Based on these requirements, we expose our vision of driving next-generation workflows from the data plane and list remaining open challenges.

Suter, Fred↗
DOE OSTI2020

Attack on Grid Event Cause Analysis: An Adversarial Machine Learning Approach

With the ever-increasing reliance on data for data-driven applications in power grids, such as event cause analysis, the authenticity of data streams has become crucially important. The data can be prone to adversarial stealthy attacks aiming to manipulate the data such that residual-based bad data detectors cannot detect them, and the perception of system operators or event classifiers changes about the actual event. This paper investigates the impact of adversarial attacks on convolutional neural network-based event cause analysis frameworks. We have successfully verified the ability of adversaries to maliciously misclassify events through stealthy data manipulations. The vulnerability assessment is studied with respect to the number of compromised measurements. Furthermore, a defense mechanism to robustify the performance of the event cause analysis is proposed. The effectiveness of adversarial attacks on changing the output of the framework is studied using the data generated by real-time digital simulator (RTDS) under different scenarios such as type of attacks and level of access to data.

Niazazari, Iman↗
DOE OSTI2021

Advanced Computing, Data Science, and Artificial Intelligence Research Opportunities for Energy-Focused Transportation Science

The Energy Efficient Mobility Systems (EEMS) technology landscape is complex and rapidly evolving, which provides both tremendous opportunities and formidable challenges. Significant alterations to the mobility landscape are underway due to the advent of vehicle and infrastructure connectivity, autonomous driving, and rapid passenger- and freight-vehicle electrification. Advanced computing will play an increasingly important role in enabling the EEMS program to understand and identify the most important levers to improve the energy productivity of future integrated mobility systems. It is also driving new approaches to mobility and the research to unlock an affordable, efficient, safe, and accessible transportation future. Driving much of this change is the collection, analysis, and strategic use of massive amounts of diverse, complex data from infrastructure and vehicles with on-board sensors and data storage and transmission capabilities. Diverse and representative data are key to implementing approaches to maximize mobility energy productivity. While high-fidelity modeling of integrated transportation networks has strengthened our understanding of dynamic movement and behavior patterns, existing tools must be expanded beyond their current focus. This work necessitates data infrastructure investments (e.g., secure-streaming data platforms driven by ubiquitous sensors and video analytics) as well as investments in critical capabilities for large-scale automated analysis and organization using modern machine learning, statistics, and artificial intelligence. Other chief needs include agile, large-scale storage that can be quickly searched and queried for relevant data to support validation and model development, data-sharing agreements, and formatting standards for key data types. The future of public transit must be explored in greater detail, research must inform design, and opportunities must be identified for improving the mobility productivity of public transit in both urban and rural America.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2026

The ABCs of On-Demand Transit (ODT)

On-demand mobility - also referred to as on-demand transit (ODT) - is a form of public mobility that is flexible with respect to where and when service is provided, and ODT deployments have increased significantly in recent years. Transit agencies are becoming increasingly interested in ODT, and due to differing definitions and various service design and business model options, it can be difficult to learn about the emerging industry. This work provides an overview of the definitions of ODT, recent trends internationally and in the U.S., ODT's benefits and challenges (particularly compared to fixed-route transit), three primary service design options, system costs and funding considerations, and a metrics framework for evaluating ODT systems to ensure continued successful performance. Identified benefits include increased service areas, short ride and wait times, increased user flexibility, potential to reduce energy consumption and emissions through shared trips and smaller, right-sized vehicles, increased safety and comfort through door-to-door service, and rich data streams including granular spatio-temporal data that can be analyzed to continuously improve the service. Challenges include scaling ODT service up as small increases in ridership require additional supply to keep service quality high, serving peak times including keeping low wait times, the lack of fixed schedule being challenging for commuters, integrating ODT services with nearby transit systems, and equity for riders without smartphones who cannot track the vehicle in a mobile app. Finally, an overview of seven ODT case studies (in Texas, Missouri, New York, and Ontario, Canada) performed by NREL and related analysis of travel time, energy and emissions, costs, and equity are presented. Initial key findings include: ODT can be cost- and energy-effective compared to fixed-route transit, ODT serves more people than other transit options, and ODT system deployments can be followed by rapid growth.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS1981

Memory-based frame synchronizer

A frame synchronizer for use in digital communications systems wherein data formats can be easily and dynamically changed is described. The use of memory array elements provide increased flexibility in format selection and sync word selection in addition to real time reconfiguration ability. The frame synchronizer comprises a serial-to-parallel converter which converts a serial input data stream to a constantly changing parallel data output. This parallel data output is supplied to programmable sync word recognizers each consisting of a multiplexer and a random access memory (RAM). The multiplexer is connected to both the parallel data output and an address bus which may be connected to a microprocessor or computer for purposes of programming the sync word recognizer. The RAM is used as an associative memory or decorder and is programmed to identify a specific sync word. Additional programmable RAMs are used as counter decoders to define word bit length, frame word length, and paragraph frame length.

Stattel, R. J.↗