Search NASA⌕ Search

SEARCH · Search NASA

Results for “Real-Time Streaming Data Architecture”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Trustworthy Autonomy for Gateway Vehicle System Manager

The Vehicle System Manager (VSM) is the highest-level software control system in the Gateway hierarchical Autonomous System Management Architecture. The VSM provides four function categories: Mission Management and Timeline Execution, Resource Management, Fault Management, Vehicle Control and Operation. VSM provides various levels of automation ranging from fully autonomous operations with no flight crew and minimal ground monitoring to advisory automation when Gateway is crewed and has full ground monitoring. Trustworthiness is achieved via verified specification, comprehensive development verification, and real-time verification using assume-guarantee contracts. Development verification includes semantic verification of the data model via peer review and testing and assume-guarantee contracts implemented using the PlusCal/TLA+ environment. VSM also uses runtime assume-guarantee contracts, implemented in R2U2 via a runtime monitor that feeds the necessary telemetry data to R2U2 and which receives and responds to the R2U2 verdict stream. The full lifecycle verification approach and use of assume-guarantee contracts provides increased trustworthiness to VSM. Preliminary results provide encouragement that VSM can be both autonomous and trustworthy.

Assume-guarantee contracts↗
NASA NTRS2002

REACH: Real-Time Data Awareness in Multi-Spacecraft Missions

Missions have been proposed that will use multiple spacecraft to perform scientific or commercial tasks. Indeed, in the commercial world, some spacecraft constellations already exist. Aside from the technical challenges of constructing and flying these missions, there is also the financial challenge presented by the tradition model of the flight operations team (FOT) when it is applied to a constellation mission. Proposed constellation missions range in size from three spacecraft to more than 50. If the current ratio of three-to-five FOT personnel per spacecraft is maintained, the size of the FOT becomes cost prohibitive. The Advanced Architectures and Automation Branch at the Goddard Space Flight Center (GSFC Code 588) saw the potential to reduce the cost of these missions by creating new user interfaces to the ground system health-and-safety data. The goal is to enable a smaller FOT to remain aware and responsive to the increased amount of ground system information in a multi-spacecraft environment. Rather than abandon the tried and true, these interfaces were developed to run alongside existing ground system software to provide additional support to the FOT. These new user interfaces have been combined in a tool called REACH. REACH-the Real-time Evaluation and Analysis of Consolidated Health-is a software product that uses advanced visualization techniques to make spacecraft anomalies easy to spot, no matter how many spacecraft are in the constellation. REACH reads a real-time stream of data from the ground system and displays it to the FOT such that anomalies are easy to pick out and investigate. Data visualization has been used in ground system operations for many years. To provide a unique visualization tool, we developed a unique source of data to visualize: the REACH Health Model Engine. The Health Model Engine is rule-based software that receives real-time telemetry information and outputs "health" information related to the subsystems and spacecraft that the telemetry belong to. The Health Engine can run out-of-the-box or can be tailored with a scripting language. Out of the box, it uses limit violations to determine the health of subsystems and spacecraft; when tailored, it determines health using equations combining the values and limits of any telemetry in the spacecraft. The REACH visualizations then "roll up" the information from the Health Engine into high level, summary displays. These summary visualizations can be "zoomed" into for increasing levels of detail. Currently REACH is installed in the Small Explorer (SMEX) lab at GSFC, and is monitoring three of their five spacecraft. We are scheduled to install REACH in the Mid-sized Explorer (MIDEX) lab, which will allow us to monitor up to six more spacecraft. The process of installing and using our "research" software in an operational environment has provided many insights into which parts of REACH are a step forward and which of our ideas are missteps. Our paper explores both the new concepts in spacecraft health-and-safety visualization, the difficulties of such systems in the operational environment, and the cost and safety issues of multi-spacecraft missions.

Maks, Lori↗
NASA NTRS2002

REACH: Real-Time Data Awareness in Multi-Spacecraft Missions

NASA's Advanced Architectures and Automation Branch at the Goddard Space Flight Center (Code 588) saw the potential to reduce the cost of constellation missions by creating new user interfaces to the ground system health-and-safety data. The goal is to enable a small Flight Operations Team (FOT) to remain aware and responsive to the increased amount of ground system information in a multi-spacecraft environment. Rather than abandon the tried and true, these interfaces were developed to run alongside existing ground system software to provide additional support to the FOT. These new user interfaces have been combined in a tool called REACH. REACH-the Real-time Evaluation and Analysis of Consolidated Health-is a software product that uses advanced visualization techniques to make spacecraft anomalies easy to spot, no matter how many spacecraft are in the constellation. REACH reads numerous real-time streams of data from the ground system(s) and displays synthesized information to the FOT such that anomalies are easy to pick out and investigate.

Maks, Lori↗
NASA NTRS2002

Novel Advancements in Internet-Based Real Time Data Technologies

AZ Technology has been working with MSFC Ground Systems Department to find ways to make it easier for remote experimenters (RPI's) to monitor their International Space Station (ISS) payloads in real-time from anywhere using standard/familiar devices. AZ Technology was awarded an SBIR Phase I grant to research the technologies behind and advancements of distributing live ISS data across the Internet. That research resulted in a product called "EZStream" which is in use on several ISS-related projects. Although the initial implementation is geared toward ISS, the architecture and lessons learned are applicable to other space-related programs. This paper presents the high-level architecture and components that make up EZStream. A combination of commercial-off-the-shelf (COTS) and custom components were used and their interaction will be discussed. The server is powered by Apache's Jakarta-Tomcat web server/servlet engine. User accounts are maintained in a My SQL database. Both Tomcat and MySQL are Open Source products. When used for ISS, EZStream pulls the live data directly from NASA's Telescience Resource Kit (TReK) API. TReK parses the ISS data stream into individual measurement parameters and performs on-the- fly engineering unit conversion and range checking before passing the data to EZStream for distribution. TReK is provided by NASA at no charge to ISS experimenters. By using a combination of well established Open Source, NASA-supplied. and AZ Technology-developed components, operations using EZStream are robust and economical. Security over the Internet is a major concern on most space programs. This paper describes how EZStream provides for secure connection to and transmission of space- related data over the public Internet. Display pages that show sensitive data can be placed under access control by EZStream. Users are required to login before being allowed to pull up those web pages. To enhance security, the EZStream client/server data transmissions can be encrypted to preclude interception. EZStream was developed to make use of a host of standard platforms and protocols. Each are discussed in detail in this paper. The I3ZStream server is written as Java Servlets. This allows different platforms (i.e. Windows, Unix, Linux . Mac) to host the server portion. The EZStream client component is written in two different flavors: JavaBean and ActiveX. The JavaBean component is used to develop Java Applet displays. The ActiveX component is used for developing ActiveX-based displays. Remote user devices will be covered including web browsers on PC#s and scaled-down displays for PDA's and smart cell phones. As mentioned. the interaction between EZStream (web/data server) and TReK (data source) will be covered as related to ISS. EZStream is being enhanced to receive and parse binary data stream directly. This makes EZStream beneficial to both the ISS International Partners and non-NASA applications (i.e. factory floor monitoring). The options for developing client-side display web pages will be addressed along with the development of tools to allow creation of display web pages by non-programmers.

Myers, Gerry↗
DOE OSTI2025

Tactical Analysis for Calculating Contextual Risk at Boundaries: Summary of Laboratory Directed Research & Development Effort

The Tactical Analysis for Calculating Contextual Risk at Boundaries (TACCRAB) tool is an innovative digital twin (DT) platform and automated risk algorithm designed to transform operational decision-making in structured screening environments, with an initial focus on Southern Border Land Ports of Entry (POEs). The invention provides integration points for advanced artificial intelligence, predictive modeling, and real-time data analysis to produce a comprehensive risk management tool that enables proactive, data-informed security strategies. The core inventive features of TACCRAB center on its unique risk algorithm, which dynamically calculates contextual risk by synthesizing historical data, near real-time streaming data from the checkpoints themselves, and AI-generated predictions. Unlike traditional risk assessment methods, TACCRAB utilizes a DT to provide comprehensive operational insights, allowing stakeholders to visualize, simulate, and optimize checkpoint configurations with unprecedented speed and contextual awareness. TACCRAB's key innovation lies in its ability to combine multiple complex inputs - including technology detection probabilities, resource availability, screening pathway characteristics, and threat actor behavioral patterns - into a unified risk calculation and update these inputs based on changing operational and environmental conditions. By leveraging a DT that continuously updates and learns from linked data, TACCRAB can suggest adaptive mitigation strategies that minimize risk while maintaining operational efficiency. Particularly novel is the platform's approach to decision support, which goes beyond static risk assessment. The DT provides dynamic metrics such as wait times, resource allocation effectiveness, and potential emerging threat scenarios, enabling users to view sophisticated, relevant what-if simulations and optimize checkpoint operations in near real-time. The system's architecture allows for generalized application across different screening environments, such as secure facilities, ports of entry, and soft targets, making it a versatile tool for security and operational management. The invention distinguishes itself through its comprehensive integration of predictive modeling, AI-driven pattern discovery, and user-friendly interface design. By combining these elements, TACCRAB transforms complex risk data into actionable insights, supporting decision-makers at various organizational levels - from booth agents making split-second screening decisions to checkpoint managers optimizing the day's resource allocation to strategic planners managing long-term investments.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
NASA NTRS1992

CCD data acquisition systems at Lick and Keck Observatories

This paper will describe and compare two distinct but related CCD data acquisition systems (DAS) currently under development at Lick and Keck Observatories. Although these two systems have a number of major architectural differences, they share a considerable amount of common hardware and software. Both of these new systems build on a large body of proven software that is the foundation of the existing CCD DAS currently in use at Lick Observatory. Both will provide support for reading up to four on-chip amplifiers per CCD and/or reading out mosaics of CCD chips. In addition, they will provide the capability for interactive, real-time adjustment of CCD waveforms for engineering purposes. Each of these two systems is composed of three major subsystems: (1) an instrument computer and its software; (2) a data capture computer and its software; and (3) a CCD/dewar controller and its software. The instrument computer is a Unix workstation, and the functions it provides include user interfaces, the interactive real-time display of CCD images, and the recording of image and FITS header data to disk and/or tape. The data capture computer is responsible for the packaging and high-speed transfer of the CCD pixel data stream into a bulk RAM, and the subsequent transfer of this data to the instrument computer. The CCD/dewar controller generates the waveforms for clocking the CCD, digitizes the pixel data, and transmits it via high-speed link to the data capture computer. It is also responsible for monitoring and controlling the dewar temperature and cryogen levels. Given the number of different types of processors and high-speed data links employed in both systems, a major emphasis of this paper will be on the various forms of interprocessor communications utilized for data transfer and distributed process synchronization.

Kibrick, R. I.↗
NASA NTRS2026

Proactive Wildfire Management: A Remote Sensing and Multimodal CNN-MLP Architecture for Ignition Risk Forecasting

As the frequency and intensity of wildfires increase, with fire seasons now starting earlier and ending later than they have over the past decades, current monitoring systems, such as lookout towers and satellites, are hindered by cloud cover, low-resolution imagery, and static data gaps that fail to track vegetation moisture levels fast enough to catch rapid pre-ignition changes. This report proposes a Machine Learning-enabled Wildfire Ignition Prediction framework that combines satellite monitoring with dynamic and high-resolution remote sensing from Unmanned Aerial Vehicle (UAV) swarms. The method would use multispectral and thermal data from the Landsat program to create a baseline for vegetation health, calculating a two-band Enhanced Vegetation Index (EVI2) and the moisture content of the vegetation. These inputs will later be fused with microscale UAV weather data, including thermal hotspots found through thick canopies, hyperspectral chemical signatures of pre-visual combustion, and local weather streams. The multispectral satellite, multispectral Light Detection and Ranging (LiDAR), and thermal data would then be processed through a Convolutional Neural Network (CNN), alongside a Multilayer Perceptron (MLP) for the micro-weather telemetry. The outputs of these networks would be fused into a single feature representation and passed through a final prediction network to generate real-time ignition risk scores and hotspot alerts. Model performance would be assessed using standard classification metrics, including a Receiver Operating Characteristic - Area Under the Curve (ROC AUC) and F1 score. This system would allow first responders to identify high-risk zones and intervene before ignition occurs, improving emergency response time compared to current approaches.

machine learning↗
NASA NTRS2008

An Overview of Starfish: A Table-Centric Tool for Interactive Synthesis

Engineering is an interactive process that requires intelligent interaction at many levels. My thesis [1] advances an engineering discipline for high-level synthesis and architectural decomposition that integrates perspicuous representation, designer interaction, and mathematical rigor. Starfish, the software prototype for the design method, implements a table-centric transformation system for reorganizing control-dominated system expressions into high-level architectures. Based on the digital design derivation (DDD) system a designer-guided synthesis technique that applies correctness preserving transformations to synchronous data flow specifications expressed as co- recursive stream equations Starfish enhances user interaction and extends the reachable design space by incorporating four innovations: behavior tables, serialization tables, data refinement, and operator retiming. Behavior tables express systems of co-recursive stream equations as a table of guarded signal updates. Developers and users of the DDD system used manually constructed behavior tables to help them decide which transformations to apply and how to specify them. These design exercises produced several formally constructed hardware implementations: the FM9001 microprocessor, an SECD machine for evaluating LISP, and the SchemEngine, garbage collected machine for interpreting a byte-code representation of compiled Scheme programs. Bose and Tuna, two of DDD s developers, have subsequently commercialized the design derivation methodology at Derivation Systems, Inc. (DSI). DSI has formally derived and validated PCI bus interfaces and a Java byte-code processor; they further executed a contract to prototype SPIDER-NASA's ultra-reliable communications bus. To date, most derivations from DDD and DRS have targeted hardware due to its synchronous design paradigm. However, Starfish expressions are independent of the synchronization mechanism; there is no commitment to hardware or globally broadcast clocks. Though software back-ends for design derivation are limited to the DDD stream-interpreter, targeting synchronous or real-time software is not substantively different from targeting hardware.

Tsow, Alex↗
DOE OSTI2025

A hybrid neural architecture: Online attosecond x-ray characterization

The emergence of high-repetition-rate x-ray free-electron lasers (XFELs), such as SLAC’s LCLS-II, serves as our canonical example for autonomous controls that necessitate high-throughput diagnostics paired with streaming computational pipelines capable of single-shot analysis with extremely low latency. We present the deterministic characterization with an integrated parallelizable hybrid resolver architecture, a hybrid machine learning framework designed for fast, accurate analysis of XFEL diagnostics using angular streaking-based sinogram images. This architecture integrates convolutional neural networks and bidirectional long short-term memory models to denoise input, identify x-ray sub-spike features, and extract sub-spike relative delays with sub-30 attosecond temporal resolution. Deployed on low-latency hardware, it achieves over 10 kHz throughput with 168.3 μs inference latency, indicating scalability to 14 kHz with field-programmable gate array integration. By transforming regression tasks into classification problems and leveraging optimized error encoding, we achieve high precision with low-latency performance that is critical for real-time streaming event selection and experimental control feedback signals. This represents a key development in real-time control pipelines for next-generation autonomous science, generally, and high repetition-rate x-ray experiments in particular.

Accelerator Physics (physics.acc-ph)↗
NASA NTRSDate not supplied

Biological Research and Space Health Enabled by Machine Learning to Support Deep Space Missions

A key science goal of the NASA “Moon to Mars” campaign is to understand how biology responds to the Lunar, Martian, and deep space environments in order to advance fundamental knowledge, reduce risk, and support safe, productive human space missions. Through the powerful emerging approaches of artificial intelligence (AI) and machine learning (ML), a paradigm shift has begun in biomedical science and engineered astronaut health systems, to enable Earth independence and autonomy of mission operations. Here we present an overview of AI/ML architecture to support deep space mission goals, developed with leaders in the field. First, we focus on the fundamental biological research that supports our understanding of physiological responses to spaceflight, and we describe current efforts to support AI/ML research including data standardization and data engineering through maximally open and FAIR (findable, accessible, interoperable, reusable) databases and the generation of AI-ready datasets for reuse and analysis. We also discuss remote data management frameworks for research data as well as environmental and health data that are generated during deep space missions. We highlight several research projects that leverage data standardization and management for fundamental biological discovery to uncover the complex effects of space travel on living systems. Next, we provide an overview of cutting-edge AI/ML approaches that can be integrated to support remote monitoring and analysis during deep space missions, including generative models and large language models to learn the underlying biomedical patterns and predict outcomes or answer questions during off world medical scenarios. We also describe current AI/ML methods to support this research and monitoring through automated cloud-based labs which enable limited human intervention and closed-loop experimentation in remote settings. These labs could support mission autonomy by analyzing environmental data streams, and would be facilitated through in situ analytics capabilities to avoid sending large raw data files through low bandwidth communications. Finally, in the context of deep space missions with limited communications or access to medical advice from Earth, we describe a solution for integrated, real-time mission biomonitoring across hierarchical levels from continuous environmental monitoring, to wearables and point-of-care devices, to molecular and physiological monitoring. We introduce a precision space health system that will ensure that the future of space health is predictive, preventative, participatory and personalized.

artificial intelligence↗
NASA NTRS2012

NASA Tech Briefs, May 2012

Topics covered include: An "Inefficient Fin" Non-Dimensional Parameter to Measure Gas Temperatures Efficiently; On-Wafer Measurement of a Multi-Stage MMIC Amplifier with 10 dB of Gain at 475 GHz; Software to Control and Monitor Gas Streams; Miniaturized Laser Heterodyne Radiometer (LHR) for Measurements of Greenhouse Gases in the Atmospheric Column; Anomaly Detection in Test Equipment via Sliding Mode Observers; Absolute Position of Targets Measured Through a Chamber Window Using Lidar Metrology Systems; Goldstone Solar System Radar Waveform Generator; Fast and Adaptive Lossless Onboard Hyperspectral Data Compression System; Iridium Interfacial Stack - IrIS; Downsampling Photodetector Array with Windowing; Optical Phase Recovery and Locking in a PPM Laser Communication Link; High-Speed Edge-Detecting Line Scan Smart Camera; Optical Communications Channel Combiner; Development of Thermal Infrared Sensor to Supplement Operational Land Imager; Amplitude-Stabilized Oscillator for a Capacitance-Probe Electrometer; Automated Performance Characterization of DSN System Frequency Stability Using Spacecraft Tracking Data; Histogrammatic Method for Determining Relative Abundance of Input Gas Pulse; Predictive Sea State Estimation for Automated Ride Control and Handling - PSSEARCH; LEGION: Lightweight Expandable Group of Independently Operating Nodes; Real-Time Projection to Verify Plan Success During Execution; Automated Performance Characterization of DSN System Frequency Stability Using Spacecraft Tracking Data; Web-Based Customizable Viewer for Mars Network Overflight Opportunities; Fabrication of a Cryogenic Terahertz Emitter for Bolometer Focal Plane Calibrations; Fabrication of an Absorber-Coupled MKID Detector; Graphene Transparent Conductive Electrodes for Next- Generation Microshutter Arrays; Method of Bonding Optical Elements with Near-Zero Displacement; Free-Mass and Interface Configurations of Hammering Mechanisms; Wavefront Compensation Segmented Mirror Sensing and Control; Long-Life, Lightweight, Multi-Roller Traction Drives for Planetary Vehicle Surface Exploration; Reliable Optical Pump Architecture for Highly Coherent Lasers Used in Space Metrology Applications; Electrochemical Ultracapacitors Using Graphitic Nanostacks; Improved Whole-Blood-Staining Device; Monitoring Location and Angular Orientation of a Pill; Molecular Technique to Reduce PCR Bias for Deeper Understanding of Microbial Diversity; Laser Ablation Electrodynamic Ion Funnel for In Situ Mass Spectrometry on Mars; High-Altitude MMIC Sounding Radiometer for the Global Hawk Unmanned Aerial Vehicle; PRTs and Their Bonding for Long-Duration, Extreme-Temperature Environments; Mid- and Long-IR Broadband Quantum Well Photodetector; 3D Display Using Conjugated Multiband Bandpass Filters; Real-Time, Non-Intrusive Detection of Liquid Nitrogen in Liquid Oxygen at High Pressure and High Flow; Method to Enhance the Operation of an Optical Inspection Instrument Using Spatial Light Modulators; Dual-Compartment Inflatable Suitlock; Large-Strain Transparent Magnetoactive Polymer Nanocomposites; Thermodynamic Vent System for an On-Orbit Cryogenic Reaction Control Engine; Time Distribution Using SpaceWire in the SCaN Testbed on ISS; and Techniques for Solution- Assisted Optical Contacting.

Source record↗
DOE OSTI2025

Packages of Distributed Energy Technologies Demonstrating Demand Flexibility at Community Scale

The combination of increased electric load growth across all sectors, deferred electrical infrastructure investment, and other factors resulting in variable electric power supply, has created technical challenges to maintaining a resilient and reliable grid. Many federal, regional, and local efforts are in play to modernize the electric grid, including advancing building technologies and distributed energy resources (DERs) that are utilizing smarter controls to become responsive to both occupant and grid needs. This report reviews ten pilot projects demonstrating how groups of buildings combined with behind-the-meter (BTM) DERs such as electric vehicle (EV) charging, battery storage, flexible HVAC and domestic hot water systems, and photovoltaic systems can reliably and cost effectively provide grid services. Each of the ten pilot projects aim to deliver both energy efficiency and demand flexibility (DF) while supporting load growth. The ten demonstration teams are piloting flexible DER packages across diverse communities of residential and commercial buildings to address a variety of regional grid needs. The outcomes of these pilot projects will be used to inform future scaling through utility program development. This paper characterizes the ten teams, showcasing the decision-making process used by each group to develop their packages (Section 2), the grid services they plan to deliver (Section 3), the types of DER packages selected for deployment within building sectors (Section 4) and trends between building sector, DER types, and grid services In order to achieve community scale benefits, the pilot projects must utilize aggregated control mechanisms for coordinating buildings and DERs together. Several types of coordinated control architectures have evolved amongst the teams, influenced by use type, existing market conditions, and integration type. Three coordinated controls architectures have been characterized, highlighting their use cases, benefits, challenges, and tradeoffs in their design. These insights can aid utilities, control vendors, and developers in scaling community-level energy systems (Paul, 2024). Ultimately, the technology packages selected by the ten teams will be coordinated to provide power system services, also known as grid services. Insights from these demonstrations will be useful for grid operators, regulators, aggregators and other stakeholders as they look to deploy demand flexible resources as grid services in the future. The grid services that each team is targeting for demonstration are described in Section 3 and Section 4. Methods for evaluating the grid services have been described in the paper Metrics for Evaluating Grid Service Provision from Communities of Grid-interactive and Efficient Buildings and other DER (MacDonald, 2023). To identify technology packages for demonstration, Section 2 shows that project teams used a range of analysis approaches, including building energy modeling, AMI data analysis, cost-benefit frameworks, and utility pilot data. Some teams emphasized technical modeling to quantify grid impacts and demand reduction potential, while others prioritized economic evaluations, stakeholder input, or exploratory pilots to inform deployment decisions. This diversity reflects the need to tailor selection methods to project goals, available data, and organizational context. Section 5 discusses trends between the DER technologies deployed and the grid service provisions from each team. Residential buildings (multifamily and single family) lean towards technologies that enhance energy efficiency (e.g. weatherization upgrades, smart thermostats) and onsite power generation integration (e.g. solar PV). Commercial building demonstrations prioritize technologies that ensure operational reliability (e.g. battery storage) and centralized energy management systems and optimization solutions. Teams that are deploying controllable storage-based technologies are more likely to provide grid services that require a near real-time response. Teams incorporating load shifting technologies like smart thermostats with HEMs are likely to include energy markets participation and customer bill management offerings. Campus demonstrations are adopting diverse sets of DERs to emphasize renewable generation, paired with centralized control. This section also describes technologies that were considered during project planning but ultimately excluded from final deployment. These demonstrations reveal that effective DER package design should be tailored to building type, customer segment, and construction vintage. Multifamily buildings benefit from centralized HVAC upgrades and supervisory controls, while single-family homes are well-suited for individualized technologies like solar, storage, and smart home energy monitors. Commercial and campus settings prioritize EMIS integration and load optimization. New construction enables cost-effective integration of DER-ready infrastructure, whereas retrofits require deployments aligned with owner and tenant value streams. For utility program planners, early coordination with developers and building owners, paired with segmented and modular program offerings, can improve adoption, scalability, and grid impact.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows

The evolving landscape of scientific computing requires seamless transitions from experimental to production HPC environments for interactive workflows. This paper presents a structured transition pathway developed at OLCF that bridges the gap between development testbeds and production systems. We address both technological and policy challenges, introducing frameworks for data streaming architectures, secure service interfaces, and adaptive resource scheduling for time-sensitive workloads and improved HPC interactivity. Our approach transforms traditional batch-oriented HPC into a more dynamic ecosystem capable of supporting modern scientific workflows that require near real-time data analysis, experimental steering, and cross-facility integration.

Etz, Brian [ORNL] (ORCID:0000000208554863)↗
DOE OSTI2026

The high level trigger and express data production at STAR

To meet the demands of the Beam Energy Scan phase-II (BES-II) program, the STAR experiment at the Relativistic Heavy Ion Collider (RHIC) developed a dual real-time framework consisting of a High Level Trigger (HLT) and an Express Data Production system (xProduction). The HLT operates online within the Data Acquisition (DAQ) chain on a dedicated multi-core CPU cluster with the option to offload compute-intensive kernels to Xeon Phi coprocessors. It uses parallelized algorithms, such as the Cellular Automaton (CA) Track Finder, to perform rapid tracking, vertexing, and event filtering. This allows it to select events of interest in real time and provide immediate feedback on detector and beam conditions. In contrast, the xProduction workflow runs concurrently and independently of the DAQ loop. It applies near offline-quality calibration and reconstruction within hours of data collection. The xProduction input is the express data stream, whose content can be enriched by HLT trigger/priority selections under DAQ/HLT resource constraints, and it uses the STAR calibration/conditions framework, incorporating online calibration/QA information when available. This enables early preliminary physics analysis, including the reconstruction of rare signals, such as hyperons and hypernuclei. It also provides collaboration-wide access to analysis-ready datasets. Together, the HLT and xProduction systems form a complementary architecture: the HLT performs online event selection while the xProduction chain delivers high-quality results within a short amount of time. This integrated framework has enabled the prompt reconstruction of the $^5_Λ$ He hypernucleus with high statistical significance and the efficient processing of hundreds of millions of heavy-ion collision events. In conclusion, its demonstrated scalability and robustness establish a model for future high-luminosity experiments requiring both online event filtering and rapid access to analysis-quality data.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

eCounter: Inline Per-IP Network Monitoring at Millisecond Resolution via eBPF

Scientific data acquisition (SciDAQ) systems are shifting from archive-based workflows to streaming paradigms, where real-time, fine-grained network monitoring becomes essential. While P4-enabled devices offer per-packet in-band observability, they require specialized switches and routers. Host-side tools like Prometheus exporters lack sufficient temporal granularity. To bridge this gap, we present eCounter, a lightweight, hardware-agnostic, inline telemetry agent built on extended Berkeley Packet Filter (eBPF). eCounter captures per-interface ingress and egress traffic, categorized by IP address and protocol, at millisecond to sub-millisecond resolution. In a 100 Gbps environment, it continuously exports up to 3,257 time-series bins per second with only 4% CPU utilization at a 35¿KiB/s data rate. We evaluate eCounter across diverse NIC MTU settings, hook types, CPU architectures and operating systems, and observed negligible impact on concurrent high-throughput streaming applications. Complexity analysis confirms that it can be readily scaled to distributed SciDAQ deployments.

Mei, Xinxin [Computational Sciences and Technology↗
DOE OSTI2026

Detector Interface for Streaming, Control, and Open-source integration (DISCO) v1.0.0

This suite consists of a multi-package ecosystem featuring detector emulators, EPICS areaDetector drivers, and remote server frameworks designed for the Advanced Light Source (ALS). Engineered for high-bandwidth devices—including VFCCD, Timepix3, Timepix4, and related pixel detectors—the software simulates hardware, wraps vendor SDKs into remote-callable servers, and integrates with open-source control systems. Key Capabilities: Distributed SDK Architecture: Server packages wrap hardware-specific SDKs, allowing areaDetector drivers to execute remote framework calls. This isolates proprietary libraries from the EPICS IOC, enhancing stability and enabling distributed computing across beamline networks. Device Support: Custom drivers for VFCCD, the Timepix family, and similar sensors optimize the data path from hardware control to high-speed transport. Full-Stack Emulation: Sophisticated emulator packages allow end-to-end pipeline testing and software development without requiring physical hardware or beam time. Integrated Workflows: Supports high-bandwidth streaming for real-time analysis and robust, metadata-rich file-based workflows (e.g., HDF5/NeXus). By standardizing interfaces across heterogeneous hardware, this suite reduces technical debt. It provides the ALS with a scalable, open-source solution to manage massive data rates within a unified control environment.

Mahl, Johannes [Lawrence Berkeley National Laborat↗
DOE OSTI2024

Time Series Foundation Models and Deep Learning Architectures for Earthquake Temporal and Spatial Nowcasting

Advancing the capabilities of earthquake nowcasting, the real-time forecasting of seismic activities, remains crucial for reducing casualties. This multifaceted challenge has recently gained attention within the deep learning domain, facilitated by the availability of extensive earthquake datasets. Despite significant advancements, the existing literature on earthquake nowcasting lacks comprehensive evaluations of pre-trained foundation models and modern deep learning architectures; each focuses on a different aspect of data, such as spatial relationships, temporal patterns, and multi-scale dependencies. This paper addresses the mentioned gap by analyzing different architectures and introducing two innovative approaches called Multi Foundation Quake and GNNCoder. We formulate earthquake nowcasting as a time series forecasting problem for the next 14 days within 0.1-degree spatial bins in Southern California. Earthquake time series are generated using the logarithm energy released by quakes, spanning 1986 to 2024. Our comprehensive evaluations demonstrate that our introduced models outperform other custom architectures by effectively capturing temporal-spatial relationships inherent in seismic data. The performance of existing foundation models varies significantly based on the pre-training datasets, emphasizing the need for careful dataset selection. However, we introduce a novel method, Multi Foundation Quake, that achieves the best overall performance by combining a bespoke pattern with Foundation model results handled as auxiliary streams.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Innovating the next generation of commercial smart building software

Nearly 30% of commercial building energy use is wasted due to equipment faults and HVAC controls problems. The result is increased emissions, compromised comfort and productivity, and less reliable coordination of building power needs with a clean grid. The energy impact alone represents $17 billion in potential savings. Today’s smart building software provides a robust solution to address these operational deficiencies. Energy management and information systems (EMIS) are saving up to 9% on average, with two-year paybacks. They are being incorporated into energy management processes, commissioning services, and utility programs. As effective as they are, two barriers prevent even deeper benefits; limited personnel to fix problems once they are identified, and the expense and time to manually implement changes in control systems. In partnership with the research community, the EMIS industry is developing new capabilities to overcome these barriers. Moving beyond siloed products for either fault detection and diagnostics, or optimal control, these new capabilities empower users to not only automatically identify faults, but also to push corrective action, and control improvements to their buildings. In this paper, several areas for enhancements are documented: ‘one-time’ correction of faults such as setpoints, schedules, and economizer lockouts; short-term active testing for automated proportional integral derivative (PID) loop tuning and functional testing; and continuous supervisory control for demand flexibility and year-round efficiency. Results are presented from a pair of partner implementations out of a dozen providers integrating these enhancements into their products, including field tests from across the country, and insights into operator acceptance and integration into operations and maintenance practices.

Casillas, Armando↗