Search NASA⌕ Search

SEARCH · Search NASA

Results for “zmq”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

3 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Arroyo Stream Processing Toolset (arroyopy) v0.1.0

Processing event or streaming data presents several technological challenges. A variety of technologies are often used by scientific user facilities. ZMQ is used to stream data and messages in a peer-to-peer fashion. Message brokers like Kafka, Redis Pubsub, EPICS PVA and RabbitMQ are often employed to route and pass messages from instruments to processing workflows. Arroyopy provides an API and structure to flexibly integrate with these tools and incorporate arbitrarily complex processing workflows, letting the hooks to the workflow code be independent of the connection code and hence reusable at a variety of instruments.

Chavez Esparza, Tanny Andrea [Lawrence Berkeley Na↗
DOE OSTI2026

Lightfall v0.0.1

Lightfall is a desktop application for synchrotron beamline instrument control, data acquisition, and live analysis at the Advanced Light Source (ALS). Built on Python and Qt, it provides a native graphical interface for operating beamline hardware, configuring and executing experimental scans, and visualizing results in real time. Key features include direct integration with EPICS control systems, a built-in electronic logbook, remote beamline access over secure tunnels, and an interprocess communication (IPC) architecture that coordinates with external analysis applications via ZMQ and EPICS process variables. This IPC approach allows Lightfall to orchestrate specialized analysis tools—including GPU-accelerated streaming correlators—without embedding them, avoiding the dependency conflicts common in monolithic scientific software platforms. Compared to prior approaches such as Xi-CAM's plugin-based architecture, Lightfall's design cleanly separates instrument control from domain-specific analysis, enabling feedback-driven acquisition where live analysis results can adjust scan parameters during an experiment. Its native Qt interface provides responsive performance for real-time data visualization that web-based alternatives struggle to match. Lightfall is designed for use by beamline scientists and staff operating synchrotron instruments at national user facilities.

Pandolfi, Ronald [Lawrence Berkeley National Labor↗
DOE OSTI2024

Streaming Large-Scale Microscopy Data to a Supercomputing Facility

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy generates data at a nominal rate of 480 Gbit s -1 (87,000 frames s -1 ⁠), producing a 700 GB dataset in 15 s. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera’s data acquisition system to supercomputing nodes at the National Energy Research Scientific Computing Center, bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a postmortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.

4D-STEM↗