SEARCH · Search NASA

Results for “memory mapping”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

DOE OSTI2021

Enabling Scalable and Extensible Memory-mapped Datastores in Userspace

Exascale workloads are expected to incorporate data-intensive processing in close coordination with traditional physics simulations. These emerging scientific, data-analytics and machine learning applications need to access a wide variety of datastores in flat files and structured databases. Programmer productivity is greatly enhanced by mapping datastores into the application process's virtual memory space to provide a unified “in-memory” interface. Currently, memory mapping is provided by system software primarily designed for generality and reliability. However, scalability at high concurrency is a formidable challenge on exascale systems. Also, there is a need for extensibility to support new datastores potentially requiring HPC data transfer services. In this article, we present UMap , a scalable and extensible userspace service for memory-mapping datastores. Furthermore, through decoupled queue management, concurrency aware adaptation, and dynamic load balancing, UMap enables application performance to scale even at high concurrency. We evaluate UMap in data-intensive applications, including sorting, graph traversal, database operations, and metagenomic analytics. Our results show that UMap as a userspace service outperforms an optimized kernel-based service across a wide range of intra-node concurrency by 1.22-1.9 × . We performed two case studies to demonstrate UMap 's extensibility. First, a new datastore residing in remote memory is incorporated into UMap as an application-specific plugin. Second, we present a persistent memory allocator Metall built atop UMap for unified storage/memory.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

UMap: An application-oriented user level memory mapping library

Exploiting the prominent role of complex memories in exascale node architecture, the UMap page fault handler offers new capabilities to access large memory-mapped data sets directly. UMap provides flexible configuration options to customize page handling to each application, including analysis of massive observational and simulation data sets. The high-performance design features I/O decoupling, dynamic load balancing, and application-level controls. Page faults triggered by application threads and processes accessing data mapped to a UMapp’ed region are handled via the Linux userfaultfd protocol, an asynchronous message-oriented kernel-user communication mechanism that avoids the context switch penalty of traditional signal fault handlers. UMap is fully open source. In this paper, we give an overview of the UMap library architecture, its extensible plugin architecture, and the use/performance of UMap in emerging heterogeneous memory hierarchies such as near-node Non-volatile Memory (NVM) and network attached memories. We highlight new capabilities in two pagefault management plugins, the NetworkStore and SparseStore. We demonstrate the integration between UMap and multiple ECP products including Caliper, Metall, ZFP, Mochi, and Ripples.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1976

A memory-mapped output interface: Omega navigation output data from the JOLT (TM) microcomputer

A hardware interface which allows both digital and analog data output from the JOLT microcomputer is described in the context of a software-based Omega Navigation receiver. The interface hardware described is designed for output of six (or eight with simple extensions) bits of binary output in response to a memory store command from the microcomputer. The interface was produced in breadboard form and is operational as an evaluation aid for the software Omega receiver.

Lilley, R. W.↗

NASA NTRS1977

A four digit memory-mapped display

A circuit for interfacing the JOLT (TM) microprocessor to four digital displays was designed and fabricated. Circuit operation and test program are described and presented in detail.

Smith, R. E.↗

NASA NTRS2011

S-Band POSIX Device Drivers for RTEMS

This is a set of POSIX device driver level abstractions in the RTEMS RTOS (Real-Time Executive for Multiprocessor Systems real-time operating system) to SBand radio hardware devices that have been instantiated in an FPGA (field-programmable gate array). These include A/D (analog-to-digital) sample capture, D/A (digital-to-analog) sample playback, PLL (phase-locked-loop) tuning, and PWM (pulse-width-modulation)-controlled gain. This software interfaces to Sband radio hardware in an attached Xilinx Virtex-2 FPGA. It uses plug-and-play device discovery to map memory to device IDs. Instead of interacting with hardware devices directly, using direct-memory mapped access at the application level, this driver provides an application programming interface (API) offering that easily uses standard POSIX function calls. This simplifies application programming, enables portability, and offers an additional level of protection to the hardware. There are three separate device drivers included in this package: sband_device (ADC capture and DAC playback), pll_device (RF front end PLL tuning), and pwm_device (RF front end AGC control).

Lux, James P.↗

DOE OSTI2025

Optimizing Management of Persistent Data Structures in High-Performance Analytics

Large-scale data analytics workflows ingest massive input data into various data structures, including graphs and key-value datastores. These data structures undergo multiple transformations and computations and are typically reused in incremental and iterative analytics workflows. Persisting in-memory views of these data structures enables reusing them beyond the scope of a single program run while avoiding repetitive raw data ingestion overheads. Memory-mapped I/O enables persisting in-memory data structures without data serialization and deserialization overheads. However, memory-mapped I/O lacks the key feature of persisting consistent snapshots of these data structures for incremental ingestion and processing. The obstacles to efficient virtual memory snapshots using memory-mapped I/O include background writebacks outside the application’s control, and the significantly high storage footprint of such snapshots. To address these limitations, we present Privateer, a memory and storage management tool that enables storage-efficient virtual memory snapshotting while also optimizing snapshot I/O performance. Here, we integrated Privateer into Metall, a state-of-the-art persistent memory allocator for C++, and the Lightning Memory-Mapped Database (LMDB), a widely-used key-value datastore in data analytics and machine learning. Privateer optimized application performance by 1.22× when storing data structure snapshots to node-local storage, and up to 16.7× when storing snapshots to a parallel file system. Privateer also optimizes storage efficiency of incremental data structure snapshots by up to 11× using data deduplication and compression.

Computer science↗

NASA NTRS1991

Fuzzy associative memories

Mappings between fuzzy cubes are discussed. This level of abstraction provides a surprising and fruitful alternative to the propositional and predicate-calculas reasoning techniques used in expert systems. It allows one to reason with sets instead of propositions. Discussed here are fuzzy and neural function estimators, neural vs. fuzzy representation of structured knowledge, fuzzy vector-matrix multiplication, and fuzzy associative memory (FAM) system architecture.

Kosko, Bart↗

DOE OSTI2024

Computing Sparse Tensor Decompositions via Chapel and C++/MPI Interoperability without Intermediate I/O

We extend an existing approach for efficient use of shared mapped memory across Chapel and C++ for graph data stored as 1-D arrays to sparse tensor data stored using a combination of 2-D and 1-D arrays. We describe the specific extensions that provide use of shared mapped memory tensor data for a particular C++ tensor decomposition tool called GentenMPI. We then demonstrate our approach on several real-world datasets, providing timing results that illustrate minimal overhead incurred using this approach. Finally, we extend our work to improve memory usage and provide convenient random access to sparse shared mapped memory tensor elements in Chapel, while still being capable of leveraging high performance implementations of tensor algorithms in C++.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1992

How to handle 6GBytes a night and not get swamped

The Macho Project has undertaken a 5 year effort to search for dark matter in the halo of the Galaxy by scanning the Magellanic Clouds for micro-lensing events. Each evening's raw image data will be reduced in real-time into the observed stars' photometric measurements. The actual search for micro-lensing events will be a post-processing operation. The theoretical prediction of the rate of such events necessitates the collection of a large number of repeated exposures. The project designed camera subsystem delivers 64 Mbytes per exposure with exposures typically occurring every 500 seconds. An ideal evening's observing will provide 6 Gbytes of raw image data and 40 Mbytes of reduced photometric measurements. Recognizing the difficulty of digging out from a snowballing cascade of raw data, the project requires the real-time reduction of each evening's data. The software team's implementation strategy centered on this non-negotiable mandate. Accepting the reality that 2 full time people needed to implement the core real-time control and data management system within 6 months, off-the-shelf vendor components were explored to provide quick solutions to the classic needs for file management, data management, and process control. Where vendor solutions were lacking, state-of-the-art models were used for hand tailored subsystems. In particular, petri nets manage process control, memory mapped bulletin boards provide interprocess communication between the multi-tasked processes, and C++ class libraries provide memory mapped, disk resident databases. The differences between the implementation strategy and the final implementation reality are presented. The necessity of validating vendor product claims are explored. Both the successful and hindsight decisions enabling the collection and processing of the nightly data barrage are reviewed.

Allsman, R.↗

NASA NTRS1984

Memory-based parallel data output controller

A memory-based parallel data output controller employs associative memories and memory mapping to decommutate multiple channels of telemetry data. The output controller contains a random access memory (RAM) which has at least as many address locations as there are channels. A word counter addresses the RAM which provides as it outputs an encoded peripheral device number and a MSB/LSB-first flag. The encoded device number and a bit counter address a second RAM which contains START and STOP flags to pick out the required bits from the specified word number. The LSB/MSB, START and STOP flags, along with the serial input digital data go to a control block which selectively fills a shift register used to drive the parallel data output bus.

Stattel, R. J.↗

DOE OSTI2022

Metall: A persistent memory allocator for data-centric analytics

Data analytics applications transform raw input data into analytics-specific data structures before performing analytics. Unfortunately, such data ingestion steps are often more expensive than analytics. In addition, various types of NVRAM devices are already used in many HPC systems today. Such devices will be useful for storing and reusing data structures beyond a single process life cycle. We developed Metall, a persistent memory allocator built on top of the memory-mapped file mechanism. Metall enables applications to transparently allocate custom C++ data structures into various types of persistent memories. Metall incorporates a concise and high-performance memory management algorithm inspired by Supermalloc and the rich C++ interface developed by Boost.Interprocess library. On a dynamic graph construction workload, Metall achieved up to 11.7x and 48.3x performance improvements over Boost.Interprocess and memkind (PMEM kind), respectively. We also demonstrate Metall’s high adaptability by integrating Metall into a graph processing framework, GraphBLAS Template Library. Here this study’s outcomes indicate that Metall will be a strong tool for accelerating future large-scale data analytics by allowing applications to leverage persistent memory efficiently.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Tiling Framework for Heterogeneous Computing of Matrix based Tiled Algorithms

Tiling matrix operations can improve the load balancing and performance of applications on heterogeneous computing resources. Writing a tile-based algorithm for each operation with a traditional, hand-tuned tiling approach that uses for loops in C/C++ is cumbersome and error prone. Moreover, it must enable and support the heterogeneous memory management of data objects and also explore architecture-supported, native, tiled-data transfer APIs instead of copying the tiled data to continuous memory before the data transfer. The tiling framework provides a tiled data structure for heterogeneous memory mapping and parameterization to a heterogeneous task specification API. We have integrated our tiled framework into MatRIS (Math kernels library using IRIS). IRIS is a heterogeneous run-time framework with a heterogeneous programming model, memory model, and task execution model. Experiments reveal that the tiled framework for BLAS operations has improved the programmability of tiled BLAS and improved performance by ~20% when compared against the traditional method that copies the data to continuous memory locations for heterogeneous computing.

Miniskar, Narasinga Rao↗

DOE OSTI2026

rustpix

rustpix is a high-performance, open-source Rust library with first-class Python bindings (via PyO3) for processing pixel-detector data in neutron imaging. It targets time-stamping detectors such as Timepix3 (TPX3) at ORNL's Spallation Neutron Source (VENUS beamline), where each detected neutron deposits charge across a cluster of pixels within a very high-rate event stream (96M+ hits/sec). rustpix parses TPX3 event data in parallel using memory-mapped I/O, offers four interchangeable clustering algorithms (ABS adjacency-based search, DBSCAN, graph/union-find connected components, and a parallel grid method), and extracts weighted, super-resolved centroids to produce neutron-event lists. A streaming architecture lets it process files larger than available memory. rustpix is distributed as a pip-installable Python package (with NumPy integration), Rust crates, a command-line tool, and an interactive GUI; it writes HDF5, Apache Arrow, and CSV; and it is designed to extend to TPX4 and other detector types. Released as open-source under the MIT License.

Zhang, Chen [Oak Ridge National Laboratory (ORNL),↗

DOE OSTI2021

UPC++ as_eager Working Group Draft, Revision 2020.6.2

This draft proposes an extension for a new future-based completion variant that can be more effectively streamlined for RMA and atomic access operations that happen to be satisfied at runtime using purely node-local resources. Many such operations are most efficiently performed synchronously using load/store instructions on shared-memory mappings, where the actual access may only require a few CPU instructions. In such cases we believe it’s critical to minimize the overheads imposed by the UPC++ runtime and completion queues, in order to enable efficient operation on hierarchical node hardware using shared-memory bypass. The new upcxx::{source,operation}_cx::as_eager_future() completion variant accomplishes this goal by relaxing the current restriction that future-returning access operations must return a non-ready future whose completion is deferred until a subsequent explicit invocation of user-level progress. This relaxation allows access operations that are completed synchronously to instead return a ready future, thereby avoiding most or all of the runtime costs associated with deferment of future completion and subsequent mandatory entry into the progress engine. We additionally propose to make this new as_eager_future() completion variant the new default completion for communication operations that currently default to returning a future. This should encourage use of the streamlined variant, and may provide performance improvements to some codes without source changes. A mechanism is proposed to restore the legacy behavior on-demand for codes that might happen to rely on deferred completion for correctness. Finally, we propose a new as_eager_promise() completion variant that extends analogous improvements to promise-based completion, and corresponding changes to the default behavior of as_promise().

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

JTAG-based PLC memory acquisition framework for industrial control systems

In industrial control systems (ICS), programmable logic controllers (PLC) are the embedded devices that directly control and monitor critical industrial infrastructure processes such as nuclear plants and power grid stations. Cyberattacks often target PLCs to sabotage a physical process. A memory forensic analysis of a suspect PLC can answer questions about an attack, including compromised firmware and manipulation of PLC control logic code and I/O devices. Given physical access to a PLC, collecting forensic information from the PLC memory at the hardware-level is risky and challenging. It may cause the PLC to crash or hang since PLCs have proprietary, legacy hardware with heterogeneous architecture. This paper addresses this research problem and proposes a novel JTAG (Joint Test Action Group)-based framework, Kyros, for reliable PLC memory acquisition. Kyros systematically creates a JTAG profile of a PLC through hardware assessment, JTAG pins identification, memory map creation, and optimizing acquisition parameters. It also facilitates the community of interest (such as ICS owners, operators, and vendors) to develop the JTAG profiles of PLCs. Further, we present a case study of Kyros implementation over Allen-Bradley 1756-A10/B to help understand the framework's application on a real-world PLC used in industry settings. The sample PLC memory dumps are shared with the research community to facilitate further research.

Rais, Muhammad Haris↗

DOE OSTI2026

Gaia: segmented germanium detector for high-energy X-ray fluorescence and spectroscopic imaging

We present Gaia, a monolithic array of 96 high-purity germanium pixel detectors integrated with a custom low-noise application-specific integrated circuit (ASIC) and a field-programmable gate array (FPGA)-based data acquisition system. The sensor operates at ∼100 K using a commercial closed-cycle cryocooler, with the in-vacuum electronics thermally isolated from the cold finger to ensure thermal stability. The system demonstrates an average energy resolution of 711 eV at 122 keV, measured using a 57 Co source, and 253 eV at 5.89 keV, measured with 55 Fe across all channels. The readout architecture incorporates a high-performance FPGA paired with a dual-core ARM processor, forming a complete embedded Linux-based computing platform. Communication between the processor and FPGA is handled via memory-mapped I/O, and data are streamed over high-speed gigabit Ethernet. A full-scale 384-pixel Gaia detector, based on this 96-element module, is currently under fabrication.

36 MATERIALS SCIENCE↗

DOE OSTI2020

Highly Asynchronous Visitor Queue Graph Toolkit

HavoqGT (Highly Asynchronous Visitor Queue Graph Toolkit) is a framework for expressing asynchronous vertex-centric graph algorithms, and executing them on High Performance Computing (HPC) systems. It provides a vertex 'visitor' interface, where actions are defined at an individual vertex level, and contains a suite of classic graph algorithms. HavoqGT is capable of processing large graphs stored in NVRAM (SSDs) using a memory mapped interface.

Reza, TahsinA.↗

DOE OSTI2020

XPlacer/Tracer

XPlacer/Tracer is a dynamic analysis tool that finds bad memory access patterns in heterogeneous codes. XPlacer/Tracer consists of two components. (1) A configurable ROSE plugin for source code instrumentation of heterogeneous code written in the C/C++ and CUDA programming languages. (2) A sample configuration that instruments memory accesses to dynamic memory in CPU and GPU codes, and a runtime library that tracks these memory accesses at runtime. The collected information is reported in form of a textual summary or as memory map that can be converted to images.

Pirkelbauer, PeterM.↗