Search NASA⌕ Search

SEARCH · Search NASA

Results for “File System”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗
DOE OSTI2020

Advanced Tri-lab Software Environment (ATSE)

The Advanced Tri-lab Software Environment (ATSE) is an effort to build an open, modular, extensible, community-engaged, and vendor-adaptable software ecosystem that enables the prototyping of new technologies for improving the ASC computing environment. The initial target for ATSE is to accelerate the maturity of the Arm ecosystem for supporting ASC computing and the high-performance computing community more broadly. ATSE provides an integrated and optimized software stack that includes: 1) Application development environment and libraries including compilers, math libraries, tools, MPI, and OpenMP, 2) Low-level system software including optimized Linux, network stack, file systems, containers, and virtual machines, 3) Job scheduling and management including workload manager, application launcher, and user tools, and 4) System administration and management tools supporting booting, monitoring, and operating system image management. SAND2020-12377 M Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Younge, Andrew J.↗
DOE OSTI2016

Impact of Burst Buffer Architectures on Application Portability

The Oak Ridge and Argonne Leadership Computing Facilities are both receiving new systems under the Collaboration of Oak Ridge, Argonne, and Livermore (CORAL) program. Because they are both part of the INCITE program, applications need to be portable between these two facilities. However, the Summit and Aurora systems will be vastly different architectures, including their I/O subsystems. While both systems will have POSIX-compliant parallel file systems, their Burst Buffer technologies will be different. This difference may pose challenges to application portability between facilities. Application developers need to pay attention to specific burst buffer implementations to maximize code portability.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Simulation-Based Validation of An Open-Source, Scalable Framework for Building Energy Management in Small and Medium-Sized Commercial Buildings

Abstract: Small and medium-sized commercial buildings (SMCBs) represent 94% of U.S. commercial buildings but encounter substantial obstacles in adopting Building Energy Management (BEM) systems. Current approaches exhibit fundamental limitations: vendor-specific API platforms restrict interoperability through proprietary ecosystems; commercial automation software demands extensive technical expertise and licensing costs; open-source IoT solutions lack native support for building automation protocols and semantic models. This paper introduces a configuration-driven web interface framework addressing the gap between smart device advancements and accessible BEM software infrastructure for SMCBs. The framework leverages VOLTTRON middleware integrated with an automated converter that processes unified YAML configurations into heterogeneous system files, reducing required configuration artifacts from six separate files to a single unified specification. The system architecture enables vendor-agnostic operation through BACnet and Modbus protocols while supporting semantic building model integration via automated Brick Schema parsing. Configuration-driven interfaces automatically adapt to diverse HVAC types without custom development. Simulation-based validation using BOPTEST demonstrates automatic interface generation between fan coil and hydronic systems, with the automated converter successfully generating all platform-specific outputs from the single YAML input. The result demonstrates the framework's capability to streamline BEM system deployment through reduced configuration complexity. This work bridges simulation capabilities with operational deployment, demonstrating how virtual testbeds validate generalizable software frameworks for real-world building automation.

Chung, Jihoon [ORNL] (ORCID:0000000184880815)↗
NASA NTRS2004

Tools for Administration of a UNIX-Based Network

Several computer programs have been developed to enable efficient administration of a large, heterogeneous, UNIX-based computing and communication network that includes a variety of computers connected to a variety of subnetworks. One program provides secure software tools for administrators to create, modify, lock, and delete accounts of specific users. This program also provides tools for users to change their UNIX passwords and log-in shells. These tools check for errors. Another program comprises a client and a server component that, together, provide a secure mechanism to create, modify, and query quota levels on a network file system (NFS) mounted by use of the VERITAS File SystemJ software. The client software resides on an internal secure computer with a secure Web interface; one can gain access to the client software from any authorized computer capable of running web-browser software. The server software resides on a UNIX computer configured with the VERITAS software system. Directories where VERITAS quotas are applied are NFS-mounted. Another program is a Web-based, client/server Internet Protocol (IP) address tool that facilitates maintenance lookup of information about IP addresses for a network of computers.

LeClaire, Stephen↗
DOE OSTI2019

End-to-end I/O portfolio for the summit supercomputing ecosystem

The I/O subsystem for the Summit supercomputer, No. 1 on the Top500 list, and its ecosystem of analysis platforms is composed of two distinct layers, namely the in-system layer and the center-wide parallel file system layer (PFS), Spider 3. The in-system layer uses node-local SSDs and provides 26.7 TB/s for reads, 9.7 TB/s for writes, and 4.6 billion IOPS to Summit. The Spider 3 PFS layer uses IBM's Spectrum Scale™ and provides 2.5 TB/s and 2.6 million IOPS to Summit and other systems. While deploying them as two distinct layers was operationally efficient, it also presented usability challenges in terms of multiple mount points and lack of transparency in data movement. To address these challenges, we have developed novel end-to-end I/O solutions for the concerted use of the two storage layers. We present the I/O subsystem architecture, the end-to-end I/O solution space, their design considerations and our deployment experience.

Oral, Sarp↗
NASA NTRS2016

Using Cloud-Based Storage Technologies for Earth Science Data

Cloud based infrastructure may offer several key benefits of scalability, built in redundancy and reduced total cost of ownership as compared with a traditional data center approach. However, most of the tools and software systems developed for NASA data repositories were not developed with a cloud based infrastructure in mind and do not fully take advantage of commonly available cloud-based technologies. Object storage services are provided through all the leading public (Amazon Web Service, Microsoft Azure, Google Cloud, etc.) and private (Open Stack) clouds, and may provide a more cost-effective means of storing large data collections online. We describe a system that utilizes object storage rather than traditional file system based storage to vend earth science data. The system described is not only cost effective, but shows superior performance for running many different analytics tasks in the cloud. To enable compatibility with existing tools and applications, we outline client libraries that are API compatible with existing libraries for HDF5 and NetCDF4. Performance of the system is demonstrated using clouds services running on Amazon Web Services.

Data↗
DOE OSTI2021

Grand unified file indexing

Systems and methods are disclosed for a unified file index for a file system. In one example, a Grand Unified File Index (GUFI) includes a tree replicating the directory hierarchy of one or more primary filesystems, and individual metadata stores for each directory. The GUFI tree permits fast traversal, efficient user space access controls, and simple tree directed operations such as renames, moves, or permission changes. In some examples, the individual metadata stores can be implemented as embedded databases on flash storage for speed. In some examples, use of summary tables at the directory or subtree level can eliminate wasteful executions, prune tree traversal, and further improve performance. In various examples, efficient operation can be achieved from laptop to supercomputer scale, across a wide mix of file distributions and filesystems.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1987

Usage analysis of user files in UNIX

Presented is a user-oriented analysis of short term file usage in a 4.2 BSD UNIX environment. The key aspect of this analysis is a characterization of users and files, which is a departure from the traditional approach of analyzing file references. Two characterization measures are employed: accesses-per-byte (combining fraction of a file referenced and number of references) and file size. This new approach is shown to distinguish differences in files as well as users, which cam be used in efficient file system design, and in creating realistic test workloads for simulations. A multi-stage gamma distribution is shown to closely model the file usage measures. Even though overall file sharing is small, some files belonging to a bulletin board system are accessed by many users, simultaneously and otherwise. Over 50% of users referenced files owned by other users, and over 80% of all files were involved in such references. Based on the differences in files and users, suggestions to improve the system performance were also made.

Devarakonda, Murthy V.↗
NASA NTRS1982

Interactive analysis of magnetic field data

The development of an interactive graphics system for speeding up the identification and analysis of magnetic field data received from spacecraft is described. A flat file system was designed with a standard header file which contained space for user notes. The system featured a minicomputer with the capacity to handle up to 20 remote terminal requests simultaneously. Time series of vector field analyses were displayed and analyzed using the B-analysis (BANAL) program, while TANAL was devised to handle trajectory analyses. BANAL permits calculations of rotations, hodograms, and power spectra, and TANAL features time series, altitude profile, and vector trajectory plots. Subsequent programs for cross-correlations and filtering are indicated.

Russell, C. T.↗
DOE OSTI2018

On Data Transfers Over Wide-Area Dedicated Connections

Dedicated wide-area network connections are employed in big data and high-performance computing scenarios, since the absence of cross-traffic promises to make it easier to analyze and optimize data transfers over them. However, nonlinear transport dynamics and end-system complexity due to multi-core hosts and distributed file systems make these tasks surprisingly challenging. We present an overview of methods to analyze memory and disk file transfers using extensive measurements over 10 Gbps physical and emulated connections with 0–366 ms round trip times (RTTs). For memory transfers, we derive performance profiles of TCP and UDT throughput as a function of RTT, which show concave regions in contrast to entirely convex regions predicted by previous models. These highly desirable concave regions can be expanded by utilizing large buffers and more parallel flows. We also present Poincar´e maps and Lyapunov exponents of TCP and UDT throughputtraces that indicate complex throughput dynamics. For disk file transfers, we show that throughput can be optimized using a combination of parallel I/O and network threads under direct I/O mode. Our initial throughput measurements of Lustre filesystems mounted over long-haul connections using LNet routers show convex profiles indicative of I/O limits.

Rao, Nageswara S.↗
NASA NTRS1995

Performance Modeling of Network-Attached Storage Device Based Hierarchical Mass Storage Systems

Network attached storage devices improve I/O performance by separating control and data paths and eliminating host intervention during the data transfer phase. Devices are attached to both a high speed network for data transfer and to a slower network for control messages. Hierarchical mass storage systems use disks to cache the most recently used files and a combination of robotic and manually mounted tapes to store the bulk of the files in the file system. This paper shows how queuing network models can be used to assess the performance of hierarchical mass storage systems that use network attached storage devices as opposed to host attached storage devices. Simulation was used to validate the model. The analytic model presented here can be used, among other things, to evaluate the protocols involved in 1/0 over network attached devices.

Menasce, Daniel A.↗
NASA NTRS1991

Aero/fluids database system

The AFAS Database System was developed to provide the basic structure of a comprehensive database system for the Marshall Space Flight Center (MSFC) Structures and Dynamics Laboratory Aerophysics Division. The system is intended to handle all of the Aerophysics Division Test Facilities as well as data from other sources. The system was written for the DEC VAX family of computers in FORTRAN-77 and utilizes the VMS indexed file system and screen management routines. Various aspects of the system are covered, including a description of the user interface, lists of all code structure elements, descriptions of the file structures, a description of the security system operation, a detailed description of the data retrieval tasks, a description of the session log, and a description of the archival system.

Reardon, John E.↗
DOE OSTI2021

Scalable augmented enumeration and metadata operations for large filesystems

Systems, apparatus, and methods are disclosed for performing scalable operations in a file system. Metadata entries in a namespace or directory tree are sharded across multiple file metadata servers. An augmented enumeration operation, such as listing a directory, is parallelized across the multiple file metadata servers, transparently to clients. Exemplary augmentation features can include filtering and sorting. Augmentation features can be executed concurrently with enumeration, prior to enumeration, after enumeration, or as a combination of these, and can utilize pre-built index structures or holding structures for intermediate results. Augmented enumeration operations can also include no-output operations such as changing file attributes or deleting a file, and cumulative operations such as counting total disk space usage. The parallelization is compatible with tree-level parallelization and storage-level parallelization. Disclosed technologies can be applied to other fields requiring scalable enumeration, such as database and network applications.

Grider, Gary A.↗
DOE OSTI2022

MEASUR - Manufacturing Energy Assessment Software for Utility Reduction

MEASUR your energy savings with the free DOE MEASUR software The Department of Energy (DOE), with Oak Ridge National Laboratory (ORNL), released version 1.0 of their energy efficiency software tool MEASUR (Manufacturing Energy Assessment Software for Utility Reduction). MEASUR has been available for several years as a beta version, being tested by industry experts and real users, and will continue to be updated and improved in the coming years. It is an integrated suite of tools to aid manufacturers in improving the efficiency of energy systems and equipment within a plant, including motors, pumps, fans, process heating, steam, and compressed air. Additionally, there are modules for wastewater energy analysis and to help perform energy treasure hunts. Several calculators are also included, allowing users to independently perform smaller calculations and analyses (such as estimating pump head, performing a fan traverse analysis, estimating waste heat recovery potential, and cataloging compressed air leaks). The MEASUR modules are based on previous DOE software tools that have been used by industry since the early 2000s (such as MotorMaster, AirMaster+, PSAT, PHAST, and FSAT). The original tools only ran on Windows operating systems, and by Windows 10, most of them were inoperable. DOE started their energy efficiency software tool revitalization effort in 2016, first with PSAT (for pumps), then began to integrate the other tools and expand their functionality and utility. The new MEASUR suite provides an extensively more user-friendly, modern, and versatile set of tools. All the assessment modules and most of the calculators have several visual components and graphs and detailed help text for every user input. To help reach international users, the tool utilizes Google translate and users can easily change unit systems, even converting existing user inputs if desired. The assessment files can be organized within the internal file system and easily shared to other users, regardless of their operating system. The entire suite is free, open-source, and can be downloaded on Windows, Mac, or Linux operating systems.

Accawi, Gina [Oak Ridge National Lab. (ORNL), Oak ↗
NASA NTRS1997

High Performance Databases For Scientific Applications

The goal for this task is to develop an Extensible File System (ELFS). ELFS attacks the problem of the following: 1. Providing high bandwidth performance architectures; 2. Reducing the cognitive burden faced by applications programmers when they attempt to optimize; and 3. Seamlessly managing the proliferation of data formats and architectural differences. The approach for ELFS solution consists of language and run-time system support that permits the specification on a hierarchy of file classes.

French, James C.↗
NASA NTRS2004

HEC Applications on Columbia Project

NASA's Columbia system consists of a cluster of twenty 512 processor SGI Altix systems. Each of these systems is 3 TFLOP/s in peak performance - approximately the same as the entire compute capability at NAS just one year ago. Each 512p system is a single system image machine with one Linunx O5, one high performance file system, and one globally shared memory. The NAS Terascale Applications Group (TAG) is chartered to assist in scaling NASA's mission critical codes to at least 512p in order to significantly improve emergency response during flight operations, as well as provide significant improvements in the codes. and rate of scientific discovery across the scientifc disciplines within NASA's Missions. Recent accomplishments are 4x improvements to codes in the ocean modeling community, 10x performance improvements in a number of computational fluid dynamics codes used in aero-vehicle design, and 5x improvements in a number of space science codes dealing in extreme physics. The TAG group will continue its scaling work to 2048p and beyond (10240 cpus) as the Columbia system becomes fully operational and the upgrades to the SGI NUMAlink memory fabric are in place. The NUMlink uprades dramatically improve system scalability for a single application. These upgrades will allow a number of codes to execute faster at higher fidelity than ever before on any other system, thus increasing the rate of scientific discovery even further

Taft, Jim↗
DOE OSTI2021

Battle of the Defaults: Extracting Performance Characteristics of HDF5 under Production Load

Popular parallel I/O libraries, such as HDF5, provide tuning parameters to obtain superior performance. However, the selection of effective parameters on production systems is complex due to the interdependence of I/O software and file system layers. Hence, application developers typically use the default parameters and often experience poor I/O performance. This work conducts a benchmarking-based analysis on the HDF5 behaviors with a wide variety of I/O patterns to extract performance characteristics under the production workload. To make the analysis well controlled, we exercise I/O benchmarks on POSIX-IO, MPI-IO, and HDF5 using the same I/O patterns and in the same jobs. To address high performance variability in production environments, we repeat the benchmarks across I/O patterns, storage devices, and time intervals. Based on the results, we identified consistent HDF5 behaviors that appropriate configurations and operations on dataset layout and file-metadata placement can improve performance significantly. We apply our findings and evaluate the tuned I/O library on two supercomputers: Summit and Cori. The results show that our tuned parameters can achieve more than 10× I/O performance speedup than that with default parameters on both systems, suggesting the effectiveness, stability, and generality of our solution.

Xie, Bing↗