Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data management systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

PANSOPHY, a JLAB SRF engineering data management system, supporting data collection, retrieval and analysis utilized by LCLS-II

Pansophy is an Engineering Data Management System that provides a comprehensive solution for managing information in the production and testing of cryomodules. It is especially suited to supporting the Data & Quality Management Systems for large projects like LCLS-II. With extensive amounts of data collected for an individual project, data retrieval to facilitate feedback and enhancement of production and processing activities is a high priority. The priority shares importance with the needs of managing the project, including production status, NCR, and Quality Management reports. Recent Pansophy enhancements have been to Data and Quality management reports and statistical analysis. Such enhancements include a database driven menu system, extended MSWord macro and preprocessing of travelers, and an extensive reporting system. The reporting system allows managers and group leaders to quickly respond to the needs of the project in areas of cavity and cryomodule production, data collection, NCR, Quality Management and schedule. Extensions include integration with the SRF inventory system PRIMeS, allowing traceability from receiving of manufactured parts to final cryomodule product.

Bookwalter, Valerie D.↗
DOE OSTI2020

LDM-148: Rubin Observatory Data Management System Design

The Rubin Observatory LSST Data Management System (DMS) is a set of services employing a variety of software components running on computational and networking infrastructure that combine to deliver science data products to the observatory's users and support observatory operations. This document describes the components, their service instances, and their deployment environments as well as the interfaces among them, the rest of the LSST system, and the outside world.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2018

The ATLAS Data Management System Rucio: Supporting LHC Run-2 and beyond

With this contribution we present some recent developments made to Rucio, the data management system of the High-Energy Physics Experiment ATLAS. Already managing 300 Petabytes of both official and user data, Rucio has seen incremental improvements throughout LHC Run-2, and is currently laying the groundwork for HEP computing in the HL-LHC era. The focus of this contribution are (a) the automations that have been put in place such as data rebalancing or dynamic replication of user data, as well as their supporting infrastructures such as real-time networking metrics or transfer time predictions; (b) the flexible approach towards inclusion of heterogeneous storage systems, including object stores, while unifying the potential access paths using generally available tools and protocols; (c) machine learning approaches to help with transfer throughput estimation; and (d) the adoption of Rucio for two other experiments, AMS and Xenon1t. We conclude by presenting operational numbers and figures to quantify these improvements, and extrapolate the necessary changes and developments for future LHC runs.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2018

MetaHCR: a web-enabled metagenome data management system for hydrocarbon resources

The ever-increasing metagenomic data necessitate appropriate cataloguing in a way that facilitates the comparison and better contextualization of the underlying investigations. To this extent, information associated with the sequencing data as well as the original sample and the environment where it was obtained from is crucial. To date, there are not any publicly available repositories able to capture environmental metadata pertaining to hydrocarbon-rich environments. As such, contextualization and comparative analysis among sequencing datasets derived from these environments is to a certain degree hindered or cannot be fully evaluated. The metagenomics data management system for hydrocarbon resources (MetaHCRs) enables the capturing of marker gene and whole metagenome sequencing data as well as over 300 contextual attributes associated with samples, organisms, environments and geological properties, among others. Moreover, MetaHCR implements the Minimum Information about any Sequence–hydrocarbon resource specification from the Genomic Standards Consortium; it integrates a user-friendly web interface and relational database model, and it enables the generation of complex custom search. MetaHCR has been tested with 36 publicly available metagenomic studies, and its modular architecture can be easily customized for other types of environmental and metagenomics studies.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Recent Experience with the CMS Data Management System

The CMS[1] experiment manages a large-scale data infrastructure, currently handling over 200 PB of disk and 500 PB of tape storage and transferring more than 1 PB of data per day on average between various WLCG[2] sites. Utilizing Rucio[3] for high-level data management, FTS[4] for data transfers, and a variety of storage and network technologies at the sites, CMS confronts inevitable challenges due to the system’s growing scale and evolving nature. Key challenges include managing transfer and storage failures, optimizing data distribution across different storages based on production and analysis needs, implementing necessary technology upgrades and migrations, and efficiently handling user requests. The data management team has established comprehensive monitoring to supervise this system and has successfully addressed many of these challenges. The team’s efforts aim to ensure data availability and protection, minimize failures and manual interventions, maximize transfer throughput and resource utilization, and provide reliable user support. This paper details the operational experience of CMS with its data management system in recent years, focusing on the encountered challenges, the effective strategies employed to overcome them and the ongoing challenges as we prepare for future demands.

Öztürk, Hasan [CERN]↗
DOE OSTI2018

A Conditions Data Management System for HEP Experiments

Conditions data infrastructure for both ATLAS and CMS have to deal with the management of several Terabytes of data. Distributed computing access to this data requires particular care and attention to manage request-rates of up to several tens of kHz. Thanks to the large overlap in use cases and requirements, ATLAS and CMS have worked towards a common solution for conditions data management with the aim of using this design for data-taking in Run 3. In the meantime other experiments, including NA62, have expressed an interest in this cross- experiment initiative. For experiments with a smaller payload volume and complexity, there is particular interest in simplifying the payload storage. The conditions data management model is implemented in a small set of relational database tables. A prototype access toolkit consisting of an intermediate web server has been implemented, using standard technologies available in the Java community. Access is provided through a set of REST services for which the API has been described in a generic way using standard Open API specications, implemented in Swagger. Such a solution allows the automatic generation of client code and server stubs and further allows changes in the backend technology transparently. An important advantage of using a REST API for conditions access is the possibility of caching identical URLs, addressing one of the biggest challenges that large distributed computing solutions impose on conditions data access, avoiding direct DB access by means of standard web proxy solutions.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2018

APS Data Management System

As the capabilities of modern X-ray detectors and acquisition technologies increase, so do the data rates and volumes produced at synchrotron beamlines. This brings into focus a number of challenges related to managing data at such facilities, including data transfer, near real-time data processing, automated processing pipelines, data storage, handling metadata and remote user access to data. The Advanced Photon Source Data Management System software is designed to help beamlines deal with these issues. Finally, this paper presents the system architecture and describes its components and functionality; the system's current usage is discussed, examples of its use have been provided and future development plans are outlined.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

MetaCat - metadata catalog for data management systems

Metadata management is one of three major areas and parts of functionality of scientific data management along with replica management and workflow management. Metadata is the information describing the data stored in a data item, a file or an object. It includes the data item provenance, recording conditions, format and other attributes. MetaCat is a metadata management database designed and developed for High Energy Physics experiments. As a component of a data management system, it’s main objectives are to provide efficient metadata storage and management and fast data items selection functionality. MetaCat is supposed to work on the scale of 100 million files (or objects) and beyond. The article will discuss the functionality of MetaCat and technological solutions used to implement the product.

Mandrichenko, Igor↗
DOE OSTI2024

LSE-61: Rubin Observatory Data Management System (DMS) Requirements

This document contains the top-level requirements for the Data Management subsystem of the LSST, when combined with the Data Management-related requirements in the LSST System Requirements (LSR, LSE-29) and Observatory System Specifications (OSS, LSE-30).

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2017

Innovative Data Management System in Support of Remedial Actions for Legacy Radioactive Contamination - 17280

The Port Hope Area Initiative (PHAI) is a project mandated and funded by the Government of Canada to remove contamination from properties with legacy low-level radioactive waste (LLRW) contamination arising from a crown-owned radium and, later, uranium refinery in the Town of Port Hope, Ontario. From 1976 to 1981, a community clean-up project was undertaken, with the specific purpose to decrease the level of historic LLRW contamination within Port Hope. Since then, residual LLRW has been managed by the Low Level Radioactive Waste Management Office (LLRWMO) (and its predecessors). The PHAI and the Port Hope Project (PHP) were initiated in 2001 with a formal agreement between affected municipalities and the federal government to develop a safe, long-term management solution for the historic LLRW remaining within Port Hope. An important activity of the PHP is the survey of about 4,800 'small scale sites' (SSS) throughout the community to confirm the presence or absence of LLRW. This involves surveys of properties based on the measurement of exterior gamma, contaminated objects and surfaces, soil concentrations and indoor radon and classification to Port Hope project specific criteria. A very large amount of information is involved in the project including; scheduling individual field visits to the properties, capture of field data and comments, laboratory sample tracking, QA/QC, report generation and internal and client project reporting at regular schedules. Field data collection of exterior gamma radiation surveys includes direct (real time) and interactive loading of data into the Information Management (IM) system to support initial QA/QC and reporting back to the field on radiation levels. Screening soil measurements including X-ray fluorescence (XRF) and gamma radiation are collected from all 20 cm increments in each borehole in addition to down-hole gamma radiation measurements and logging of soil characteristics which are used in documentation and interpretation of the property condition. The IM system outlined in this paper was developed to respond to the requirements for radiological surveys in Port Hope. The IM system continues to evolve as a part of continuing review and improvement. The IM system incorporates Web-enabled field tools and cloud hosted data and applications, provides increased field efficiency, timeliness in reporting, real time QA/QC analysis, and consistency in tracking project advances and changes made to the data for decision-making and reporting on an ongoing basis. Examples of specific applications are provided. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2026

GenomeDepot: data management system for microbial comparative genomics

Summary GenomeDepot is an open-source web-based platform for annotation, management, and comparative analysis of microbial genomic sequences and associated data including ortholog families, protein domains, operons, regulatory interactions, strain taxonomy, and sample metadata. GenomeDepot supports rapid creation of websites for user-defined genome collections that include bioinformatic tools for interactive genome browsing, Basic Local Alignment Search Tool (BLAST) search, annotation search, comparative genomic neighborhood visualization, and sequence download. Gene function annotations are generated by a customizable annotation pipeline. The pipeline runs annotation tools in Conda environments and can be easily extended with additional user-specified tools. Availability and implementation GenomeDepot is open source and distributed under the GNU General Public License via GitHub (https://github.com/aekazakov/genome-depot). GenomeDepot is implemented in Python and was tested in Ubuntu Linux. Full installation instructions and documentation are available at https://aekazakov.github.io/genome-depot/. GenomeDepot demo server is freely accessible at https://iseq.lbl.gov/demogd/.

Kazakov, Alexey [Lawrence Berkeley National Labora↗
DOE OSTI2018

Data Import Manager

The Data Import Manager is a part of PRISM, the Proteomics Research Information and Management System. The Data Import Manager monitors a central folder for new dataset trigger files, which are XML files that specify metadata for new datasets to import into the Data Management System (DMS). The Data Import Manager validates the metadata in the XML trigger file, notifying the instrument operator by e-mail if invalid metadata is encountered. After processing each XML trigger file, the file is moved into either a success or failure folder.

Monroe, Matthew↗
DOE OSTI2022

Exploring Legacy Document Management Systems to Increase Access to Project Data

The current Electronic Data Management System (EDMS) makes it difficult to efficiently access information in critical time frames. This project explored ways to allow Environmental Support and Services staff the ability to efficiently access documents with necessary historic information while leveraging existing data classification methods and the capabilities of EDMS.

99 GENERAL AND MISCELLANEOUS↗