Results for “Data access” — Page 15

DOE OSTI2025

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗

DOE OSTI2025

Community Requirements Meta-Analysis: Characterizing Needs and Opportunities for HPDF

This High Performance Data Facility (HPDF) Project is creating a new scientific user facility to provide advanced infrastructure for data-intensive science, supporting the DOE’s Office of Science (SC) community. HPDF’s mission is to enable and accelerate scientific discovery by delivering state-of-the-art data management infrastructure, capabilities, and tools. This meta-analysis examines the needs of the breadth of the SC community, captured in publicly available community reports or mission documents. The meta-analysis identifies and provides initial characterization of fifteen core requirements for the HPDF Project team to consider during the conceptual design phase. The fifteen requirements illustrate how scientific work among SC communities requires modern, seamless user experiences across the ASCR Ecosystem to advance the use of large volumes of heterogeneous data. The scientific community requires support for the missing middle of compute between local and HPC to interactively and collaboratively use growing datasets. Data producers and end users will benefit from enhanced data catalogs and portals that improve data access through advanced search of well curated data. The fifteen requirements are examined here organized across five themes for discussion. Examples in each theme illustrate the array of scientific needs that convey the important role that the fully realized and operational High Performance Data Facility will be able to play as an integral part of the evolving ASCR Ecosystem. Our amalgamated data tables from ESnet reports demonstrate ranges to the volumes of data HPDF must be concerned with, but limitations are inherent to this meta-analysis (see Key Challenges & Limitations). Feedback and validation of these requirements along with additional details and emergent community requirements will be gathered through user research and design activities.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1981

Preliminary evaluation of the FGGE special effort for data enhancement

Man-computer Interactive Data Access System (McIDAS) terminals were utilized for data evaluation, quality assessment, and satellite data enhancement.

Atlas, R.↗

NASA NTRS1986

Rdesign: A data dictionary with relational database design capabilities in Ada

Data Dictionary is defined to be the set of all data attributes, which describe data objects in terms of their intrinsic attributes, such as name, type, size, format and definition. It is recognized as the data base for the Information Resource Management, to facilitate understanding and communication about the relationship between systems applications and systems data usage and to help assist in achieving data independence by permitting systems applications to access data knowledge of the location or storage characteristics of the data in the system. A research and development effort to use Ada has produced a data dictionary with data base design capabilities. This project supports data specification and analysis and offers a choice of the relational, network, and hierarchical model for logical data based design. It provides a highly integrated set of analysis and design transformation tools which range from templates for data element definition, spreadsheet for defining functional dependencies, normalization, to logical design generator.

Lekkos, Anthony A.↗

NASA NTRS2012

Analysis of Water and Energy Budgets and Trends Using the NLDAS Monthly Data Sets

The North American Land Data Assimilation System (NLDAS) is a collaborative project between NASA GSFC, NOAA, Princeton University, and the University of Washington. NLDAS has created surface meteorological forcing data sets using the best-available observations and reanalyses. The forcing data sets are used to drive four separate land-surface models (LSMs), Mosaic, Noah, VIC, and SAC, to produce data sets of soil moisture, snow, runoff, and surface fluxes. NLDAS hourly data, accessible from the NASA GES DISC Hydrology Data Holdings Portal, http://disc.sci.gsfc.nasa.gov/hydrology/data-holdings, are widely used by various user communities in modeling, research, and applications, such as drought and flood monitoring, watershed and water quality management, and case studies of extreme events. More information is available at http://ldas.gsfc.nasa.gov/. To further facilitate analysis of water and energy budgets and trends, NLDAS monthly data sets have been recently released by NASA GES DISC.

Vollmer, Bruce E.↗

NASA NTRS2018

Technologies and Methods Used at the Laboratory for Atmospheric and Space Physics (LASP) to Serve Solar Irradiance Data

The Laboratory for Atmospheric and Space Physics (LASP) at the University of Colorado in Boulder, USA operates the Solar Radiation and Climate Experiment (SORCE) NASA mission, as well as several other NASA spacecraft and instruments. Dozens of Solar Irradiance data sets are produced, managed, and disseminated to the science community. Data are made freely available to the scientific immediately after they are produced using a variety of data access interfaces, including the LASP Interactive Solar Irradiance Datacenter (LISIRD), which provides centralized access to a variety of solar irradiance data sets using both interactive and scriptable/programmatic methods. This poster highlights the key technological elements used for the NASA SORCE mission ground system to produce, manage, and disseminate data to the scientific community and facilitate long-term data stewardship. The poster presentation will convey designs, technological elements, practices and procedures, and software management processes used for SORCE and their relationship to data quality and data management standards, interoperability, NASA data policy, and community expectations.

Pankratz, Chris↗

NASA NTRSDate not supplied

Memory Optimizations for Sparse Linear Algebra on GPU Hardware

An effort to maximize memory bandwidth utilization for a sparse linear algebra kernel executing on NVIDIA® Tesla V100 and A100 Graphics Processing Units (GPUs) is described. The kernel consists of a block-sparse matrix-vector product and a series of forward/backward triangular solves. The computation is memory-bound and exhibits low arithmetic intensity. Along with a relatively small block size, the data layout poses a challenge to effectively utilize the available memory bandwidth on common GPU architectures. An earlier implementation using a warp to process a single row of the matrix was found to yield good memory performance on the V100 architecture. However, anew approach, which assigns a warp to six rows of the matrix, is proposed for the A100. In addition, two new features offered by the A100 architecture are explored.L2residency control enables a portion of theL2cache to be used for persistent data access, and the asynchronous copy instruction allows data to be loaded directly from main memory into shared memory. Demonstrations show that the new implementation improves memory bandwidth utilization from 71.5% to 81.2% of the peak available on theA100 architecture.

GPU↗

NASA NTRSDate not supplied

Air Quality (AQ) Monitoring From Space By NASA Using Tempo

In an era marked by escalating environmental concerns, understanding the Earth's atmosphere and its complex interactions is paramount. The Tropospheric Emission Monitoring of POllution (TEMPO) instrument is a cutting-edge venture by NASA that stands at the forefront of Earth observation technology and exemplifies NASA's commitment to unraveling the intricacies of the air we breathe. TEMPO was launched with the primary objective of monitoring air quality. This story map explores the innovative technology behind the TEMPO instrument, its mission objectives, tools and services provided by the Atmospheric Science Data Center (ASDC) for data access and the potential impact of TEMPO data findings on our health and Earth's environmental future.

Hazem Mahmoud↗

NASA NTRS2019

OPeNDAP and HDF5 in the Cloud: Techniques and Best Practices for Serving HDF5 Data

In our talk we will discuss considerations and best practices for organizing data and metadata in HDF5 files when serving ESDIS products in S3 with OPeNDAP. The first part of the talk will be devoted to the best practices for organizing data and user-defined metadata including CF conventions in HDF5. We will also address interoperability with netCDF-4 and its role in accessing data in HDF5. In the second part of the talk we will go over OPeNDAP considerations for S3 access to HDF5 data.

HDF5↗

NASA NTRS1983

VAS operational procedures and results at the Kansas City Satellite Field Services Station

An operational assessment of VAS data by using a Man-computer Interactive Data Access System (McIDAS) terminal linked by a 9600 band telephone line is discussed. Seven hours of VAS data were processed and edited daily. Data was scheduled 16 hours a day, 7 days a week; however, during this time period there were very few days with 16 hours of data to evalute. The McIDAS terminal, which has 10 display frames and 5 graphics, provide access to the sounding data processed. These data are processed using two procedures. The dwell sounding data are generated by using all 12 spectral channels with a spin budget of 39. To provide coverage for most of the United States, soundings are made starting at 18 minutes after the hour from approximately 49 deg N to 36 deg N and at 48 minutes after the hour from 36 deg N to 26 deg N. The dwell imaging mode uses 11 channels but the spin budge is 17. With the reduced spin budget, retrievals can be made at 18 or 48 minutes after the hour for approximately 44 deg N to 27 deg N. With these constraints a schedule, of data sets was proposed to use the schedule and how the data set could be used are shown.

Heckman, B.↗

NASA NTRS1984

The Pilot Land Data System: Report of the Program Planning Workshops

An advisory report to be used by NASA in developing a program plan for a Pilot Land Data System (PLDS) was developed. The purpose of the PLDS is to improve the ability of NASA and NASA sponsored researchers to conduct land-related research. The goal of the planning workshops was to provide and coordinate planning and concept development between the land related science and computer science disciplines, to discuss the architecture of the PLDs, requirements for information science technology, and system evaluation. The findings and recommendations of the Working Group are presented. The pilot program establishes a limited scale distributed information system to explore scientific, technical, and management approaches to satisfying the needs of the land science community. The PLDS paves the way for a land data system to improve data access, processing, transfer, and analysis, which land sciences information synthesis occurs on a scale not previously permitted because of limits to data assembly and access.

Source record↗

NASA NTRS2006

Wireless augmented reality communication system

The system of the present invention is a highly integrated radio communication system with a multimedia co-processor which allows true two-way multimedia (video, audio, data) access as well as real-time biomedical monitoring in a pager-sized portable access unit. The system is integrated in a network structure including one or more general purpose nodes for providing a wireless-to-wired interface. The network architecture allows video, audio and data (including biomedical data) streams to be connected directly to external users and devices. The portable access units may also be mated to various non-personal devices such as cameras or environmental sensors for providing a method for setting up wireless sensor nets from which reported data may be accessed through the portable access unit. The reported data may alternatively be automatically logged at a remote computer for access and viewing through a portable access unit, including the user's own.

Devereaux, Ann↗

NASA NTRS2014

Wireless Augmented Reality Communication System

The system of the present invention is a highly integrated radio communication system with a multimedia co-processor which allows true two-way multimedia (video, audio, data) access as well as real-time biomedical monitoring in a pager-sized portable access unit. The system is integrated in a network structure including one or more general purpose nodes for providing a wireless-to-wired interface. The network architecture allows video, audio and data (including biomedical data) streams to be connected directly to external users and devices. The portable access units may also be mated to various non-personal devices such as cameras or environmental sensors for providing a method for setting up wireless sensor nets from which reported data may be accessed through the portable access unit. The reported data may alternatively be automatically logged at a remote computer for access and viewing through a portable access unit, including the user's own.

Devereaux, Ann↗

NASA NTRS2014

Wireless Augmented Reality Communication System

The system of the present invention is a highly integrated radio communication system with a multimedia co-processor which allows true two-way multimedia (video, audio, data) access as well as real-time biomedical monitoring in a pager-sized portable access unit. The system is integrated in a network structure including one or more general purpose nodes for providing a wireless-to-wired interface. The network architecture allows video, audio and data (including biomedical data) streams to be connected directly to external users and devices. The portable access units may also be mated to various non-personal devices such as cameras or environmental sensors for providing a method for setting up wireless sensor nets from which reported data may be accessed through the portable access unit. The reported data may alternatively be automatically logged at a remote computer for access and viewing through a portable access unit, including the user's own.

Devereaux, Ann↗

NASA NTRS2016

Wireless Augmented Reality Communication System

The system of the present invention is a highly integrated radio communication system with a multimedia co-processor which allows true two-way multimedia (video, audio, data) access as well as real-time biomedical monitoring in a pager-sized portable access unit. The system is integrated in a network structure including one or more general purpose nodes for providing a wireless-to-wired interface. The network architecture allows video, audio and data (including biomedical data) streams to be connected directly to external users and devices. The portable access units may also be mated to various non-personal devices such as cameras or environmental sensors for providing a method for setting up wireless sensor nets from which reported data may be accessed through the portable access unit. The reported data may alternatively be automatically logged at a remote computer for access and viewing through a portable access unit, including the user's own.

Devereaux, Ann↗

DOE OSTI2026

Energy Consumption, Performance, and Cost Estimates for Medium and Heavy-Duty Vehicles Based on 2022 Assumptions

Assumptions for this work was collected and the analysis was completed in FY22. This contains information for more than 20 types of medium and heavy duty vehicles. Vehicles with various levels of hybridization, electric and fuel cell powertrains are considered in this work. More details are available in the report published by Argonne accessible from https://vms.taps.anl.gov/research-highlights/u-s-doe-vto-hfto-r-d-benefits/. TechScape, a convenient data visualization tool is also provided by Argonne for this data, accessible from [TechScape Web](https://vms.taps.anl.gov/data/techscape-web-2023/).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2024

Object storage model for CMS data

In CMS, data access and management is organized around the data-tier model: a static definition of what subset of event information is available in a particular dataset, realized as a collection of files. In previous work, we have proposed a novel data management model that obviates the need for data tiers by exploding files into individual event data product objects. In this work, we estimate the potential savings in data volume based on user analysis patterns.

Smith, Nick↗

DOE OSTI2021

Hydrogen-Based Energy Storage Systems for Large-Scale Data Center Applications

Global demand for data and data access has spurred the rapid growth of the data center industry. To meet demands, data centers must provide uninterrupted service even during the loss of primary power. Service providers seeking ways to eliminate their carbon footprint are increasingly looking to clean and sustainable energy solutions, such as hydrogen technologies, as alternatives to traditional backup generators. In this viewpoint, a survey of the current state of data centers and hydrogen-based technologies is provided along with a discussion of the hydrogen storage and infrastructure requirements needed for large-scale backup power applications at data centers.

08 HYDROGEN↗