Search NASA⌕ Search

SEARCH · Search NASA

Results for “data access”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

A Qualitative Readiness-Requirements Assessment Model for Enterprise Big-Data Infrastructure Investment

In the last three decades, there has been an exponential growth in the area of information technology providing the information processing needs of data-driven businesses in government, science, and private industry in the form of capturing, staging, integrating, conveying, analyzing, and transferring data that will help knowledge workers and decision makers make sound business decisions. Data integration across enterprise warehouses is one of the most challenging steps in the big data analytics strategy. Several levels of data integration have been identified across enterprise warehouses: data accessibility, common data platform, and consolidated data model. Each level of integration has its own set of complexities that requires a certain amount of time, budget, and resources to implement. Such levels of integration are designed to address the technical challenges inherent in consolidating the disparate data sources. In this paper, we present a methodology based on industry best practices to measure the readiness of an organization and its data sets against the different levels of data integration. We introduce a new Integration Level Model (ILM) tool, which is used for quantifying an organization and data system s readiness to share data at a certain level of data integration. It is based largely on the established and accepted framework provided in the Data Management Association (DAMA-DMBOK). It comprises several key data management functions and supporting activities, together with several environmental elements that describe and apply to each function. The proposed model scores the maturity of a system s data governance processes and provides a pragmatic methodology for evaluating integration risks. The higher the computed scores, the better managed the source data system and the greater the likelihood that the data system can be brought in at a higher level of integration.

Big data↗

A Science Data Gateway for Environmental Management

Science data gateways are effective in providing complex science data collections to the world-wide user Q1 communities. In this paper we describe a gateway for the Advanced Simulation Capability for Environmental Management (ASCEM) framework. Built on top of established web service technologies, the ASCEM data gateway is specifically designed for environmental modeling applications. Its key distinguishing feature sinclude (1) handling of complex spatiotemporal data, (2) offering a variety of selective data access mechanisms, (3) providing state-of-the-art plotting and visualization of spatiotemporal data records, and (4) integrating seamlessly with a distributed workflow system using a RESTful interface. ASCEM projectscientists have been using this data gateway since 2011.

ASCEM, Akuna, Database Management↗

The Data Mine model for accessible partnerships in data science

Abstract The Data Mine at Purdue University is a pioneering experiential learning community for undergraduate and graduate students of any background to learn data science. The first data‐intensive experience embedded in a large learning community, The Data Mine had nearly 1300 students in academic year (AY) 2022–2023 and nearly 1700 students for AY 2023–2024. The Data Mine embodies data‐infused education, research, and collaboration. Students learn Python, R, SQL, and shell‐scripting, while working on weekly projects within a high‐performance computing (HPC) cluster. In the Corporate Partners cohort, students work on teams of 5–15 students, led by a paid student team leader. Each cohort follows an Agile approach, working on data‐intensive projects provided by industry partners and mentored by company employees. Students develop professional and data skills throughout the academic year, from August through April. Many students return in subsequent years to the program, increasing their tenure with a Corporate Partner. Student teams are inherently interdisciplinary; students from 133 different majors are involved in the program, ranging from new incoming students through PhD level students. These interdisciplinary teams of students bring new perspectives to challenging problems in which data science is a key part of the solution. The interdisciplinary teams foster an environment of synthesis with ideas and solutions. Students come together with different life experiences, different levels of technical skill, but also varying ways they navigate paths to solutions because of the variety of majors represented, resulting in a more creative and robust solution than a traditional data science program. This article is categorized under: Applications of Computational Statistics > Education in Computational Statistics

Betz, Margaret A.↗

An Autonomous MCP Bridge to Rucio: Enhancing Data Management Accessibility for High Energy Physics

The Rucio Data Management System [1] is an important tool used by High Energy Physics experiments, including those at Fermi National Accelerator Laboratory, to store and manage exabyte-scale scientific datasets. Despite its central role in coordinating data across globally distributed storage sites, Rucio's command line interface (CLI) presents a steep learning curve, and makes it difficult for scientists to navigate through. To solve this issue, a containerized Model Context Protocol (MCP) [2] server was built that connects Large Language Models directly to Rucio, allowing AI agents to handle data tasks by using simple, natural language rather than memorized terminal commands. The core engineering focus of this project was moving the server away from slow terminal commands that require text parsing and replacing them with a native Python Client API toolset and a planned REST API framework. Moving to the Python API handles data operations directly in memory, which helps clear up formatting errors, provides the AI with clean, structured JSON data and speeds up tool execution. To prove that the system actually works, a benchmarking pipeline was also built with various questions to test the AI across four different model configurations. The questions included finding data scopes, tracking down specific datasets, and checking replication rules. Through benchmarking, early runs showed that with raw terminal text, the model would get confused and stuck, whereas switching to the Python API to feed the AI clean, structured data yielded massive improvement. By creating an intelligent and autonomous bridge to a storage network, this project shows how AI can be implemented in scientific data management, which ultimately helps scientists at Fermilab spend less time sorting through data and more time focusing on their experiments and analysis.

Akella, Kashyap [William Rainey Harper Coll.]↗

Methods Data Qualification Interim Report

The overall goal of the Next Generation Nuclear Plant (NGNP) Data Management and Analysis System (NDMAS) is to maintain data provenance for all NGNP data including the Methods component of NGNP data. Multiple means are available to access data stored in NDMAS. A web portal environment allows users to access data, view the results of qualification tests and view graphs and charts of various attributes of the data. NDMAS also has methods for the management of the data output from VHTR simulation models and data generated from experiments designed to verify and validate the simulation codes. These simulation models represent the outcome of mathematical representation of VHTR components and systems. The methods data management approaches described herein will handle data that arise from experiment, simulation, and external sources for the main purpose of facilitating parameter estimation and model verification and validation (V&V). A model integration environment entitled ModelCenter is used to automate the storing of data from simulation model runs to the NDMAS repository. This approach does not adversely change the why computational scientists conduct their work. The method is to be used mainly to store the results of model runs that need to be preserved for auditing purposes or for display to the NDMAS web portal. This interim report demonstrates the currently development of NDMAS for Methods data and discusses data and its qualification that is currently part of NDMAS.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Fixing Amdahl's Law within the Limits of Accelerated Systems: FALLACY

Closeout report for FALLACY project. The performance of Data Model Convergence Initiative (DMC) applications on parallel machines is far below the limit set by Amdahl’s law. Whether the machine is based on many-core, GPUs, FPGAs, or a heterogeneous combination, usually the most significant bottleneck is accessing data from the memory system. Aligning with DMC’s HW/architecture thrust, this project developed a set of memory-centric tools called ‘MemGaze’ that inform the HW/SW stack about an application’s memory behavior, including data access latency and diagnosing poor data layout and data composition. Our approach uses architectural modeling and analysis of workload data accesses.

97 MATHEMATICS AND COMPUTING↗

International Arctic Systems for Observing the Atmosphere (IASOA) Metadata Harvest for Oliktok Point Observations Field Campaign Report

The International Arctic Systems for Observing the Atmosphere (IASOA) was initiated as an International Polar Year (IPY) project (Uttal et al. 2015, on-line release) to address key atmospheric science questions through coordinating the considerable atmospheric observing assets at 10 pan-arctic observatories (http://iasoa.org). The mission of IASOA is to advance cross-site research objectives from independent arctic atmospheric observatories through: (1) strategically developing comprehensive observational capacity; (2) facilitating data access and usability through a single gateway; and (3) mobilizing contributions to synergistic science and socially relevant services derived from IASOA assets and expertise. In April 2015 the IASOA science steering committee invited the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) user facility’s Oliktok Point observatory to contribute to consortium activities. The IASOA steering committee identified enhanced, cross-site data sharing through an IASOA-specific data access portal as a focus for ongoing consortium activities. The design intent of the portal has been to display the data inventories of all IASOA observatories in a succinct summary (Data-at-a-Glance) and provide the most proximal access links available to those data sets. The IASOA data access portal is not intended to replace the functions of existing archives. This succinct summary of data availability supports cross-site and interdisciplinary synthesis science activities. Efforts to enhance the legacy IASOA data access portal functionality have focused on two areas: developing a comprehensive and coherent data vocabulary for IASOA data and implementing an interoperable metadata standard for all IASOA data. The technical goal of this proposal is to work with ARM data managers to operationalize the export of metadata for observational data sets for Oliktok Point from the ARM metadata database. IASOA harvests data in the commonly used International Organization for Standardization (ISO) format. ARM has successfully generated metatdata matching this format in the past to provide metadata from its North Slope of Alaska (NSA) Barrow observatory.

54 ENVIRONMENTAL SCIENCES↗

Querying Large Scientific Data Sets with Adaptable IO System ADIOS

When working with a large dataset, a relatively small fraction of data records are of interest in each analysis operation. For example, while examining a billion-particle dataset from an accelerator model, the scientists might focus on a few thousand fastest particles, or on the particle farthest from the beam center. In general, this type of selective data access is challenging because the selected data records could be anywhere in the dataset and require a significant amount of time to locate and retrieve. In this paper, we report our experience of addressing this data access challenge with the Adaptable IO System ADIOS. More specifically, we design a query interface for ADIOS to allow arbitrary combinations of range conditions on known variables, implement a number of different mechanisms for resolving these selection conditions, and devise strategies to reduce the time needed to retrieve the scattered data records. In many cases, the query mechanism can retrieve the selected data records orders of magnitude faster than the brute-force approach.Our work relies heavily on the in situ data processing feature of ADIOS to allow user functions to be executed in the data transport pipeline. This feature allows us to build indexes for efficient query processing, and to perform other intricate analyses while the data is in memory.

Gu, Junmin↗

Using Xrootd to federate regional storage

While the LHC data movement systems have demonstrated the ability to move data at the necessary throughput, we have identified two weaknesses: the latency for physicists to access data and the complexity of the tools involved. To address these, both ATLAS and CMS have begun to federate regional storage systems using Xrootd. Xrootd, referring to a protocol and implementation, allows us to provide data access to all disk-resident data from a single virtual endpoint. This redirector discovers the actual location of the data and redirects the client to the appropriate site. The approach is particularly advantageous since typically the redirection requires much less than 500 milliseconds and the Xrootd client is conveniently built into LHC physicists analysis tools. Currently, there are three regional storage federations - a US ATLAS region, a European CMS region, and a US CMS region. The US ATLAS and US CMS regions include their respective Tier 1, Tier 2 and some Tier 3 facilities, a large percentage of experimental data is available via the federation. Additionally, US ATLAS has begun studying low-latency regional federations of close-by sites. From the base idea of federating storage behind an endpoint, the implementations and use cases diverge. The CMS software framework is capable of efficiently processing data over high-latency links, so using the remote site directly is comparable to accessing local data. The ATLAS processing model allows a broad spectrum of user applications with varying degrees of performance with regard to latency, a particular focus has been optimizing n-tuple analysis. Both VOs use GSI security. ATLAS has developed a mapping of VOMS roles to specific file system authorizations, while CMS has developed callouts to the site's mapping service. Each federation presents a global namespace to users. For ATLAS, the global-to-local mapping is based on a heuristic-based lookup from the site's local file catalog, while CMS does the mapping based on translations given in a configuration file. We will also cover the latest usage statistics and interesting use cases that have developed over the previous 18 months.

Bauerdick, L.↗

Performance of Julia for High Energy Physics Analyses

We argue that the Julia programming language is a compelling alternative to currently more common implementations in Python and C++ for common data analysis workflows in high energy physics. We compare the speed of implementations of different workflows in Julia with those in Python and C++. Furthermore, our studies show that the Julia implementations are competitive for tasks that are dominated by computational load rather than data access. For work that is dominated by data access, we demonstrate an application with concurrent file reading and parallel data processing.

97 MATHEMATICS AND COMPUTING↗

A review of privacy in energy applications

As the distribution system continues to experience an increase in distributed energy resource (DER) and electric vehicle (EV) penetration, so does the need for new solutions that can help grid operators manage and leverage their capabilities. This will undoubtedly lead to new operational schemes and business opportunities that will transform the traditional consumer into a prosumer who will be more actively engaged in grid operations. Although the field is still under active development, many of the potential use cases presented in literature or industry are built upon edge computing, two-way communications, and other innovative computational constructs to attain their goals. However, at their core, many use cases assume a great level of data access to aid with the decision-making process, an assumption that may need to be revised to ensure fair and equitable operational processes are maintained. This may be particularly true as edge resources are predicted to participate in retail-side, many-to-many, or peer-to-peer markets and thus may lead to financial impacts if data access considerations are ignored. The need to revise data access mechanisms can be further justified by the introduction of new participants into the operational process, who do not have the same level of trust, nor the incentives to focus on energy delivery as their primary objective. At the same time, more consumers are becoming aware of their own data, and the potential impacts of its abuse. To help solution developers better understand these risks, this report has been developed to offer an initial introduction to the topic of privacy. This is achieved by 1) Highlighting the need for privacy-aware solutions; 2) Encouraging system designers to be inquisitive about the status quo; 3) Documenting the existing threat space; 4) Presenting and evaluating tools that may be helpful towards enabling better privacy postures; and 5) Making recommendations to encourage the adoption of better practices. From a technical perspective, the report focuses on evaluating two potential techniques by applying them to the Transactive Energy Space. Based on the obtained results, it can be established that differential privacy (DP) methods may have limited applicability when highly correlated, time-series data records need to be protected. However, DP may be a powerful tool when it is used to aggregate and analyze mid-size and large-size data sets in a more traditional statistical environment. The second tool under evaluation is threshold cryptography, which can guarantee complete secrecy (and thus privacy) but requires the establishment of key management procedures and dedicated communication channels for key coordination. Therefore, due to its increased computational overhead, the use of threshold cryptography must be weighted using a cost/benefit analysis on a per-application basis.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Key Accomplishments & Results: Energy Data Accelerator

This fact sheet outlines the results of DOE’s Better Buildings Energy Data Accelerator. Key accomplishments of partners are outlined in this fact sheet, profiling the historic expansion of data accessibility and increases in building energy benchmarking.

data accessibility, data access, utilities, buildi↗

Accessing global data from accelerator devices

An aspect includes a table of contents (TOC) that was generated by a compiler being received at an accelerator device. The TOC includes an address of global data in a host memory space. The global data is copied from the address in the host memory space to an address in the device memory space. The address in the host memory space is obtained from the received TOC. The received TOC is updated to indicate that global data is stored at the address in the device memory space. A kernel that accesses the global data from the address in the device memory space is executed. The address in the device memory space is obtained based on contents of the updated TOC. When the executing is completed, the global data from the address in the device memory space is copied to the address in the host memory space.

97 MATHEMATICS AND COMPUTING↗

Accessing global data from accelerator devices

An aspect includes a table of contents (TOC) that was generated by a compiler being received at an accelerator device. The TOC includes an address of global data in a host memory space. The global data is copied from the address in the host memory space to an address in the device memory space. The address in the host memory space is obtained from the received TOC. The received TOC is updated to indicate that global data is stored at the address in the device memory space. A kernel that accesses the global data from the address in the device memory space is executed. The address in the device memory space is obtained based on contents of the updated TOC. When the executing is completed, the global data from the address in the device memory space is copied to the address in the host memory space.

97 MATHEMATICS AND COMPUTING↗

Efficient Unsteady Flow Visualization with High-Order Access Dependencies

We present a novel high-order access dependencies based model for efficient pathline computation in unsteady flow visualization. By taking longer access sequences into account to model more sophisticated data access patterns in particle tracing, our method greatly improves the accuracy and reliability in data access prediction. In our work, high-order access dependencies are calculated by tracing uniformly-seeded pathlines in both forward and backward directions in a preprocessing stage. The effectiveness of our proposed approach is demonstrated through a parallel particle tracing framework with high-order data prefetching. Results show that our method achieves higher data locality and hence improves the efficiency of pathline computation.

Zhang, Jiang↗

Solar Bankability Data to Advance Transactions and Access (SB-DATA) Final Technical Report

The DOE project “Solar Bankability Data to Advance Transactions and Access”, known in the solar industry and referred to in this report as “Orange Button”, had project objectives of convening key stakeholder participants in the solar project lifecycle to develop a data taxonomy that meets to the solar industry’s needs for standardized data types for use in solar projects. Additional Orange Button deliverables included the development of an online data exchange to act as a clearinghouse of data that conformed to the data standard and additional solar industry data, and a Solar Data Translation Platform that could transform legacy solar project data to conform to the Orange Button data standard. All of these objectives were achieved. The Smart Electric Power Alliance (SEPA) provided program management, industry participant recruiting and convening, worked with the solar industry to define the scope of the Orange Button data taxonomy, and documented and delivered the taxonomy data requirements to funded project partners to develop both the taxonomy and the data conversion tools. Further efforts included performing extensive industry outreach and awareness to encourage adoption of the Orange Button taxonomy by the solar industry by enumerating the cost saving benefits of using the taxonomy and the public benefit of the increased use of solar energy due to increased bankability and viability of solar development projects. Solar PV panel and component prices have fallen in the last decade, while soft costs have remained stubbornly high and can account for as much as 64% of the total cost of a new solar system. Part of the U.S. Department of Energy SunShot Initiative, Orange Button enabled the creation and adoption of industry-led open data standards for rapid and seamless information exchange across the solar value chain, from origination to decommissioning. Standardizing data will reduce soft costs by making it easier to share solar data and speeding up processes. The taxonomy is a product of solar industry participants convening to identify the pain points caused by incompatible data, and to agree on and standardize data types to be used by the actors in the solar project lifecycle. This report focuses on the SEPA led first phase of Orange Button project that included industry outreach and recruiting participants for five industry working groups, widespread industry outreach to increase awareness and encourage eventual adoption of the taxonomy, and the definition of the scope, use cases, and supporting data types of the Orange Button taxonomy. The solar industry has been receptive of the completed Orange Button taxonomy. In SEPA’s work, 401 solar industry participants facilitated by SEPA defined the scope, use cases, and requirements for the taxonomy leading to over 150 taxonomy early adopters incorporating the taxonomy into their software products serving the solar industry, and several large investors and banks have started projects to enable them to utilize the taxonomy to gain back-office efficiencies. Moving forward, as the solar industry more widely embraces the Orange Button taxonomy and begins realizing the tangible benefits of adoption at scale, we expect further pressure on solar soft costs and increases in the bankability of solar projects.

14 SOLAR ENERGY↗