SEARCH · Search NASA
Results for “Data access”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Data Model and Data Access Architecture in the Geophysical Monitoring System (Lightning Talk)
Explore the source record for details and available documents.
Data Model and Data Access Architecture in the Geophysical Monitoring System
Explore the source record for details and available documents.
Efforts and Innovations to Promote Data Sharing and Data Accessibility in the EGS Collab Experiments
Explore the source record for details and available documents.
Efforts and Innovations to Promote Data Sharing and Data Accessibility in the EGS Collab Experiments.
Abstract not provided.
pvlib iotools—Open-source Python functions for seamless access to solar irradiance data
Access to accurate solar resource data is critical for numerous applications, including estimating the yield of solar energy systems, developing radiation models, and validating irradiance datasets. However, lack of standardization in data formats and access interfaces across providers constitutes a major barrier to entry for new users. pvlib python’s iotools subpackage aims to solve this issue by providing standardized Python functions for reading local files and retrieving data from external providers. All functions follow a uniform pattern and return convenient data outputs, allowing users to seamlessly switch between data providers and explore alternative datasets. The pvlib package is community-developed on GitHub: https://github.com/pvlib/pvlib-python. As of pvlib python version 0.9.5, the iotools subpackage supports 12 different datasets, including ground measurement, reanalysis, and satellite-derived irradiance data. The supported ground measurement networks include the Baseline Surface Radiation Network (BSRN), NREL MIDC, SRML, SOLRAD, SURFRAD, and the US Climate Reference Network (CRN). Additionally, satellite-derived and reanalysis irradiance data from the following sources are supported: PVGIS (SARAH & ERA5), NSRDB PSM3, and CAMS Radiation Service (including McClear clear-sky irradiance).
Data Placement Optimization for ATLAS in a Multi-Tiered Storage System within a Data Center
Scientific experiments and computations, especially in High Energy Physics, are generating and accumulating data at an unprecedented rate. Effectively managing this vast volume of data while ensuring efficient data analysis poses a significant challenge for data centers, which must integrate various storage technologies. This paper proposes addressing this challenge by designing and developing a precise data popularity prediction model utilizing state-of-theart AI/ML techniques. This model is crafted from the analysis of ATLAS data and access patterns. It enables us to migrate infrequently accessed data to more economical storage media, such as tape drives, while storing frequently accessed data on faster yet costlier storage media like HDD or SSD. This strategic approach ensures data is placed optimally into the appropriate storage classes, thereby maximizing storage capacity while minimizing data access latency for end-users. Furthermore, the paper includes a performance evaluation of the prediction model using various key metrics such as F1 score, accuracy, precision and recall. Finally, we present a prototype use case, leveraging real-world file access data to assess the model’s impact on performance.
Dynamic adaptation of memory page management policy
Systems, apparatuses, and methods for determining preferred memory page management policies by software are disclosed. Software executing on one or more processing units generates a memory request. Software determines the preferred page management policy for the memory request based at least in part on the data access size and data access pattern of the memory request. Software conveys an indication of a preferred page management policy to a memory controller. Then, the memory controller accesses memory for the memory request using the preferred page management policy specified by software.
Accessing protected data by a high-performance computing cluster
A data protection system is provided that allows applications to access protected data in a way that restricts applications from outputting to unauthorized targets any unprotected data derived from the protected data and that ensures that the applications do not have access to a key that allows access to the unprotected data. The data protection system provides a policy server that may execute on a service node of a high performance computing system and a data encryption process that may execute on each compute node that is allocated to an application or batch job. The policy server maintains policies of entities specifying access control for protected data. The data encryption process generates a secure execution environment for an application process and interfaces with the policy server to retrieve keys for decrypting protected data in accordance with a policy, and it decrypts and provides the decrypted data to the application process.
Effectiveness and predictability of in-network storage cache for Scientific Workflows
Large scientific collaborations often have multiple scientists accessing the same set of files while doing different analyses, which create repeated accesses to the large amounts of shared data located far away. These data accesses have long latency due to distance and occupy the limited bandwidth available over the wide-area network. To reduce the wide-area network traffic and the data access latency, regional data storage caches have been installed as a new networking service. To study the effectiveness of such a cache system in scientific applications, we examine the Southern California Petabyte Scale Cache for a high-energy physics experiment. By examining about 3TB of operational logs, we show that this cache removed 67.6% of file requests from the wide-area network and reduced the traffic volume on wide-area network by 12. 3TB (or 35.4%) an average day. The reduction in the traffic volume (35.4%) is less than the reduction in file counts (67.6%) because the larger files are less likely to be reused. Due to this difference in data access patterns, the cache system has implemented a policy to avoid evicting smaller files when processing larger files. We also build a machine learning model to study the predictability of the cache behavior. Tests show that this model is able to accurately predict the cache accesses, cache misses, and network throughput, making the model useful for future studies on resource provisioning and planning.
A use-case-driven approach for demonstrating the added value of digitalisation in wind energy
Digitalisation is one of the key drivers for reducing the costs and risks of wind energy. When considering whether to embark on a digitalisation initiative, two key questions arise. The first is what business or operational opportunities might feasibly be addressed and the second is which of the many potential aspects of digitalisation are relevant to those opportunities. In this work, we show how these questions can be answered with a use-case-driven approach, based around a survey aiming to collect and collate the main "pain points" (or everyday challenges) of people in the wind energy sector. Although the relatively low number of participants of the survey (46) means that the results should only be used indicatively, it is still possible to make some general recommendations for priorities for digitalisation efforts in the wind energy sector. Firstly, digitalisation efforts should focus both on supporting people carrying out cross-lifecycle tasks, in particular sharing data, managing data, undertaking general data analyses and accessing data. Tools to do this should deal with varying data formats and naming conventions, make metadata more accessible, define data and metadata standards, make more data publicly available and improve the quality of data. Secondly, efforts should also focus on supporting people in the wind farm operational phase, in particular with failure detection, fault diagnosis, failure rate modelling and predictive maintenance. Solutions to do this should focus on accessible and validated tools for fault detection, cloud or other data pipeline solutions for SCADA data and tools for exhaustive data documentation. Finally, digitalisation efforts should focus on better communicating and helping people become aware of existing solutions and tools, as well as on helping people to exert a stronger influence on possible solutions.
The Data Mine model for accessible partnerships in data science
Abstract The Data Mine at Purdue University is a pioneering experiential learning community for undergraduate and graduate students of any background to learn data science. The first data‐intensive experience embedded in a large learning community, The Data Mine had nearly 1300 students in academic year (AY) 2022–2023 and nearly 1700 students for AY 2023–2024. The Data Mine embodies data‐infused education, research, and collaboration. Students learn Python, R, SQL, and shell‐scripting, while working on weekly projects within a high‐performance computing (HPC) cluster. In the Corporate Partners cohort, students work on teams of 5–15 students, led by a paid student team leader. Each cohort follows an Agile approach, working on data‐intensive projects provided by industry partners and mentored by company employees. Students develop professional and data skills throughout the academic year, from August through April. Many students return in subsequent years to the program, increasing their tenure with a Corporate Partner. Student teams are inherently interdisciplinary; students from 133 different majors are involved in the program, ranging from new incoming students through PhD level students. These interdisciplinary teams of students bring new perspectives to challenging problems in which data science is a key part of the solution. The interdisciplinary teams foster an environment of synthesis with ideas and solutions. Students come together with different life experiences, different levels of technical skill, but also varying ways they navigate paths to solutions because of the variety of majors represented, resulting in a more creative and robust solution than a traditional data science program. This article is categorized under: Applications of Computational Statistics > Education in Computational Statistics
An Autonomous MCP Bridge to Rucio: Enhancing Data Management Accessibility for High Energy Physics
The Rucio Data Management System [1] is an important tool used by High Energy Physics experiments, including those at Fermi National Accelerator Laboratory, to store and manage exabyte-scale scientific datasets. Despite its central role in coordinating data across globally distributed storage sites, Rucio's command line interface (CLI) presents a steep learning curve, and makes it difficult for scientists to navigate through. To solve this issue, a containerized Model Context Protocol (MCP) [2] server was built that connects Large Language Models directly to Rucio, allowing AI agents to handle data tasks by using simple, natural language rather than memorized terminal commands. The core engineering focus of this project was moving the server away from slow terminal commands that require text parsing and replacing them with a native Python Client API toolset and a planned REST API framework. Moving to the Python API handles data operations directly in memory, which helps clear up formatting errors, provides the AI with clean, structured JSON data and speeds up tool execution. To prove that the system actually works, a benchmarking pipeline was also built with various questions to test the AI across four different model configurations. The questions included finding data scopes, tracking down specific datasets, and checking replication rules. Through benchmarking, early runs showed that with raw terminal text, the model would get confused and stuck, whereas switching to the Python API to feed the AI clean, structured data yielded massive improvement. By creating an intelligent and autonomous bridge to a storage network, this project shows how AI can be implemented in scientific data management, which ultimately helps scientists at Fermilab spend less time sorting through data and more time focusing on their experiments and analysis.
Fixing Amdahl's Law within the Limits of Accelerated Systems: FALLACY
Closeout report for FALLACY project. The performance of Data Model Convergence Initiative (DMC) applications on parallel machines is far below the limit set by Amdahl’s law. Whether the machine is based on many-core, GPUs, FPGAs, or a heterogeneous combination, usually the most significant bottleneck is accessing data from the memory system. Aligning with DMC’s HW/architecture thrust, this project developed a set of memory-centric tools called ‘MemGaze’ that inform the HW/SW stack about an application’s memory behavior, including data access latency and diagnosing poor data layout and data composition. Our approach uses architectural modeling and analysis of workload data accesses.
Performance of Julia for High Energy Physics Analyses
We argue that the Julia programming language is a compelling alternative to currently more common implementations in Python and C++ for common data analysis workflows in high energy physics. We compare the speed of implementations of different workflows in Julia with those in Python and C++. Furthermore, our studies show that the Julia implementations are competitive for tasks that are dominated by computational load rather than data access. For work that is dominated by data access, we demonstrate an application with concurrent file reading and parallel data processing.
A review of privacy in energy applications
As the distribution system continues to experience an increase in distributed energy resource (DER) and electric vehicle (EV) penetration, so does the need for new solutions that can help grid operators manage and leverage their capabilities. This will undoubtedly lead to new operational schemes and business opportunities that will transform the traditional consumer into a prosumer who will be more actively engaged in grid operations. Although the field is still under active development, many of the potential use cases presented in literature or industry are built upon edge computing, two-way communications, and other innovative computational constructs to attain their goals. However, at their core, many use cases assume a great level of data access to aid with the decision-making process, an assumption that may need to be revised to ensure fair and equitable operational processes are maintained. This may be particularly true as edge resources are predicted to participate in retail-side, many-to-many, or peer-to-peer markets and thus may lead to financial impacts if data access considerations are ignored. The need to revise data access mechanisms can be further justified by the introduction of new participants into the operational process, who do not have the same level of trust, nor the incentives to focus on energy delivery as their primary objective. At the same time, more consumers are becoming aware of their own data, and the potential impacts of its abuse. To help solution developers better understand these risks, this report has been developed to offer an initial introduction to the topic of privacy. This is achieved by 1) Highlighting the need for privacy-aware solutions; 2) Encouraging system designers to be inquisitive about the status quo; 3) Documenting the existing threat space; 4) Presenting and evaluating tools that may be helpful towards enabling better privacy postures; and 5) Making recommendations to encourage the adoption of better practices. From a technical perspective, the report focuses on evaluating two potential techniques by applying them to the Transactive Energy Space. Based on the obtained results, it can be established that differential privacy (DP) methods may have limited applicability when highly correlated, time-series data records need to be protected. However, DP may be a powerful tool when it is used to aggregate and analyze mid-size and large-size data sets in a more traditional statistical environment. The second tool under evaluation is threshold cryptography, which can guarantee complete secrecy (and thus privacy) but requires the establishment of key management procedures and dedicated communication channels for key coordination. Therefore, due to its increased computational overhead, the use of threshold cryptography must be weighted using a cost/benefit analysis on a per-application basis.
ESS-DIVE Unoccupied Aerial Systems (UAS) Reporting Format v1
Here we present documentation of the ESS-DIVE reporting format for Unoccupied Aerial System (UAS) data and metadata. This reporting format provides guidance to data contributors on how to store data to maximize their discoverability, facilitate their efficient reuse, and add value to individual datasets. For data users, the reporting format will better allow data repositories to optimize data search and extraction, and more readily integrate similar data into harmonized synthesis products. The reporting format provides templates and guidance for the reporting of metadata for UAS experimental campaigns, individual flights, platform and sensor description. To improve data access and discoverability, the reporting format proposes a data description scheme of Levels based on the degree of processing, where Level 0 includes raw data, through to Level 3 being derived data end products. A range of examples of data types for each Level are given, with suggested file naming schemes. The reporting format presented here is intended to form a foundation for future development that will accommodate new UAS technologies and approaches to data access and use in the future. The reporting format documentation is maintained and updated on the ESS-DIVE Community Space GitHub at https://github.com/ess-dive-community/essdive-uas. This data package is the first published version of this reporting format, and comprises a zip file of the complete content of https://github.com/ess-dive-community/essdive-uas v1.0. The zip contains the reporting format description, instructions and variable definitions in GitHub markdown language (*.md) and metadata templates in csv format. The reporting format is designed to be compatible with other ESS-DIVE formats, and it is specifically recommended that this reporting format be used in conjunction with the File-level metadata (FLMD) and comma separated values (csv) reporting formats for submission to the ESS-DIVE repository.