Search NASA⌕ Search

SEARCH · Search NASA

Results for “Scientific data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21

ALACRITY: Analytics-Driven Lossless Data Compression for Rapid In-Situ Indexing, Storing, and Querying. In: Transactions on Large-Scale Data- and Knowledge-Centered Systems X. Lecture Notes in Computer Science, vol 8220

High-performance computing architectures face nontrivial data processing challenges, as computational and I/O components further diverge in performance trajectories. For scientific data analysis in particular, methods based on generating heavyweight access acceleration structures, e.g. indexes, are becoming less feasible for ever-increasing dataset sizes. We present ALACRITY, demonstrating the effectiveness of a fused data and index encoding of scientific, floating-point data in generating lightweight data structures amenable to common types of queries used in scientific data analysis. We exploit the representation of floating-point values by extracting significant bytes, using the resulting unique values to bin the remaining data along fixed-precision boundaries. To optimize query processing, we use an inverted index, mapping each generated bin to a list of records contained within, allowing us to optimize query processing with attribute range constraints. Overall, the storage footprint for both index and data is shown to be below numerous configurations of bitmap indexing, while matching or outperforming query performance.

Compression Ratio↗

Architecture of a Data Portal for Publishing and Delivering Open Data for Atmospheric Measurement

Atmospheric data are collected by researchers every day. Campaigns such as GOAmazon 2014/2015 and the Amazon Tall Tower Observatory collect essential data on aerosols, gases, cloud properties, and meteorological parameters in the Brazilian Amazon basin. These data products provide insights and essential information for analyzing and predicting natural processes. However, in Brazil, it is estimated that more than 80% of the scientific data collected are not published due to the lack of web portals that collect and store these data. This makes it difficult, or even impossible, to access and integrate the data, which can result in the loss of significant amounts of information and significantly affect the understanding of the overall data. To address this problem, we propose a data portal architecture and open data deployment that enable Big Data processing, human interaction, and download-oriented approaches with tools that help users catalog, publish and visualize atmospheric data. Thus, we describe the architecture developed, based on the experience of the Atmospheric Radiation Measurement Data Center, which incorporates the principles of FAIR, the infrastructure and content management system for managing scientific data. The portal partial results were tested with environmental data from contaminated areas at the University of São Paulo. Overall, this data portal creates more shared knowledge about atmospheric processes by providing users with access to open environmental data.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Services + Components = Data Intensive Scientific Workflow Applications with MeDICi

Scientific applications are often structured as workflows that execute a series of distributed software modules to analyze large data sets. Such workflows are typically constructed using general-purpose scripting languages to coordinate the execution of the various modules and to exchange data sets between them. While such scripts provide a cost-effective approach for simple workflows, as the workflow structure becomes complex and evolves, the scripts quickly become complex and difficult to modify. This makes them a major barrier to easily and quickly deploying new algorithms and exploiting new, scalable hardware platforms. In this paper, we describe the MeDICi Workflow technology that is specifically designed to reduce the complexity of workflow application development, and to efficiently handle data intensive workflow applications. MeDICi integrates standard component-based and service-based technologies, and employs an efficient integration mechanism to ensure large data sets can be efficiently processed. We illustrate the use of MeDICi with a climate data processing example that we have built, and describe some of the new features

99 GENERAL AND MISCELLANEOUS↗

Automated Design of Neuromorphic Networks for Scientific Applications at the Edge

Designing spiking neural networks for neuromorphic deployment is a non-trivial task. It is further complicated when there are resource constraints for the neuromorphic implementation, such as size or power constraints, that may be present in edge applications. In this work, we utilize a previously presented approach, EONS, to design spiking neural networks for a memristive neuromorphic implementation for scientific data applications. We specifically use a multi-objective approach in EONS to maximize network accuracy on the scientific data application task, but also to minimize network size and energy. We illustrate that EONS determines both the network structure and the parameters, removing the burden from the user on determining the appropriate spiking neural network structure, and we show that the resulting networks are very different from the layered structure of typical neural networks. Finally, we show that the multi-objective approach produces smaller, more energy efficient networks than the original EONS approach and produces comparable accuracy to a back-propagation style training approach.

Schuman, Catherine↗

Laboratory for Atmospheres 2002 Technical Highlights

How can we improve our ability to predict the weather-tomorrow, next week, and into the future? How is the Earth's climate changing? What causes such change? And what are its costs? What can the atmospheres of distant planets teach us about our own planet and its evolution? The Laboratory for Atmospheres is helping to answer these and other scientific questions about our planet and its neighbors. The Laboratory conducts a broad theoretical and experimental research program studying all aspects of the atmospheres of the Earth and other planets, including their structural, dynamical, radiative, and chemical properties, with the overarching goal to provide better understanding and to improve prediction of the Earth's climate. Vigorous research is central to NASA's exploration of the frontiers of knowledge. NASA scientists play a key role in conceiving new space missions, providing mission requirements, and carrying out research to explore the behavior of planetary systems, including, notably, the Earth's. Our Laboratory's scientists also supply outside scientists with technical assistance and scientific data to further investigations not immediately addressed by NASA itself. Laboratory scientists submit competitive research proposals with diverse scientific or technological approaches to NASA and other Federal agencies to acquire research support. The Laboratory management strives to provide a working environment that promotes creativity, competition, and openness. The Laboratory for Atmospheres is a vital participant in NASA's research program. Our Laboratory often has relatively large programs, sizable satellite missions, or observational campaigns that require the cooperative and collaborative efforts of many scientists. We ensure an appropriate balance between our scientists' responsibility for these large collaborative projects and their need for an active individual research agenda. This balance allows members of the Laboratory to continuously improve their scientific credentials. The Laboratory places high importance on promoting and measuring quality in its scientific research. We strive to assure high quality through peer-review funding processes that support approximately 90% of the work in the Laboratory. The overall quality of our scientific efforts is evaluated periodically by committees of advisors from the external scientific community, as detailed in Appendix 2 of this document. Members of the Laboratory interact with the general public to support a wide range of interests in the atmospheric sciences. Among other activities, the Laboratory raises the public's awareness of atmospheric science by presenting public lectures and demonstrations, by making scientific data available to wide audiences, by teaching, and by mentoring students and teachers. Section 6 presents details of the Laboratory's outreach activities during 2002. The Laboratory is also committed to addressing the demographic imbalances that exist today in the atmospheric and space sciences. We must address these imbalances for our field to enjoy the full benefit of all of the Nation's talent. The Laboratory makes substantial efforts to attract new scientists to the fields of atmospheric and space sciences. We strongly encourage the establishment of partnerships with Federal and state agencies that have operational responsibilities to promote the societal application of Earth sciences.

Steven E. Platnick↗

Creating Training Data for Scientific Named Entity Recognition with Minimal Human Effort

Scientific Named Entity Referent Extraction is often more complicated than traditional Named Entity Recognition (NER). For example, in polymer science, chemical structure may be encoded in a variety of nonstandard naming conventions, and authors may refer to polymers with conventional names, commonly used names, labels (in lieu of longer names), synonyms, and acronyms. As a result, accurate scientific NER methods are often based on task-specific rules, which are difficult to develop and maintain, and are not easily generalized to other tasks and fields. Machine learning models require substantial expert-annotated data for training. Here we propose polyNER: a semi-automated system for efficient identification of scientific entities in text. PolyNER applies word embedding models to generate entity-rich corpora for productive expert labeling, and then uses the resulting labeled data to bootstrap a context-based word vector classifier. Evaluation on materials science publications shows that the polyNER approach enables improved precision or recall relative to a state-of-the-art chemical entity extraction system at a dramatically lower cost: it required just two hours of expert time, rather than extensive and expensive rule engineering, to achieve that result. This result highlights the potential for human-computer partnership for constructing domain-specific scientific NER systems.

Crowdsourcing↗

JANUS: Resilient and Adaptive Data Transmission for Enabling Timely and Efficient Cross-Facility Scientific Workflows

In modern science, the growing complexity of large-scale scientific projects has led to an increasing reliance on cross-facility scientific workflows, where resources and expertise from multiple institutions and geographic locations are leveraged to accelerate scientific discovery. These workflows often require transmitting huge amounts of scientific data through wide-area networks. Although high-speed networks like ESnet and transfer services such as Globus have improved data mobility, several challenges remain. The sheer volume of data can overwhelm network bandwidth, widely used transport protocols such as TCP suffer from inefficiencies due to retransmissions triggered by packet loss, and existing fault-tolerance mechanisms like erasure coding introduce substantial overhead. In this paper, we propose Janus, a resilient and adaptable data transmission approach designed for cross-facility scientific workflows. Unlike traditional TCP-based methods, Janus leverages UDP, integrates erasure coding for fault tolerance, and combines it with error-bounded lossy compression to reduce overhead. This novel design allows users to balance data transmission time and accuracy, optimizing transfer performance based on specific scientific requirements. Additionally, Janus dynamically adjusts erasure coding parameters in response to real-time network conditions, ensuring efficient data transfers even in fluctuating environments. We develop optimization models for determining ideal configurations and implement adaptive data transfer protocols to enhance reliability. Through extensive simulations and real-network experiments, we demonstrate that Janus significantly improves transfer efficiency while maintaining data fidelity.

Esaulov, Vladislav [Georgia State University, Atla↗

PLUSWIND Derived Data

This dataset consists of annual CSV files containing multiple sources of modeled, hourly wind speeds and generation. For complete information about this dataset, including validation of modeled generation versus recorded generation, please see the Scientific Data article: Millstein, D., Jeong, S., Ancell, A., & Wiser, R. (2023). A database of hourly wind speed and modeled generation for US wind plants based on three meteorological models. Scientific Data, 10(1), 883. https://doi.org/10.1038/s41597-023-02804-w

17 WIND ENERGY↗

Final report- UFL - RAPIDS2: A SciDAC Institute for Computer Science, Data, and Artificial Intelligence

The research initiatives supported by the U.S. Department of Energy (DOE) Grant DE-SC0022265 are fundamentally aimed at pioneering advanced machine learning (ML) techniques for scientific data compression within high-performance computing (HPC) environments. This comprehensive body of work addresses the critical challenge posed by the exponential growth of data generated by scientific simulations in domains such as fusion energy, climate modeling, and computational fluid dynamics (CFD). A core objective is to develop compression algorithms that achieve substantial data reduction—often by orders of magnitude—while rigorously ensuring the fidelity of both the primary data (PD) and scientifically crucial derived quantities of interest (QoI). The methodologies deployed under this grant integrate sophisticated deep learning architectures, prominently featuring autoencoders, advanced generative models like conditional diffusion, and hybrid learning techniques. Key innovations include the development of Guaranteed Autoencoders (GAE) and the Guaranteed Conditional Diffusion with Tensor Correction (GCDTC) framework, which provide explicit, instance-level error bounds on reconstructed data. Furthermore, specialized strategies such as nonlinear constraint satisfaction are employed to preserve the integrity of QoI, a vital requirement for the trustworthiness of downstream scientific analyses. This research also focuses on the design and implementation of scalable, GPU-accelerated software pipelines that seamlessly integrate into existing HPC workflows, ensuring both computational efficiency and practical applicability. The CAESAR framework, for example, unifies foundation and generative models to create an adaptive and efficient compression solution for spatio-temporal scientific data. Collectively, these efforts represent a significant advancement in mitigating the scientific data deluge, enabling more effective data management, accelerated scientific discovery, and optimized utilization of HPC resources.

97 MATHEMATICS AND COMPUTING↗

ColorMoves: Real-time Interactive Colormap Construction for Scientific Visualization

This article presents ColorMoves, an interactive tool that promotes exploration of scientific data through artist-driven color methods in a unique and transformative way. Here, we discuss the power of contrast in scientific visualization, the design of the ColorMoves tool, and the tools application in several science domains.

97 MATHEMATICS AND COMPUTING↗

A Delay Tolerant Networking-Based Approach to a High Data Rate Architecture for Spacecraft

Historically, it has been the case that SWaP placed such severe constraints on radios that the links between spacecraft and the ground were relatively slow. This meant that the radio link was normally a significant bottleneck in returning scientific data. Over recent years, however, a combination of more efficient radio design, intelligent waveforms, and highly directed, high-frequency RF / optical systems have led to a rapid increase in the amount of data that can be pushed through radio and optical links. This has led to some cases where the radio links are capable of moving data much more quickly than the spacecraft and instruments are capable of actually generating it! In some instances, scientific data can therefore be lost not because the downlink is too slow to support the data rate, but instead because the spacecraft was not designed in a way that would let it fully utilize both the radio and the networking services available to it.The High Data Rate Architecture (HiDRA) project describes a packet-based approach to building modern, distributed spacecraft systems. It presents a means for spacecraft and other assets to participate in both present and future Delay Tolerant Networks (DTN), while simultaneously ensuring that the asset is able to fully utilize the new, high-speed links that have been seeing more widespread development and deployment in recent years. With this in mind, this paper begins with a discussion regarding HiDRA's evolution. Next, it discusses the capabilities and limitations of NASA's present DTN-enabled networks. Of particular note is the way in which principles of network design at the terrestrial level (e.g. use of programmable networks / software-defined networks, separation between data and control plane, infusion of COTS Ethernet switch chips, etc.) can all be translated into the space environment as well. After this, the paper discusses the design and implementation of a present prototype reference implementation of High-Rate DTN (HDTN), which is intended to demonstrate future high-rate networking concepts as part of a coherent demonstration on the International Space Station (ISS). The goal, of both the research and of this implementation, is to help develop a ready-made toolbox of ideas, approaches, and examples from which mission designers can draw when putting together new missions. Assuming all goes as planned, this should not only work to reduce the cost of individual mission design, but also improve the rate at which science data can be returned for mission participants to review.

Hylton, Alan↗

Improving NASA Earth Science Data and Information Access Through Natural Language Processing Based Data Analysis and Visualization

NASA: The Research Access initiative is part of the agency's framework for increasing public access to scientific publications and digital scientific data. The initiative follows the release of White House Office of Science and Technology Policy's (OSTP) memorandum "Increasing Access to the Results of Federally Funded Research," to ensure federally funded research is available to the public within one year of publication. NASA answered the mandate by creating an agency plan entitled "NASA Plan for Increasing Access to the Results of Scientific Research" and associated policy, NPD 2230.1, Research Data and Publication Access. Principles in NASA SMD Strategic Plan for Scientific Data and Computing: Continued free and open access to scientific data for any use. Improved ease of use and discoverability. Enhanced science applications and new use cases. Incorporates best practices and "state of the art" through partnerships. Earth Data and Systems are Evolving: Increasing archive and file sizes. More complicated data structures. More user-friendly and data services. What is the future direction?

Liu, Zhong↗

Interdisciplinary Research Scenario Testing of EOSDIS

During the reporting period, the Principle Investigator (PI) has continued to serve on numerous review panels, task forces and committees with the goal of providing input and guidance for the Earth Observing System Data and Information System (EOSDIS) program at NASA Headquarters and NASA GSFC. In addition, the PI has worked together with personnel at the University of Virginia and the subcontractor (Simpson Weather Associates (SWA)) to continue to evaluate the latest releases of various versions of the user interfaces to the EOSDIS. Finally, as part of the subcontract, SWA has created an on-line Hierarchial Data Format (HDF) tutorial for non-HDF experts, particularly those that will be using EOSDIS and future EOS data products. A summary of these three activities is provided. The topics include: 1) Participation on EODIS Panels and Committees; 2) Evaluation and Tire Kicking of EODIS User Interfaces; and 3) An On-line HDF Tutorial. The report also includes attachments A, B, and C. Attachment A: Report From the May 1999 Science Data Panel. The topics include: 1) Summary of Data Panel Meeting; and 2) Panel's Comments/Recommendations. Attachment B: Survey Requesting Integrated Design Systems (IDS) Teams Input on the Descoping and Rescoping of the EODIS; and Attachment C: An HDF Tutorial for Beginners: EODIS Users and Small Data Providers (HTML Version). The topics include: 1) Tutorial Overview; 2) An introduction to HDF; 3) The HDF Library: Software and Hardware; 4) Methods of Working with HDF Files; 5) Scientific Data API; 6) Attributes and Metadata; 7) Writing a SDS to an HDF file; 8) Obtaining Information on Existing HDF Files; 9) Reading a Scientific Data Set from an HDF file: 10) Example Programs; 11) Browsing and Visualizing HDF Data; and 12) Laboratory (Question and Answer).

Emmitt, G. D.↗

A Delay Tolerant Networking-Based Approach to a High Data Rate Architecture for Spacecraft

Historically, it has been the case that SWaP (Size, Weigh a lot, And consume Power) placed such severe constraints on radios that the links between spacecraft and the ground were relatively slow. This meant that the radio link was normally a significant bottleneck in returning scientific data. Over recent years, however, a combination of more efficient radio design, intelligent waveforms, and highly directed, high-frequency RF / optical systems have led to a rapid increase in the amount of data that can be pushed through radio and optical links. This has led to some cases where the radio links are capable of moving data much more quickly than the spacecraft and instruments are capable of actually generating it! In some instances, scientific data can therefore be lost not because the downlink is too slow to support the data rate, but instead because the spacecraft was not designed in a way that would let it fully utilize both the radio and the networking services available to it. The High Data Rate Architecture (HiDRA) project describes a packet-based approach to building modern, distributed spacecraft systems. It presents a means for spacecraft and other assets to participate in both present and future Delay Tolerant Networks (DTN), while simultaneously ensuring that the asset is able to fully utilize the new, high-speed links that have been seeing more widespread development and deployment in recent years. With this in mind, this paper begins with a discussion regarding HiDRA's evolution. Next, it discusses the capabilities and limitations of NASA's present DTN-enabled networks. Of particular note is the way in which principles of network designate the terrestrial level (e.g., use of programmable networks / software-defined networks, separation between data and control plane, infusion of COTS (Commercial Off-The-Shelf) Ethernet switch chips, etc.) can all be translated into the space environment as well. After this, the paper discusses the design and implementation of a present prototype reference implementation of High-Rate DTN (HDTN),which is intended to demonstrate future high-rate networking concepts as part of a coherent demonstration on the International Space Station (ISS). The goal, of both the research and of this implementation, is to help develop a ready-made toolbox of ideas, approaches, and examples from which mission designers can draw when putting together new missions. Assuming all goes as planned, this should not only work to reduce the cost of individual mission design, but also improve the rate at which science data can be returned for mission participants to review.

Hylton, Alan↗

Exploring Future Storage Options for ATLAS at the BNL/SDCC facility

The ATLAS experiment is expected to deliver an unprecedented amount of scientific data in the High Luminosity(HL-LHC) era. As the demand for disk storage capacity in ATLAS continues to rise steadily, the BNL Scientific Data and Computing Center (SDCC) faces challenges in terms of cost implications for maintaining multiple disk copies and adapting to the coming ATLAS storage requirements. To address these challenges, the SDCC Storage team has undertaken a thorough analysis of the ATLAS experiment's requirements, matching them to suitable storage options and strategies, and has explored alternatives to enhance or replace the current storage solution. This paper aims to present the main challenges encountered while supporting big data experiments such as ATLAS. We describe the experiment's specific requirements and priorities, particularly focusing on the critical storage system characteristics of the high-luminosity run and how the key storage components provided by the Storage team work together: the dCache disk storage system; its archival back-end, HPSS; and its OS-level backend Storage. Specifically, we investigate a novel approach to integrate Lustre and XRootD. In this setup, Lustre serves as backend storage and XRootD acts as an access layer frontend, supporting various grid access protocols. Additionally, we also describe the validation and commissioning tests, including the performance comparison between dCache and XRootd. Furthermore, we provide a performance and cost analysis comparing OpenZFS and LINUX MD RAID, evaluate different storage software stacks, and showcase stress tests conducted to validate Third Party Copy (TPC) functionality.

99 GENERAL AND MISCELLANEOUS↗

Qresp, a tool for curating, discovering and exploring reproducible scientific papers

We propose a strategy and present a simple tool to facilitate scientific data reproducibility by making available, in a distributed manner, all data and procedures presented in scientific papers, together with metadata to render them searchable and discoverable. In particular, we describe a graphical user interface (GUI), Qresp, to curate papers (i.e. generate metadata) and to explore curated papers and automatically access the data presented in scientific publications.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Department of Energy's Biological and Environmental Research Strategic Data Roadmap for Earth System Science

Rapid advances in experimental, sensor, and computational technologies and techniques are driving exponential growth in the volume, acquisition rate, variety, and complexity of scientific data. This wealth of scientifically meaningful data has tremendous potential to lead to scientific discovery. However, to achieve scientific breakthroughs, these data must be exploitable—they must be analyzed effectively and efficiently and the results shared and communicated easily within the wider Department of Energy’s (DOE’s) Biological and Environmental Research (BER) Climate and Environmental Sciences Division (CESD) community. The explosion in data complexity and scale makes these tasks exceedingly difficult to achieve, particularly given that an increasing number of disciplines are working across techniques, integrating simulation and experimental or observational results (see Table 5 in Appendix 2). Consequently, we need new approaches to data management, analysis, and visualization that provide research teams with easy-to-use and scalable end-to-end solutions. These solutions must facilitate (and where feasible, automate and capture) every stage in the data lifecycle (shown in Figure 1), from collection to management, annotation, sharing, discovery, analysis, and visualization. In addition, the core functionalities are the same across climate science communities, but they require customization to adapt to specific needs and fit into research and analysis workflows. To this end, the mission of CESD’s Data and Informatics Program is to integrate all existing and future distributed CESD data holdings into a seamless and unified environment for the acceleration of Earth system science.

54 ENVIRONMENTAL SCIENCES↗

Data Reduction Techniques for Simulation, Visualization and Data Analysis

Data reduction is increasingly being applied to scientific data for numerical simulations, scientific visualizations, and data analyses. It is most often used to lower I/O and storage costs, and sometimes to lower in-memory data size as well. With this work, we consider five categories of data reduction techniques based on their information loss: 1) truly lossless, 2) near lossless, 3) lossy, 4) mesh reduction, and 5) derived representations. We then survey available techniques in each of these categories, summarize their properties from a practical point of view, and discuss relative merits within a category. We believe, in total, this work will enable simulation scientists and visualization/data analysis scientists to decide which data reduction techniques will be most helpful for their needs.

97 MATHEMATICS AND COMPUTING↗