SEARCH · Search NASA

Results for “parquet”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

NASA NTRS2017

Use of Schema on Read in Earth Science Data Archives

Traditionally, NASA Earth Science data archives have file-based storage using proprietary data file formats, such as HDF and HDF-EOS, which are optimized to support fast and efficient storage of spaceborne and model data as they are generated. The use of file-based storage essentially imposes an indexing strategy based on data dimensions. In most cases, NASA Earth Science data uses time as the primary index, leading to poor performance in accessing data in spatial dimensions. For example, producing a time series for a single spatial grid cell involves accessing a large number of data files. With exponential growth in data volume due to the ever-increasing spatial and temporal resolution of the data, using file-based archives poses significant performance and cost barriers to data discovery and access. Storing and disseminating data in proprietary data formats imposes an additional access barrier for users outside the mainstream research community. At the NASA Goddard Earth Sciences Data Information Services Center (GES DISC), we have evaluated applying the schema-on-read principle to data access and distribution. We used Apache Parquet to store geospatial data, and have exposed data through Amazon Web Services (AWS) Athena, AWS Simple Storage Service (S3), and Apache Spark. Using the schema-on-read approach allows customization of indexing spatially or temporally to suit the data access pattern. The storage of data in open formats such as Apache Parquet has widespread support in popular programming languages. A wide range of solutions for handling big data lowers the access barrier for all users. This presentation will discuss formats used for data storage, frameworks with This presentation will discuss formats used for data storage, frameworks with support for schema-on-read used for data access, and common use cases covering data usage patterns seen in a geospatial data archive.

cloud applications↗

DOE OSTI2016

Storage and Read-Optimized Data Placement Structures for High-Performance Analysis [Book Chapter]

Here, we present state-of-art structures and methods for efficient data preparation and representation for analysis. Our intent is to introduce the data science and analytics communities to open source data placements, structures, and methods. These practices can make the foundational processes of data preparation and access dramatically more efficient than typical raw file or database representations and use more conservative storage. To illustrate this, we introduce two highly efficient data placement structures. We then present a tutorial, supported by step-by-step examples, of how to create, use, and access data, structured by Parquet or ORC (Optimized Row Columnar), using Apache Spark. Finally, we illustrate the benefits of using these structures with computational and storage volume benchmarks.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

HERO WEC Belt Test Data

The following submission includes raw and processed data from the 2024 Hydraulic and Electric Reverse Osmosis Wave Energy Converter (HERO WEC) belt tests conducted using NREL's Large Amplitude Motion Platform (LAMP). A description of the motion profiles run during testing can be found in the run log document. Data was collected using NREL's Modular Ocean Data AcQuisition (MODAQ) system in the form of TDMS files. Data was then processed using Python and MATLAB and converted to MATLAB workspace, parquet, and csv file formats. During Data processing, a low pass filter was applied to each array and the arrays were then resampled to common 10Hz timestamps. A MATLAB data viewer script is provided to quickly visualize these data sets. The following arrays are contained in each test data file: - Time: Unix seconds timestamp - Test_Time: Time in seconds since beginning of test - POS_OS_1001: Encoder position in degrees (the encoder is located on the secondary shaft of the spring return and is driven by the winch after a 4.5:1 gear reduction) - LC_ST_1001: Anchor load cell data in lbf - PRESS_OS_2002: Air spring pressure in psi This data set has been developed by the National Renewable Energy Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. Funding provided by the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy Water Power Technologies Office.

16 TIDAL AND WAVE POWER↗

DOE OSTI2016

Real-Time Discovery Services over Large, Heterogeneous and Complex Healthcare Datasets Using Schema-Less, Column-Oriented Methods

We present a service platform for schema-leess exploration of data and discovery of patient-related statistics from healthcare data sets. The architecture of this platform is motivated by the need for fast, schema-less, and flexible approaches to SQL-based exploration and discovery of information embedded in the common, heterogeneously structured healthcare data sets and supporting components (electronic health records, practice management systems, etc.) The motivating use cases described in the paper are clinical trials candidate discovery, and a treatment effectiveness analysis. Following the use cases, we discuss the key features and software architecture of the platform, the underlying core components (Apache Parquet, Drill, the web services server), and the runtime profiles and performance characteristics of the platform. We conclude by showing dramatic speedup with some approaches, and the performance tradeoffs and limitations of others.

Apache Drill Apache Parquet Column Oriented Stores↗

DOE OSTI2019

Orbital selectivity versus Pomeranchuk instability in the iron–chalcogenide superconductors: A two-loop renormalization group study

We perform a two-loop renormalization group (RG) analysis of a 2D effective multiband model, which is relevant for describing the low-energy properties of some iron–chalcogenide superconducting materials. Crucial ingredients in this analysis are the calculation of higher-order contributions in the RG scheme that go beyond the widely-used parquet approximation and the consequent inclusion of nontrivial self-energy effects of the model that yield an anisotropic renormalization of the quasiparticle weight in the system. The motivation of our work is the experimental discovery by Sprau et al. (2017) that orbitally-selective renormalization of the quasiparticle weight in the Hund’s metal phase at moderate temperatures underpins the highly unusual gap in the superconducting phase of the FeSe compound at lower temperatures. One prediction we arrive here is that the underlying origin of nematicity in these systems may indeed come from orbital-selectivity, instead of a Pomeranchuk instability in the d{sub ±} channel. This orbital selectivity is driven by the presence of stripe-type antiferromagnetic fluctuations in the model. Therefore, we argue that the present RG results may provide a scenario from a weak-to-moderate coupling perspective, in which the role of orbital selectivity to describe the physical properties of some iron–chalcogenide superconductors is emphasized.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

DOE OSTI2025

High-performance data format for scientific data storage and analysis

Here, in this article, we present the High-Performance Output (HiPO) data format developed at Jefferson Laboratory for storing and analyzing data from Nuclear Physics experiments. The format was designed to efficiently store large amounts of experimental data, utilizing modern fast compression algorithms. The purpose of this development was to provide organized data in the output, facilitating access to relevant information within the large data files. The HiPO data format has features that are suited for storing raw detector data, reconstruction data, and the final physics analysis data efficiently, eliminating the need to do data conversions through the lifecycle of experimental data. The HiPO data format is implemented in C++ and JAVA, and provides bindings to FORTRAN, Python, and Julia, providing users with the choice of data analysis frameworks to use. In this paper, we will present the general design and functionalities of the HiPO library and compare the performance of the library with more established data formats used in data analysis in High Energy and Nuclear Physics (such as ROOT and Parquete). In columnar data analysis, HiPO surpasses established data formats in performance and can be effectively applied to data analysis in other scientific fields.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2014

Nanostructured rocksalt-type solid solution series (Ge{sub 1−x}Sn{sub x}Te){sub n}Sb{sub 2}Te{sub 3} (n=4, 7, 12; 0≤x≤1): Thermal behavior and thermoelectric properties

Solid solutions (Ge{sub 1−x}Sn{sub x}Te){sub n}Sb{sub 2}Te{sub 3} (n=4, 7, 12; 0≤x≤1) represent stable high-temperature phases and can be obtained as metastable compounds by quenching. High-resolution transmission electron microscopy reveals that the quenched (pseudo-)cubic materials exhibit parquet-like nanostructures comparable to, but especially for n=4 more pronounced than in (GeTe){sub n}Sb{sub 2}Te{sub 3} (GST materials). The temperature-dependent phase transitions are comparable; however, substitution with Sn significantly lowers the transition temperatures between cubic high-temperature phase and the long range ordered layered phases that are stable at ambient conditions. In addition, the metrics of the quenched Sn-containing materials remains closer to cubic, especially for samples with n=7 or 12. For samples with high defect concentrations (n=4, 7), Sn-substituted samples exhibit electrical conductivities up to 3 times higher than those of corresponding GST materials. Since the difference in thermal conductivity is much less pronounced, this results in a doubling of the thermoelectric figure of merit (ZT) at high temperatures for (Ge{sub 0.5}Sn{sub 0.5}Te){sub 4}Sb{sub 2}Te{sub 3} as compared to (GeTe){sub 4}Sb{sub 2}Te{sub 3}. Sn doping in (GeTe){sub 7}Sb{sub 2}Te{sub 3} increases the ZT value by a factor of up to 4 which is also due to an increased Seebeck coefficient. - Graphical abstract: High-resolution transmission electron micrographs of (GeTe){sub 4}Sb{sub 2}Te{sub 3} (top) and (Ge{sub 0.5}Sn{sub 0.5}Te){sub 4}Sb{sub 2}Te{sub 3} (bottom) with different nanostructures and thermoelectric figures of merit (ZT) of these samples. - Highlights: • Representative samples of solid solutions of (Ge{sub 1−x}Sn{sub x}Te){sub n}Sb{sub 2}Te{sub 3} were synthesized. • Sn substitution leads to more pronounced nanostructures in defect-rich compounds. • Phase transitions are comparable to (GeTe){sub n}Sb{sub 2}Te{sub 3} but occur at lower temperatures. • Sn substitution tends to increase electrical conductivity and Seebeck coefficient. • ZT of (GeTe){sub 7}Sb{sub 2}Te{sub 3} increases from 0.2 to 0.7 (at 400 °C) upon Sn substitution.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2020

Disorderly Conduct of Benzamide IV: Crystallographic and Computational Analysis of High Entropy Polymorphs of Small Molecules

Benzamide, a simple derivative of benzoic acid and a common intermediate of pharmaceutical compounds, was reported to form two polymorphs in 1832, but the single crystal structure of the more stable form was not solved until 1959. Nearly 50 years later, the second form was characterized by powder diffraction, followed shortly thereafter by characterization of a third form, a polytype of the most thermodynamically stable Form I. These two new forms, Forms II and III, are metastable. Herein, we describe a fourth polymorph, Form IV, discovered by melt crystallization concurrently with its crystallization under confinement at small length scales (<10 nm), where it is stable indefinitely. Form III exists under confinement in larger pores, and melting point data for different pore sizes corroborate the existence of Form IV below 10 nm. Form IV is highly disordered, precluding indexing of powder diffraction data other than hk0 reflections. Nonetheless, a combination of powder X-ray diffraction and computational crystal structure prediction reveals that Form IV contains a 2D motif resembling that of Form II, but with longrange order in the third dimension masked by ubiquitous stacking faults. This approach relies on distilling a large number of candidate structures to a few possible disorder models based on benzamide tetrads that organize in 2D parquet-like tiles, with organization along the third dimension, that can be modeled with various stacking fault configurations having distinct intermolecular interactions and translations in the dimension orthogonal to the tiling planes. These observations reveal a bewildering crystallographic complexity for such a simple molecule. Nonetheless, the approach described herein demonstrates that challenging structures that may be abandoned prematurely because of poor crystallinity, twinning, or disorder can be solved.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2020

Striped Data Analysis Framework

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. We have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

Gutsche, Oliver↗

DOE OSTI2018

Striped Data Server for Scalable Parallel Data Analysis

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. As a result, we have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2025

Unified Picture of Superconductivity and Magnetism in CeRh 2 ⁢As 2

Here, we propose a theory for the microscopic origin of the multiple superconducting and magnetic phases observed in CeRh 2 ⁢As 2 based on the existence of Van Hove singularities near the Fermi energy. The nonsymmorphic symmetry of this material implies that these singularities are located away from high-symmetry momenta; i.e., they have so-called type-II character. This allows us to include the significant Rashba spin-orbit coupling in CeRh 2 ⁢As 2 in a parquet renormalization group approach. When Fermi-surface nesting is strong, our analysis reveals two closely competing superconducting states with opposite parities, as well as an instability toward spin-density wave states that support both of them, consistent with the phase diagram of CeRh 2 ⁢As 2 . Type-II Van Hove singularities are generic to nonsymmorphic space groups, and so our theory implies that many other compounds may support closely competing even- and odd-parity superconductivity.

Rashba coupling↗

DOE OSTI2025

Preliminary Study on Fine-Grained Power and Energy Measurements on Grace Hopper GH200 with Open-Source Performance Tools

The increasing adoption of tightly integrated, heterogeneous architectures, combined with the slowdown of Moore’s law, has made application power and energy-driven optimizations critical to efficiently use high-performance computing systems. This paper introduces a newly developed open-source toolkit that seamlessly integrates the Linux real-time hardware monitoring program hwmon with the Performance Application Programming Interface and the Score-P performance measurement system, thereby enabling fine-grained power and energy measurements for high-performance computing applications. Our primary target platform is the Wombat test bed, which is a system based on the NVIDIA GH200 superchip. The toolkit can capture transient power peaks with high temporal resolution (50 ms) and, thanks to Score-P integration, can map power metrics to specific code regions, thereby providing actionable information on power-intensive operations and inefficiencies. The toolkit also provides a holistic view of both the power and the energy consumption of the entire GH200 superchip by covering all major components: the Grace CPU, the Hopper GPU, and the I/O subsystem. Experiments that use Locally Self-consistent Multiple Scattering, which is an application for first-principles calculations of materials developed at Oak Ridge National Laboratory, have demonstrated the tool’s ability to identify transient power spikes and uncover opportunities for energy-aware optimizations. Additionally, we introduce a Python-based utility for converting Open Trace Format 2 traces to Parquet format, thus enabling advanced data analysis for numerical integration methods applied to power data for accurate energy profiling.

Hernandez Mendoza, Oscar [ORNL] (ORCID:00000002538↗

DOE OSTI2021

webspinner

Python utilities for working with data source types used by NREL's dsgrid (Demand-Side Grid Model) project (i.e. AWS, PostgreSQL, and .parquet). https://www.nrel.gov/analysis/dsgrid.html

Hale, Elaine↗

DOE OSTI2023

BiG-SLiCE 2 v1.0.0

BiG-SLiCE was originally an open source Python-based command line bioinformatics software that offers a highly scalable clustering analysis on biosynthetic gene clusters (BGC) data. It allows a simultaneous analysis of millions of BGCs, exceeding the capability of other existing tools (around one hundred thousands). As a tradeoff, the clustering accuracy is relatively lower and sometimes fall short in corner cases and specific BGC classes such as the RiPPs (Ribosomally-translated, Post-translationally modified Peptides). In BiG-SLiCE V2 (developed in LBNL), the clustering algorithm has been significantly improved to deliver a much accurate result even for RiPPs and other previous corner case classes. Moreover, the speed of the overall pipeline has been improved by 50-100%. Finally, additional features were implemented to support downstream analyses of BiG-SLiCE results, such as customized tabular (TSV/CSV) and columnar (Parquet) outputs.

Kautsar, Satria↗

DOE OSTI2025

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗

DOE OSTI2026

polars-dovmed (dovmed) v0.1.0

polars-dovmed is a python package for text search and extraction from NCBI's PubMed Central Open Access subset. It is powered by the polars dataframe library and leverages modern file formats (parquet) to efficiently scan public literature.

Roux, Simon [Lawrence Berkeley National Laboratory↗

DOE OSTI2026

datasight [SWR-26-045]

This software is an AI-powered data exploration with natural language. datasight connects an AI agent to your database and provides a web UI where you can ask questions in natural language. The agent writes SQL, runs queries, and generates interactive Plotly visualizations. Supports DuckDB, PostgreSQL, SQLite, and Flight SQL databases. Also queries local CSV and Parquet files directly — no database setup required. Supports Anthropic Claude (default), GitHub Models (open source), and Ollama (local) as LLM backends.

Thom, Daniel [National Laboratory of the Rockies (↗

DOE OSTI2025

Electricity Baseline 2022 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2022 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilized the appdirs Python dependency (https://pypi.org/project/appdirs/). This submission includes the background data used to generate the 2022 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: `python -c "import appdirs; print(appdirs.user_data_dir())"`). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2022 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; data inventory↗