Search NASA⌕ Search

SEARCH · Search NASA

Results for “parquet”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

Real-Time Discovery Services over Large, Heterogeneous and Complex Healthcare Datasets Using Schema-Less, Column-Oriented Methods

We present a service platform for schema-leess exploration of data and discovery of patient-related statistics from healthcare data sets. The architecture of this platform is motivated by the need for fast, schema-less, and flexible approaches to SQL-based exploration and discovery of information embedded in the common, heterogeneously structured healthcare data sets and supporting components (electronic health records, practice management systems, etc.) The motivating use cases described in the paper are clinical trials candidate discovery, and a treatment effectiveness analysis. Following the use cases, we discuss the key features and software architecture of the platform, the underlying core components (Apache Parquet, Drill, the web services server), and the runtime profiles and performance characteristics of the platform. We conclude by showing dramatic speedup with some approaches, and the performance tradeoffs and limitations of others.

Apache Drill Apache Parquet Column Oriented Stores↗
DOE OSTI2019

Orbital selectivity versus Pomeranchuk instability in the iron–chalcogenide superconductors: A two-loop renormalization group study

We perform a two-loop renormalization group (RG) analysis of a 2D effective multiband model, which is relevant for describing the low-energy properties of some iron–chalcogenide superconducting materials. Crucial ingredients in this analysis are the calculation of higher-order contributions in the RG scheme that go beyond the widely-used parquet approximation and the consequent inclusion of nontrivial self-energy effects of the model that yield an anisotropic renormalization of the quasiparticle weight in the system. The motivation of our work is the experimental discovery by Sprau et al. (2017) that orbitally-selective renormalization of the quasiparticle weight in the Hund’s metal phase at moderate temperatures underpins the highly unusual gap in the superconducting phase of the FeSe compound at lower temperatures. One prediction we arrive here is that the underlying origin of nematicity in these systems may indeed come from orbital-selectivity, instead of a Pomeranchuk instability in the d{sub ±} channel. This orbital selectivity is driven by the presence of stripe-type antiferromagnetic fluctuations in the model. Therefore, we argue that the present RG results may provide a scenario from a weak-to-moderate coupling perspective, in which the role of orbital selectivity to describe the physical properties of some iron–chalcogenide superconductors is emphasized.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2025

High-performance data format for scientific data storage and analysis

Here, in this article, we present the High-Performance Output (HiPO) data format developed at Jefferson Laboratory for storing and analyzing data from Nuclear Physics experiments. The format was designed to efficiently store large amounts of experimental data, utilizing modern fast compression algorithms. The purpose of this development was to provide organized data in the output, facilitating access to relevant information within the large data files. The HiPO data format has features that are suited for storing raw detector data, reconstruction data, and the final physics analysis data efficiently, eliminating the need to do data conversions through the lifecycle of experimental data. The HiPO data format is implemented in C++ and JAVA, and provides bindings to FORTRAN, Python, and Julia, providing users with the choice of data analysis frameworks to use. In this paper, we will present the general design and functionalities of the HiPO library and compare the performance of the library with more established data formats used in data analysis in High Energy and Nuclear Physics (such as ROOT and Parquete). In columnar data analysis, HiPO surpasses established data formats in performance and can be effectively applied to data analysis in other scientific fields.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2014

Nanostructured rocksalt-type solid solution series (Ge{sub 1−x}Sn{sub x}Te){sub n}Sb{sub 2}Te{sub 3} (n=4, 7, 12; 0≤x≤1): Thermal behavior and thermoelectric properties

Solid solutions (Ge{sub 1−x}Sn{sub x}Te){sub n}Sb{sub 2}Te{sub 3} (n=4, 7, 12; 0≤x≤1) represent stable high-temperature phases and can be obtained as metastable compounds by quenching. High-resolution transmission electron microscopy reveals that the quenched (pseudo-)cubic materials exhibit parquet-like nanostructures comparable to, but especially for n=4 more pronounced than in (GeTe){sub n}Sb{sub 2}Te{sub 3} (GST materials). The temperature-dependent phase transitions are comparable; however, substitution with Sn significantly lowers the transition temperatures between cubic high-temperature phase and the long range ordered layered phases that are stable at ambient conditions. In addition, the metrics of the quenched Sn-containing materials remains closer to cubic, especially for samples with n=7 or 12. For samples with high defect concentrations (n=4, 7), Sn-substituted samples exhibit electrical conductivities up to 3 times higher than those of corresponding GST materials. Since the difference in thermal conductivity is much less pronounced, this results in a doubling of the thermoelectric figure of merit (ZT) at high temperatures for (Ge{sub 0.5}Sn{sub 0.5}Te){sub 4}Sb{sub 2}Te{sub 3} as compared to (GeTe){sub 4}Sb{sub 2}Te{sub 3}. Sn doping in (GeTe){sub 7}Sb{sub 2}Te{sub 3} increases the ZT value by a factor of up to 4 which is also due to an increased Seebeck coefficient. - Graphical abstract: High-resolution transmission electron micrographs of (GeTe){sub 4}Sb{sub 2}Te{sub 3} (top) and (Ge{sub 0.5}Sn{sub 0.5}Te){sub 4}Sb{sub 2}Te{sub 3} (bottom) with different nanostructures and thermoelectric figures of merit (ZT) of these samples. - Highlights: • Representative samples of solid solutions of (Ge{sub 1−x}Sn{sub x}Te){sub n}Sb{sub 2}Te{sub 3} were synthesized. • Sn substitution leads to more pronounced nanostructures in defect-rich compounds. • Phase transitions are comparable to (GeTe){sub n}Sb{sub 2}Te{sub 3} but occur at lower temperatures. • Sn substitution tends to increase electrical conductivity and Seebeck coefficient. • ZT of (GeTe){sub 7}Sb{sub 2}Te{sub 3} increases from 0.2 to 0.7 (at 400 °C) upon Sn substitution.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Disorderly Conduct of Benzamide IV: Crystallographic and Computational Analysis of High Entropy Polymorphs of Small Molecules

Benzamide, a simple derivative of benzoic acid and a common intermediate of pharmaceutical compounds, was reported to form two polymorphs in 1832, but the single crystal structure of the more stable form was not solved until 1959. Nearly 50 years later, the second form was characterized by powder diffraction, followed shortly thereafter by characterization of a third form, a polytype of the most thermodynamically stable Form I. These two new forms, Forms II and III, are metastable. Herein, we describe a fourth polymorph, Form IV, discovered by melt crystallization concurrently with its crystallization under confinement at small length scales (<10 nm), where it is stable indefinitely. Form III exists under confinement in larger pores, and melting point data for different pore sizes corroborate the existence of Form IV below 10 nm. Form IV is highly disordered, precluding indexing of powder diffraction data other than hk0 reflections. Nonetheless, a combination of powder X-ray diffraction and computational crystal structure prediction reveals that Form IV contains a 2D motif resembling that of Form II, but with longrange order in the third dimension masked by ubiquitous stacking faults. This approach relies on distilling a large number of candidate structures to a few possible disorder models based on benzamide tetrads that organize in 2D parquet-like tiles, with organization along the third dimension, that can be modeled with various stacking fault configurations having distinct intermolecular interactions and translations in the dimension orthogonal to the tiling planes. These observations reveal a bewildering crystallographic complexity for such a simple molecule. Nonetheless, the approach described herein demonstrates that challenging structures that may be abandoned prematurely because of poor crystallinity, twinning, or disorder can be solved.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Striped Data Analysis Framework

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. We have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

Gutsche, Oliver↗
DOE OSTI2018

Striped Data Server for Scalable Parallel Data Analysis

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. As a result, we have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2025

Unified Picture of Superconductivity and Magnetism in CeRh 2 ⁢As 2

Here, we propose a theory for the microscopic origin of the multiple superconducting and magnetic phases observed in CeRh 2 ⁢As 2 based on the existence of Van Hove singularities near the Fermi energy. The nonsymmorphic symmetry of this material implies that these singularities are located away from high-symmetry momenta; i.e., they have so-called type-II character. This allows us to include the significant Rashba spin-orbit coupling in CeRh 2 ⁢As 2 in a parquet renormalization group approach. When Fermi-surface nesting is strong, our analysis reveals two closely competing superconducting states with opposite parities, as well as an instability toward spin-density wave states that support both of them, consistent with the phase diagram of CeRh 2 ⁢As 2 . Type-II Van Hove singularities are generic to nonsymmorphic space groups, and so our theory implies that many other compounds may support closely competing even- and odd-parity superconductivity.

Rashba coupling↗
DOE OSTI2025

Preliminary Study on Fine-Grained Power and Energy Measurements on Grace Hopper GH200 with Open-Source Performance Tools

The increasing adoption of tightly integrated, heterogeneous architectures, combined with the slowdown of Moore’s law, has made application power and energy-driven optimizations critical to efficiently use high-performance computing systems. This paper introduces a newly developed open-source toolkit that seamlessly integrates the Linux real-time hardware monitoring program hwmon with the Performance Application Programming Interface and the Score-P performance measurement system, thereby enabling fine-grained power and energy measurements for high-performance computing applications. Our primary target platform is the Wombat test bed, which is a system based on the NVIDIA GH200 superchip. The toolkit can capture transient power peaks with high temporal resolution (50 ms) and, thanks to Score-P integration, can map power metrics to specific code regions, thereby providing actionable information on power-intensive operations and inefficiencies. The toolkit also provides a holistic view of both the power and the energy consumption of the entire GH200 superchip by covering all major components: the Grace CPU, the Hopper GPU, and the I/O subsystem. Experiments that use Locally Self-consistent Multiple Scattering, which is an application for first-principles calculations of materials developed at Oak Ridge National Laboratory, have demonstrated the tool’s ability to identify transient power spikes and uncover opportunities for energy-aware optimizations. Additionally, we introduce a Python-based utility for converting Open Trace Format 2 traces to Parquet format, thus enabling advanced data analysis for numerical integration methods applied to power data for accurate energy profiling.

Hernandez Mendoza, Oscar [ORNL] (ORCID:00000002538↗
DOE OSTI2021

webspinner

Python utilities for working with data source types used by NREL's dsgrid (Demand-Side Grid Model) project (i.e. AWS, PostgreSQL, and .parquet). https://www.nrel.gov/analysis/dsgrid.html

Hale, Elaine↗
DOE OSTI2023

BiG-SLiCE 2 v1.0.0

BiG-SLiCE was originally an open source Python-based command line bioinformatics software that offers a highly scalable clustering analysis on biosynthetic gene clusters (BGC) data. It allows a simultaneous analysis of millions of BGCs, exceeding the capability of other existing tools (around one hundred thousands). As a tradeoff, the clustering accuracy is relatively lower and sometimes fall short in corner cases and specific BGC classes such as the RiPPs (Ribosomally-translated, Post-translationally modified Peptides). In BiG-SLiCE V2 (developed in LBNL), the clustering algorithm has been significantly improved to deliver a much accurate result even for RiPPs and other previous corner case classes. Moreover, the speed of the overall pipeline has been improved by 50-100%. Finally, additional features were implemented to support downstream analyses of BiG-SLiCE results, such as customized tabular (TSV/CSV) and columnar (Parquet) outputs.

Kautsar, Satria↗
DOE OSTI2025

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗
DOE OSTI2026

polars-dovmed (dovmed) v0.1.0

polars-dovmed is a python package for text search and extraction from NCBI's PubMed Central Open Access subset. It is powered by the polars dataframe library and leverages modern file formats (parquet) to efficiently scan public literature.

Roux, Simon [Lawrence Berkeley National Laboratory↗
DOE OSTI2026

datasight [SWR-26-045]

This software is an AI-powered data exploration with natural language. datasight connects an AI agent to your database and provides a web UI where you can ask questions in natural language. The agent writes SQL, runs queries, and generates interactive Plotly visualizations. Supports DuckDB, PostgreSQL, SQLite, and Flight SQL databases. Also queries local CSV and Parquet files directly — no database setup required. Supports Anthropic Claude (default), GitHub Models (open source), and Ollama (local) as LLM backends.

Thom, Daniel [National Laboratory of the Rockies (↗
DOE OSTI2025

Electricity Baseline 2022 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2022 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilized the appdirs Python dependency (https://pypi.org/project/appdirs/). This submission includes the background data used to generate the 2022 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: `python -c "import appdirs; print(appdirs.user_data_dir())"`). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2022 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; data inventory↗
DOE OSTI2025

Electricity Baseline 2021 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2021 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilizes the appdirs Python dependency (https://pypi.org/project/appdirs/). An overview of the ElectricityLCI data stores may be found on the README (https://github.com/USEPA/ElectricityLCI/blob/v2.0/README.md#data-store). This submission includes the background data used to generate the 2021 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: python -c "import appdirs; print(appdirs.user_data_dir())"). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2021 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; LCI; Life Cycle; data inventory↗
DOE OSTI2025

Electricity Baseline 2020 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2020 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilizes the appdirs Python dependency (https://pypi.org/project/appdirs/). An overview of the ElectricityLCI data stores may be found on the README (https://github.com/USEPA/ElectricityLCI/blob/v2.0/README.md#data-store). This submission includes the background data used to generate the 2020 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: python -c "import appdirs; print(appdirs.user_data_dir())"). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2020 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; LCI; Life Cycle; data inventory↗
DOE OSTI2021

Machine Learning Guided Operational Intelligence from Synchrophasors (Final Report)

Schweitzer Engineering Laboratories (SEL) and Oregon State University (OSU) received over 27 terabytes of electrical power system phasor measurement unit (PMU) data for the Eastern, Western, and ERCOT interconnections. The dataset includes measurements spread across 446 PMUs from early 2016 to mid 2018 depending on the interconnect. The full dataset was split into a training and test (holdout) dataset by PNNL. All data was received in the Apache Parquet format. The overarching goal of this project is to develop and execute a strategy to mitigate data anomalies, perform analysis on the dataset, and detect anomalous events in the data.

24 POWER TRANSMISSION AND DISTRIBUTION↗