Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data refactoring”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Energy Exascale Earth System Model v2.0.1

First patch release of v2.0.0 Changes since v2.0.0 [Important change] Fix ocean threading bug seen in debug cases on Chrysalis with Intel 20.0.4. Was introduced around time of v2.0.0 tag. Does not change v2.0.0 answers on Chrysalis because those didn't use threading or debugging. [EAM] Add semi-lagrangian tracer transport for theta-l (F90 and C++), add new algorithm for finding tropopause, add DSCREAM to allow v2 and SCREAM settings in same code such as adjust_ps [EAM-MMF] 60L default, allow C++ back end of RRTMGP (EAM too). [EAMxx] add nu-top functionality, fix forcing functor, add ttype9 and dcmip2012 tests 2.1, 2.2, and 3 HOMME: remove obsolete remap algs, option to specify dynamics alg indep of tracer, new sponge layer, add imex tests [ELM] Add topography-based subgrid (topounits), add FATES-ELM Nitro., Phos. and CH4 coupling, add land-use ts for NARRM, add lulc for SSP3 RCP7, Fix nutrient fertilization exp test and carbon isotope flux, Fix xactive lnd dry deposition, add lake water storage option, fix plant hydraulics 2d params, fix carbon budget calc, fix soil nutrient conc. bug, fix mosart dam bug, add test for new ELM, MOSART features, fix bug in O3 dry dep stomatal resistances, fix plant hydraulics restart BFB error, update mkmapdata. [MOSART] fix bug for reading the latitude from an unstructured input file, fix oversat in bubble test. [MPAS-ocean] Add CFC11, CFC12 tracers, add 2D spherical transport tests, fix del4 tracer mixing, add MARBL ocean tracer mixing, modify harmonic analysis options, add GPU port of vmix routines, fix calc of ML-averaged BV freq. [MPAS-seaice] Change extents of initial polar disks for oRRS18to6v3 grid, fix ice BGC with MARBL, update spherical test cases, fix DON coupling, Remove Cf from sea ice constants. [MPAS-landice] add CRYO1850-4xCO2 compset [CIME] add GCP, ANL GCE, Spock, Perlmutter, deprecate config_compilers.xml, fix and clean-up cmake macros, fix slurm bindings, refactor CIME internal testing, cleanup SCORPIO perf data, allow position independent compset naming, [also] update v2 benchmarking suite, extend e3sm_prod with throughput and memory checks

E3SM Project, DOE↗

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

NASA collects, processes and distributes petabytes of Earth Observation (EO) data from satellites, aircraft, in situ instruments and model output, with an order of magnitude increase expected by 2024. Cloud-based web object storage (WOS) of these data can simplify the execution of such an increase. More importantly, it can also facilitate user analysis of those volumes by making the data available to the massively parallel computing power in the cloud. However, storing EO data in cloud WOS has a ripple effect throughout the NASA archive system with unexpected challenges and opportunities. One challenge is modifying data servicing software (such as Web Coverage Service servers) to access and subset data that are no longer on a directly accessible file system, but rather in cloud WOS. Opportunities include refactoring of the archive software to a cloud-native architecture; virtualizing data products by computing on demand; and reorganizing data to be more analysis-friendly.

Lynnes, Christopher↗

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

NASA collects, processes and distributes petabytes of Earth Observation (EO) data from satellites, aircraft, in situ instruments and model output, with an order of magnitude increase expected by 2024. Cloud-based web object storage (WOS) of these data can simplify the execution of such an increase. More importantly, it can also facilitate user analysis of those volumes by making the data available to the massively parallel computing power in the cloud. However, storing EO data in cloud WOS has a ripple effect throughout the NASA archive system with unexpected challenges and opportunities. One challenge is modifying data servicing software (such as Web Coverage Service servers) to access and subset data that are no longer on a directly accessible file system, but rather in cloud WOS. Opportunities include refactoring of the archive software to a cloud-native architecture; virtualizing data products by computing on demand; and reorganizing data to be more analysis-friendly. Reviewed by Mark McInerney ESDIS Deputy Project Manager.

Lynnes, Christopher↗

Tuning the Interpolation Basis in a Multigrid Decomposition for Local Error Control

In the compression of scientific data, error-controlled compressors enable to considerably decrease the size of the dataset while maintaining adequate levels of accuracy. In this paper, we note that multi-level refactoring scheme such as MGARD i) rely on an approximation of the data based on the interpolation of coefficients, ii) estimate the resulting error with global metrics on the dataset. To improve on these two aspects, we propose a method that aims to divide the original dataset into blocks based on their smoothness and refactors each block separately with the most relevant interpolation order. We show the relevance of such a method on tailored datasets and the benefits and challenges when applying it to large scientific data.

Vidal, Nicolas [ORNL]↗

Boosting RDataFrame performance with transparent bulk event processing

RDataFrame is ROOT’s high-level interface for Python and C++ data analysis. Since it first became available, RDataFrame adoption has grown steadily and it is now poised to be a major component of analysis software pipelines for LHC Run 3 and beyond. Thanks to its design inspired by declarative programming principles, RDataFrame enables the development of highperformance, highly parallel analyses without requiring expert knowledge of multi-threading and I/O: user logic is expressed in terms of self-contained, small computation kernels tied together by a high-level API. This design completely decouples analysis logic from its actual execution, and opens several interesting avenues for workflow optimization. In particular, in this work we explore the benefits of moving internal data processing from an event-by-event to a bulkby-bulk loop. This refactoring dramatically reduces the framework’s runtime overheads; in collaboration with the I/O layer it improves data access patterns; it exposes information that optimizing compilers might use to auto-vectorize the invocation of user-defined computations; finally, while existing user-facing interfaces remain unaffected, it becomes possible to additionally offer interfaces that explicitly expose bulks of events, useful e.g. for the injection of GPU kernels into the analysis workflow. In order to inform similar future R&D, design challenges will be presented, as well as an investigation of the relevant timememory trade-off backed by novel performance benchmarks.

Guiraud, Enrico↗

Data Services for Visualization and Analysis - ASC Level II Milestone (7186)

A new in transit Data Service is presented and compared to the traditional file-based workflow and the newly refactored in situ Catalyst workflow. Each workflow is enabled by the IOSS mesh interface equipped with data management layers for Exodus and CGNS (file-based), Catalyst (in situ), and FAODEL (in transit). FAODEL is a distributed object store that can transmit data across MPI allocations. Catalyst is a Para View-based visualization capability developed as part of the CSSE Data Services effort. The workflows considered here take SPARC data into Catalyst for visualization post-processing. Although still in unoptimized form, we show that the in transit approach is a viable alternative to file-based and in situ workflows and offers several advantages to both simulation and post-processing developers. Since IOSS is a mature interface with wide adoption across Sandia and externally, each workflow can be reconfigured to use different simulations that generate mesh data and post-processing tools that consume it.

97 MATHEMATICS AND COMPUTING↗

Introducing Object-Oriented Concepts into GSI

Enhancements are now being made to the Gridpoint Statistical Interpolation (GSI) data assimilation system to expand its capabilities. This effort opens the way for broadening the scope of GSI's applications by using some standard object-oriented features in Fortran, and represents a starting point for the so-called GSI refactoring, as a part of the Joint Effort for Data-assimilationI ntegration (JEDI) project of JCSDA.

GSI↗

Refactoring the elastic–viscous–plastic solver from the sea ice model CICE v6.5.1 for improved performance

This study focuses on the performance of the elastic–viscous–plastic (EVP) dynamical solver within the sea ice model, CICE v6.5.1. The study has been conducted in two steps. First, the standard EVP solver was extracted from CICE for experiments with refactored versions, which are used for performance testing. Second, one refactored version was integrated and tested in the full CICE model to demonstrate that the new algorithms do not significantly impact the physical results. The study reveals two dominant bottlenecks, namely (1) the number of Message Parsing Interface (MPI) and Open Multi-Processing (OpenMP) synchronization points required for halo exchanges during each time step combined with the irregular domain of active sea ice points and (2) the lack of single-instruction, multiple-data (SIMD) code generation. The standard EVP solver has been refactored based on two generic patterns. The first pattern exposes how general finite differences on masked multi-dimensional arrays can be expressed in order to produce significantly better code generation by changing the memory access pattern from random access to direct access. The second pattern takes an alternative approach to handle static grid properties. The measured single-core performance improvement is more than a factor of 5 compared to the standard implementation. The refactored implementation of strong scales on the Intel® Xeon® Scalable Processors series node until the available bandwidth of the node is used. For the Intel® Xeon® CPU Max series, there is sufficient bandwidth to allow the strong scaling to continue for all the cores on the node, resulting in a single-node improvement factor of 35 over the standard implementation. This study also demonstrates improved performance on GPU processors.

58 GEOSCIENCES↗

An NAFP Project: Use of Object Oriented Methodologies and Design Patterns to Refactor Software Design

In the early problem-solution era of software programming, functional decompositions were mainly used to design and implement software solutions. In functional decompositions, functions and data are introduced as two separate entities during the design phase, and are followed as such in the implementation phase. Functional decompositions make use of refactoring through optimizing the algorithms, grouping similar functionalities into common reusable functions, and using abstract representations of data where possible; all these are done during the implementation phase. This paper advocates the usage of object-oriented methodologies and design patterns as the centerpieces of refactoring software solutions. Refactoring software is a method of changing software design while explicitly preserving its external functionalities. The combined usage of object-oriented methodologies and design patterns to refactor should also benefit the overall software life cycle cost with improved software.

Shaykhian, Gholam Ali↗

ISiCLE: A Quantum Chemistry Pipeline for Establishing in Silico Collision Cross Section Libraries

High throughput, comprehensive, and confident identifications of metabolites and other chemicals in biological and envi-ronmental samples will revolutionize our understanding of the role of these intriguing and chemically diverse molecules in biological systems. Despite recent technological advances, a metabolomics study still results in the detection of a dispropor-tionate number of features than cannot be confidently assigned to a chemical structure. This inadequacy is driven by the sin-gle most significant limitation in metabolomics: the reliance on reference libraries constructed through the analysis of au-thentic reference chemicals and the limited commercial availability of these compounds. To this end, we have developed the in silico chemical library engine (ISiCLE), a high-performance computing-friendly cheminformatics workflow for generating chemical properties. In the instantiation described here, we predict probable three-dimensional molecular conformers (i.e. conformational isomers) using chemical identifiers as input, from which collision cross sections (CCS) are derived. The ap-proach employs state-of-the-art first-principles simulation, distinguished by use of molecular dynamics, quantum chemistry, and ion mobility calculations to generate predictions, all without training data. Importantly, optimization of ISiCLE includ-ed a refactoring of the popular MOBCAL code for trajectory-based mobility calculations, improving its computational effi-ciency by over two orders of magnitude. Calculated CCS values were validated against 1,983 experimental CCS values and compared to previously reported CCS calculation approaches. Average CCS calculation absolute error is 3.2% with standard deviation 3.1%. An online database is introduced for sharing both calculated and experimental CCS values (metabolom-ics.pnnl.gov), initially including a CCS library with over 0.8M entries. Finally, three successful applications of molecule characterization using calculated CCS are described, including the identification of an environmental degradation product, the separation of molecular isomers, and the decoding of complex blinded mixtures of exposure chemicals. This work represents a promising method to address the limitations of small molecule identification, and offers alternatives to standards-based chemical identification amenable to high-performance computation.

metabolomics, standards-free, collision cross sect↗

High-performance computing in water resources hydrodynamics

In this work, we present a vision of future water resources hydrodynamics codes that can fully utilize the strengths of modern high-performance computing. The advances to computing power, formerly driven by the improvement of central processing unit processors, now focus on parallel computing and, in particular, the use of graphics processing units (GPUs). However, this shift to a parallel framework requires refactoring the code to make efficient use of the data as well as changing even the nature of the algorithm that solves the system of equations. These concepts along with other features such as the precision for the computations, dry regions management, and input/output data are analyzed in this paper. A 2D multi-GPU flood code applied to a large-scale test case is used to corroborate our statements and ascertain the new challenges for the next-generation parallel water resources codes.

54 ENVIRONMENTAL SCIENCES↗

ISiCLE: A Quantum Chemistry Pipeline for Establishing in Silico Collision Cross Section Libraries

High throughput, comprehensive, and confident identifications of metabolites and other chemicals in biological and envi-ronmental samples will revolutionize our understanding of the role of these intriguing and chemically diverse molecules in biological systems. Despite recent technological advances, a metabolomics study still results in the detection of a dispropor-tionate number of features than cannot be confidently assigned to a chemical structure. This inadequacy is driven by the sin-gle most significant limitation in metabolomics: the reliance on reference libraries constructed through the analysis of au-thentic reference chemicals and the limited commercial availability of these compounds. To this end, we have developed the in silico chemical library engine (ISiCLE), a high-performance computing-friendly cheminformatics workflow for generating chemical properties. In the instantiation described here, we predict probable three-dimensional molecular conformers (i.e. conformational isomers) using chemical identifiers as input, from which collision cross sections (CCS) are derived. The ap-proach employs state-of-the-art first-principles simulation, distinguished by use of molecular dynamics, quantum chemistry, and ion mobility calculations to generate predictions, all without training data. Importantly, optimization of ISiCLE includ-ed a refactoring of the popular MOBCAL code for trajectory-based mobility calculations, improving its computational effi-ciency by over two orders of magnitude. Calculated CCS values were validated against 1,983 experimental CCS values and compared to previously reported CCS calculation approaches. Average CCS calculation absolute error is 3.2% with standard deviation 3.1%. An online database is introduced for sharing both calculated and experimental CCS values (metabolom-ics.pnnl.gov), initially including a CCS library with over 0.8M entries. Finally, three successful applications of molecule characterization using calculated CCS are described, including the identification of an environmental degradation product, the separation of molecular isomers, and the decoding of complex blinded mixtures of exposure chemicals. This work represents a promising method to address the limitations of small molecule identification, and offers alternatives to standards-based chemical identification amenable to high-performance computation.

Colby, Sean M↗

Comprehensive Measurement and Analysis of the User-Perceived I/O Performance in a Production Leadership-Class Storage System

With the increase of the scale and intensity of the parallel I/O workloads generated by those scientific applications running on high performance computing facilities, understanding the I/O dynamics, especially the root cause of the I/O performance variability and degradation in HPC environment, have become extremely critical to the HPC community. In this paper, we run extensive I/O measuring tests on a production leadership-class storage system to capture the performance variabilities of large-scale parallel I/O. Analyzing these results and its statistic correlation revealed some valuable insights into the characteristics of the storage system and the root cause of I/O performance variability. Further, we leverage these findings and propose an I/O middleware design refactoring which can improve the performance of the parallel I/O by optimizing the data striping and placement. Our preliminary evaluation results demonstrate the proposed approach can reduce the average per-process write latency by at least 80% and the maximum per-process write latency by at least 20%.

Wan, Lipeng↗

Python for Development of OpenMP and CUDA Kernels for Multidimensional Data

Design of data structures for high performance computing (HPC) is one of the principal challenges facing researchers looking to utilize heterogeneous computing machinery. Heterogeneous systems derive cost, power, and speed efficiency by being composed of the appropriate hardware for the task. Yet, each type of processor requires a specific organization of the application state in order to achieve peak performance. Discovering this and refactoring the code can be a challenging and time-consuming task for the researcher, as the data structures and the computational model must be co-designed. We present a methodology that uses Python as the environment for which to explore tradeoffs in both the data structure design as well as the code executing on the computation accelerator. Our method enables multi-dimensional arrays to be used effectively in any target environment. We have chosen to focus on OpenMP and CUDA environments, thus exploring the development of optimized kernels for the two most common classes of computing hardware available today: multi-core CPU and GPU. Python s large palette of file and network access routines, its associative indexing syntax and support for common HPC environments makes it relevant for diverse hardware ranging from laptops through computing clusters to the highest performance supercomputers. Our work enables researchers to accelerate the development of their codes on the computing hardware of their choice.

43 PARTICLE ACCELERATORS↗

Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities

We present algorithmic improvements to the loading operations of certain reduced data ensembles produced from neutron scattering experiments at Oak Ridge National Laboratory (ORNL) facilities. Ensembles from multiple measurements are required to cover a wide range of the phase space of a sample material of interest. They are stored using the standard NeXus schema on individual HDF5 files. This makes it a scalability challenge, as the number of experiments stored increases in a single ensemble file. The present work follows up on our previous efforts on data management algorithms, to address identified input output (I/O) bottlenecks in Mantid, an open-source data analysis framework used across several neutron science facilities around the world. We reuse an in-memory binary-tree metadata index that resembles data access patterns, to provide a scalable search and extraction mechanism. In addition, several memory operations are refactored and optimized for the current common use cases, ranging most frequently from 10 to 180, and up to 360 separate measurement configurations. Results from this work show consistent speed ups in wall-clock time on the Mantid LoadMD routine, ranging from 19% to 23% on average, on ORNL production computing systems. The latter depends on the complexity of the targeted instrument-specific data and the system I/O and compute variability for the shared computational resources available to users of ORNL’s Spallation Neutron Source (SNS) and the High Flux Isotope Reactor (HFIR) instruments. Nevertheless, we continue to highlight the need for more research to address reduction challenges as experimental data volumes, user time and processing costs increase.

Godoy, William↗

Kelpie: FY2014 Project Update

The ASC CSSE project Kelpie is a research and development project focused on developing a distributed, in-memory data management system that can be leveraged in a number of high-performance computing (HPC) applications. After FY13s demonstration that a key/value data store could be implemented on top of the Nessie RDMA/RPC library, we began refactoring Kelpie in FY14 in order to make it more usable by other research teams that need it for upcoming milestones. This report provides a summary of the different efforts in FY14 that took place to make Kelpie a more usable system.

97 MATHEMATICS AND COMPUTING↗

Recent progress with the top to bottom approach to vectorization in GeantV

SIMD acceleration can potentially boost by factors the application throughput. Achieving efficient SIMD vectorization for scalar code with complex data flow and branching logic, goes however way beyond breaking some loop dependencies and relying on the compiler. Since the refactoring effort scales with the number of lines of code, it is important to understand what kind of performance gains can be expected in such complex cases. We started to investigate a couple of years ago a top to bottom vectorization approach to particle transport simulation. Percolating vector data to algorithms was mandatory since not all the components can internally vectorize. Vectorizing low-level algorithms is certainly necessary, but not sufficient to achieve relevant SIMD gains. In addition, the overheads for maintaining the concurrent vector data flow and copy data have to be minimized. In the context of a vectorization R&D for simulation we developed a framework to allow different categories of scalar and vectorized components to co-exist, dealing with data flow management and real-time heuristic optimizations. The paper describes our approach on coordinating SIMD vectorization at framework level, making a detailed quantitative analysis of the SIMD gain versus overheads, with a breakdown by components in terms of geometry, physics and magnetic field propagation. We also present the more general context of this R&D work and goals for 2018.

97 MATHEMATICS AND COMPUTING↗

building-physics/refrigeration-system

The existing refrigeration system for OpenStudio was originally designed for a prototype supermarket model. To support the development of new modular space types, it is crucial to refactor the system. This process includes refining the design by utilizing the manufacturer's existing data, along with incorporating the latest data from manufacturers, to enhance the system’s adaptability and functionality. This repository currently contains the refrigeration system database and Python-based automation for generating OpenStudio-compatible refrigeration JSON files.

Bae, Yeonjin [Oak Ridge National Laboratory (ORNL)↗