Search NASA⌕ Search

SEARCH · Search NASA

Results for “high performance and distributed computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

Counting Kmers for Biological Sequences at Large Scale

Counting the abundance of all the distinct kmers in biological sequence data is a fundamental step in bioinformatics. These applications include de novo genome assembly, error correction, etc. With the development of sequencing technology, the sequence data in a single project can reach Petabyte-scale or Terabyte-scale nucleotides. Counting demand for the abundance of these sequencing data is beyond the memory and computing capacity of single computing node, and how to process it efficiently is a challenge on a high-performance computing cluster. As such, we propose SWAPCounter, a highly scalable distributed approach for kmer counting. This approach is embedded with an MPI streaming I/O module for loading huge data set at high speed, and a counting bloom filter module for both memory and communication efficiency. By overlapping all the counting steps, SWAPCounter achieves high scalability with high parallel efficiency. In this work, the experimental results indicate that SWAPCounter has competitive performance with two other tools on shared memory environment, KMC 2 , and MSPKmerCounter. Moreover, SWAPCounter also shows the highest scalability under strong scaling experiments. In our experiment on Cetus supercomputer, SWAPCounter scales to 32,768 cores with 79% parallel efficiency (using 2048 cores as baseline) when processing 4 TB sequence data of 1000 Genomes. The source code of SWAPCounter is publicly available at https://github.com/mengjintao/SWAPCounter.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Regen: An object layout regenerator on large-scale production HPC systems

This article proposes an object layout regenerator called Regen which regenerates and removes the object layout dynamically to improve the read performance of applications. Regen first detects frequent access patterns from the I/O requests of the applications. Second, Regen reorganizes the objects and regenerates or preallocates new object layouts according to the identified access patterns. Finally, Regen removes or reuses the obsolete or regenerated object layouts as necessary. As a result, Regen accelerates access to objects by providing a flexible object layout. We implement Regen as a framework on top of Proactive Data Container (PDC) and evaluate it on Cori supercomputer, a production-scale HPC system, by using realistic HPC I/O benchmarks. The experimental results show that Regen improves the I/O performance by up to 16.92 × compared with an existing system.

Distributed file system↗
DOE OSTI2023

MILK : a Python scripting interface to MAUD for automation of Rietveld analysis

Modern diffraction experiments ( e.g. in situ parametric studies) present scientists with many diffraction patterns to analyze. Interactive analyses via graphical user interfaces tend to slow down obtaining quantitative results such as lattice parameters and phase fractions. Furthermore, Rietveld refinement strategies ( i.e. the parameter turn-on-off sequences) tend to be instrument specific or even specific to a given dataset, such that selection of strategies can become a bottleneck for efficient data analysis. Managing multi-histogram datasets such as from multi-bank neutron diffractometers or caked 2D synchrotron data presents additional challenges due to the large number of histogram-specific parameters. To overcome these challenges in the Rietveld software Material Analysis Using Diffraction ( MAUD ), the MAUD Interface Language Kit ( MILK ) is developed along with an updated text batch interface for MAUD . The open-source software MILK is computer-platform independent and is packaged as a Python library that interfaces with MAUD . Using MILK , model selection ( e.g. various texture or peak-broadening models), Rietveld parameter manipulation and distributed parallel batch computing can be performed through a high-level Python interface. A high-level interface enables analysis workflows to be easily programmed, shared and applied to large datasets, and external tools to be integrated with MAUD . Through modification to the MAUD batch interface, plot and data exports have been improved. The resulting hierarchical folders from Rietveld refinements with MILK are compatible with Cinema: Debye–Scherrer , a tool for visualizing and inspecting the results of multi-parameter analyses of large quantities of diffraction data. In this manuscript, the combined Python scripting and visualization capability of MILK is demonstrated with a quantitative texture and phase analysis of data collected at the HIPPO neutron diffractometer.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Virtual Time III, Part 2: Combining Conservative and Optimistic Synchronization

This is Part 2 of a trio of works intended to provide a unifying framework in which conservative and optimistic synchronization for parallel discrete event simulations can be freely and transparently combined in the same logical process on an event-by-event basis. Here, in this article, we continue the outline of an approach called Unified Virtual Time (UVT) that was introduced in Part 1, showing in detail via two extended examples how conservative synchronization can be refactored and combined with optimistic synchronization in the UVT framework. We describe UVT versions of both a basic time windowing algorithm called Unified Simple Time Windows and a refactored version of the Chandy-Misra-Bryant Null Message algorithm called Unified CMB.

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

Impact of Emerging Computing Architectures and Opportunities for Process Systems Engineering Applications

Moore’s “law” was the observation that the number of transistors in an integrated circuit doubled approximately every two years. This trend has distinctly failed to hold in recent years. The death of Moore’s law has left researchers and practitioners in the computational sciences searching for technologies to provide the speedups formerly supported by Moore’s law. Previously overlooked chip architectures and other computing technologies are now receiving more development resources. Critically, these technologies are gaining more mature software support, opening their adoption by researchers in algorithms and applications. In this article, we review some of these computing technologies, their relationship with various algorithms and applications, and their potential benefits (or pitfalls). We close with recommendations for future work by the process systems engineering community specifically.

Emerging hardware↗
NASA NTRS2004

QuakeSim and the Solid Earth Research Virtual Observatory

We are developing simulation and analysis tools in order to develop a solid Earth science framework for understanding and studying active tectonic and earthquake processes. The goal of QuakeSim and its extension, the Solid Earth Research Virtual Observatory (SERVO), is to study the physics of earthquakes using state-of-the-art modeling, data manipulation, and pattern recognition technologies. We are developing clearly defined accessible data formats and code protocols as inputs to simulations, which are adapted to high-performance computers. The solid Earth system is extremely complex and nonlinear resulting in computationally intensive problems with millions of unknowns. With these tools it will be possible to construct the more complex models and simulations necessary to develop hazard assessment systems critical for reducing future losses from major earthquakes. We are using Web (Grid) service technology to demonstrate the assimilation of multiple distributed data sources (a typical data grid problem) into a major parallel high-performance computing earthquake forecasting code. Such a linkage of Geoinformatics with Geocomplexity demonstrates the value of the Solid Earth Research Virtual Observatory (SERVO) Grid concept, and advances Grid technology by building the first real-time large-scale data assimilation grid.

virtual observatory↗
DOE OSTI2022

Prototype Integrated Hydrogen Fuel Cell Powered Data Center (Cooperative Research and Development Final Report)

Data centers are the backbone of the modern economy-from the server rooms that power small- to medium-sized organizations, to the enterprise-class data centers that support American corporations, and the server farms that run cloud computing services. Rather than a traditional grid-centric data center approach, work under the CRADA will be directed towards implementing a proof-of-concept "hydrogen-based" data center utilizing hydrogen fuel cells, electrolyzers, and solar photovoltaics as key building blocks. This integrated proof-of-concept builds on preliminary work done by HPE and its partners Mercedes-Benz Research & Development North America, Inc. (MBRDNA) and Power Innovations, to dramatically simplify the electrical infrastructure in the data center, leverage the latest automotive-scale hydrogen fuel cell technologies, and tightly couple renewably generated DC power directly to the Information Technology (IT). While some limited aspects of this concept have been demonstrated previously the entire integrated proof of concept proposed here has not yet been realized.

08 HYDROGEN↗
NASA NTRS2004

Complexity Computational Environment: Data Assimilation SERVOGrid

We are using Web (Grid) service technology to demonstrate the assimilation of multiple distributed data sources (a typical data grid problem) into a major parallel high-performance computing earthquake forecasting code. Such a linkage of Geoinformatics with Geocomplexity demonstrates the value of the Solid Earth Research Virtual Observatory (SERVO) Grid concept, and advance Grid technology by building the first real-time large-scale data assimilation grid Here we develop the next steps for both the SERVO concept and the identified need for a Solid Earth problem-solving environment. We use a challenging motivating problem of importance to NASA namely integrating NASA space geodetic observations with numerical simulations of a changing earth.

data assimiliation↗
NASA NTRS1996

Cooperative high-performance storage in the accelerated strategic computing initiative

The use and acceptance of new high-performance, parallel computing platforms will be impeded by the absence of an infrastructure capable of supporting orders-of-magnitude improvement in hierarchical storage and high-speed I/O (Input/Output). The distribution of these high-performance platforms and supporting infrastructures across a wide-area network further compounds this problem. We describe an architectural design and phased implementation plan for a distributed, Cooperative Storage Environment (CSE) to achieve the necessary performance, user transparency, site autonomy, communication, and security features needed to support the Accelerated Strategic Computing Initiative (ASCI). ASCI is a Department of Energy (DOE) program attempting to apply terascale platforms and Problem-Solving Environments (PSEs) toward real-world computational modeling and simulation problems. The ASCI mission must be carried out through a unified, multilaboratory effort, and will require highly secure, efficient access to vast amounts of data. The CSE provides a logically simple, geographically distributed, storage infrastructure of semi-autonomous cooperating sites to meet the strategic ASCI PSE goal of highperformance data storage and access at the user desktop.

Gary, Mark↗
NASA NTRS2022

Computational Analysis of the X-57 Maxwell Airplane at Unpowered Conditions (Preliminary Fuselage)

The X-57 Maxwell is an all-electric airplane that implements a distributed electric propulsion system to demonstrate that high-efficiency electric propulsion can be integrated with aerodynamics to increase the performance of an airplane. To this end, distributed electric fans were installed on the wing to provide increased flow over the wing at the low takeoff and landing speeds of the X-57. The low-speed lift augmentation allows for a reduction in wing area for cruise optimization. The X-57 wing area was reduced to 42 percent of the wing area of the baseline aircraft, a Tecnam P2006T. With this reduced wing area and the electric propulsion system, it is estimated that the X-57 will cruise on less than one-third the total energy compared to the baseline aircraft. To meet the cruise performance goal at a Mach number of 0.233 at an altitude of 8000 feet, the X-57 has a cruise lift coefficient of 0.7516 and needs to have a cruise drag coefficient of 0.05423 or less. The USM3D computational solver was used to investigate the X-57 performance, without the distributed electric propulsion high-lift system operating. The unpowered X-57 performance is of interest to quantify if the X-57 can meet the cruise drag performance goal, and to document the lift performance of the very small wing at takeoff and landing conditions. The primary configurations investigated in this paper include the cruise configuration with no flap deflection, a takeoff configuration with a 10◦ flap deflection, and a landing configuration with a 30◦ flap deflection. The conditions for the cruise configuration were a flight unit Reynolds number of 1.32E+06 per foot, an altitude of 8000 feet, a Mach number of 0.233, and angles of attack from −2° to 24° . At the cruise lift coefficient of 0.7516, the computed drag coefficient is 0.05275. This computed drag is less than the drag coefficient of 0.05423 that is required to meet the X-57 airplane performance goal. However, the computational airplane is a completely smooth geometry and does not account for protuberance drag, nor the drag from steps and gaps in the actual X-57 airplane. Therefore, based upon the CFD drag calculation there is a 10-percent margin to account for some of the differences between the as-built metal fuselage and empennage construction, and the smooth computational geometry. The computed cruise drag also does not account for an induced drag reduction due to the wing-tip propellers and a drag reduction due to laminar flow achieved on the wing. The computed lift to drag ratio is 14.14 at the cruise lift coefficient of 0.7516, and the maximum computed lift to drag ratio is 15.8. The maximum lift coefficient for the cruise configuration was 2.13 at an angle of attack of 15°. The conditions for the takeoff configuration with a 10° flap deflection were a flight unit Reynolds number of 0.986E+06 per foot, an altitude of 2500 feet, a Mach number of 0.149, and angles of attack from −2° to 22°. The maximum lift coefficient for the takeoff configuration was 2.21 at an angle of attack of 16°. The conditions for the landing configuration with a 30° flap deflection were a flight unit Reynolds number of 0.922E+06 per foot, an altitude of 2500 feet, a Mach number of 0.139, and angles of attack from −2° to 24°. The maximum lift coefficient for the landing configuration was 2.58 and occurred at two angles of attack, 10° and 14°. Based on the unpowered maximum lift coefficient of 2.58 for the 30° flap deflection, along with computations of the distributed electric propulsion lift augmentation (not shown in this paper), the X-57 Maxwell is estimated to meet its powered landing goal of a maximum lift coefficient of 4.0.

X-57 Maxwell Airplane↗
NASA NTRS2024

From Blood Platelets Classification to Earth System Digital Twins: A Career in Computer Vision

IEEE SIgnal Processing Society (SPS) and Women in Signal Processing (WiSP) Webinar (https://signalprocessingsociety.org/blog/webinar-wisp-blood-platelets-classification-earth-system-digital-twins-career-computer-vision): Join us for an interactive session with Dr. Jacqueline Le Moigne, Manager of the NASA Science Mission Directorate Earth Science Technology Office’s Advanced Information Systems Technology Program. In this webinar, Dr. Le Moigne will share her journey through academia, the private sector, and her pivotal roles at NASA, emphasizing her work in signal processing, computer vision, and related technologies. The webinar will conclude with an interactive Q&A session, providing attendees the opportunity to engage directly with Dr. Le Moigne about her experiences and insights. Dr. Jacqueline Le Moigne manages NASA's Advanced Information Systems Technology Program, focusing on novel technologies that support Earth Science missions. She previously held roles at NASA Goddard, including Assistant Chief for Technology, and has worked on NASA's Space Technology Research Grants Program and Technology Roadmaps. Before NASA, she was a Research Scientist at the University of Maryland's Computer Vision Laboratory and also gained experience in the private sector. Jacqueline earned her Ph.D. in Computer Science from the University Pierre and Marie Curie in Paris. Her research interests include Image Registration, Computer Vision, Artificial Intelligence, Autonomous Systems, Distributed Spacecraft Missions (DSM), and Earth System Digital Twins (ESDT), as well as high-performance and onboard computing. She has authored over 180 publications, including books and patents, and has received multiple awards, including the NASA Exceptional Service Medal and the Goddard Information Science and Technology Award in 2012.

Computer Vision; Image Processing; Earth Science R↗
DOE OSTI2019

ATLAS Data Carousel

The ATLAS experiment at CERN's LHC stores detector and simulation data in raw and derived data formats across more than 150 Grid sites world-wide, currently in total about 200PB on disk and 250PB on tape. Data have different access characteristics due to various computational workflows, and can be accessed from different media, such as remote I/O, disk cache on hard disk drives or SSDs. Also, larger data centers provide the majority of offline storage capability via tape systems. For the High-Luminosity LHC (HL-LHC), the estimated data storage requirements are several factors bigger than the present forecast of available resources, based on a flat budget assumption. On the computing side, ATLAS Distributed Computing was very successful in the last years with high performance and high throughput computing integration and in using opportunistic computing resources for the Monte Carlo simulation. On the other hand, equivalent opportunistic storage does not exist. ATLAS started the "Data Carousel" project to increase the usage of less expensive storage, i.e. tapes or even commercial storage, so it is not limited to tape technologies exclusively. Data Carousel orchestrates data processing between workload management, data management, and storage services with the bulk data resident on offline storage. The processing is executed by staging and promptly processing a sliding window of inputs onto faster buffer storage, such that only a small percentage of input data are available at any one time. With this project, we aim to demonstrate that this is the natural way to dramatically reduce our storage cost. The first phase of the project was started in the fall of 2018 and was related to I/O tests of the sites archiving systems. Phase II now requires a tight integration of the workload and data management systems. Additionally, the Data Carousel studies the feasibility to run multiple computing workflows from tape. The project is progressing very well and the results presented in this document will be used before the LHC Run 3.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2017

Position Paper - pFLogger: The Parallel Fortran Logging framework for HPC Applications

In the context of high performance computing (HPC), software investments in support of text-based diagnostics, which monitor a running application, are typically limited compared to those for other types of IO. Examples of such diagnostics include reiteration of configuration parameters, progress indicators, simple metrics (e.g., mass conservation, convergence of solvers, etc.), and timers. To some degree, this difference in priority is justifiable as other forms of output are the primary products of a scientific model and, due to their large data volume, much more likely to be a significant performance concern. In contrast, text-based diagnostic content is generally not shared beyond the individual or group running an application and is most often used to troubleshoot when something goes wrong. We suggest that a more systematic approach enabled by a logging facility (or logger) similar to those routinely used by many communities would provide significant value to complex scientific applications. In the context of high-performance computing, an appropriate logger would provide specialized support for distributed and shared-memory parallelism and have low performance overhead. In this paper, we present our prototype implementation of pFlogger a parallel Fortran-based logging framework, and assess its suitability for use in a complex scientific application.

Fortran↗
NASA NTRS2017

POSITION PAPER - pFLogger: The Parallel Fortran Logging Framework for HPC Applications

In the context of high performance computing (HPC), software investments in support of text-based diagnostics, which monitor a running application, are typically limited compared to those for other types of IO. Examples of such diagnostics include reiteration of configuration parameters, progress indicators, simple metrics (e.g., mass conservation, convergence of solvers, etc.), and timers. To some degree, this difference in priority is justifiable as other forms of output are the primary products of a scientific model and, due to their large data volume, much more likely to be a significant performance concern. In contrast, text-based diagnostic content is generally not shared beyond the individual or group running an application and is most often used to troubleshoot when something goes wrong. We suggest that a more systematic approach enabled by a logging facility (or 'logger') similar to those routinely used by many communities would provide significant value to complex scientific applications. In the context of high-performance computing, an appropriate logger would provide specialized support for distributed and shared-memory parallelism and have low performance overhead. In this paper, we present our prototype implementation of pFlogger - a parallel Fortran-based logging framework, and assess its suitability for use in a complex scientific application.

Clune, Thomas L.↗
NASA NTRS2017

pFlogger: The Parallel Fortran Logging Utility

In the context of high performance computing (HPC), software investments in support of text-based diagnostics, which monitor a running application, are typically limited compared to those for other types of IO. Examples of such diagnostics include reiteration of configuration parameters, progress indicators, simple metrics (e.g., mass conservation, convergence of solvers, etc.), and timers. To some degree, this difference in priority is justifiable as other forms of output are the primary products of a scientific model and, due to their large data volume, much more likely to be a significant performance concern. In contrast, text-based diagnostic content is generally not shared beyond the individual or group running an application and is most often used to troubleshoot when something goes wrong. We suggest that a more systematic approach enabled by a logging facility (or 'logger)' similar to those routinely used by many communities would provide significant value to complex scientific applications. In the context of high-performance computing, an appropriate logger would provide specialized support for distributed and shared-memory parallelism and have low performance overhead. In this paper, we present our prototype implementation of pFlogger - a parallel Fortran-based logging framework, and assess its suitability for use in a complex scientific application.

Clune, Tom↗
NASA NTRSDate not supplied

The Future of NASA Earth Science in the Commercial Cloud: Challenges and Opportunities

NASA produces a large volume and variety of data products that are used every day to support research, decision making, and education. The widespread use of NASA’s Earth Science data is enabled by NASA’s Earth Science Data System (ESDS) program, which oversees the archiving and distribution of these data and invests in the development of new data systems and tools. However, NASA’s current approach to Earth Science data distribution — based on distributed institutional archives with individual on-premises high-performance computing capabilities — faces some significant challenges, including massive increases in data volume from upcoming missions, a greater need for transdisciplinary science that synthesizes many different kinds of observations, and a push to make science more open, inclusive, and accessible. To address these challenges, NASA is aggressively migrating its Earth Science data and related tools and services into the commercial cloud. Migration of data into the commercial cloud can significantly improve NASA’s existing data system capabilities by (1) providing more flexible options for storage and compute (including rapid, as-needed access to state-of-the-art capabilities); (2) by centralizing and standardizing data access, which gives all of NASA’s institutional data centers access to all of each other’s datasets; and (3) by facilitating “analysis-in-place”, whereby users can bring their own computational workflows and tools to the data rather than having to maintain their own copies of NASA datasets. However, migration to the commercial cloud also poses some significant challenges, including (1) managing costs under a “pay-as-you-go” model; (2) incompatibility with existing tools and data formats with object-based storage and network access; (3) vendor lock-in; (4) challenges with data access for workflows that mix on-premise and cloud computing; and (5) standardization for highly diverse data as is present in NASA’s data archive. I conclude with two examples of recent NASA activities showcasing capabilities enabled by the commercial cloud: An interactive analysis and development platform for analyzing airborne imaging spectroscopy data, and a new collection of tools and services for data discovery, analysis, publication, and data-driven storytelling (Visualization, Exploration, and Data Analysis, VEDA).

Alexey N Shiklomanov↗
NASA NTRS1996

Performance Evaluation of Communication Software Systems for Distributed Computing

In recent years there has been an increasing interest in object-oriented distributed computing since it is better quipped to deal with complex systems while providing extensibility, maintainability, and reusability. At the same time, several new high-speed network technologies have emerged for local and wide area networks. However, the performance of networking software is not improving as fast as the networking hardware and the workstation microprocessors. This paper gives an overview and evaluates the performance of the Common Object Request Broker Architecture (CORBA) standard in a distributed computing environment at NASA Ames Research Center. The environment consists of two testbeds of SGI workstations connected by four networks: Ethernet, FDDI, HiPPI, and ATM. The performance results for three communication software systems are presented, analyzed and compared. These systems are: BSD socket programming interface, IONA's Orbix, an implementation of the CORBA specification, and the PVM message passing library. The results show that high-level communication interfaces, such as CORBA and PVM, can achieve reasonable performance under certain conditions.

Fatoohi, Rod↗
DOE OSTI2020

Massively Parallel Quantum Chemistry: A high-performance research platform for electronic structure

The Massively Parallel Quantum Chemistry (MPQC) program is a 30-year-old project that enables facile development of electronic structure methods for molecules for efficient deployment to massively parallel computing architectures. Here, we describe the historical evolution of MPQC’s design into its latest (fourth) version, the capabilities and modular architecture of today’s MPQC, and how MPQC facilitates rapid composition of new methods as well as its state-of-the-art performance on a variety of commodity and high-end distributed-memory computer platforms.

Peng, Chong↗