Search NASA⌕ Search

SEARCH · Search NASA

Results for “data intensive computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26

Analysis of low energy electrons

Simultaneous observations of low energy electrons in the plasma sheet and in the auroral zone were analyzed. Data from the MIT plasma experiment on the OGO-3 satellite and from the Lockheed experiment on the OV1-18 satellite were processed and compared. The OV1-18 carried thirteen magnetic electron spectrometers designed to measure the intensity, angular, and energy distributions of the auroral electrons and protons in the energy range below 50 keV. Two computer programs were developed for reduction of the OV1-18 data. One program computed the various plasma properties at one second intervals as a function of Universal Time and pitch angle; the other program produced survey plots showing the outputs of the various detectors on the satellite as a function of time on a scale of approximately 100 seconds per cm. The OV1-18 data exhibit the high degree of variability associated with substorm controlled phenomena.

Sharp, R. D.↗

Building near-real-time processing pipelines with the spark-MPI platform

Advances in detectors and computational technologies provide new opportunities for applied research and the fundamental sciences. Concurrently, dramatic increases in the three V's (Volume, Velocity, and Variety) of experimental data and the scale of computational tasks produced the demand for new real-time processing systems at experimental facilities. Recently, this demand was addressed by the Spark-MPI approach connecting the Spark data-intensive platform with the MPI high-performance framework. In contrast with existing data management and analytics systems, Spark introduced a new middleware based on resilient distributed datasets (RDDs), which decoupled various data sources from high-level processing algorithms. The RDD middleware significantly advanced the scope of data-intensive applications, spreading from SQL queries to machine learning to graph processing. Spark-MPI further extended the Spark ecosystem with the MPI applications using the Process Management Interface. The paper explores this integrated platform within the context of online ptychographic and tomographic reconstruction pipelines.

97 MATHEMATICS AND COMPUTING↗

Geodata Modeling and Query in Geographic Information Systems

Geographic information systems (GIS) deal with collecting, modeling, man- aging, analyzing, and integrating spatial (locational) and non-spatial (attribute) data required for geographic applications. Examples of spatial data are digital maps, administrative boundaries, road networks, and those of non-spatial data are census counts, land elevations and soil characteristics. GIS shares common areas with a number of other disciplines such as computer- aided design, computer cartography, database management, and remote sensing. None of these disciplines however, can by themselves fully meet the requirements of a GIS application. Examples of such requirements include: the ability to use locational data to produce high quality plots, perform complex operations such as network analysis, enable spatial searching and overlay operations, support spatial analysis and modeling, and provide data management functions such as efficient storage, retrieval, and modification of large datasets; independence, integrity, and security of data; and concurrent access to multiple users. It is on the data management issues that we devote our discussions in this monograph. Traditionally, database management technology have been developed for business applications. Such applications require, among other things, capturing the data requirements of high-level business functions and developing machine- level implementations; supporting multiple views of data and yet providing integration that would minimize redundancy and maintain data integrity and security; providing a high-level language for data definition and manipulation; allowing concurrent access to multiple users; and processing user transactions in an efficient manner. The demands on database management systems have been for speed, reliability, efficiency, cost effectiveness, and user-friendliness. Significant progress have been made in all of these areas over the last two decades to the point that many generalized database platforms are now available for developing data intensive applications that run in real-time. While continuous improvement is still being made at a very fast-paced and competitive rate, new application areas such as computer aided design, image processing, VLSI design, and GIS have been identified by many as the next generation of database applications. These new application areas pose serious challenges to the currently available database technology. At the core of these challenges is the nature of data that is manipulated. In traditional database applications, the database objects do not have any spatial dimension, and as such, can be thought of as point data in a multi-dimensional space. For example, each instance of an entity EMPLOYEE will have a unique value corresponding to every attribute such as employee id, employee name, employee address and so on. Thus, every Employee instance can be thought of as a point in a multi-dimensional space where each dimension is represented by an attribute. Furthermore, all operations on such data are one-dimensional. Thus, users may retrieve all entities satisfying one or more constraints. Examples of such constraints include employees with addresses in a certain area code, or salaries within a certain range. Even though constraints can be specified on multiple attributes (dimensions), the search for such data is essentially orthogonal across these dimensions.

Adam, Nabil↗

Compiler-Directed Transformation for Higher-Order Stencils

As the cost of data movement increasingly dominates performance, developers of finite-volume and finite-difference solutions for partial differential equations (PDEs) are exploring novel higher-order stencils that increase numerical accuracy and computational intensity. This paper describes a new compiler reordering transformation applied to stencil operators that performs partial sums in buffers, and reuses the partial sums in computing multiple results. This optimization has multiple effect son improving stencil performance that are particularly important to higher-order stencils: exploits data reuse, reduces floating-point operations, and exposes efficient SIMD parallelism to backend compilers. We study the benefit of this optimization in the context of Geometric Multigrid (GMG), a widely used method to solvePDEs, using four different Jacobi smoothers built from 7-, 13-, 27-and 125-point stencils. We quantify performance, speedup, andnumerical accuracy, and use the Roofline model to qualify our results. Ultimately, we obtain over 4× speedup on the smoothers themselves and up to a 3× speedup on the multigrid solver. Finally, we demonstrate that high-order multigrid solvers have the potential of reducing total data movement and energy by several orders of magnitude.

97 MATHEMATICS AND COMPUTING↗

The Data Mine model for accessible partnerships in data science

Abstract The Data Mine at Purdue University is a pioneering experiential learning community for undergraduate and graduate students of any background to learn data science. The first data‐intensive experience embedded in a large learning community, The Data Mine had nearly 1300 students in academic year (AY) 2022–2023 and nearly 1700 students for AY 2023–2024. The Data Mine embodies data‐infused education, research, and collaboration. Students learn Python, R, SQL, and shell‐scripting, while working on weekly projects within a high‐performance computing (HPC) cluster. In the Corporate Partners cohort, students work on teams of 5–15 students, led by a paid student team leader. Each cohort follows an Agile approach, working on data‐intensive projects provided by industry partners and mentored by company employees. Students develop professional and data skills throughout the academic year, from August through April. Many students return in subsequent years to the program, increasing their tenure with a Corporate Partner. Student teams are inherently interdisciplinary; students from 133 different majors are involved in the program, ranging from new incoming students through PhD level students. These interdisciplinary teams of students bring new perspectives to challenging problems in which data science is a key part of the solution. The interdisciplinary teams foster an environment of synthesis with ideas and solutions. Students come together with different life experiences, different levels of technical skill, but also varying ways they navigate paths to solutions because of the variety of majors represented, resulting in a more creative and robust solution than a traditional data science program. This article is categorized under: Applications of Computational Statistics > Education in Computational Statistics

Betz, Margaret A.↗

Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON

For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organisations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analysed by researchers, or archived for safe long-term storage. To fulfil these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS at LHC to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown, many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities that arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments Belle II and CMS, the neutrino experiment DUNE, the scattering radar experiment EISCAT3D, the gravitational wave observatories LIGO and VIRGO, the SKA radio telescope, and the dark matter search experiment XENON.

Lassnig, Mario↗

Surface plasmon coupled emission studies on engineered thin film hybrids of nano α−Al{sub 2}O{sub 3} on silver

We report the first time engineering and fabrication of a novel thin film hybrid of nano α-alumina doped in a polyvinyl alcohol (PVA) matrix along with rhodamine b (Rh.B) on a silver thin film. Silver films of 50 nm thickness on glass slides were fabricated by thermal evaporation. Nano α-alumina was synthesized through the combustion route and characterized by XRD. The α-alumina was dispersed in the PVA-Rh.B matrix by tip sonication. The resultant solution was spin coated on the Ag thin film at 3000 rpm to generate an overcoat of ∼30 nm. We have designed and constructed an opto-mechanical setup for performing the SPCE studies. Excitation with a 532 nm continuous laser, led to the coupling of the energy of Rh.B emission to the surface plasmon modes of silver. The emission @ 580 nm was recorded using an Ocean Optics(copyright, serif) fiber optic spectrometer. Calculation of the ratio of signal intensity between the directional SPCE and isotropic fluorescence gives us the factor of signal enhancements which SPCE offers. We report an '8 fold' signal enhancement attributed to SPCE arising from the metal oxide doped thin film hybrid. We observed only a '5 fold' signal enhancement in the case of a thin film hybrid without α-alumina. The emission was also 92% P-polarized which is in coherence with the theory of SPCE. The greater degree of signal enhancement observed in the α-alumina doped thin film substrate can be attributed to the surface roughness which alumina offers to silver, which along with the porous nature of alumina enables a greater degree of adsorption of Rh.B which results in a higher emission intensity. Computational modeling was also performed, based on surface plasmon resonance (SPR) calculations to provide theoretical background to observed experimental data. The α-alumina thin film hybrid can be extended as an economical sensing platform towards the high sensitive detection of analytes.

36 MATERIALS SCIENCE↗

The 'last mile' of data handling: Fermilab's IFDH tools

IFDH (Intensity Frontier Data Handling), is a suite of tools for data movement tasks for Fermilab experiments and is an important part of the FIFE[2] (Fabric for Intensity Frontier [1] Experiments) initiative described at this conference. IFDH encompasses moving input data from caches or storage elements to compute nodes (the 'last mile' of data movement) and moving output data potentially to those caches as part of the journey back to the user. IFDH also involves throttling and locking to ensure that large numbers of jobs do not cause data movement bottlenecks. IFDH is realized as an easy to use layer that users call in their job scripts (e.g. 'ifdh cp'), hiding the low level data movement tools. One advantage of this layer is that the underlying low level tools can be selected or changed without the need for the user to alter their scripts. Logging and performance monitoring can also be added easily. This system will be presented in detail as well as its impact on the ease of data handling at Fermilab experiments.

Lyon, Adam L.↗

Electron-Hole Theory of the Effect of Quantum Nuclei on the X-Ray Absorption Spectra of Liquid Water

Electron-hole excitation theory is used to unveil the role of nuclear quantum effects on the x-ray absorption spectral signatures of water, whose structure is computed via path-integral molecular dynamics with the MB-pol intermolecular potential model. Compared to spectra generated from the classically modeled water, quantum nuclei introduce important effects on the spectra in terms of both the energies and line shapes. Fluctuations due to delocalized protons influence the short-range ordering of the hydrogen bond network via changes in the intramolecular covalence, which broaden the preedge spectra. For intermediate-range and long-range ordering, quantum nuclei approach the neighboring oxygen atoms more closely than classical protons, promoting an “icelike” spectral feature with the intensities shifted from the main edge to the postedge. Computed spectra are in nearly quantitative agreement with the available experimental data.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

GreenThrift: Optimizing Carbon and Cost for Flexible Residential Loads

Reducing buildings’ carbon emissions is an important sustainability challenge. While scheduling flexible building loads has been previously used for a variety of grid and energy optimizations, carbon footprint reduction using such flexible loads poses new challenges since such methods need to balance both energy and carbon costs while also reducing user inconvenience from delaying such loads. This paper highlights the potential conflict between electricity prices and carbon emissions and the resulting trade-offs in carbon-aware and cost-aware load scheduling. To address this trade-off, we propose GreenThrift, a home automation system that leverages the scheduling capabilities of smart appliances and knowledge of future carbon intensity and cost to reduce both the carbon emissions and costs of flexible energy loads. At the heart of GreenThrift is an optimization technique that automatically computes schedules based on user configurations and preferences. We evaluate the effectiveness of GreenThrift using real-world carbon intensity data, electricity prices, and load traces from multiple locations and across different scenarios and objectives. Our results show that GreenThrift can replicate the offline optimal and retains 97% of the savings when optimizing the carbon emissions. Moreover, we show how GreenThrift can balance the conflict between carbon and cost and retain 95.3% and 85.5% of the potential carbon and cost savings, respectively.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

A Software Defined Storage Approach to Exascale Storage Services (Final Technical Report)

The Mochi project began in 2015 in recognition of a set of changes that were set to alter the landscape of storage and I/O for DOE science. On the one hand, an influx of new classes of applications including an initial set of data intensive and machine learning applications were operating with new types of data and in new ways, as compared to traditional computational science simulation codes. Simultaneously, new memory, storage, and networking technologies were driving a need to adapt software designs to a rapidly changing design space. The goal of the three-year Mochi project was to develop and demonstrate an approach for composing storage services which provides new levels of functionality, performance, and reliability for science applications at extreme scale. The team, including a mix of Laboratory, university, and industry collaborators, focused on systems and applications of interest to DOE Office of Science, tracking developments at NERSC and the LCFs as well as application trends within ECP and SciDAC. Technical work focused on three themes: composition (understand the building blocks needed to provide storage services that meet DOE mission needs and how these components are flexibly composed), application interfacing (what service compositions are most effective for common DOE application use cases), and consistency and synchronization (taking advantage of knowledge about, and from, the system and scientific applications to provide better service than would otherwise be possible). The project was highly successful, producing 13 publications and tens of thousands of lines of open source software that is now being used by multiple ECP teams, HPC software vendors, and international collaborators. The uptake of Mochi technologies speaks to both the value of the service composition model and the quality of architecture and software developed by the team. Through continued funding under ASCR and ECP, the Mochi team and its technologies are on track to further contribute to DOE science successes into the exascale timeframe.

97 MATHEMATICS AND COMPUTING↗

Results of precision processing (scene correction) of ERTS-1 images using digital image processing techniques

ERTS-1 MSS and RBV data recorded on computer compatible tapes have been analyzed and processed, and preliminary results have been obtained. No degradation of intensity (radiance) information occurred in implementing the geometric correction. The quality and resolution of the digitally processed images are very good, due primarily to the fact that the number of film generations and conversions is reduced to a minimum. Processing times of digitally processed images are about equivalent to the NDPF electro-optical processor.

Bernstein, R.↗

Moored rainfall measurements during COARE

This presentation discusses mini-ORG rainfall estimates collected from an array of six moornings in the western equatorial Pacific during the TOGA-COARE experiment. The moorings were clustered in the vicinity of the COARE intensive flux array (IFA) centered near 2 deg S, 156 deg E. The basic data set consisted of hourly means computed from 5-second samples.

Mcphaden, Michael J.↗

Modeling of Flow Transition Using an Intermittency Transport Equation

A new transport equation for intermittency factor is proposed to model transitional flows. The intermittent behavior of the transitional flows is incorporated into the computations by modifying the eddy viscosity, mu(sub t), obtainable from a turbulence model, with the intermittency factor, gamma: mu(sub t, sup *) = gamma.mu(sub t). In this paper, Menter's SST model (Menter, 1994) is employed to compute mu(sub t) and other turbulent quantities. The proposed intermittency transport equation can be considered as a blending of two models - Steelant and Dick (1996) and Cho and Chung (1992). The former was proposed for near-wall flows and was designed to reproduce the streamwise variation of the intermittency factor in the transition zone following Dhawan and Narasimha correlation (Dhawan and Narasimha, 1958) and the latter was proposed for free shear flows and was used to provide a realistic cross-stream variation of the intermittency profile. The new model was used to predict the T3 series experiments assembled by Savill (1993a, 1993b) including flows with different freestream turbulence intensities and two pressure-gradient cases. For all test cases good agreements between the computed results and the experimental data are observed.

Suzen, Y. B.↗

The Science DMZ: A Network Design Pattern for Data-Intensive Science

The ever-increasing scale of scientific data has become a significant challenge for researchers that rely on networks to interact with remote computing systems and transfer results to collaborators worldwide. Despite the availability of high-capacity connections, scientists struggle with inadequate cyberinfrastructure that cripples data transfer performance, and impedes scientific progress. The Science DMZ paradigm comprises a proven set of network design patterns that collectively address these problems for scientists. We explain the Science DMZ model, including network architecture, system configuration, cybersecurity, and performance tools, that creates an optimized network environment for science. We describe use cases from universities, supercomputing centers and research laboratories, highlighting the effectiveness of the Science DMZ model in diverse operational settings. In all, the Science DMZ model is a solid platform that supports any science workflow, and flexibly accommodates emerging network technologies. As a result, the Science DMZ vastly improves collaboration, accelerating scientific discovery.

Dart, Eli↗

TOPEX/POSEIDON tides estimated using a global inverse model

Altimetric data from the TOPEX/POSEIDON mission will be used for studies of global ocean circulation and marine geophysics. However, it is first necessary to remove the ocean tides, which are aliased in the raw data. The tides are constrained by the two distinct types of information: the hydrodynamic equations which the tidal fields of elevations and velocities must satisfy, and direct observational data from tide gauges and satellite altimetry. Here we develop and apply a generalized inverse method, which allows us to combine rationally all of this information into global tidal fields best fitting both the data and the dynamics, in a least squares sense. The resulting inverse solution is a sum of the direct solution to the astronomically forced Laplace tidal equations and a linear combination of the representers for the data functionals. The representer functions (one for each datum) are determined by the dynamical equations, and by our prior estimates of the statistics or errors in these equations. Our major task is a direct numerical calculation of these representers. This task is computationally intensive, but well suited to massively parallel processing. By calculating the representers we reduce the full (infinite dimensional) problem to a relatively low-dimensional problem at the outset, allowing full control over the conditioning and hence the stability of the inverse solution. With the representers calculated we can easily update our model as additional TOPEX/POSEIDON data become available. As an initial illustration we invert harmonic constants from a set of 80 open-ocean tide gauges. We then present a practical scheme for direct inversion of TOPEX/POSEIDON crossover data. We apply this method to 38 cycles of geophysical data records (GDR) data, computing preliminary global estimates of the four principal tidal constituents, M(sub 2), S(sub 2), K(sub 1) and O(sub 1). The inverse solution yields tidal fields which are simultaneously smoother, and in better agreement with altimetric and ground truth data, than previously proposed tidal models. Relative to the 'default' tidal corrections provided with the TOPEX/POSEIDON GDR, the inverse solution reduces crossover difference variances significantly (approximately 20-30%), even though only a small number of free parameters (approximately equal to 1000) are actually fit to the crossover data.

Egbert, Gary D.↗

Compiling and editing agricultural strata boundaries with remotely sensed imagery and map attribute data using graphics workstations

The USDA presently uses labor-intensive photographic interpretation procedures to delineate large geographical areas into manageable size sampling units for the estimation of domestic crop and livestock production. Computer software to automate the boundary delineation procedure, called the computer-assisted stratification and sampling (CASS) system, was developed using a Hewlett Packard color-graphics workstation. The CASS procedures display Thematic Mapper (TM) satellite digital imagery on a graphics display workstation as the backdrop for the onscreen delineation of sampling units. USGS Digital Line Graph (DLG) data for roads and waterways are displayed over the TM imagery to aid in identifying potential sample unit boundaries. Initial analysis conducted with three Missouri counties indicated that CASS was six times faster than the manual techniques in delineating sampling units.

Cheng, Thomas D.↗

Effect of Electronic Shot Noise on Dynamic Measurements using Optical Techniques: Examples from Rayleigh Scattering and Unsteady PSP

Electronic shot noise is an unavoidable reality in all optical techniques that depend on measuring light intensity. For steady-state, time-averaged measurements the impact of shot noise can be easily reduced by increasing the exposure time, or by averaging over multiple-exposures. That is not the case for unsteady measurements, where time histories of light intensity variations need to be created either by high-speed photography (unsteady PSP) or via photoelectron counting (spectrally resolved Rayleigh scattering) over short time durations using a photo-multiplier tube (PMT) and photon counting electronics. Electronic shot noise introduces a fixed amount of random error, which can overwhelm the light intensity variation caused by turbulent fluctuations. Spectrum computed from such time series shows a fixed noise floor that is independent of the number of data points in the time series. For a fixed optical system, where the collected power of the luminescent light (uPSP), or the scattered light (Rayleigh) is fixed, one needs to resort to special techniques to obtain enough signal-to noise ratio (SNR). For the uPSP application it is shown that averaging of the adjacent pixels improves SNR; although this may lead to a sacrifice of spatial resolution. A second means is to increase the exposure time, which leads to a lowering of the frequency range. For the Rayleigh application, improvements of SNR can be achieved via two different cross-correlation based approaches. The first involves measuring the light intensity using two PMTs using short, contiguous gates; thereby, creating two time-series of data. The second one involves collecting one long time-series of data using one set of measurement device, followed by an odd-even splitting into two time series. When the two time-series are cross-correlated, and a power spectrum is calculated, a significant reduction in the shot noise floor can be achieved. Examples from measurements of density and velocity fluctuations spectra from two different Rayleigh setup are presented to demonstrate the process.

uPSP↗