Search NASA⌕ Search

SEARCH · Search NASA

Results for “Methods:data analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 577 records · Page 32

Storage and Read-Optimized Data Placement Structures for High-Performance Analysis [Book Chapter]

Here, we present state-of-art structures and methods for efficient data preparation and representation for analysis. Our intent is to introduce the data science and analytics communities to open source data placements, structures, and methods. These practices can make the foundational processes of data preparation and access dramatically more efficient than typical raw file or database representations and use more conservative storage. To illustrate this, we introduce two highly efficient data placement structures. We then present a tutorial, supported by step-by-step examples, of how to create, use, and access data, structured by Parquet or ORC (Optimized Row Columnar), using Apache Spark. Finally, we illustrate the benefits of using these structures with computational and storage volume benchmarks.

97 MATHEMATICS AND COMPUTING↗

LDRD final report : leveraging multi-way linkages on heterogeneous data.

This report is a summary of the accomplishments of the 'Leveraging Multi-way Linkages on Heterogeneous Data' which ran from FY08 through FY10. The goal was to investigate scalable and robust methods for multi-way data analysis. We developed a new optimization-based method called CPOPT for fitting a particular type of tensor factorization to data; CPOPT was compared against existing methods and found to be more accurate than any faster method and faster than any equally accurate method. We extended this method to computing tensor factorizations for problems with incomplete data; our results show that you can recover scientifically meaningfully factorizations with large amounts of missing data (50% or more). The project has involved 5 members of the technical staff, 2 postdocs, and 1 summer intern. It has resulted in a total of 13 publications, 2 software releases, and over 30 presentations. Several follow-on projects have already begun, with more potential projects in development.

97 MATHEMATICS AND COMPUTING↗

ALACRITY: Analytics-Driven Lossless Data Compression for Rapid In-Situ Indexing, Storing, and Querying. In: Transactions on Large-Scale Data- and Knowledge-Centered Systems X. Lecture Notes in Computer Science, vol 8220

High-performance computing architectures face nontrivial data processing challenges, as computational and I/O components further diverge in performance trajectories. For scientific data analysis in particular, methods based on generating heavyweight access acceleration structures, e.g. indexes, are becoming less feasible for ever-increasing dataset sizes. We present ALACRITY, demonstrating the effectiveness of a fused data and index encoding of scientific, floating-point data in generating lightweight data structures amenable to common types of queries used in scientific data analysis. We exploit the representation of floating-point values by extracting significant bytes, using the resulting unique values to bin the remaining data along fixed-precision boundaries. To optimize query processing, we use an inverted index, mapping each generated bin to a list of records contained within, allowing us to optimize query processing with attribute range constraints. Overall, the storage footprint for both index and data is shown to be below numerous configurations of bitmap indexing, while matching or outperforming query performance.

Compression Ratio↗

Simulated JWST Data Sets for Multispectral and Hyperspectral Image Fusion

The James Webb Space Telescope (JWST) will provide multispectral and hyperspectral infrared images of a large number of astrophysical scenes. Multispectral images will have the highest angular resolution, while hyperspectral images (e.g., with integral field unit spectrometers) will provide the best spectral resolution. This paper aims at providing a comprehensive framework to generate an astrophysical scene and to simulate realistic hyperspectral and multispectral data acquired by two JWST instruments, namely, NIRCam Imager and NIRSpec IFU. We want to show that this simulation framework can be resorted to assess the benefits of fusing these images to recover an image of high spatial and spectral resolutions. To do so, we make a synthetic scene associated with a canonical infrared source, the Orion Bar. We develop forward models including corresponding noises for the two JWST instruments based on their physical features. JWST observations are then simulated by applying the forward models to the aforementioned synthetic scene. We test a dedicated fusion algorithm we developed on these simulated observations. We show that the fusion process reconstructs the high spatio-spectral resolution scene with a good accuracy on most areas, and we identify some limitations of the method to be tackled in future works. The synthetic scene and observations presented in the paper can be used, for instance, to evaluate instrument models, pipelines, or more sophisticated algorithms dedicated to JWST data analysis. Besides, fusion methods such as the one presented in this paper are shown to be promising tools to fully exploit the unprecedented capabilities of the JWST.

79 ASTRONOMY AND ASTROPHYSICS↗

Data Unfolding with Wiener-SVD Method

Here, data unfolding is a common analysis technique used in HEP data analysis. Inspired by the deconvolution technique in the digital signal processing, a new unfolding technique based on the SVD technique and the well-known Wiener filter is introduced. The Wiener-SVD unfolding approach achieves the unfolding by maximizing the signal to noise ratios in the effective frequency domain given expectations of signal and noise and is free from regularization parameter. Through a couple examples, the pros and cons of the Wiener-SVD approach as well as the nature of the unfolded results are discussed.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Author Correction: US oil and gas system emissions from nearly one million aerial site measurements

Correction to: Naturehttps://doi.org/10.1038/s41586-024-07117-5 Published online 13 March 2024 In the version of the article initially published, several errors were present and have been corrected in the HTML and PDF versions of the article and Supplementary Information. The main results, conclusions, and our interpretations of the data remain unchanged. See the new Supplementary Information Section S15 for a more detailed description of the errors corrected and the resulting effects on the analysis. Data processing and methods corrections Overflight count correction: We previously used pre-computed source coverage data for some Carbon Mapper campaigns that was computed differently than was required for our analysis. We have re-computed Carbon Mapper source coverage based on flightline polygons and source coordinates. Transition point computation, well sites: The updated version now correctly compares the cumulative emissions distribution of simulated well site emissions with that of aerially detected sources (rather than plumes) when computing the transition point. Transition point computation, midstream: Additionally, the transition point calculation has been corrected to exclude aerially detected midstream emissions below the transition point, which was previously leading to double counting of these emissions. This error was not present for upstream (well site) emissions. Calculation errors Unit error: We corrected a specific unit conversion error affecting well site emissions in the Kairos Fort Worth dataset. Across all datasets, we also correct the conversion factor for converting from standard volume to mass for midstream emissions. Sorting error: We correct code that was applying incorrect sorting when computing correction factors to account for partial detection at well sites. Small typographical corrections were made in Fig. 1b and SI Section S4.1. Data processing and methods corrections Overflight count correction: We previously used pre-computed source coverage data for some Carbon Mapper campaigns that was computed differently than was required for our analysis. We have re-computed Carbon Mapper source coverage based on flightline polygons and source coordinates. Transition point computation, well sites: The updated version now correctly compares the cumulative emissions distribution of simulated well site emissions with that of aerially detected sources (rather than plumes) when computing the transition point. Transition point computation, midstream: Additionally, the transition point calculation has been corrected to exclude aerially detected midstream emissions below the transition point, which was previously leading to double counting of these emissions. This error was not present for upstream (well site) emissions. Calculation errors Unit error: We corrected a specific unit conversion error affecting well site emissions in the Kairos Fort Worth dataset. Across all datasets, we also correct the conversion factor for converting from standard volume to mass for midstream emissions. Sorting error: We correct code that was applying incorrect sorting when computing correction factors to account for partial detection at well sites. Small typographical corrections were made in Fig. 1b and SI Section S4.1. The following practices may help researchers conducting similar analyses avoid making similar errors: 1, Clear, accessible documentation explaining the interpretation of all columns in data input tables and all internal variables within the model, 2, Simple cross-check calculations computed before and after unit conversions.

Sherwin, Evan D↗

Electron Tomography and Machine Learning for Understanding the Highly Ordered Structure of Leafhopper Brochosomes

Insects known as leafhoppers (Hemiptera: Cicadellidae) produce hierarchically structured nanoparticles known as brochosomes that are exuded and applied to the insect cuticle, thereby providing camouflage and anti-wetting properties to aid insect survival. Although the physical properties of brochosomes are thought to depend on the leafhopper species, the structure–function relationships governing brochosome behavior are not fully understood. Brochosomes have complex hierarchical structures and morphological heterogeneity across species, due to which a multimodal characterization approach is required to effectively elucidate their nanoscale structure and properties. In this work, we study the structural and mechanical properties of brochosomes using a combination of atomic force microscopy (AFM), electron microscopy (EM), electron tomography, and machine learning (ML)-based quantification of large and complex scanning electron microscopy (SEM) image data sets. This suite of techniques allows for the characterization of internal and external brochosome structures, and ML-based image analysis methods of large data sets reveal correlations in the structure across several leafhopper species. Our results show that brochosomes are relatively rigid hollow spheres with characteristic dimensions and morphologies that depend on leafhopper species. Nanomechanical mapping AFM is used to determine a characteristic compression modulus for brochosomes on the order of 1–3 GPa, which is consistent with crystalline proteins. Altogether, this work provides an improved understanding of the structural and mechanical properties of leafhopper brochosomes using a new set of ML-based image classification tools that can be broadly applied to nanostructured biological materials.

Chemical structure↗

Data-driven method for electric vehicle charging demand analysis: Case study in Virginia

Electric vehicle (EV) adoption in the U.S. will be accelerated by the historic $7.5 billion public investments in EV charging infrastructure. Careful analysis of EV charging demands plays a vital role in understanding the energy requirements, power grid impact, and smart charging management opportunities of EVs. To this end, this paper develops a data-driven trip-chaining-based modeling framework including five steps: Trip data acquisition and preprocessing, EV adoption modeling, travel itinerary synthesis, EV charging demand simulation and EV load profile generation. The developed analysis framework was demonstrated using real-world data for one region in Virginia, U.S. The results show that the proposed modeling framework can work effectively. For the study region in 2040, the predicted number of plug-in EVs is 470,114, resulting in a weekly charging demand of 38,078,127 kWh (55% home, 9% work, and 36% public) in September and 45,920,358 kWh (61% home, 9% work, and 30% public) in February.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Multimodal X-ray nano-spectromicroscopy analysis of chemically heterogeneous systems

Abstract Understanding the nanoscale chemical speciation of heterogeneous systems in their native environment is critical for several disciplines such as life and environmental sciences, biogeochemistry, and materials science. Synchrotron-based X-ray spectromicroscopy tools are widely used to understand the chemistry and morphology of complex material systems owing to their high penetration depth and sensitivity. The multidimensional (4D+) structure of spectromicroscopy data poses visualization and data-reduction challenges. This paper reports the strategies for the visualization and analysis of spectromicroscopy data. We created a new graphical user interface and data analysis platform named XMIDAS (X-ray multimodal image data analysis software) to visualize spectromicroscopy data from both image and spectrum representations. The interactive data analysis toolkit combined conventional analysis methods with well-established machine learning classification algorithms (e.g. nonnegative matrix factorization) for data reduction. The data visualization and analysis methodologies were then defined and optimized using a model particle aggregate with known chemical composition. Nanoprobe-based X-ray fluorescence (nano-XRF) and X-ray absorption near edge structure (nano-XANES) spectromicroscopy techniques were used to probe elemental and chemical state information of the aggregate sample. We illustrated the complete chemical speciation methodology of the model particle by using XMIDAS. Next, we demonstrated the application of this approach in detecting and characterizing nanoparticles associated with alveolar macrophages. Our multimodal approach combining nano-XRF, nano-XANES, and differential phase-contrast imaging efficiently visualizes the chemistry of localized nanostructure with the morphology. We believe that the optimized data-reduction strategies and tool development will facilitate the analysis of complex biological and environmental samples using X-ray spectromicroscopy techniques.

36 MATERIALS SCIENCE↗

Learning Spectral Templates for Photometric Redshift Estimation from Broadband Photometry

Estimating redshifts from broadband photometry is often limited by how accurately we can map the colors of galaxies to an underlying spectral template. Current techniques utilize spectrophotometric samples of galaxies or spectra derived from spectral synthesis models. Both of these approaches have their limitations: either the sample sizes are small and often not representative of the diversity of galaxy colors, or the model colors can be biased (often as a function of wavelength), which introduces systematics in the derived redshifts. In this paper, we learn the underlying spectral energy distributions from an ensemble of ∼100 K galaxies with measured redshifts and colors. We show that we are able to reconstruct emission and absorption lines at a significantly higher resolution than the broadband filters used to measure the photometry for a sample of 20 spectral templates. We find that our training algorithm reduces the fraction of outliers in the derived photometric redshifts by up to 28%, bias up to 91%, and scatter up to 25%, when compared to estimates using a standard set of spectral templates. We discuss the current limitations of this approach and its applicability for recovering the underlying properties of galaxies. Our derived templates and the code used to produce these results are publicly available in a dedicated Github repository: https://github.com/dirac-institute/photoz-template-learning.

79 ASTRONOMY AND ASTROPHYSICS↗

Highly Selective and Stable Multivariable Gas Sensors for Enhanced Robustness and Reliability of SOFC Operation

GE Global Research, in partnership with SUNY Polytechnic Institute (SUNY Poly) and GE–Fuel Cells LLC, designed, built, and tested multi-gas sensor prototypes for in situ monitoring of gases produced during steam reforming in solid oxide fuel cell (SOFC) systems. To be widely adopted and to deliver a lower operating cost for customers, SOFC systems should enhance their operation reliability and improve stack robustness. These needs can be achieved by early diagnostics of potential upsets by using in-situ gas sensors for the real-time monitoring of gas mixture compositions. Thus, this project objective was to achieve selectivity and stability of sensing of gases for SOFC applications by implementing a new generation of gas sensors, known as multivariable sensors. Such sensors are intended to complement the mature traditional analytical instruments based on sensor performance and design characteristics. The key sensor-performance characteristics include accurate detection and quantitation of several gases in their mixtures and stability of operation in unattended applications. The key design characteristics of sensors include their unobtrusive form factor and cost-effectiveness. The duration of the project was 18 months with the project structure that included three technical tasks such as (1) design and laboratory validation of multivariable sensors for selective detection of H 2 and CO gases, (2) laboratory validation of sensor stability over a two-week testing, and (3) field-validation of the developed multivariable sensor system prototype at GE–Fuel Cells factory. Multi-gas detection was achieved with multivariable photonic sensors designed to operate at elevated temperature. The sensor design was based on three-dimensional (3D) bio-inspired photonic nanostructure that was fabricated using materials responsive to gases of interest such as H 2 and CO. The gas discrimination and the rejection of interferences such as CH 4 , H 2 O, and others was performed by optimizing the design parameters of the 3D photonic nanostructures such as their composition and morphology. Planar sol-gel and sputtered sensing films were fabricated as controls. As a result of the fabrication, functionalization, excitation, and performance testing of 3D photonic sensors, we developed initial design rules for high temperature multivariable sensors. These design rules pave the way for the new generation of cost-effective industrial sensors. Stability of sensor operation was tested in the laboratory conditions under variations of the ambient environments. Stability tests were designed in the order of enhanced complexity and realism of sensor operation and included quantitation of a single gas, quantitation of a single gas in the presence of interferences, and quantitation of two and three gases. We have found that conventional machine learning data analysis stools were unable to correct for the sensor drift. Thus, we have developed new methods of machine learning data analysis that provided us with needed performance stability. Such performance improvement in gas sensing was a significant milestone in bringing our multivariable sensors to practical applications. The project culminated with initial field validation of developed multivariable sensor prototypes at GE–Fuel Cells LLC. The in situ data generated by the sensor system in the lab and field conditions provided inputs that advanced our fundamental understanding of gas sensing at high temperatures to enable sensors for SOFC applications and allowed development of the recommendations for further deliverables.

47 OTHER INSTRUMENTATION↗

Surface analysis insight note: Differentiation methods applicable to noisy data for determination of sp2‐ versus sp3‐hybridization of carbon allotropes and AES signal strengths

The derivatives of the spectra are commonly used for quantification in Auger Electron Spectroscopy (AES) spectra, while the derivative of the KLL C Auger line has proven to be valuable in obtaining a measure of the relative proportions of sp 2 ‐ and sp 3 ‐hybridization using the D‐parameter in both AES and X‐ray Photoelectron Spectroscopy (XPS). Differentiation of X‐ray Photoelectron Spectroscopy (XPS) and Auger Electron Spectroscopy (AES) spectra by numerical means is presented and illustrated for polymeric, such as PEEK and Nylon, as well as for graphitic materials including highly ordered pyrolytic graphite and graphene oxide. The most commonly available Savitzky–Golay method is explained mathematically and developed through the case of constructing a 5‐point quadratic polynomial convolution kernel suitable for differentiating spectra of adequate signal to noise. The concept of differentiation of spectra where signal to noise is less than adequate is also developed. Two alternative strategies to Savitzky–Golay differentiation are presented, which fit curves to data that allow derivatives to be obtained where Savitzky–Golay would otherwise fail. These alternative methods involve constructing a parametric curve that fits data over the entire energy interval of interest. Derivatives of spectra are then obtained by differentiating these parametric curves directly. A comparison of results for different materials for which specific sp 2 ‐ vs sp 3 ‐hybridized carbon proportions are of interest is used to emphasize the importance of characterizing methods used to differentiate spectra and understanding the characteristics of instrumentation used to measure spectra. The case for using Principal Component Analysis noise reduction with C KLL spectra is made for spectra collected from a heterogeneous graphene oxide sample.

Fairley, Neal↗

EINSTEIN-HOME DISCOVERY OF 24 PULSARS IN THE PARKES MULTI-BEAM PULSAR SURVEY

We have conducted a new search for radio pulsars in compact binary systems in the Parkes multi-beam pulsar survey (PMPS) data, employing novel methods to remove the Doppler modulation from binary motion. This has yielded unparalleled sensitivity to pulsars in compact binaries. The required computation time of Almost-Equal-To 17, 000 CPU core years was provided by the distributed volunteer computing project Einstein-Home, which has a sustained computing power of about 1 PFlop s{sup -1}. We discovered 24 new pulsars in our search, 18 of which were isolated pulsars, and 6 were members of binary systems. Despite the wide filterbank channels and relatively slow sampling time of the PMPS data, we found pulsars with very large ratios of dispersion measure (DM) to spin period. Among those is PSR J1748-3009, the millisecond pulsar with the highest known DM ( Almost-Equal-To 420 pc cm{sup -3}). We also discovered PSR J1840-0643, which is in a binary system with an orbital period of 937 days, the fourth largest known. The new pulsar J1750-2536 likely belongs to the rare class of intermediate-mass binary pulsars. Three of the isolated pulsars show long-term nulling or intermittency in their emission, further increasing this growing family. Our discoveries demonstrate the value of distributed volunteer computing for data-driven astronomy and the importance of applying new analysis methods to extensively searched data.

79 ASTRONOMY AND ASTROPHYSICS↗

OPTIMAL STRATEGIES FOR CONTINUOUS GRAVITATIONAL WAVE DETECTION IN PULSAR TIMING ARRAYS

Supermassive black hole binaries (SMBHBs) are expected to emit a continuous gravitational wave signal in the pulsar timing array (PTA) frequency band (10{sup -9} to 10{sup -7} Hz). The development of data analysis techniques aimed at efficient detection and characterization of these signals is critical to the gravitational wave detection effort. In this paper, we leverage methods developed for LIGO continuous wave gravitational searches and explore the use of the F-statistic for such searches in pulsar timing data. Babak and Sesana have used this approach in the context of PTAs to show that one can resolve multiple SMBHB sources in the sky. Our work improves on several aspects of prior continuous wave search methods developed for PTA data analysis. The algorithm is implemented fully in the time domain, which naturally deals with the irregular sampling typical of PTA data and avoids spectral leakage problems associated with frequency domain methods. We take into account the fitting of the timing model and have generalized our approach to deal with both correlated and uncorrelated colored noise sources. We also develop an incoherent detection statistic that maximizes over all pulsar-dependent contributions to the likelihood. To test the effectiveness and sensitivity of our detection statistics, we perform a number of Monte Carlo simulations. We produce sensitivity curves for PTAs of various configurations and outline an implementation of a fully functional data analysis pipeline. Finally, we present a derivation of the likelihood maximized over the gravitational wave phases at the pulsar locations, which results in a vast reduction of the search parameter space.

79 ASTRONOMY AND ASTROPHYSICS↗

Search for Low-Mass Dark Matter with CDMSlite Using a Profile Likelihood Fit

The Cryogenic Dark Matter Search low ionization threshold experiment (CDMSlite) searches for interactions between dark matter particles and germanium nuclei in cryogenic detectors. The experiment has achieved a low energy threshold with improved sensitivity to low-mass (<10 GeV/c2) dark matter particles. We present an analysis of the final CDMSlite data set, taken with a different detector than was used for the two previous CDMSlite data sets. This analysis includes a data "salting" method to protect against bias, improved noise discrimination, background modeling, and the use of profile likelihood methods to search for a dark matter signal in the presence of backgrounds. We achieve an energy threshold of 70 eV and significantly improve the sensitivity for dark matter particles with masses between 2.5 and 10 GeV/c2 compared to previous analyses. We set an upper limit on the dark matter-nucleon scattering cross section in germanium of 5.4×10-42 cm2 at 5 GeV/c2, a factor of ~2.5 improvement over the previous CDMSlite result.

Agnese, R↗

Machine-learning-assisted automation of single-crystal neutron diffraction

Neutron scattering is a powerful but expensive technique to study materials and discover new matter. Advanced detector technology has significantly improved the efficiency of neutron experiments, increasing the complexity of neutron data reduction and analysis. Machine learning (ML) brings new directions for neutron diffraction data reduction and experiment operation. Here, this work presents an ML-assisted data reduction and analysis method for precise recognition of Bragg peaks and the corresponding regions of interest; it can then automatically screen and align a measured crystal using the recognized peaks, and subsequently plan and optimize the data collection with user-provided information and uncertainty quantification values of detected peaks. This method shows robust performance in different complex sample environments and enables automated single-crystal neutron diffraction.

47 OTHER INSTRUMENTATION↗