Search NASA⌕ Search

SEARCH · Search NASA

Results for “large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19

Searches for New Physics at the Tevatron and LHC

This is an auspicious moment in experimental particle physics - there are large data samples at the Tevatron and a new energy regime being explored at the Large Hadron Collider with ever larger data samples. The coincidence of these two events suggests that we will soon be able to address the question, what lies beyond the standard model? Particle physics's current understanding of the universe is embodied in it. The model has been tested to extreme precision - better than a part in ten thousand - but we suspect that it is only an approximation, and that physics beyond this standard model will appear in the data of the Tevatron and LHC in the near future. This brief review touches on the status of searches for new physics at the time of the conference.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Improving the reliability of automated non-destructive inspection

In automated NDE a region of an inspected component is often interrogated several times, be it within a single data channel, across multiple channels or over the course of repeated inspections. The systematic combination of these diverse readings is recognized to provide a means to improve the reliability of the inspection, for example by enabling noise suppression. Specifically, such data fusion makes it possible to declare regions of the component defect-free to a very high probability whilst readily identifying indications. Registration, aligning input datasets to a common coordinate system, is a critical pre-computation before meaningful data fusion takes place. A novel scheme based on a multiobjective optimization is described. The developed data fusion framework, that is able to identify and rate possible indications in the dataset probabilistically, based on local data statistics, is outlined. The process is demonstrated on large data sets from the industrial ultrasonic testing of aerospace turbine disks, with major improvements in the probability of detection and probability of false call being obtained.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Accessing Data Federations with CVMFS

Data federations have become an increasingly common tool for large collaborations such as CMS and Atlas to efficiently distribute large data files. Unfortunately, these typically are implemented with weak namespace semantics and a non-POSIX API. On the other hand, CVMFS has provided a POSIX-compliant read-only interface for use cases with a small working set size (such as software distribution). The metadata required for the CVMFS POSIX interface is distributed through a caching hierarchy, allowing it to scale to the level of about a hundred thousand hosts. In this paper, we will describe our contributions to CVMFS that merges the data scalability of XRootD-based data federations (such as AAA) with metadata scalability and POSIX interface of CVMFS. We modified CVMFS so it can serve unmodified files without copying them to the repository server. CVMFS 2.2.0 is also able to redirect requests for data files to servers outside of the CVMFS content distribution network. Finally, we added the ability to manage authorization and authentication using security credentials such as X509 proxy certificates. We combined these modifications with the OSGs StashCache regional XRootD caching infrastructure to create a cached data distribution network. Here, we will show performance metrics accessing the data federation through CVMFS compared to direct data federation access. Additionally, we will discuss the improved user experience of providing access to a data federation through a POSIX filesystem.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

VisIO: enabling interactive visualization of ultra-scale, time-series data via high-bandwidth distributed I/O systems

Petascale simulations compute at resolutions ranging into billions of cells and write terabytes of data for visualization and analysis. Interactive visuaUzation of this time series is a desired step before starting a new run. The I/O subsystem and associated network often are a significant impediment to interactive visualization of time-varying data; as they are not configured or provisioned to provide necessary I/O read rates. In this paper, we propose a new I/O library for visualization applications: VisIO. Visualization applications commonly use N-to-N reads within their parallel enabled readers which provides an incentive for a shared-nothing approach to I/O, similar to other data-intensive approaches such as Hadoop. However, unlike other data-intensive applications, visualization requires: (1) interactive performance for large data volumes, (2) compatibility with MPI and POSIX file system semantics for compatibility with existing infrastructure, and (3) use of existing file formats and their stipulated data partitioning rules. VisIO, provides a mechanism for using a non-POSIX distributed file system to provide linear scaling of 110 bandwidth. In addition, we introduce a novel scheduling algorithm that helps to co-locate visualization processes on nodes with the requested data. Testing using VisIO integrated into Para View was conducted using the Hadoop Distributed File System (HDFS) on TACC's Longhorn cluster. A representative dataset, VPIC, across 128 nodes showed a 64.4% read performance improvement compared to the provided Lustre installation. Also tested, was a dataset representing a global ocean salinity simulation that showed a 51.4% improvement in read performance over Lustre when using our VisIO system. VisIO, provides powerful high-performance I/O services to visualization applications, allowing for interactive performance with ultra-scale, time-series data.

97 MATHEMATICS AND COMPUTING↗

ASCR Workshop on In Situ Data Management: Enabling Scientific Discovery from Diverse Data Sources

In January 2019, the U.S. Department of Energy, Office of Science program in Advanced Scientific Computing Research, convened a workshop to identify priority research directions for in situ data management (ISDM). The workshop defined ISDM as the practices, capabilities, and procedures to control the organization of data and enable the coordination and communication among heterogeneous tasks, executing simultaneously in a high-performance computing system, cooperating toward a common objective. The workshop revealed two primary, interdependent motivations for processing and managing data in situ. The first motivation is that the in situ methodology enables scientific discovery from a broad range of data sources over a wide scale of computing platforms: leadership-class systems, clusters, clouds, workstations, and embedded devices at the edge. The successful development of ISDM capabilities will benefit real-time decision-making, design optimization, and data-driven scientific discovery. The second motivation is the need to decrease data volumes. ISDM can make critical contributions to managing large data volumes from computations and experiments to minimize data movement, save storage space, and boost resource efficiency, often while simultaneously increasing scientific precision. A fundamental finding of this workshop is that the methodologies used to manage data among a variety of tasks in situ can be used to facilitate scientific discovery from many different data sources—simulation, experiment, and sensors, for example—and that being able to do so at numerous computing scales will benefit real-time decision-making, design optimization, and data-driven scientific discovery across the Office of Science mission space. Applications wanting to use the in situ capabilities include those where data analysis feeds back to the simulation, decisions are made autonomously, big data or machine learning is among the tasks to be coordinated, and computations need to be completed in real time. The workshop identified six priority research directions that highlight the components and capabilities needed for ISDM to be successful for the wide variety of applications discussed: making ISDM capabilities more pervasive, controllable, composable, and transparent, with a focus on greater coordination with the software stack and a diversity of fundamentally new data algorithms.

97 MATHEMATICS AND COMPUTING↗

ASCR Workshop on In Situ Data Management

In January 2019, the U.S. Department of Energy, Office of Science program in Advanced Scientific Computing Research, convened a workshop to identify priority research directions for in situ data management (ISDM). The workshop defined ISDM as the practices, capabilities, and procedures to control the organization of data and enable the coordination and communication among heterogeneous tasks, executing simultaneously in a high-performance computing system, cooperating toward a common objective. The workshop revealed two primary, interdependent motivations for processing and managing data in situ. The first motivation is that the in situ methodology enables scientific discovery from a broad range of data sources over a wide scale of computing platforms: leadership-class systems, clusters, clouds, workstations, and embedded devices at the edge. The successful development of ISDM capabilities will benefit real-time decision-making, design optimization, and data-driven scientific discovery. The second motivation is the need to decrease data volumes. ISDM can make critical contributions to managing large data volumes from computations and experiments to minimize data movement, save storage space, and boost resource efficiency, often while simultaneously increasing scientific precision.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Hybrid Storage Solution

With the rise of artificial intelligence and machine learning, data sets used to train models have become increasingly large. The availability, accessibility and integrity of large data sets has become important to the research conducted at Los Alamos National Laboratory. Ceph is a storage solution suitable for use with critical data because of its distributed nature and ability to keep multiple copies of a file in different locations. The amount of data means that bandwidth, latency, and cost are important factors and the reason most storage solutions are on-premises. However, there are distinct advantages to hosting services in the cloud, namely scalability and ease-of-use. In this paper, we explore the possibility of provisioning a hybrid Ceph cluster that leverages the benefits of both cloud architectures and on-premise performance.

97 MATHEMATICS AND COMPUTING↗

Tau (or no) leptons in top quark decays at hadron colliders

Measurements in the final states with taus or with no-leptons are among the most challenging as they are those with the smallest signal-to-background ratio. However, these final states are of particular interest as they can be important probes of new physics. Tau identification techniques and cross section measurements in top quark decays in these final states are discussed. The results, limited by systematical uncertainties, are consistent with standard model predictions, and are used to setstringent limits on new physics searches. The large data samples available at the Fermilab and at the Large Hadron Collider may help further improving the measurements.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

A Comprehensive Characterization of Galaxy-cool CGM Connections at z < 0.4 with DESI Year 1 Data

We investigate the relationships between the cool circumgalactic medium (CGM), traced by Ca II absorption lines, and galaxy properties at z < 0.4 using ∼900,000 galaxy–quasar pairs within 200 kpc from the Year 1 data of the Dark Energy Spectroscopic Instrument (DESI). This large data set enables us to obtain composite spectra with sensitivity reaching to the mÅ level and to explore the Ca II absorption as a function of stellar mass, star formation rate (SFR), redshift, and galaxy types, including active galactic nuclei (AGNs). Our results show a positive correlation between the absorption strength and stellar mass of star-forming galaxies with $\langle$$W$$^{Ca II}_{0}$$\rangle$ α $M$$^{0.5}_{*}$ over 3 orders of magnitude in stellar mass from ∼10 8 to 10 11 M ⊙ , while such a mass dependence is weaker for quiescent galaxies. At a fixed mass, Ca II absorption is stronger around star-forming galaxies than quiescent ones especially within impact parameters <30 kpc. Among star-forming galaxies, the Ca II absorption further correlates with SFR, following ∝SFR 0.3 . However, in contrast to the results at higher redshifts, stronger absorption is not preferentially observed along the minor axis of star-forming galaxies, indicating a possible redshift evolution of CGM dynamics resulting from galactic feedback. Moreover, no significant difference between the properties of the cool gas around AGNs and galaxies is detected. Finally, we measure the absorption profiles with respect to the virial radius of dark matter halos and show that the total Ca II mass in the CGM is comparable to the Ca mass in the ISM of galaxies.

Circumgalactic medium↗

Digital processing of radiographic images

Some techniques are presented and the software documentation for the digital enhancement of radiographs. Both image handling and image processing operations are considered. The image handling operations dealt with are: (1) conversion of format of data from packed to unpacked and vice versa; (2) automatic extraction of image data arrays; (3) transposition and 90 deg rotations of large data arrays; (4) translation of data arrays for registration; and (5) reduction of the dimensions of data arrays by integral factors. Both the frequency and the spatial domain approaches are presented for the design and implementation of the image processing operation. It is shown that spatial domain recursive implementation of filters is much faster than nonrecursive implementations using fast fourier transforms (FFT) for the cases of interest in this work. The recursive implementation of a class of matched filters for enhancing image signal to noise ratio is described. Test patterns are used to illustrate the filtering operations. The application of the techniques to radiographic images of metallic structures is demonstrated through several examples.

Bond, A. D.↗

Generic functional requirements for a NASA general-purpose data base management system

Generic functional requirements for a general-purpose, multi-mission data base management system (DBMS) for application to remotely sensed scientific data bases are detailed. The motivation for utilizing DBMS technology in this environment is explained. The major requirements include: (1) a DBMS for scientific observational data; (2) a multi-mission capability; (3) user-friendly; (4) extensive and integrated information about data; (5) robust languages for defining data structures and formats; (6) scientific data types and structures; (7) flexible physical access mechanisms; (8) ways of representing spatial relationships; (9) a high level nonprocedural interactive query and data manipulation language; (10) data base maintenance utilities; (11) high rate input/output and large data volume storage; and adaptability to a distributed data base and/or data base machine configuration. Detailed functions are specified in a top-down hierarchic fashion. Implementation, performance, and support requirements are also given.

Lohman, G. M.↗

Pilot study of the domestic information display system in state and local government

An interactive computer based system that can retrieve a wide range of data (demographic, environmental, socio-economic, etc.,) from a large data base and display these data for different geographic units in the form of choropleth maps was developed. The system was designed to display statistical information in a geographic format for national policy makers.

Source record↗

Imaging spectrometry for earth remote sensing

Initial results of the novel remote earth sensing technique of imaging spectrometry, which is technically feasible from both spacecraft and aircraft platforms, indicate that the direct identification of surface materials on a picture-element basis is possible through proper sampling of absorption features in the reflectance spectrum. Sensors of this type are able to acquire images simultaneously in 100-200 contiguous spectral bands. Computerized data reduction and storage techniques are available for the large data sets thus generated, and novel analytic techniques are under development to maximize information content extraction.

Goetz, A. F. H.↗

SAR image data compression for an on-line archive system

This paper summarizes the investigation of SAR image data compression for an on-line archive data distribution system. This system is planned for the ground processing system of Alaska SAR Facility (ASF) and Shuttle Imaging Radar (SIR-C). The objective of the SAR image data compression is to enable the data archive system to provide the remote users a large data base with good image quality, short response time, low transfer cost, and minimal decoding complexity. The requirements and limitations of the on-line archive data distribution system are presented. The effects of SAR image data characteristics on data compression are addressed. The users' survey results suggest that compression ratios between 10:1 and 20:1 appear suitable. Based on the algorithm evaluation results, the two-level tree-searched vector quantization technique has been recommended as the SAR image data compression algorithm for the on-line archive data distribution system.

Chang, C. Y.↗

Online Time Series Analysis of Land Products over Asia Monsoon Region via Giovanni

Time series analysis is critical to the study of land cover/land use changes and climate. Time series studies at local-to-regional scales require higher spatial resolution, such as 1km or less, data. MODIS land products of 250m to 1km resolution enable such studies. However, such MODIS land data files are distributed in 10ox10o tiles, due to large data volumes. Conducting a time series study requires downloading all tiles that include the study area for the time period of interest, and mosaicking the tiles spatially. This can be an extremely time-consuming process. In support of the Monsoon Asia Integrated Regional Study (MAIRS) program, NASA GES DISC (Goddard Earth Sciences Data and Information Services Center) has processed MODIS land products at 1 km resolution over the Asia monsoon region (0o-60oN, 60o-150oE) with a common data structure and format. The processed data have been integrated into the Giovanni system (Goddard Interactive Online Visualization ANd aNalysis Infrastructure) that enables users to explore, analyze, and download data over an area and time period of interest easily. Currently, the following regional MODIS land products are available in Giovanni: 8-day 1km land surface temperature and active fire, monthly 1km vegetation index, and yearly 0.05o, 500m land cover types. More data will be added in the near future. By combining atmospheric and oceanic data products in the Giovanni system, it is possible to do further analyses of environmental and climate changes associated with the land, ocean, and atmosphere. This presentation demonstrates exploring land products in the Giovanni system with sample case scenarios.

Shen, Suhung↗

Onboard Classification of Hyperspectral Data on the Earth Observing One Mission

Remote-sensed hyperspectral data represents significant challenges in downlink due to its large data volumes. This paper describes a research program designed to process hyperspectral data products onboard spacecraft to (a) reduce data downlink volumes and (b) decrease latency to provide key data products (often by enabling use of lower data rate communications systems). We describe efforts to develop onboard processing to study volcanoes, floods, and cryosphere, using the Hyperion hyperspectral imager and onboard processing for the Earth Observing One (EO-1) mission as well as preliminary work targeting the Hyperspectral Infrared Imager (HyspIRI) mission.

cryosphere↗

Advanced Analytics and Big Earth Data

NASA's Earth Science Data Systems process, archive and distribute petabytes of Earth Observation data to a variety of end users. These end users will face dramatically increased data size in the near future, bringing about new challenges and opportunities in analyzing those data. One area of particular ferment currently is Machine Learning. Many Machine Learning methods are black boxes, limiting direct insight into the data's properties. However, they can be used for a variety of data enhancement purposes, such as parameter retrieval, data fusion and image classification and segmentation. The Earth Observing System Data and Information System is also evolving to host large data volumes in the cloud, enabling data proximal analysis. As part of this effort, an Analytics framework is being developed to support and enhance user analysis of the data. By using standards based services in the framework, diverse user communities can be served, while also allowing inter-system collaboration in the analysis process.

Cloud Computing↗

SMAnalytics Automated Shape Memory Test Analysis Software User's Manual: UCFTC, UPER, and DSC Test Methods, Version 1.0.2

Over the course of 60 years of shape memory alloy research and development, the properties of these alloys have been measured using various testing methods, which are often customized by the organization performing the test. However, commercial adoption of shape memory alloys in aeronautic actuator applications requires reducing property uncertainty through standardizing test methods. Historically, differential scanning calorimetry (DSC) has been used to measure transformation temperatures under stress-free conditions. This testing method is well established, and tests are run under the applicable standard, ASTM F2004-17: Standard Test Method for Transformation Temperature of Nickel-Titanium Alloys by Thermal Analysis. Two other test methods for measuring strains and transformation temperatures under constant load or free recovery after prestraining have only recently been standardized—ASTM E3097: Standard Test Method for Mechanical Uniaxial Constant Force Thermal Cycling of SMAs (UCFTC) and ASTM E3098: Standard Test Method for Mechanical Uniaxial Pre-Strain and Thermal Free Recovery of SMAs (UPFR). These standards represent a critical step forward in producing reliable material property data for the use of these alloys in aeronautics and other commercial areas. However, no standard programs or software packages were previously available for postprocessing UCFTC- and UPFR-type test data to extract property data. Each organization that performed the testing analyzed data using its own custom manual methods, such as using a plot and a straight edge, spreadsheet plotting methods, or custom code routines, to extract the required values for strains, stresses, and transformation temperatures. SMAnalytics represents the first publicly available uniform software package for processing data generated using the DSC, UCFTC, and UPFR test methods, as well as many modifications of these methods, including loading in martensite versus austenite and performing multiple thermal cycles at stress. In doing so, it provides a tool for reducing potential error or variability in the measured values due to the precision of the technique used and person-to-person variability. It is also expected that this automated data-parsing tool can accelerate the data analysis phase of the test campaign, especially for large data files. This User’s Manual describes how the SMAnalytics software works and the method for its use.

Shape Memory Alloy↗