Search NASA⌕ Search

SEARCH · Search NASA

Results for “Large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Clusters of galaxies up to z = 1.5 identified from photometric data of the Dark Energy Survey and unWISE

ABSTRACT Using photometric data from the Dark Energy Survey and the Wide-field Infrared Survey Explorer, we estimate photometric redshifts for 105 million galaxies using the nearest-neighbour algorithm. From such a large data base, 151 244 clusters of galaxies are identified in the redshift range of 0.1 < z ≲ 1.5 based on the overdensity of the total stellar mass of galaxies within a given photometric redshift slice, among which 76 826 clusters are newly identified and 30 477 clusters have a redshift z > 1. We cross-match these clusters with those in the catalogues identified from the X-ray surveys and the Sunyaev–Zel’dovich (SZ) effect by the Planck, South Pole Telescope and Atacama Cosmology Telescope surveys, and get the redshifts for 45 X-ray clusters and 56 SZ clusters. More than 95 per cent SZ clusters in the sky region have counterparts in our catalogue. We find multiple optical clusters in the line of sight towards about 15 per cent of SZ clusters.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

TEACHING AN OLD ACCELERATOR NEW TRICKS

The Argonne Tandem Linac Accelerator System (ATLAS) has been a National User Facility since 1985. In that time, many of the systems that help operators retrieve, modify, and store beamline parameters have not kept pace with the advancement of technology. Development of a new method of storing and retrieving beamline parameters resulted in the testing and installation of a time-series database as a potential replacement for the traditional relational database. InfluxDB was selected due to its self-hosted Open-Source version availability as well as the simplicity of installation and setup. A program was written to periodically gather all accelerator parameters in the control system and store them in the time-series database. This resulted in over 13,000 distinct data points, captured at 5-minute intervals. A second test captured 35 channels on a 1-minute cadence. Graphing of the captured data is being done on Grafana, an Open-Source version is available that co-exists well with InfluxDB as the back-end. Grafana made visualizing the data simple and flexible. The testing has allowed for the use of modern graphing tools to generate new insights into operating the accelerator, as well as opened the door to building large data sets suitable for Artificial Intelligence and Machine Learning applications.

Novak, D.↗
NASA NTRS1974

An analysis of pilot error-related aircraft accidents

A multidisciplinary team approach to pilot error-related U.S. air carrier jet aircraft accident investigation records successfully reclaimed hidden human error information not shown in statistical studies. New analytic techniques were developed and applied to the data to discover and identify multiple elements of commonality and shared characteristics within this group of accidents. Three techniques of analysis were used: Critical element analysis, which demonstrated the importance of a subjective qualitative approach to raw accident data and surfaced information heretofore unavailable. Cluster analysis, which was an exploratory research tool that will lead to increased understanding and improved organization of facts, the discovery of new meaning in large data sets, and the generation of explanatory hypotheses. Pattern recognition, by which accidents can be categorized by pattern conformity after critical element identification by cluster analysis.

Kowalsky, N. B.↗
NASA NTRS1979

Recombination of NO/+/ in the mid-latitude trough and the polar ionization hole

The dissociative recombination of NO(+) with electrons is further examined using a sample of Atmosphere Explorer data from the midlatitude ionization trough. The electron temperature for this data sample covered a range of 2500 K, thus allowing an accurate determination of the temperature dependence of the rate coefficient. The results confirm earlier determinations which were made from a large data base, and the rate coefficient alpha is found to be inversely proportional to the electron temperature. The possibility of a neglected process in the NO(+) chemistry is investigated which, if it were included, might result in a smaller temperature dependence of alpha. Such a source would require a direct dependence on electron temperature. A possible candidate is associative recombination of energetic N(D-2) with O. The results confirm recent measurements which find that this process cannot be significant. The NO(+) chemistry is a very sensitive indicator of chemical equilibrium conditions. By using measurements of the relevant parameters made in the polar ionization hole it is found that horizontal drifts do not appear to be significant in this region.

Torr, M. R.↗
NASA NTRS1987

Contextual classification on the massively parallel processor

Classifiers are often used to produce land cover maps from multispectral Earth observation imagery. Conventionally, these classifiers have been designed to exploit the spectral information contained in the imagery. Very few classifiers exploit the spatial information content of the imagery, and the few that do rarely exploit spatial information content in conjunction with spectral and/or temporal information. A contextual classifier that exploits spatial and spectral information in combination through a general statistical approach was studied. Early test results obtained from an implementation of the classifier on a VAX-11/780 minicomputer were encouraging, but they are of limited meaning because they were produced from small data sets. An implementation of the contextual classifier is presented on the Massively Parallel Processor (MPP) at Goddard that for the first time makes feasible the testing of the classifier on large data sets.

Tilton, James C.↗
NASA NTRS2002

Telescope Scientist on the Advanced X-Ray Astrophysics Observatory

This period included many scientific observations made with the Chandra Observatory. The results, as is well known, are spectacular. Fortunately, the HRMA performance continues to be essentially identical to that predicted from ground calibration data. The Telescope Scientist Team has improved the mirror model to provide a more accurate description to the Chandra observers and enable them to reduce the systematic errors and uncertainties in their data reduction. There also has been progress in the scientific program. At this time 47 distant clusters of galaxies have been observed. We are performing a systematic analysis of this rather large data set for the purpose of determining absolute distances utilizing the Sunyaev Zel'dovich effect.

VanSpeybroeck, L.↗
NASA NTRS2009

Data Mining at NASA: From Theory to Applications

This slide presentation demonstrates the data mining/machine learning capabilities of NASA Ames and Intelligent Data Understanding (IDU) group. This will encompass the work done recently in the group by various group members. The IDU group develops novel algorithms to detect, classify, and predict events in large data streams for scientific and engineering systems. This presentation for Knowledge Discovery and Data Mining 2009 is to demonstrate the data mining/machine learning capabilities of NASA Ames and IDU group. This will encompass the work done re cently in the group by various group members.

Srivastava, Ashok N.↗
NASA NTRS1985

Extended testing of a general contextual classifier using the massively parallel processor - Preliminary results and test plans

Earlier encouraging test results of a contextual classifier that combines spatial and spectral information employing a general statistical approach are expanded. The earlier results were of limited meaning because they were produced from small (50-by-50 pixel) data sets. An implementation of the contextual classifier on NASA Goddard's Massively Parallel Processor (MPP) is presented; for the first time the MPP makes feasible the testing of the classifier on large data sets (a 12-hour test on a VAX-11/780 minicomputer now takes 5 minutes on the MPP). The MPP is a Single-Instruction, Multiple Data Stream computer, consisting of 16,384 bit serial microprocessors connected in a 128-by-128 mesh array with each element having data transfer connections with its four nearest neighbors so that the MPP is capable of billions of operations per second. Preliminary results are given (with more expected for the conference) and plans are mentioned for extended testing of the contextual classifier on Thematic Mapper data sets.

Tilton, J. C.↗
NASA NTRS2019

Ask-The-Expert: Minimizing Human Review for Big Data Analytics Through Active Learning

In this CIF project, we worked toward semi-automating knowledge discovery from anomaly detection algorithms through the use of active learning. Active learning is an area of research within machine learning that uses an "expert in the loop" to learn from large data sets that have very few annotations or labels available, and where providing such labels is expensive. In our case, the task can be defined as the identification of safety events from flight operational data. Since traditional anomaly detection algorithms cannot differentiate between operationally relevant and irrelevant statistical anomalies, Subject Matter Experts (SMEs) have a lengthy and expensive burden of investigating every example identified by the detection algorithm, classifying and labeling them as relevant or irrelevant. Active learningidentifies the unlabeled example for which a label would most improve the classifier, asks the domain expert for a label, and repeats this process until there are no more resources (time, budget) available for labeling or a minimum required performance is reached. A positive label indicates an operationally significant safety event whereas a negative label indicates otherwise. Based on these few labels we propose to build an active learning system that utilizes the SME's time in the most effective manner by iteratively asking for labels for as few informative instances as possible. Our work was proposed to be a stepping stone toward implementation and deployment of the system with user interface to be pursued by the Aviation Operations and Safety Program (AOSP) given its interest in safety monitoring and discovery of safety incidents.

aviation safety↗
DOE OSTI2023

Data and Analysis Preservation in the PHENIX Experiment at RHIC

The PHENIX experiment (the Pioneering High Energy Nuclear Interaction eXperiment) is the largest of the four experiments that have operated at the Relativistic Heavy Ion Collider (RHIC), taking data in 2000-2016. PHENIX has made fundamental contributions to the discovery and study of the Quark-Gluon Plasma, advancement of spin physics and other areas. Currently, the PHENIX Collaboration is analyzing large data samples previously collected, while facing challenges in the area of Data and Analysis Preservation. We describe the strategy and practices employed by the PHENIX Collaboration to meet these challenges by leveraging state-of-the-art platforms and tools created and maintained by the High Energy and Nuclear Physics communities, including Zenodo, HEPData, OpenData, REANA and others.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2018

Examining Dense Data Usage near the Regions with Severe Storms in All-Sky Microwave Radiance Data Assimilation and Impacts on GEOS Hurricane Analyses

Many numerical weather prediction (NWP) centers assimilate radiances affected by clouds and precipitation from microwave sensors, with the expectation that these data can provide critical constraints on meteorological parameters in dynamically sensitive regions to make significant impacts on forecast accuracy for precipitation. The Global Modeling and Assimilation Office (GMAO) at NASA Goddard Space Flight Center assimilates all-sky microwave radiance data from various microwave sensors such as all-sky GPM Microwave Imager (GMI) radiance in the Goddard Earth Observing System (GEOS) atmospheric data assimilation system (ADAS), which includes the GEOS atmospheric model, the Gridpoint Statistical Interpolation (GSI) atmospheric analysis system, and the Goddard Aerosol Assimilation System (GAAS). So far, most of NWP centers apply same large data thinning distances, that are used in clear-sky radiance data to avoid correlated observation errors, to all-sky microwave radiance data. For example, NASA GMAO is applying 145 km thinning distances for most of satellite radiance data including microwave radiance data in which all-sky approach is implemented. Even with these coarse observation data usage in all-sky assimilation approach, noticeable positive impacts from all-sky microwave data on hurricane track forecasts were identified in GEOS-5 system. The motivation of this study is based on the dynamic thinning distance method developed in our all-sky framework to use of denser data in cloudy and precipitating regions due to relatively small spatial correlations of observation errors. To investigate the benefits of all-sky microwave radiance on hurricane forecasts, several hurricane cases selected between 2016-2017 are examined. The dynamic thinning distance method is utilized in our all-sky approach to understand the sources and mechanisms to explain the benefits of all-sky microwave radiance data from various microwave radiance sensors like Advanced Microwave Sounder Unit (AMSU-A), Microwave Humidity Sounder (MHS), and GMI on GEOS-5 analyses and forecasts of various hurricanes.

Kim, Min-Jeong↗
NASA NTRS2019

Accelerated Simulation of Air Pollution Using NVIDIA RAPIDS

Atmospheric chemistry models are a central tool to study and forecast the impact of air pollution on the environment, vegetation, and human health. However, the numerical simulation of chemical kinetics is computationally expensive due to the stiffness of the system of ordinary differential equations that describes atmospheric chemistry. Here we present an alternative approach to the computation of atmospheric chemistry based on machine learning. Our training data set is produced using the NASA Goddard Earth Observing System (GEOS) model with GEOS-Chem chemistry, run on the NASA Center for Climate Simulation (NCCS) Discover supercomputing cluster on 384 Intel Xeon Haswell cores. This model spends more than 50% of total run time on solving atmospheric chemistry. The data set contains as input features the air pollution concentrations before solving the differential equations, together with some key physical parameters such as temperature and sun intensity. As target variables we define the air pollution concentrations after solving the differential equations. Using Dask-cuDF and Dask-XGBoost on the NVIDIA RAPIDS platform on 8 Tesla V100 GPUs, we generate from this training set gradient boosted decision tree models that can reproduce the simulation of chemical kinetics. We do this on the NCCS Advanced Data Analytics Platform (ADAPT) science cloud environment. Our application takes full advantage of recent advances in Dask-XGBoost, such as multi-node and multi-GPU scaling for distributed training with large data sets. The increase in training data size enabled by this is critical to capture the full range of chemical environments encountered across the globe and all annual seasons.The boosted tree models offer good predictability and show many of the features of the full chemistry reference simulation. Further improvements can be achieved through mass balance considerations and by accounting for error correlations. We incorporate the boosted tree models into the GEOS reference model using XGBoost's C API. This enables a seamless integration of the GPU trained models into GEOS-Chem, which is written in Fortran and optimized for use in a massively parallel CPU environment. We show the benefits of this approach and discuss the potential speedup of this machine learning accelerated atmospheric chemistry model.

Keller, Christoph A.↗
NASA NTRS2012

The Cabauw Intercomparison Campaign for Nitrogen Dioxide Measuring Instruments (CINDI): Design, Execution, and Early Results

From June to July 2009 more than thirty different in-situ and remote sensing instruments from all over the world participated in the Cabauw Intercomparison campaign for Nitrogen Dioxide measuring Instruments (CINDI). The campaign took place at KNMI's Cabauw Experimental Site for Atmospheric Research (CESAR) in the Netherlands. Its main objectives were to determine the accuracy of state-ofthe- art ground-based measurement techniques for the detection of atmospheric nitrogen dioxide (both in-situ and remote sensing), and to investigate their usability in satellite data validation. The expected outcomes are recommendations regarding the operation and calibration of such instruments, retrieval settings, and observation strategies for the use in ground-based networks for air quality monitoring and satellite data validation. Twenty-four optical spectrometers participated in the campaign, of which twenty-one had the capability to scan different elevation angles consecutively, the so-called Multi-axis DOAS systems, thereby collecting vertical profile information, in particular for nitrogen dioxide and aerosol. Various in-situ samplers and lidar instruments simultaneously characterized the variability of atmospheric trace gases and the physical properties of aerosol particles. A large data set of continuous measurements of these atmospheric constituents has been collected under various meteorological conditions and air pollution levels. Together with the permanent measurement capability at the CESAR site characterizing the meteorological state of the atmosphere, the CINDI campaign provided a comprehensive observational data set of atmospheric constituents in a highly polluted region of the world during summertime. First detailed comparisons performed with the CINDI data show that slant column measurements of NO2, O4 and HCHO with MAX-DOAS agree within 5 to 15%, vertical profiles of NO2 derived from several independent instruments agree within 25% of one another, and MAX-DOAS aerosol optical thickness agrees within 20-30% with AERONET data. For the in-situ NO2 instrument using a molybdenum converter, a bias was found as large as 5 ppbv during day time, when compared to the other in-situ instruments using photolytic converters.

Atmospheric composition↗
NASA NTRS1989

Satellite/rocket ozone comparisons at Natal, Brazil

Comparisons are presented of satellite, rocket, and balloon ozone profiles near Natal, Brazil (5.9 deg S, 35.2 deg W). The low variability of stratospheric ozone at Natal during March and April of 1985 has allowed intercomparisons of reasonably large data sets, rather than a small number of paired satellite/in situ comparisons. There are sharp differences between the profile from the SBUV instrument on Nimbus 7 and the in situ measurements. These results support the conclusions of the NASA Ozone Trends Panel that there is an instrumental cause for the very large changes in upper stratospheric ozone seen by SBUV. Along with other comparisons, these results are being used in a reassessment of the SBUV instrument and its data reduction procedures. The agreement between the ozone profiles from the SAGE II instrument on the ERBS satellite and the rocket values is excellent over the full range of comparisons. Both SAGE II and ROCOZ-A must convert from altitude to pressure for intercomparisons with SME and with SBUV-type instruments. The conversion between pressure and altitude is as important as the ozone measurements, especially in the upper stratosphere where the scale height for ozone is approximately half that for pressure.

Barnes, Robert A.↗
NASA NTRS2004

GPS FOM Chimney Analysis using Generalized Extreme Value Distribution

Many a time an objective of a statistical analysis is to estimate a limit value like 3-sigma 95% confidence upper limit from a data sample. The generalized Extreme Value Distribution method can be profitably employed in many situations for such an estimate. . .. It is well known that according to the Central Limit theorem the mean value of a large data set is normally distributed irrespective of the distribution of the data from which the mean value is derived. In a somewhat similar fashion it is observed that many times the extreme value of a data set has a distribution that can be formulated with a Generalized Distribution. In space shuttle entry with 3-string GPS navigation the Figure Of Merit (FOM) value gives a measure of GPS navigated state accuracy. A GPS navigated state with FOM of 6 or higher is deemed unacceptable and is said to form a FOM 6 or higher chimney. A FOM chimney is a period of time during which the FOM value stays higher than 5. A longer period of FOM of value 6 or higher causes navigated state to accumulate more error for a lack of state update. For an acceptable landing it is imperative that the state error remains low and hence at low altitude during entry GPS data of FOM greater than 5 must not last more than 138 seconds. I To test the GPS performAnce many entry test cases were simulated at the Avionics Development Laboratory. Only high value FoM chimneys are consequential. The extreme value statistical technique is applied to analyze high value FOM chimneys. The Maximum likelihood method is used to determine parameters that characterize the GEV distribution, and then the limit value statistics are estimated.

Ott, Rick↗
NASA NTRS1991

Physical processes in comets

Post-Halley comets are known to be irregular objects with most nucleus activity very localized and with the dust coma capable of fragmentation and apparently being a source of gas. Older, 1-D strategies which assume steady isotropic outflow of material can give poor time-and-space averaged results, at best. With 2-D data, images through interference filters, it is hoped that: dust structures can be seen that give evidence of the proper geometry for data reduction; gradients along the axes of symmetry and evidence of fragmentation can be studied; and that evidence for gas abundance gradients associated with the dust can be found. High quality data from brighter comets can then be used to suggest improved data reduction procedures for fainter ones. To obtain such data, large image-quality interference filters were procured for use with a charge coupled device (CCD) camera at Lick Observatory, where the scale of the 1 m Nickel reflector is ideal for brighter comets. Whenever possible, data is taken simultaneously with other telescopes and equipment, especially spectroscopy at the Lick 3 m or infrared photometry at the Infrared Telescope Facility (IRTF) on Mauna Kea.

Newburn, Ray L., Jr.↗
DOE OSTI2024

Randomized low-rank decompositions of nuclear three-body interactions

First-principles simulations of many-fermion systems are commonly limited by the computational requirements of processing large data objects. As a remedy, we propose the use of low-rank approximations of three-body interactions, which are the dominant such limitation in nuclear physics. We introduce a randomized decomposition technique to handle the excessively large matrix dimensions and study the sensitivity of low-rank properties to interaction details. The developed low-rank three-nucleon interactions are benchmarked in ab initio simulations of few- and many-body systems. Exploiting low-rank properties provides a promising route to extend the microscopic description of atomic nuclei to large systems where storage requirements exceed the computational capacities of the most advanced high-performance computing facilities.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

Robust Online Sequential RVFLNs for Data Modeling of Dynamic Time-Varying Systems with Application of an Ironmaking Blast Furnace

In a world where the increasing complexity of modern industrial processes brings difficulties for accurate mathematical modeling, taking advantage of data has become an efficient solution to complex dynamic process modeling issue. In this paper, we develop a novel robust online sequential version of random vector functional-link networks (RVFLNs) for data-driven modeling of dynamic time-varying system and applied it in a blast furnace (BF) ironmaking process. First, to overcome the time-varying dynamics of process and to enable the RVFLNs to learn online with avoiding data saturation, an improved online sequential version of RVFLNs (OS-RFVLNs) is first presented by online sequential learning with forgetting factor. This improved OS-RVFLNs algorithm is not only suitable for the real-time and large data transfer situation, but also can adjust the sensitivity of the algorithm to different samples with the help of the introduced forgetting factor. Second, since the output weights of the improved OS-RVFLNs as well as other RVFLNs algorithms are obtained by the least squares approach, a robustness problem may occur when the training dataset is contaminated with various outliers. To solve this problem, a Cauchy distribution weighted M-estimator is introduced to improve the robustness of the improved OS- RVFLNs. For this proposed robust OS-RVFLNs (R-OS- RVFLNs), since the weights of different outlier data are properly determined by the Cauchy distribution function, their corresponding contribution on modeling can be properly distinguished. Thus robust and better modeling results can be achieved. Experiments using actual industrial data of BF ironmaking process and comparative studies have demonstrated that the proposed method produces a better estimation accuracy and stronger robustness than other methods.

Blast furnace (BF), Modelling, Dynamic systems↗