Search NASA⌕ Search

SEARCH · Search NASA

Results for “large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,441 records · Page 80

Data indexing techniques for the EUVE all-sky survey

This poster describes techniques developed for manipulating large full-sky data sets for the Extreme Ultraviolet Explorer project. The authors have adapted the quatrilateralized cubic sphere indexing algorithm to allow us to efficiently store and process several types of large data sets, such as full-sky maps of photon counts, exposure time, and count rates. A variation of this scheme is used to index sparser data such as individual photon events and viewing times for selected areas of the sky, which are eventually used to create EUVE source catalogs.

Lewis, J.↗

The CCD/Transit Instrument (CTI) data-analysis system

The automated software system for archiving, analyzing, and interrogating data from the CCD/Transit Instrument (CTI) is described. The CTI collects up to 450 Mbytes of image-data each clear night in the form of a narrow strip of sky observed in two colors. The large data-volumes and the scientific aims of the project make it imperative that the data are analyzed within the 24-hour period following the observations. To this end a fully automatic and self evaluating software system has been developed. The data are collected from the telescope in real-time and then transported to Tucson for analysis. Verification is performed by visual inspection of random subsets of the data and obvious cosmic rays are detected and removed before permanent archival is made to the optical disc. The analysis phase is performed by a pair of linked algorithms, one operating on the absolute pixel-values and the other on the spatial derivative of the data. In this way both isolated and merged images are reliably detected in a single pass. In order to isolate the latter algorithm from the effects of noise spikes a 3x3 Hanning filter is applied to the raw data before the analysis is run. The algorithms reduce the input pixel-data to a database of measured parameters for each image which has been found. A contrast filter is applied in order to assign a detection-probability to each image and then x-y calibration and intensity calibration are performed using known reference stars in the strip. These are added to as necessary by secondary standards boot-strapped from the CTI data itself. The final stages involve merging the new data into the CTI Master-list and History-list and the automatic comparison of each new detection with a set of pre-defined templates in parameter-space to find interesting objects such as supernovae, quasars and variable stars. Each stage of the processing from verification to interesting image selection is performed under a data-logging system which both controls the pipe-lining of data through the system and records key performance monitor parameters which are built into the software. Furthermore, the data from each stage are stored in databases to facilitate evaluation, and all stages offer the facility to enter keyword-indexed free-format text into the data-logging system. In this way a large measure of certification is built into the system to provide the necessary confidence in the end results.

Cawson, M. G. M.↗

Large-Scale Grid Models: Synthetic Data and Co-Simulation

Power grid modernization and wide-spread distributed energy resource (DER) integration are driving an explosion of interest in advanced algorithms and new optimization schemes. However, there has historically been a lack of open-access realistically sized and detailed datasets-particularly for the distribution system-and simulation tools able to model these systems. This talk will present recent advances with very large-scale (>>1M nodes) distribution data sets and the use of co-simulation to tractably model large integrated transmission-distribution grids and their interactions with new controls/optimization.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Fuzzy Linguistic Knowledge Based Behavior Extraction for Building Energy Management Systems

Significant portion of world energy production is consumed by building Heating, Ventilation and Air Conditioning (HVAC) units. Thus along with occupant comfort, energy efficiency is also an important factor in HVAC control. Modern buildings use advanced Multiple Input Multiple Output (MIMO) control schemes to realize these goals. However, since the performance of HVAC units is dependent on many criteria including uncertainties in weather, number of occupants, and thermal state, the performance of current state of the art systems are sub-optimal. Furthermore, because of the large number of sensors in buildings, and the high frequency of data collection, large amount of information is available. Therefore, important behavior of buildings that compromise energy efficiency or occupant comfort is difficult to identify. This paper presents an easy to use and understandable framework for identifying such behavior. The presented framework uses human understandable knowledge-base to extract important behavior of buildings and present it to users via a graphical user interface. The presented framework was tested on a building in the Pacific Northwest and was shown to be able to identify important behavior that relates to energy efficiency and occupant comfort.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

GOES satellite data maps areas of extreme cold in Colorado

Geostationary Operational Environmental Satellite (GOES) enhanced infrared (IR) imagery depicted very cold temperatures over Colorado on the morning of 8 December 1978. The situation was unusual because skies were clear and the cold temperatures were not associated with high cloud tops. Instead, satellite data mapped large areas that were experiencing extremely cold surface temperatures. The GOES data were also examined using the Colorado State University interactive data processing system and it was found that the cold IR readings corresponded well with early morning low temperatures over the state. GOES data can be of use in monitoring surface temperatures and can, in certain situations, provide detailed spatial and temporal information over regions experiencing extreme temperatures.

Maddox, R. A.↗

Exploratory Data analysis ENvironment eXtreme scale (EDENx)

EDENx is a multivariate data visualization tool that allows interactive user driven analysis of large-scale data sets with high dimensionality. EDENx builds on our earlier system, called EDEN to enable analysis of more dimensions and larger scale data sets. EDENx provides an initial overview of summary statistics for each variable in the data set under investigation. EDENx allows the user to interact with graphical summary plots of the data to investigate subsets and their statistical associations. These plots include histograms, binned scatterplots, binned parallel coordinate plots, timeline plots, and graphical correlation indicators. From the EDENx interface, a user can select a subsample of interest and launch a more detailed data visualization via the EDEN system. EDENx is best suited for high-level, aggregate analysis tasks while EDEN is more appropriate for detail data investigations.

Steed, Chad Allen↗

Evolution of HEP Processing Frameworks

HEP data-processing software must support the disparate physics needs of many experiments. For both collider and neutrino environments, HEP experiments typically use data-processing frameworks to manage the computational complexities of their large-scale data processing needs. Data-processing frameworks are being faced with new challenges this decade. The computing landscape has changed from the past three decades of homogeneous single-core x86 batch jobs running on grid sites. Frameworks must now work on a heterogeneous mixture of different platforms: multi-core machines, different CPU architectures, and computational accelerators; and different computing sites: grid, cloud, and high-performance computing. We describe these challenges in more detail and how frameworks may confront them. Given their historic success, frameworks will continue to be critical software systems that enable HEP experiments to meet their computing needs. Frameworks have weathered computing revolutions in the past; they will do so again with support from the HEP community

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

A case study on parallel HDF5 dataset concatenation for high energy physics data analysis

In High Energy Physics (HEP), experimentalists generate large volumes of data that, when analyzed, helps us better understand the fundamental particles and their interactions. This data is often captured in many files of small size, creating a data management challenge for scientists. In order to better facilitate data management, transfer, and analysis on large scale platforms, it is advantageous to aggregate data further into a smaller number of larger files. However, this translation process can consume significant time and resources, and if performed incorrectly the resulting aggregated files can be inefficient for highly parallel access during analysis on large scale platforms. In this paper, we present our case study on parallel I/O strategies and HDF5 features for reducing data aggregation time, making effective use of compression, and ensuring efficient access to the resulting data during analysis at scale. We focus on NOvA detector data in this case study, a large-scale HEP experiment generating many terabytes of data. Here, the lessons learned from our case study inform the handling of similar datasets, thus expanding community knowledge related to this common data management task.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Fractals and Spatial Methods for Mining Remote Sensing Imagery

The rapid increase in digital remote sensing and GIS data raises a critical problem -- how can such an enormous amount of data be handled and analyzed so that useful information can be derived quickly? Efficient handling and analysis of large spatial data sets is central to environmental research, particularly in global change studies that employ time series. Advances in large-scale environmental monitoring and modeling require not only high-quality data, but also reliable tools to analyze the various types of data. A major difficulty facing geographers and environmental scientists in environmental assessment and monitoring is that spatial analytical tools are not easily accessible. Although many spatial techniques have been described recently in the literature, they are typically presented in an analytical form and are difficult to transform to a numerical algorithm. Moreover, these spatial techniques are not necessarily designed for remote sensing and GIS applications, and research must be conducted to examine their applicability and effectiveness in different types of environmental applications. This poses a chicken-and-egg problem: on one hand we need more research to examine the usability of the newer techniques and tools, yet on the other hand, this type of research is difficult to conduct if the tools to be explored are not accessible. Another problem that is fundamental to environmental research are issues related to spatial scale. The scale issue is especially acute in the context of global change studies because of the need to integrate remote-sensing and other spatial data that are collected at different scales and resolutions. Extrapolation of results across broad spatial scales remains the most difficult problem in global environmental research. There is a need for basic characterization of the effects of scale on image data, and the techniques used to measure these effects must be developed and implemented to allow for a multiple scale assessment of the data before any useful process-oriented modeling involving scale-dependent data can be conducted. Through the support of research grants from NASA, we have developed a software module called ICAMS (Image Characterization And Modeling System) to address the need to develop innovative spatial techniques and make them available to the broader scientific communities. ICAMS provides new spatial techniques, such as fractal analysis, geostatistical functions, and multiscale analysis that are not easily available in commercial GIS/image processing software. By bundling newer spatial methods in a user-friendly software module, researchers can begin to test and experiment with the new spatial analysis methods and they can gauge scale effects using a variety of remote sensing imagery. In the following, we describe briefly the development of ICAMS and present application examples.

Lam, Nina↗

Visualizing Electrical Power Systems as Flow Fields

We describe a method for visualizing data flows on large networks. We transform data flow on fixed networks into a vector field, which can be directly visualized using scientific flow visualization techniques. We evaluated the method on power flowing through two transmission power networks: a small, regional, IEEE test system (RTS-96) and a large national-scale system (the Eastern Interconnection). For the larger and more complex transmission system, the method illustrates features of the power flow that are not accessible when visualizing the power transmission with traditional network visualization techniques.

97 MATHEMATICS AND COMPUTING↗

Stride search: A general algorithm for storm detection in high-resolution climate data

This study discusses the problem of identifying extreme climate events such as intense storms within large climate data sets. The basic storm detection algorithm is reviewed, which splits the problem into two parts: a spatial search followed by a temporal correlation problem. Two specific implementations of the spatial search algorithm are compared: the commonly used grid point search algorithm is reviewed, and a new algorithm called Stride Search is introduced. The Stride Search algorithm is defined independently of the spatial discretization associated with a particular data set. Results from the two algorithms are compared for the application of tropical cyclone detection, and shown to produce similar results for the same set of storm identification criteria. Differences between the two algorithms arise for some storms due to their different definition of search regions in physical space. The physical space associated with each Stride Search region is constant, regardless of data resolution or latitude, and Stride Search is therefore capable of searching all regions of the globe in the same manner. Stride Search's ability to search high latitudes is demonstrated for the case of polar low detection. Wall clock time required for Stride Search is shown to be smaller than a grid point search of the same data, and the relative speed up associated with Stride Search increases as resolution increases.

54 ENVIRONMENTAL SCIENCES↗

A parallel data management system for large-scale NASA datasets

The past decade has experienced a phenomenal growth in the amount of data and resultant information generated by NASA's operations and research projects. A key application is the reprocessing problem which has been identified to require data management capabilities beyond those available today (PRAT93). The Intelligent Information Fusion (IIF) system (ROEL91) is an ongoing NASA project which has similar requirements. Deriving our understanding of NASA's future data management needs based on the above, this paper describes an approach to using parallel computer systems (processor and I/O architectures) to develop an efficient parallel database management system to address the needs. Specifically, we propose to investigate issues in low-level record organizations and management, complex query processing, and query compilation and scheduling.

Srivastava, Jaideep↗

Arm DevSummit Keynote: Environment for Data Engineering in Virtual Reality: Ethical Considerations

Currently, the US Government is going through a large-scale data Transformation effort, where the GSA playbook is guiding all agencies to make their processes data-driven with the help of emerging technologies. To address concerns about algorithm sharing, AI adoption, and vendor lock-in, the NASA Langley Research Center Digital Transformation Group has developed the Environment for Data Engineering in Virtual Reality (EnDEVR), a data science ecosystem that allows users to command and investigate customizable data analyses from a VR environment. The system has been evaluated in the Oculus Rift S and Quest environments, two popular VR systems powered by the ARM architecture. Current and future development will employ several AI capabilities to guide research and automation within the environment and to facilitate algorithm sharing. In this talk, we will discuss the results of an initial ethical investigation and recommended considerations for the use of AI within the system.

Artificial Intelligence↗

Using Public Participation to Improve MELs Energy Data Collection

Miscellaneous and electronic loads (MELs) comprise an increasing share of building energy consumption. Large-scale data collection is needed to inform meaningful energy reduction strategies because of the diversity of MELs and our lack of understanding about how people use them. Traditional methods of data collection, however, usually incur high labor and metering equipment expenses. As an alternative, this paper investigates the feasibility of crowdsourcing data collection to satisfy at least part of the data collection needs with acceptable accuracy. We assessed the reliability and accuracy of crowd-sourced data by recruiting 18 volunteers and testing our crowdsourcing protocol. The protocol asked volunteers to perform measurement tasks for three MELs devices of increasing complexity 1) record power meter and MELs product characteristics, 2) identify and measure all power modes available, and 3) report the measured power. Volunteers performed reasonably well for devices with functionalities with which they were familiar, but many could not correctly identify all available power modes in complex devices. Accuracy may improve when participants measure the power used by familiar devices in their home, or by providing more specific instructions, e.g. videos. Furthermore, crowdsourcing data collection from individual homeowners has the potential to generate valuable information about MELs energy use in homes when integrated with existing programs such as Home Energy Saver and Building America.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Improving the LandScan USA Non-Obligate Population Estimate (NOPE)

Where do people go when they have nowhere to be? Nonobligate activities are a significant part of our social and cultural lives, but there are no existing large scale data which characterize spatial variability in population allocation for these activities. As large scale population estimates have ever-finer resolutions, gaps in our ability to estimate this population segment have an increasingly large impact on high resolution population estimates. In this paper, we demonstrate an improved method for estimating the spatial allocation of the non-obligate population - people who are not at work, school, or in another residential institution. This method builds upon on anonymized and aggregate data on visits to public places, allocating the non-obligate population proportionally to worker population while accounting for the estimated ratio of visitors to workers in public places.

Brelsford, Christa↗

Non-negative Tensor Factorization for Robust Exploratory Big-Data Analytics

Currently, large multidimensional datasets are being accumulated in almost every field. Data are: (1) collected by distributed sensor networks in real-time all over the globe, (2) produced by large-scale experimental measurements or engineering activities, (3) generated by high-performance simulations, and (4) gathered by electronic communications and socialnetwork activities, etc. Simultaneous analysis of these ultra-large heterogeneous multidimensional datasets is often critical for scientific discoveries, decision-making, emergency response, and national and global security. The importance of such analyses mandates the development of the next-generation of robust machine learning (ML) methods and tools for bigdata exploratory analysis.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Policy Considerations When Federating Facilities for Experimental and Observational Data Analysis

Today’s computational, experimental, and observational facilities afford us tremendous opportunities to couple theory and experiment at increasingly large scales. Empirical sensing capabilities are growing dramatically with beam line and detector improvements, and with advances in our ability to deploy large-scale data gathering observations of the natural world. The coupling of computational simulations and analysis to process the data from experimental and observational facilities is giving rise to cross-facility workflows. Such federations of facilities are in fact becoming an explicit requirement for large-scale scientific discovery. As we scale up these pipelines of scientific discovery, each participating facility needs to establish and align policies so that the federation can work seamlessly in an end-to-end manner. This chapter outlines specific policy considerations in enabling the federation of facilities for data analysis. Design choices and vital policy decisions cover the areas of data acquisition and storage, data transfer, computational resource allocation and co-scheduling, seamless federated user access, and cross-cutting governance. By highlighting the explicit and implicit interdependencies between facilities, we aim to provide facility designers and policymakers the information on policy issues to address early in a facility’s operations, thus enabling successful cross-facility federation and improved experimental and observational data analysis outcomes.

Shankar, Mallikarjun (Arjun)↗