Search NASA⌕ Search

SEARCH · Search NASA

Results for “large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14

perf-dump

perf-dump is a library for dumping performance data in much the same way physics simulations dump checkpoints. It records per-process, per-timestep, per-phase, and per-thread performance counter data and dumps this large data periodically into an HDF5 data file.

Gamblin, T.↗

Roundtable on Producing and Managing Large Scientific Data with Artificial Intelligence and Machine Learning (Brochure)

The US Department of Energy’s (DOE’s) twelve Basic Energy Sciences (BES) scientific user facilities provide access to the world’s most advanced research instruments, annually serving over 16,000 users with impact reported in nearly 7,000 publications and producing unprecedented quantities of scientific data. While impressive, reaching the full potential of these rapidly growing facilities will require new innovations to solve a variety of technical challenges in data acquisition, control, modeling, and analysis. Artificial intelligence and machine learning (AI/ML) have opened corresponding new avenues in optimization, efficient surrogate models, data analytics, and inverse problems. These intriguing capabilities suggest that AI/ML can greatly accelerate the quest to probe and understand fundamental phenomena across a vast range of lengths and timescales, potentially leading to transformative advances across scientific disciplines.

97 MATHEMATICS AND COMPUTING↗

Integrated Large-Scale Data Management Platform for Photovoltaic Power Conversion Equipment (PCE) Reliability Data: Preprint

To meet the demand for accuracy and real-time capability of PV system degradation evaluation, massive volume data is needed to run high-fidelity and high-efficiency simulations and perform advanced data analysis. However, PV farm operators have a series of difficulties with PV inverter data, such as data collection from multiple channels, massive data storage, data management and massive data analysis. To address these challenges, we developed an integrated data management platform capable of data acquisition, processing, storage, query, and performing big data analysis utilizing AI algorithms. The platform can also achieve data correctness verification and provide an effective distributed data management solution to retrieve massive data and establish a connection to distributed computational frameworks.

data management↗

Data management for Large Space Telescope

The data management system for the Large Space Telescope (LST) must be capable of meeting requirements of 160 million bits per 95 min orbit with a bit error rate of less than 0.00001 for data. The system will be supported by the Tracking and Data Relay Satellite System of the Space Tracking and Data Network. The on-board system includes a general purpose computer that controls the vehicle as a stable observation platform and the array of instruments used for data collection. The ground-based system comprises a Mission Operations Center (MOC), a Science Institute where instrument data is processed, and a communications service for the space and point-to-point data flow. The allocation of hardware and software between on-board and ground-based components to achieve design objectives of maximum flexibility at minimum cost is discussed.

Hope, G. R., Jr.↗

Complex Network Analysis and Intelligent Monitoring Platform

Data Products has particular interest in developing an efficient network analysis and monitoring system that utilizes advanced machine learning methods to mine complex network data of the type generated within large data centers, with an objective to provide tangible business-usable intelligence. The Laboratory is currently the lead institution for a U.S. Department of Energy Office of Science (DOE/SC) network research project called BigData Express (BDE). The project seeks to provide an orchestration service for high-performance data movement of data sets for large-scale science. This collaboration is believed to potentially be a highly symbiotic one, with Data Products platform development benefiting from the rich sets of network data available in the BDE development environment, and BDE benefiting from the Data Products machine learning analysis of BDE-generated data movement.

97 MATHEMATICS AND COMPUTING↗

Lossy Data Compression Effects on Wall-bounded Turbulence: Bounds on Data Reduction

Postprocessing and storage of large data sets represent one of the main computational bottlenecks in computational fluid dynamics. We assume that the accuracy necessary for computation is higher than needed for postprocessing. Therefore, in the current work we assess thresholds for data reduction as required by the most common data analysis tools used in the study of fluid flow phenomena, specifically wall-bounded turbulence. These thresholds are imposed a priori by the user in L 2 -norm, and we assess a set of parameters to identify the minimum accuracy requirements. The method considered in the present work is the discrete Legendre transform (DLT), which we evaluate in the computation of turbulence statistics, spectral analysis and resilience for cases highly-sensitive to the initial conditions. Maximum acceptable compression ratios of the original data have been found to be around 97%, depending on the application purpose. Furthermore, the new method outperforms downsampling, as well as the previously explored data truncation method based on discrete Chebyshev transform (DCT).

97 MATHEMATICS AND COMPUTING↗

The Social Life of a Data Base

This paper presents the complex social life of a large data base. The topics include: 1) Social Construction of Mechanisms of Memory; 2) Data Bases: The Invisible Memory Mechanism; 3) The Human in the Machine; 4) Data of the Study: A Large-Scale Problem Reporting Data Base; 5) The PRACA Study; 6) Description of PRACA; 7) PRACA and Paper; 8) Multiple Uses of PRACA; 9) The Work of PRACA; 10) Multiple Forms of Invisibility; 11) Such Systems are Everywhere; and 12) Two Morals to the Story. This paper is in viewgraph form.

Linde, Charlotte↗

A Network Management System for Handling Scientific Data Flows

Large scientific data transfers often occur at high rates causing increased burstiness in Internet traffic. To limit the adverse effects of these high-rate large-sized flows, which are referred to as $\alpha $ flows, on delay-sensitive audio/video flows, a network management system called Alpha Flow Traffic Engineering System (AFTES) is proposed for intra-domain traffic engineering. An offline approach is used in which AFTES analyzes NetFlow records collected by routers, extracts source–destination address prefixes of $\alpha $ flows, and uses these prefixes to configure firewall filters at ingress routers of a provider’s network to redirect future $\alpha $ flows to traffic-engineered paths and isolated queues. The effectiveness of this scheme was evaluated through an analysis of 7 months of NetFlow data obtained from an ESnet router. For this data set, 91 % of bytes generated by $\alpha $ flows during high-rate intervals would have been directed had AFTES been deployed. The negative aspect of using address prefixes in firewall filters, i.e., the redirection of $\beta $ flows to $\alpha $-flow paths/queues, was also quantified.

97 MATHEMATICS AND COMPUTING↗

Development of the Model of Galactic Interstellar Emission for Standard Point-Source Analysis of Fermi Large Area Telescope Data

Most of the celestial γ rays detected by the Large Area Telescope (LAT) aboard the Fermi Gamma-ray Space Telescope originate from the interstellar medium when energetic cosmic rays interact with interstellar nucleons and photons. Conventional point and extended source studies rely on the modeling of this diffuse emission for accurate characterization. We describe here the development of the Galactic Interstellar Emission Model (GIEM) that is the standard adopted by the LAT Collaboration and is publicly available. The model is based on a linear combination of maps for interstellar gas column density in Galactocentric annuli and for the inverse Compton emission produced in the Galaxy. We also include in the GIEM large-scale structures like Loop I and the Fermi bubbles. The measured gas emissivity spectra con rm that the cosmic-ray proton density decreases with Galactocentric distance beyond 5 kpc from the Galactic Center. The measurements also suggest a softening of the proton spectrum with Galactocentric distance. We observe that the Fermi bubbles have boundaries with a shape similar to a catenary at latitudes below 20° and we observe an enhanced emission toward their base extending in the North and South Galactic direction and located within ~4° of the Galactic Center.

79 ASTRONOMY AND ASTROPHYSICS↗

SU-E-T-235: Data Mining for Evaluating Treatment Performances Over a Large Quantity of Data to Monitor and Improve SBRT Workflow

Purpose: To quality assure a large quantity of retrospective treatment cases for treatment performances by randomly sampling is inefficient. Here we provide a method to efficiently monitor and investigate the QA of SBRT workflow over Mosaiq. Methods: The code developed with Microsoft SQL Server Management Studio 2008R2 and VBA was used for retrieving and sorting data from Mosaiq (version 2.3–2.6 during 2012–2015). SBRT patients were filtered by fractional dose over 350cGy and total fraction number less than 6, which SBRT prescriptions were defined. The quality assurance on the SBRT workflow was focused on the treatment deliveries such as patient positioning setup, CBCT indicated offsets and couch shifted corrections. The treatment delivery were done by Varian Truebeam systems and the record/verify by Mosaiq. Results: Total 82 SBRT patients corresponding to 103 courses and 854 CBCT images were found by the retrieval query. Most centers record daily pre-treatment (Pre-Tx: before treatment shift) image-guided shifts along treatment course for inter-fraction motion record, and it is useful to also verify it with post-treatment imaging (Post-Tx: after treatment CBCT verification) to verify intra-fraction motion. Analyzing the details of daily recorded shifts can reveals the information of patient-setup and staff’s record/verify behaviors. 3 examples were provided as solid evidences and on-going rectification for preventing future mistakes. Conclusions: The report gave feasible examples for inspector to verify a large amount of data during site investigation. This program can also be extended to a scheduled data mining with software to periodical analyze the timely records in Mosaiq, for example, a various control charts for different QA purposes. As the current trend of automation in radiation therapy field, the data mining would be a necessary tool in the future, just as the automatic plan quality evaluation has been under development in Eclipse.

60 APPLIED LIFE SCIENCES↗

Using expert systems to implement a semantic data model of a large mass storage system

The successful development of large volume data storage systems will depend not only on the ability of the designers to store data, but on the ability to manage such data once it is in the system. The hypothesis is that mass storage data management can only be implemented successfully based on highly intelligent meta data management services. There now exists a proposed mass store system standard proposed by the IEEE that addresses many of the issues related to the storage of large volumes of data, however, the model does not consider a major technical issue, namely the high level management of stored data. However, if the model were expanded to include the semantics and pragmatics of the data domain using a Semantic Data Model (SDM) concept, the result would be data that is expressive of the Intelligent Information Fusion (IIF) concept and also organized and classified in context to its use and purpose. The results are presented of a demonstration prototype SDM implemented using the expert system development tool NEXPERT OBJECT. In the prototype, a simple instance of a SDM was created to support a hypothetical application for the Earth Observing System, Data Information System (EOSDIS). The massive amounts of data that EOSDIS will manage requires the definition and design of a powerful information management system in order to support even the most basic needs of the project. The application domain is characterized by a semantic like network that represents the data content and the relationships between the data based on user views and the more generalized domain architectural view of the information world. The data in the domain are represented by objects that define classes, types and instances of the data. In addition, data properties are selectively inherited between parent and daughter relationships in the domain. Based on the SDM a simple information system design is developed from the low level data storage media, through record management and meta data management to the user interface.

Roelofs, Larry H.↗

Additional Metadata Guidelines to Improve the Structure and Usability of HDF and NetCDF Files

The Hierarchical Data Format (HDF) and Network Common Data Form (NetCDF) are data file formats created to aid users in the creation or use of scientific data. These file formats are useful for handling large data volumes and hosting extensive metadata as global attributes or variables and are popular with the modeling community. HDF and NetCDF files are largely used with remote sensing data and have been used to support measurements from numerous campaigns, from satellite to aircraft or ground and mobile based measurements. The files from airborne field studies, however, vary greatly in terms of the file structure and the amount and content of metadata. Information relevant to the file that can be useful to the user such as the data producer, location where data was taken, variable descriptions, or information about the instrument might not be included in the file. This metadata might be present in another file in the dataset containing the same data using the International Consortium for Atmospheric Research on Transport and Transformation (ICARTT) format. Recently, the Aerosols, Clouds, and their Interactions for Earth System Models (MACIE) group started a grassroots effort to develop a set of requirements for the HDF and NetCDF files for field studies, aiming to make the data products more interoperable and usable. Particularly, these requirements seek to make the files more compliant to Climate and Forecast (CF) metadata conventions and to standardize the file structure and the global and variable attributes. These requirements would help to ensure that HDF and NetCDF files contain adequate metadata to better support their use for research, e.g., the modeling community, and to enhance the usability and interoperability of data for research communities at large. To be presented are the details of the MACIE requirements as well as examples of the implementation of these requirements for merge files and lidar observation data files.

Sean Leavor↗