Results for “Data manipulation” — Page 10

NASA NTRS1989

An interactive, discipline-independent data visualization system

The data visualization techniques used by National Space Science Data Center Graphics System are discussed. The system includes a self-describing data abstraction for the storage and manipulation of multidimensional data for discipline-independent scientific applications. The system's data format and graphics display are described. Consideration is given to techniques developed for the rapid display and manipulation of large, complex data sets and for manipulating algorithms with portable implementations which can operate on any data object or geometry. The processes of operating the system are given and examples are presented from applying the system to various types of data, including TOMS data, data from the International Satellite Cloud Climatology Project, and IRAS data.

Treinish, Lloyd A.↗

NASA NTRS1994

The design of a petabyte archive and distribution system for the NASA ECS project

The NASA EOS Data and Information System (EOSDIS) Core System (ECS) will contain one of the largest data management systems ever built - the ECS Science and Data Processing System (SDPS). SDPS is designed to support long term Global Change Research by acquiring, producing, and storing earth science data, and by providing efficient means for accessing and manipulating that data. The first two releases of SDPS, Release A and Release B, will be operational in 1997 and 1998, respectively. Release B will be deployed at eight Distributed Active Archiving Centers (DAAC's). Individual DAAC's will archive different collections of earth science data, and will vary in archive capacity. The storage and management of these data collections is the responsibility of the SDPS Data Server subsystem. It is anticipated that by the year 2001, the Data Server subsystem at the Goddard DAAC must support a near-line data storage capacity of one petabyte. The development of SDPS is a system integration effort in which COTS products will be used in favor of custom components in very possible way. Some software and hardware capabilities required to meet ECS data volume and storage management requirements beyond 1999 are not yet supported by available COTS products. The ECS project will not undertake major custom development efforts to provide these capabilities. Instead, SDPS and its Data Server subsystem are designed to support initial implementations with current products, and provide an evolutionary framework that facilitates the introduction of advanced COTS products as they become available. This paper provides a high-level description of the Data Server subsystem design from a COTS integration standpoint, and discussed some of the major issues driving the design. The paper focuses on features of the design that will make the system scalable and adaptable to changing technologies.

Caulk, Parris M.↗

NASA NTRS1977

Concatenated algebraic decoder

Technique for manipulating digital data consists of mating two separate coding/decoding methods to produce hybrid inner-code/outer-code system. Interactive digital circuitry is used to manipulate casually related digital data.

Source record↗

NASA NTRS1992

System Collects And Displays Demultiplexed Data

Electronic system collects, manipulates, and displays in real time results of manipulation of streams of data transmitted from remote scientific instrumentation. Interface circuit shifts data-and-clock signal from differential logic levels of multiplexer to single-ended logic levels of computer. System accommodates nonstandard data-transmission protocol. Software useful in applications where Macintosh computers used in real-time display and recording of data.

Reschke, Millard F.↗

DOE OSTI2023

Another Set of Python Tools for Visualizing and Manipulating Small-Angle Neutron Scattering Data: Descriptions and Examples

The GP-SANS, Bio-SANS and EQ-SANS instruments at ORNL utilize drtsans for data reduction. drtsans is built on Python, and it can be run using python scripts and Jupyter notebooks. The flexibility afforded by Python makes it possible to incorporate additional actions into the scripts used for data reduction, such as analysis and visualization. Here, a new set of tools for visualizing and manipulating SANS data that can be incorporated into the data reduction scripts for the ORNL SANS instruments, or employed during post–processing, is presented that expands the capabilities of the two previously-released tool sets.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2024

Pinus ponderosa physiology data from the 2022 shade manipulation field experiment at Game Ridge, Missoula County, Montana, USA

This dataset contains physiology data on Pinus ponderosa saplings, including: leaf water potential (MPa, megapascals), leaf osmotic potential (MPa, megapascals), leaf pressure potential (MPa, megapascals), leaf relative water content (% saturated mass), as well as soluble sugars (% dry mass),starch (% dry mass), and total nonstructural carbohydrates (% dry mass) found in branch phloem and xylem. Data files and data dictionary(ies) are uploaded as .csv files and .xlsx files. The Users Guide is a .pdf file. Location data can be found in the Google Earth file GameRidge_SitePlotCoordinates.kmz.kml included here. These datasets were collected for Plant Carbohydrate Depletion, Mycorrhizal Networks, and Vulnerability to Drought: An Experimental Test in the Field. This experiment examined the interdependency between plant hydraulics and carbohydrate availability and sought to develop ways to incorporate interactions with below ground symbiotic organisms to better model and quantify forest response to drought.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

Pinus ponderosa physiology data from the 2023 girdling manipulation field experiment at Game Ridge, Missoula County, Montana, USA

This dataset contains physiology data on Pinus ponderosa saplings, including: leaf water potential (MPa, megapascals), leaf osmotic potential (MPa, megapascals), leaf pressure potential (MPa, megapascals), leaf relative water content (% saturated mass), as well as soluble sugars (% dry mass),starch (% dry mass), and total nonstructural carbohydrates (% dry mass) found in branch phloem and xylem. Data files and data dictionary(ies) are uploaded as .csv files and .xlsx files. The Users Guide is a .pdf file. Location data can be found in the Google Earth file GameRidge_SitePlotCoordinates.kmz.kml included here. These datasets were collected for Plant Carbohydrate Depletion, Mycorrhizal Networks, and Vulnerability to Drought: An Experimental Test in the Field. This experiment examined the interdependency between plant hydraulics and carbohydrate availability and sought to develop ways to incorporate interactions with below ground symbiotic organisms to better model and quantify forest response to drought.

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS1986

PCDS overview

A brief overview of the philosophy, design, and organization of the Pilot Climate Data System (PCDS) is presented. The design has evolved, over time, to deal with numerous data sets, provide appropriate tools for the manipulation and display of such data, and support a wide variety of users. The user interface to the system is provided by the Transportable Applications Executive (TAE). A user can choose one of the five major subsystems from the root menu display. These subsystems are CATALOG, INVENTORY, DATA ACCESS, DATA MANIPULATION, and GRAPHICS. Each subsystem was discussed and examples were given, demonstrating the flexibility provided by the existing PCDS software, Version 3.3. Three of these subsystems are presently being redesigned to provide a more flexible and friendly structure, to allow more consistency and facilitate the ability to support new data sets. Ongoing developmental efforts to migrate from the Version 3.3 prototype phase of PCDS to the operational Version 4.0 stress increased flexibility and data independence.

Treinish, L.↗

DOE OSTI2021

Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs

Rapid growth in scientific data and a widening gap between computational speed and I/O bandwidth make it increasingly infeasible to store and share all data produced by scientific simulations. Instead, we need methods for reducing data volumes: ideally, methods that can scale data volumes adaptively so as to enable negotiation of performance and fidelity tradeoffs in different situations. Multigrid-based hierarchical data representations hold promise as a solution to this problem, allowing for flexible conversion between different fidelities so that, for example, data can be created at high fidelity and then transferred or stored at lower fidelity via logically simple and mathematically sound operations. However, the effective use of such representations has been hindered until now by the relatively high costs of creating, accessing, reducing, and otherwise operating on such representations. We describe here highly optimized data refactoring kernels for GPU accelerators that enable efficient creation and manipulation of data in multigrid-based hierarchical forms. We demonstrate that our optimized design can achieve up to 250 TB/s aggregated data refactoring throughput—83% of theoretical peak—on 1024 nodes of the Summit supercomputer. We showcase our optimized design by applying it to a large-scale scientific visualization workflow and the MGARD lossy compression software.

Data refactoring↗

DOE OSTI2021

Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs

Rapid growth in scientific data and a widening gap between computational speed and I/O bandwidth make it increasingly infeasible to store and share all data produced by scientific simulations. Instead, we need methods for reducing data volumes: ideally, methods that can scale data volumes adaptively so as to enable negotiation of performance and fidelity tradeoffs in different situations. Multigrid-based hierarchical data representations hold promise as a solution to this problem, allowing for flexible conversion between different fidelities so that, for example, data can be created at high fidelity and then transferred or stored at lower fidelity via logically simple and mathematically sound operations. However, the effective use of such representations has been hindered until now by the relatively high costs of creating, accessing, reducing, and otherwise operating on such representations. We describe here highly optimized data refactoring kernels for GPU accelerators that enable efficient creation and manipulation of data in multigrid-based hierarchical forms. We demonstrate that our optimized design can achieve up to 250 TB/s aggregated data refactoring throughput—83% of theoretical peak—on 1024 nodes of the Summit supercomputer. We showcase our optimized design by applying it to a large-scale scientific visualization workflow and the MGARD lossy compression software.

Chen, Jieyang↗

NASA NTRS2003

Generalized Symbolic Execution for Model Checking and Testing

Modern software systems, which often are concurrent and manipulate complex data structures must be extremely reliable. We present a novel framework based on symbolic execution, for automated checking of such systems. We provide a two-fold generalization of traditional symbolic execution based approaches: one, we define a program instrumentation, which enables standard model checkers to perform symbolic execution; two, we give a novel symbolic execution algorithm that handles dynamically allocated structures (e.g., lists and trees), method preconditions (e.g., acyclicity of lists), data (e.g., integers and strings) and concurrency. The program instrumentation enables a model checker to automatically explore program heap configurations (using a systematic treatment of aliasing) and manipulate logical formulae on program data values (using a decision procedure). We illustrate two applications of our framework: checking correctness of multi-threaded programs that take inputs from unbounded domains with complex structure and generation of non-isomorphic test inputs that satisfy a testing criterion. Our implementation for Java uses the Java PathFinder model checker.

Khurshid, Sarfraz↗

DOE OSTI2021

Effects of raster terrain representation on GIS shortest path analysis

Spatial analysis extracts meaning and insights from spatially referenced data, where the results are highly dependent on the quality of the data used and the manipulations on the data when preparing it for analysis. Users should understand the impacts that data representations may have on their results in order to prevent distortions in their outcomes. We study the consequences of two common data preparations when locating a linear feature performing shortest path analysis on raster terrain data: 1) the connectivity of the network generated by connecting raster cells to their neighbors, and 2) the range of the attribute scale for assigning costs. Such analysis is commonly used to locate transmission lines, where the results could have major implications on project cost and its environmental impact. Experiments in solving biobjective shortest paths show that results are highly dependent on the parameters of the data representations, with exceedingly variable results based on the choices made in reclassifying attributes and generating networks from the raster. Based on these outcomes, we outline recommendations for ensuring geographic information system (GIS) data representations maintain analysis results that are accurate and unbiased.

Medrano, F. Antonio (ORCID:000000025913632X)↗

DOE OSTI2020

The Nuclear Data Interface (NDI) [Slides]

The Nuclear Data Interface (NDI) can be described as: Standardized methodology for accessing and manipulating nuclear data; An API that accesses nuclear data while hiding underlying details from the user; Created to enable codes to access nuclear data without knowledge of a specific library format, content, or location; Both a code and ASCII data format (just to be confusing); Not continuous energy (that’s what NDATK is for).

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

NASA NTRS1981

Computer program documentation for the patch subsampling processor

The programs presented are intended to provide a way to extract a sample from a full-frame scene and summarize it in a useful way. The sample in each case was chosen to fill a 512-by-512 pixel (sample-by-line) image since this is the largest image that can be displayed on the Integrated Multivariant Data Analysis and Classification System. This sample size provides one megabyte of data for manipulation and storage and contains about 3% of the full-frame data. A patch image processor computes means for 256 32-by-32 pixel squares which constitute the 512-by-512 pixel image. Thus, 256 measurements are available for 8 vegetation indexes over a 100-mile square.

Nieves, M. J.↗

NASA NTRS1983

A flexible grid embedding technique with application to the Euler equations

An automated grid embedding procedure for solution of flows about complex geometries is described. The physical domain is subdivided into regions that can accommodate easily generated grids. The grids are organized in a hierarchical structure, and communication among grids is accomplished by interpolation of the flow variables at mesh boundaries. Algorithms for locating embedded boundaries, special treatment of the embedded grids, and the data structures required for manipulating the solution data are described. The method is demonstrated by solution of the Euler equations for transonic flow about a supercritical airfoil and a flapped airfoil.

Benek, J. A.↗

NASA NTRS2011

Situational Lightning Climatologies for Central Florida: Phase V

The AMU added three years of data to the POR from the previous work resulting in a 22-year POR for the warm season months from 1989-2010. In addition to the flow regime stratification, moisture and stability stratifications were added to separate more active from less active lighting days within the same flow regime. The parameters used for moisture and stability stratifications were PWAT and TI which were derived from sounding data at four Florida radiosonde sites. Lightning data consisted of NLDN CG lightning flashes within 30 NM of each airfield. The AMU increased the number of airfields from nine to thirty-six which included the SLF, CCAFS, PAFB and thirty-three airfields across Florida. The NWS MLB requested the AMU calculate lightning climatologies for additional airfields that they support as a backup to NWS TBW which was then expanded to include airfields supported by NWS JAX and NWS MFL. The updated climatologies of lightning probabilities are based on revised synoptic-scale flow regimes over the Florida peninsula (Lambert 2007) for 5-, 10-, 20- and 30-NM radius range rings around the thirty-six airfields in 1-, 3- and 6-hour increments. The lightning, flow regime, moisture and stability data were processed in S-PLUS software using scripts written by the AMU to automate much of the data processing. The S-PLUS data files were exported to Excel to allow the files to be combined in Excel Workbooks for easier data handling and to create the tables and charts for the Gill. The AMU revised the Gill developed in the previous phase (Bauman 2009) with the new data and provided users with an updated HTML tool to display and manipulate the data and corresponding charts. The tool can be used with most web browsers and is computer operating system independent. The AMU delivered two Gills - one with just the PWAT stratification and one with both the PWAT and TI stratifications due to insufficient data in some of the PWATITI stratification combinations. This will allow the forecasters to choose a moisture-only or moisture/stability stratification depending on the flow regime and available data.

Bauman, William H., III↗

NASA NTRS2010

Issues and Solutions for Bringing Heterogeneous Water Cycle Data Sets Together

The water cycle research community has generated many regional to global scale products using data from individual NASA missions or sensors (e.g., TRMM, AMSR-E); multiple ground- and space-based data sources (e.g., Global Precipitation Climatology Project [GPCP] products); and sophisticated data assimilation systems (e.g., Land Data Assimilation Systems [LDAS]). However, it is often difficult to access, explore, merge, analyze, and inter-compare these data in a coherent manner due to issues of data resolution, format, and structure. These difficulties were substantiated at the recent Collaborative Energy and Water Cycle Information Services (CEWIS) Workshop, where members of the NASA Energy and Water cycle Study (NEWS) community gave presentations, provided feedback, and developed scenarios which illustrated the difficulties and techniques for bringing together heterogeneous datasets. This presentation reports on the findings of the workshop, thus defining the problems and challenges of multi-dataset research. In addition, the CEWIS prototype shown at the workshop will be presented to illustrate new technologies that can mitigate data access roadblocks encountered in multi-dataset research, including: (1) Quick and easy search and access of selected NEWS data sets. (2) Multi-parameter data subsetting, manipulation, analysis, and display tools. (3) Access to input and derived water cycle data (data lineage). It is hoped that this presentation will encourage community discussion and feedback on heterogeneous data analysis scenarios, issues, and remedies.

Acker, James↗

DOE OSTI2021

UNH TDP - ADCP Raw Data and Processing Scripts - Fall 2021

This submission contains raw Acoustic Doppler Current Profiler (ADCP) data and processing scripts associated with MHKDR submission 394 (UNH TDP - Concurrent Measurements of Inflow, Power Performance, and Loads for a Grid-Synchronized Vertical Axis Cross-Flow Turbine Operating in a Tidal Estuary, DOI: 10.15473/1973860) from the University of New Hampshire and Atlantic Marine Energy Center (AMEC) turbine deployment platform. The user is directed to the MHKDR submission 394 for relevant context and detail of this deployment; see link below. The 394_READ_ME file here provides the description from that submission for quick reference. The READ_ME file for this specific instrument from the 394 submission is also available here. This submission contains a zipped folder structure containing raw data in its original format and MATLAB (2019a) processing scripts used to process and manipulate the data into its final form. The final data products are submitted in the 394 submission.

16 TIDAL AND WAVE POWER↗