Results for “data ordering” — Page 3

DOE OSTI2020

Navigation and Meteorological Data from Multiple Sensors on Airborne Platform (NAVMET-AIR) Value-Added Product Report

The Navigation and Meteorological Data from Multiple Sensors on Airborne Platform (NAVMET-AIR) value-added product encompasses the aafnaviwg data set in the ARM Data Center (ADC), known as the IWG file, named for the Inter-Agency Working Group for Airborne Data and Telemetry Systems (IWGADTS). Its purpose is to produce a suite of tools to promote standardization of instrument interface, data format, and data processing. This data set’s contents are navigational and meteorological state variables at 1 Hz. It also contains higher-order data flags created by ARM Aerial Facility (AAF) scientists to aid analysis such as periods when the aircraft is flying level, operating in maneuvers, or flying through cloud. Due to the nature of research flights, the payload of the aircraft consists of many duplicate and overlapping instruments to ensure, in case of instrument failure, there is a backup to record the data. Measurements from multiple instruments are consolidated into a single file. Each variable of this data set is carefully chosen from the onboard instrumentation and quality checked by AAF scientists to create this wholistic and most accurate data set for airborne research. This document is intended to be a hub towards the individual “read me” files produced for each campaign. For reference about what instrument was used for calculations on specific days, the read me files are located in the ARM IOP archive at the addresses listed, or access the IOP (intensive operational period) database at https://www.archive.arm.gov/.

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS2020

Recovery, Restoration and Archiving of Previously Lost Data and Metadata from the Apollo Lunar Surface Experiments Package (ALSEP)

The Apollo Lunar Surface Experiments Package (ALSEP) is the name used to collectively represent the geophysical instruments deployed on the lunar surface by the astronauts on Apollo 12, 14, 15, 16, and 17. These instruments were active from the times of their deployment (November 1969 – December 1972) to September 1977. During that time, fourteen types of experiments were conducted, and their data were transmitted to Earth. The experiment PIs processed them. At the conclusion of the experiments, some of these data were submitted to the NASA Space Science Data Coordinated Archive (NSSDCA) for archiving, while others were not. The raw instrument data received from the Moon prior to March 1976 were not archived, either. The unarchived data, resided on open-reel magnetic tapes, became lost in the decades since, along with much of the metadata (the information necessary/useful in properly processing/analyzing the data). This article retraces the history of the ALSEP data archiving efforts in the 1970s, the subsequent loss of the data tapes, and the search, recovery, and restoration of the lost data by contemporary researchers in the 21st century. In 2006, NSSDCA began reformatting some of the ALSEP data archived in the 1970s to conform with the current Planetary Data System (PDS). In 2010, 440 of the previously lost magnetic tapes containing the raw ALSEP data were recovered. From these tapes, the data were extracted, re-packaged for individual experiments, and, for those with sufficient metadata, processed into higher order data readily usable by researchers. All of these data products have been recently archived with either PDS or NSSDCA. These newly restored data fill a number of gaps in the previously existing archive of the ALSEP data. In addition, tens of thousands of pages of Apollo era documents have been optically scanned and compiled into an online searchable catalog. This article also describes the content, organization, and usage of the restored raw ALSEP data and metadata.

S Nagihara↗

DOE OSTI2023

BEYONDPLANCK XIV. Polarized foreground emission between 30 and 70 GHz

We constrain polarized foreground emission between 30 and 70 GHz with the Planck Low Frequency Instrument (LFI) and WMAP data within the global Bayesian BeyondPlanck framework. We combine for the first time full-resolution Planck LFI time-ordered data with low-resolution WMAP sky maps at 33, 40 and 61 GHz. Spectral parameters are fit with a likelihood defined at the native resolution of each frequency channel. This analysis represents the first implementation of true multi-resolution component separation applied to CMB observations for both amplitude and spectral energy distribution (SED) parameters. For synchrotron emission, we approximate the SED as a power-law in frequency and find that the low signal-to-noise ratio of the current data strongly limits the number of free parameters that may be robustly constrained. We partition the sky into four large disjoint regions (High Latitude; Galactic Spur; Galactic Plane; and Galactic Center), each associated with its own power-law index. We find that the High Latitude region is prior-dominated, while the Galactic Center region is contaminated by residual instrumental systematics. The two remaining regions appear to be signal-dominated, and for these we derive spectral indices of $β$ S Spur = -3.17 ± 0.06 and $β$ S Plane = -3.03 ± 0.07, in good agreement with previous results. For thermal dust emission we assume a modified blackbody model and we fit a single power-law index across the full sky. We find $β$ d = 1.64 ± 0.03, which is slightly steeper than reported from Planck HFI data, but still statistically consistent at the 2σ confidence level.

79 ASTRONOMY AND ASTROPHYSICS↗

NASA NTRS1975

EMGAN: A computer program for time and frequency domain reduction of electromyographic data

An experiment in electromyography utilizing surface electrode techniques was developed for the Apollo-Soyuz test project. This report describes the computer program, EMGAN, which was written to provide first order data reduction for the experiment. EMG signals are produced by the membrane depolarization of muscle fibers during a muscle contraction. Surface electrodes detect a spatially summated signal from a large number of muscle fibers commonly called an interference pattern. An interference pattern is usually so complex that analysis through signal morphology is extremely difficult if not impossible. It has become common to process EMG interference patterns in the frequency domain. Muscle fatigue and certain myopathic conditions are recognized through changes in muscle frequency spectra.

Hursta, W. N.↗

DOE OSTI2022

Near real-time streaming analysis of big fusion data

Experiments on fusion plasmas produce high-dimensional data time series with ever-increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article, we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, Delta allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describes the modular and expandable software architecture of Delta and presents performance benchmarks of individual components as well as of an example workflow. Focusing on a streaming analysis workflow where electron cyclotron emission imaging (ECEi) data is measured at KSTAR on the National Energy Research Scientific Computing Center's (NERSC's) supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. In NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes them in under 5 min. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. Here, by providing session leaders and physics operators, results of higher-order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2021

pyDRESCALk

Modern data scientists are tasked to analyze ever-growing data sets with increasingly complex relationships. Tensor decompositions have come to play a central role in identifying underlying latent structures in higher-order data. The problem of fitting tensor models to different distributions is complicated by the combinations of size, dimensionality, and sparsity present in real world data. The situation demands efficient algorithms designed for shared-memory and distributed systems. This work will present new research that tackles these challenges on several different fronts, leveraging optimizations in numerical algorithms and sparse tensor representations in heterogeneous high performance computing environments.

Bhattarai, Manish↗

NASA NTRS1989

Nimbus-7 data product summary

Data sets resulting from the first nine years of operations of the Nimbus-7 Satellite are briefly described. After a brief description of the Nimbus-7 Mission, each of the eight experiments on-board the satellite (Coastal Zone Color Scanner (CZCS), Earth Radiation Budget (ERB), Limb Infrared Monitor of the Stratosphere (MIMS), Stratospheric Aerosol Measurement II (SAM II), Stratospheric and Mesospheric Sounder (SAMS), Solar Backscatter Ultraviolet/Total Ozone Mapping Spectrometer (SBUV/TOMS), Scanning Multichannel Microwave Radiometer (SMMR) and the Temperature Humidity Infrared Radiometer (THIR) are introduced and their respective data products are described in terms of media, general format, and suggested applications. Extensive references are provided. Instructions for obtaining further information, and for ordering data products are given.

Oakes, Arnold G.↗

NASA NTRSDate not supplied

MISR Browse data (MISBR_V3)

A MISR Browse Tool is available to allow easy access to ellipsoid-derived, true-color images for each camera reduced to 2.2 km resolution. The MISR red, green and blue bands are used to create the true-color image in JPEG format. The image is intentionally clipped and gamma-stretched to make cloud, ocean and land features visible. The user may find the Browse Tool helpful before ordering data. The tool's purpose is to search and view images in the on-line user interface. However, the browse product can be ordered separately. The Browse Tool page has Orbit-to-Date and Lat/Lon-to-Path/Block conversion tools to help identify particular MISR geographic locations and data files. [Location=GLOBAL] [Temporal_Coverage: Start_Date=2000-02-24; Stop_Date=] [Spatial_Coverage: Southernmost_Latitude=-90; Northernmost_Latitude=90; Westernmost_Longitude=-180; Easternmost_Longitude=180] [Data_Resolution: Latitude_Resolution=2.2 km; Longitude_Resolution=2.2 km; Horizontal_Resolution_Range=1 km - < 10 km or approximately .01 degree - < .09 degree; Temporal_Resolution=about 15 orbits/day; Temporal_Resolution_Range=Daily - < Weekly].

EOSDIS↗

NASA NTRSDate not supplied

MISR Browse data (MISBR_V4)

A MISR Browse Tool is available to allow easy access to ellipsoid-derived, true-color images for each camera reduced to 2.2 km resolution. The MISR red, green and blue bands are used to create the true-color image in JPEG format. The image is intentionally clipped and gamma-stretched to make cloud, ocean and land features visible. The user may find the Browse Tool helpful before ordering data. The tool's purpose is to search and view images in the on-line user interface. However, the browse product can be ordered separately. The Browse Tool page has Orbit-to-Date and Lat/Lon-to-Path/Block conversion tools to help identify particular MISR geographic locations and data files. [Location=GLOBAL] [Temporal_Coverage: Start_Date=2000-02-24; Stop_Date=] [Spatial_Coverage: Southernmost_Latitude=-90; Northernmost_Latitude=90; Westernmost_Longitude=-180; Easternmost_Longitude=180] [Data_Resolution: Latitude_Resolution=2.2 km; Longitude_Resolution=2.2 km; Horizontal_Resolution_Range=1 km - < 10 km or approximately .01 degree - < .09 degree; Temporal_Resolution=about 15 orbits/day; Temporal_Resolution_Range=Daily - < Weekly].

IMAGE↗

NASA NTRSDate not supplied

MISR Browse data (MISBR_V5)

A MISR Browse Tool is available to allow easy access to ellipsoid-derived, true-color images for each camera reduced to 2.2 km resolution. The MISR red, green and blue bands are used to create the true-color image in JPEG format. The image is intentionally clipped and gamma-stretched to make cloud, ocean and land features visible. The user may find the Browse Tool helpful before ordering data. The tool's purpose is to search and view images in the on-line user interface. However, the browse product can be ordered separately. The Browse Tool page has Orbit-to-Date and Lat/Lon-to-Path/Block conversion tools to help identify particular MISR geographic locations and data files. [Temporal_Coverage: Start_Date=2000-02-24; Stop_Date=] [Spatial_Coverage: Southernmost_Latitude=-90; Northernmost_Latitude=90; Westernmost_Longitude=-180; Easternmost_Longitude=180] [Data_Resolution: Latitude_Resolution=2.2 km; Longitude_Resolution=2.2 km; Horizontal_Resolution_Range=1 km - < 10 km or approximately .01 degree - < .09 degree; Temporal_Resolution=about 15 orbits/day; Temporal_Resolution_Range=Daily - < Weekly].

EOSDIS↗

NASA NTRS2000

Parallel Conjugate Gradient: Effects of Ordering Strategies, Programming Paradigms, and Architectural Platforms

The Conjugate Gradient (CG) algorithm is perhaps the best-known iterative technique to solve sparse linear systems that are symmetric and positive definite. A sparse matrix-vector multiply (SPMV) usually accounts for most of the floating-point operations within a CG iteration. In this paper, we investigate the effects of various ordering and partitioning strategies on the performance of parallel CG and SPMV using different programming paradigms and architectures. Results show that for this class of applications, ordering significantly improves overall performance, that cache reuse may be more important than reducing communication, and that it is possible to achieve message passing performance using shared memory constructs through careful data ordering and distribution. However, a multi-threaded implementation of CG on the Tera MTA does not require special ordering or partitioning to obtain high efficiency and scalability.

Oliker, Leonid↗

NASA NTRS2012

The Atacama Cosmology Telescope: Data Characterization and Map Making

We present a description of the data reduction and mapmaking pipeline used for the 2008 observing season of the Atacama Cosmology Telescope (ACT). The data presented here at 148 GHz represent 12% or the 90 TB collected by ACT from 2007 to 2010. In 2008 we observed for 136 days, producing a total of 142h of data (11 TB for the 148 GHz band only), with a daily average of 10.5 h of observation. From these, 108.5 h were devoted to 850 sq deg stripe (11.2 h by 9 deg.1) centered on a declination of -52 deg.7, while 175 h were devoted to a 280 square deg stripe (4.5 h by 4 deg.8) centered at the celestial equator. We discuss sources of statistical and systematic noise, calibration, telescope pointing and data selection. Out of 1260 survey hours and 1024 detectors per array, 816 h and 593 effective detectors remain after data selection for this frequency band, yielding a 38 % survey efficiency. The total sensitivity in 2008, determined from the noise level between 5 Hz and 20 Hz in the time-ordered data stream (TOD), is 32 muK square root of s in CMB units. Atmospheric brightness fluctuations constitute the main contaminant in the data and dominate the detector and noise covariance at low frequencies in the TOD. The maps were made by solving the lease squares problem using the Preconditioned Conjugate Gradient method, incorporating the details of the detector and noise correlations. Cross-correlation with WMAP sky maps as well as analysis from simulations reveal the our maps are unbiased at l > 300. This paper accompanies the public release of the 148 GHz southern stripe maps from 2008. The techniques described here will be applied to future maps and data releases.

Duenner, Rolando↗

NASA NTRS2003

Difference in Icosahedral Short-Range Order in Early and Late Transition Metals Liquids

New short-range order data are presented for equilibrium and undercooled liquids of Ti and Ni. These were obtained from in-situ synchrotron x-ray diffraction measurements of electrostatically-levitated droplets. While the short-range order of liquid Ni is icosahedral, consistent with Frank's hypothesis, significantly distorted icosahedral order is observed in liquid Ti. This is the first experimental observation of distorted icosahedral short-range order in any liquid, although this has been predicted by theoretical studies on atomic clusters.

Lee, G. W.↗

NASA NTRS2004

Difference in Icosahedral Short-Range Order in Early and Late Transition Metal Liquids

New short-range order data are presented for equilibrium and undercooled liquids of Ti and Ni. These were obtained from in-situ synchrotron x-ray diffraction measurements of electrostatically-levitated droplets. While the short-range order of liquid Ni is icosahedral, consistent with Frank's hypothesis, significantly distorted icosahedral order is observed in liquid Ti. This is the first experimental observation of distorted icosahedral short-range order in any liquid. although this has been predicted by theoretical studies on atomic clusters.

Lee, G. W.↗

NASA NTRS1991

Version 0 EOSDIS - An overview

Attention is given to NASA's Earth Observing System Data and Information System (EOSDIS), which is to be a single, distributed but internally consistent evolutionary system to support the planning and execution of EOS data acquisitions and to process, archive, and distribute EOS data products and selected non-EOS data to enable interdisciplinary studies of the earth. V0 EOSDIS, a logical step in this evolutionary process, is to address both technical and managerial challenges. Technical challenges include developing a multidiscipline, distributed system for searching and ordering data in a heterogeneous environment, and standardizing data formats and distribution techniques among differing communities and organizations. Managerial challenges include establishing and maintaining a structure consisting of geographically distributed entities such that cooperative development is carried out effectively despite organizational differences, maintaining interactions with the scientific community to ensure its close involvement despite its size and diversity, and keeping the expectations for V0 consistent with its schedules and resources.

Ramapriyan, H. K.↗

NASA NTRS2002

Effects of Ordering Strategies and Programming Paradigms on Sparse Matrix Computations

The Conjugate Gradient (CG) algorithm is perhaps the best-known iterative technique to solve sparse linear systems that are symmetric and positive definite. For systems that are ill-conditioned, it is often necessary to use a preconditioning technique. In this paper, we investigate the effects of various ordering and partitioning strategies on the performance of parallel CG and ILU(O) preconditioned CG (PCG) using different programming paradigms and architectures. Results show that for this class of applications: ordering significantly improves overall performance on both distributed and distributed shared-memory systems, that cache reuse may be more important than reducing communication, that it is possible to achieve message-passing performance using shared-memory constructs through careful data ordering and distribution, and that a hybrid MPI+OpenMP paradigm increases programming complexity with little performance gains. A implementation of CG on the Cray MTA does not require special ordering or partitioning to obtain high efficiency and scalability, giving it a distinct advantage for adaptive applications; however, it shows limited scalability for PCG due to a lack of thread level parallelism.

Oliker, Leonid↗

NASA NTRS2024

Tracking Community Building in Open Science

Open Science is enabled by a vibrant community of researchers who regularly engage with the data, from its production to its organization, curation, archiving, dissemination, analysis, and publication. This presentation will examine community building in open science. The NASA Open Science Data Repository (OSDR) makes data available to the public following the FAIR (Findability, Accessibility, Interoperability, and Reusability) principles. OSDR takes open science further with the OS Analysis Working Groups (AWGs) that facilitate community development and promotion. The primary activity of each AWG is to establish and validate analytical processes to generate higher-order data from data housed in OSDR. There are a number of these groups on various topics, including the Animal AWG, Plant AWG, Microbial AWG, Multi-Omics AWG, AI/ML AWG, and the Ames Life Sciences Data Archive (ALSDA) AWG. The international volunteers participating in these AWGs come from academia, citizen science initiatives, industry, and government. They include researchers, principal investigators, professors, trained hobbyists, and students from various domains and disciplines. Anyone may request to join the AWGs, and membership requests are vetted monthly by the group organizers before granting admission. Core to membership is demonstrated expertise through records of training, integrity, work in the professed domain(s), and good community standing. Regular virtual meetings are held for each AWG, with a varying cadence depending on the group's needs and goals. AWG communities share their expertise in research including cutting edge tools, software, frameworks, data formats, and libraries accelerating research collectively. This collaborative approach helps community members cross technology gaps and identify emerging challenges. These diverse communities encompass a wide range of individuals hailing from various sectors within the Science Mission Directorate and beyond. They serve as a means to promote and enhance transparency, accessibility, and inclusion. An annual AWG Symposium brings contributors together in person. Participation in AWGs can be synchronous or asynchronous, with some groups performing most of their work in off hours. Participants gain valuable skills and connections that allow them to add value to their communities and new organizations that they join, resulting in an expanded return on investment for the space life science community. Open science is increasingly a federal mandate and initiatives like NASA's Transform to Open Science and instruments like the Decadal Survey of Biological and Physical Sciences in Space demonstrate the need to carefully consider best practices in this domain. Here, we present greater detail about the makeup and participation metrics of the various AWGs affiliated with OSDR and details of successful peer-reviewed publication campaigns.

Christina M Johnson↗

NASA NTRSDate not supplied

Tracking Community Building in Open Science

Open Science is enabled by a vibrant community of researchers who regularly engage with the data, from its production to its organization, curation, archiving, dissemination, analysis, and publication. This presentation will examine community building in open science. The NASA Open Science Data Repository (OSDR) makes data available to the public following the FAIR (Findability, Accessibility, Interoperability, and Reusability) principles. OSDR takes open science further with the OS Analysis Working Groups (AWGs) that facilitate community development and promotion. The primary activity of each AWG is to establish and validate analytical processes to generate higher-order data from data housed in OSDR. There are a number of these groups on various topics, including the Animal AWG, Plant AWG, Microbial AWG, Multi-Omics AWG, AI/ML AWG, and the Ames Life Sciences Data Archive (ALSDA) AWG. The international volunteers participating in these AWGs come from academia, citizen science initiatives, industry, and government. They include researchers, principal investigators, professors, trained hobbyists, and students from various domains and disciplines. Anyone may request to join the AWGs, and membership requests are vetted monthly by the group organizers before granting admission. Core to membership is demonstrated expertise through records of training, integrity, work in the professed domain(s), and good community standing. Regular virtual meetings are held for each AWG, with a varying cadence depending on the group's needs and goals. AWG communities share their expertise in research including cutting edge tools, software, frameworks, data formats, and libraries accelerating research collectively. This collaborative approach helps community members cross technology gaps and identify emerging challenges. These diverse communities encompass a wide range of individuals hailing from various sectors within the Science Mission Directorate and beyond. They serve as a means to promote and enhance transparency, accessibility, and inclusion. An annual AWG Symposium brings contributors together in person. Participation in AWGs can be synchronous or asynchronous, with some groups performing most of their work in off hours. Participants gain valuable skills and connections that allow them to add value to their communities and new organizations that they join, resulting in an expanded return on investment for the space life science community. Open science is increasingly a federal mandate and initiatives like NASA's Transform to Open Science and instruments like the Decadal Survey of Biological and Physical Sciences in Space demonstrate the need to carefully consider best practices in this domain. Here, we present greater detail about the makeup and participation metrics of the various AWGs affiliated with OSDR and details of successful peer-reviewed publication campaigns.

Christina M Johnson↗