Results for “Data migration” — Page 3

NASA NTRS1992

An analysis of file migration in a UNIX supercomputing environment

The super computer center at the National Center for Atmospheric Research (NCAR) migrates large numbers of files to and from its mass storage system (MSS) because there is insufficient space to store them on the Cray supercomputer's local disks. This paper presents an analysis of file migration data collected over two years. The analysis shows that requests to the MSS are periodic, with one day and one week periods. Read requests to the MSS account for the majority of the periodicity; as write requests are relatively constant over the course of a week. Additionally, reads show a far greater fluctuation than writes over a day and week since reads are driven by human users while writes are machine-driven.

Miller, Ethan L.↗

NASA NTRS1993

Volume serving and media management in a networked, distributed client/server environment

The E-Systems Modular Automated Storage System (EMASS) is a family of hierarchical mass storage systems providing complete storage/'file space' management. The EMASS volume server provides the flexibility to work with different clients (file servers), different platforms, and different archives with a 'mix and match' capability. The EMASS design considers all file management programs as clients of the volume server system. System storage capacities are tailored to customer needs ranging from small data centers to large central libraries serving multiple users simultaneously. All EMASS hardware is commercial off the shelf (COTS), selected to provide the performance and reliability needed in current and future mass storage solutions. All interfaces use standard commercial protocols and networks suitable to service multiple hosts. EMASS is designed to efficiently store and retrieve in excess of 10,000 terabytes of data. Current clients include CRAY's YMP Model E based Data Migration Facility (DMF), IBM's RS/6000 based Unitree, and CONVEX based EMASS File Server software. The VolSer software provides the capability to accept client or graphical user interface (GUI) commands from the operator's console and translate them to the commands needed to control any configured archive. The VolSer system offers advanced features to enhance media handling and particularly media mounting such as: automated media migration, preferred media placement, drive load leveling, registered MediaClass groupings, and drive pooling.

Herring, Ralph H.↗

NASA NTRS1996

Use of HSM with Relational Databases

Hierarchical storage management (HSM) systems have evolved to become a critical component of large information storage operations. They are built on the concept of using a hierarchy of storage technologies to provide a balance in performance and cost. In general, they migrate data from expensive high performance storage to inexpensive low performance storage based on frequency of use. The predominant usage characteristic is that frequency of use is reduced with age and in most cases quite rapidly. The result is that HSM provides an economical means for managing and storing massive volumes of data. Inherent in HSM systems is system managed storage, where the system performs most of the work with minimum operations personnel involvement. This automation is generally extended to include: backup and recovery, data duplexing to provide high availability, and catastrophic recovery through use of off-site storage.

Breeden, Randall↗

NASA NTRS1999

A New Approach to Parallel Dynamic Partitioning for Adaptive Unstructured Meshes

Classical mesh partitioning algorithms were designed for rather static situations, and their straightforward application in a dynamical framework may lead to unsatisfactory results, e.g., excessive data migration among processors. Furthermore, special attention should be paid to their amenability to parallelization. In this paper, a novel parallel method for the dynamic partitioning of adaptive unstructured meshes is described. It is based on a linear representation of the mesh using self-avoiding walks.

Heber, Gerd↗

NASA NTRS1994

Beyond a Terabyte File System

The Numerical Aerodynamics Simulation Facility's (NAS) CRAY C916/1024 accesses a "virtual" on-line file system, which is expanding beyond a terabyte of information. This paper will present some options to fine tuning Data Migration Facility (DMF) to stretch the online disk capacity and explore the transitions to newer devices (STK 4490, ER90, RAID).

Powers, Alan K.↗

NASA NTRS2001

Parallel Processing of Adaptive Meshes with Load Balancing

Many scientific applications involve grids that lack a uniform underlying structure. These applications are often also dynamic in nature in that the grid structure significantly changes between successive phases of execution. In parallel computing environments, mesh adaptation of unstructured grids through selective refinement/coarsening has proven to be an effective approach. However, achieving load balance while minimizing interprocessor communication and redistribution costs is a difficult problem. Traditional dynamic load balancers are mostly inadequate because they lack a global view of system loads across processors. In this paper, we propose a novel and general-purpose load balancer that utilizes symmetric broadcast networks (SBN) as the underlying communication topology, and compare its performance with a successful global load balancing environment, called PLUM, specifically created to handle adaptive unstructured applications. Our experimental results on an IBM SP2 demonstrate that the SBN-based load balancer achieves lower redistribution costs than that under PLUM by overlapping processing and data migration.

Das, Sajal K.↗

NASA NTRS1996

Experiences From NASA/Langley's DMSS Project

There is a trend in institutions with high performance computing and data management requirements to explore mass storage systems with peripherals directly attached to a high speed network. The Distributed Mass Storage System (DMSS) Project at the NASA Langley Research Center (LaRC) has placed such a system into production use. This paper will present the experiences, both good and bad, we have had with this system since putting it into production usage. The system is comprised of: 1) National Storage Laboratory (NSL)/UniTree 2.1, 2) IBM 9570 HIPPI attached disk arrays (both RAID 3 and RAID 5), 3) IBM RS6000 server, 4) HIPPI/IPI3 third party transfers between the disk array systems and the supercomputer clients, a CRAY Y-MP and a CRAY 2, 5) a "warm spare" file server, 6) transition software to convert from CRAY's Data Migration Facility (DMF) based system to DMSS, 7) an NSC PS32 HIPPI switch, and 8) a STK 4490 robotic library accessed from the IBM RS6000 block mux interface. This paper will cover: the performance of the DMSS in the following areas: file transfer rates, migration and recall, and file manipulation (listing, deleting, etc.); the appropriateness of a workstation class of file server for NSL/UniTree with LaRC's present storage requirements in mind the role of the third party transfers between the supercomputers and the DMSS disk array systems in DMSS; a detailed comparison (both in performance and functionality) between the DMF and DMSS systems LaRC's enhancements to the NSL/UniTree system administration environment the mechanism for DMSS to provide file server redundancy the statistics on the availability of DMSS the design and experiences with the locally developed transparent transition software which allowed us to make over 1.5 million DMF files available to NSL/UniTree with minimal system outage

Source record↗

NASA NTRSDate not supplied

Applications of MERRA-2 data for avian migration, biomass burning, and dusty atmospheric rivers

Three different applications of MERRA-2 data are presented. 1) Using radar data, we introduced a new concept for spatial patterns of bird migration across the contiguous U.S. This approach allowed us to use MERRA-2 data and learn that remote forcing in the tropical Pacific—through a chain of processes including atmospheric Rossby wave trains— controls the climatic conditions, associated with bird migration in North America. 2) We showed that emissions from biomass burning in the Congo Basin are partly controlled by the low-level winds, which are in turn associated with the intensity of the subtropical high in the Indian Ocean. Using back-trajectory analysis, we found that these emissions combined with their transport mechanism explain the interannual variability of black carbon in West Africa. 3) Our analysis showed that atmospheric rivers in the Middle East contribute to both heavy flood and dust transport within their corridor. We also found that warm advection and rain-on-snow effect of dusty atmospheric rivers further enhance the chance of flood through rapid snowmelt processes.

Amin Dezfuli↗

NASA NTRSDate not supplied

Applications of MERRA-2 Data for Avian Migration, Biomass Burning, and Dusty Atmospheric Rivers

Three different applications of MERRA-2 data are presented. 1) Using radar data, we introduced a new concept for spatial patterns of bird migration across the contiguous U.S. This approach allowed us to use MERRA-2 data and learn that remote forcing in the tropical Pacific—through a chain of processes including atmospheric Rossby wave trains— controls the climatic conditions, associated with bird migration in North America. 2) We showed that emissions from biomass burning in the Congo Basin are partly controlled by the low-level winds, which are in turn associated with the intensity of the subtropical high in the Indian Ocean. Using back-trajectory analysis, we found that these emissions combined with their transport mechanism explain the interannual variability of black carbon in West Africa. 3) Our analysis showed that atmospheric rivers in the Middle East contribute to both heavy flood and dust transport within their corridor. We also found that warm advection and rain-on-snow effect of dusty atmospheric rivers further enhance the chance of flood through rapid snowmelt processes.

Amin Dezfuli↗

DOE OSTI2022

Replication Data for: Hydrogen migration at restructuring palladium-silver oxide boundaries dramatically enhances reduction rate of silver oxide

The data underlying this published work have been made publicly available in this repository as part of the IMASC Data Management Plan. This work was supported as part of the Integrated Mesoscale Architectures for Sustainable Catalysis (IMASC), an Energy Frontier Research Center funded by the U.S. Department of Energy, Office of Science, Basic Energy Sciences under Award # DE-SC0012573.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

NASA NTRSDate not supplied

Hydrological Data at the NASA GES DISC: Current Capabilities and New Opportunities

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is one of twelve NASA Earth science data centers that document, process, archive and distribute data from Earth observation missions and projects. GES DISC maintains an archive of several hydrology datasets, including the Land Data Assimilation Systems (LDAS) and the Gravity Recovery and Climate Experiment (GRACE) Data Assimilation for Drought Monitoring (GRACE-DA-DM) data products. These datasets include model output of heat fluxes, rain, snow, soil temperature, soil moisture, and runoff; and observational forcing data, including surface pressure, temperature, precipitation, downward shortwave and longwave radiation, humidity, and wind. The temporal resolution of the hydrology data at GES DISC ranges from hourly to monthly, and spatial resolutions range from 0.1° to 1.0°. The GES DISC provides services which enable users to aggregate, temporally and spatially subset, regrid, and visualize archived data including the GES DISC Subsetter, Hydrology Data Rods, and the Geospatial Interactive Online Visualization and Analysis Infrastructure (GIOVANNI). The Hydrology Data Rods service optimally reorganizes large hydrological data sets as extended time series, providing more efficient access for the hydrological community. The time series data (aka “data rods”) were integrated into hydrology community tools, such as the Data Rods Explorer on HydroShare. Furthermore, the GES DISC is in the process of migrating its data and services to the cloud. Hydrological data available at the GES DISC are now available in the Amazon Web Services (AWS) cloud (us-west-2 region) providing users Direct S3 data access and the capability for cloud computing operations. In this presentation, the hydrology data products and services currently available at the GES DISC will be summarized. Also discussed are the migration to the cloud, user support through this transition, and the status of migrating the data rods service to the cloud.

Ashley Heath↗

NASA NTRS1996

A Note on Interfacing Object Warehouses and Mass Storage Systems for Data Mining Applications

Data mining is the automatic discovery of patterns, associations, and anomalies in data sets. Data mining requires numerically and statistically intensive queries. Our assumption is that data mining requires a specialized data management infrastructure to support the aforementioned intensive queries, but because of the sizes of data involved, this infrastructure is layered over a hierarchical storage system. In this paper, we discuss the architecture of a system which is layered for modularity, but exploits specialized lightweight services to maintain efficiency. Rather than use a full functioned database for example, we use light weight object services specialized for data mining. We propose using information repositories between layers so that components on either side of the layer can access information in the repositories to assist in making decisions about data layout, the caching and migration of data, the scheduling of queries, and related matters.

Grossman, Robert L.↗

DOE OSTI2023

Microseismic Monitoring at the Farnsworth CO 2 -EOR Field

The Farnsworth Unit in northern Texas is a field site for studying geologic carbon storage during enhanced oil recovery (EOR) using CO 2 . Microseismic monitoring is essential for risk assessment by detecting fluid leakage and fractures. We analyzed borehole microseismic data acquired during CO 2 injection and migration, including data denoising, event detection, event location, magnitude estimation, moment tensor inversion, and stress field inversion. We detected and located two shallow clusters, which occurred during increasing injection pressure. The two shallow clusters were also featured by large b values and tensile cracking moment tensors that are obtained based on a newly developed moment tensor inversion method using single-borehole data. The inverted stress fields at the two clusters showed large deviations from the regional stress field. The results provide evidence for microseismic responses to CO 2 /fluid injection and migration.

02 PETROLEUM↗

NASA NTRS1999

Technology Assessment of High Capacity Data Storage Systems: Can We Avoid a Data Survivability Crisis?

In a recent address at the California Science Center in Los Angeles, Vice President Al Gore articulated a Digital Earth Vision. That vision spoke to developing a multi-resolution, three-dimensional visual representation of the planet into which we can roam and zoom into vast quantities of embedded geo-referenced data. The vision was not limited to moving through space, but also allowing travel over a time-line, which can be set for days, years, centuries, or even geological epochs. A working group of Federal Agencies, developing a coordinated program to implement the Vice President's vision, developed the definition of the Digital Earth as a visual representation of our planet that enables a person to explore and interact with the vast amounts of natural and cultural geo-referenced information gathered about the Earth. One of the challenges identified by the agencies was whether the technology existed that would be available to permanently store and deliver all the digital data that enterprises might want to save for decades and centuries. Satellite digital data is growing by Moore's Law as is the growth of computer generated data. Similarly, the density of digital storage media in our information-intensive society is also increasing by a factor of four every three years. The technological bottleneck is that the bandwidth for transferring data is only growing at a factor of four every nine years. This implies that the migration of data to viable long-term storage is growing more slowly. The implication is that older data stored on increasingly obsolete media are at considerable risk if they cannot be continuously migrated to media with longer life times. Another problem occurs when the software and hardware systems for which the media were designed are no longer serviced by their manufacturers. Many instances exist where support for these systems are phased out after mergers or even in going out of business. In addition, survivability of older media can suffer from physical breakdown of components (e.g. tapes simply lose their magnetic properties after a long time in storage). As a result, a potential data survivability crisis is emerging. The scale of the crisis is comparable to that facing the Social Security System. Sometime in one or two decades, the exponential growth of data will become so great that many enterprises will not be able to migrate through their data to more permanent media during the lifetime of the media on which it resides. This will result in significant losses of data and their resultant impacts. To avoid this crisis, we need to plan and devote greater financial and intellectual resources are needed for the development and refinement of new storage media and migration technologies in order to preserve all data any organization determines worth saving permanently. This talk will explore technological solutions and suggested recommendations to address this technological data crisis.

Halem, Milton↗

NASA NTRS2017

Technical Challenges and Lessons from the Migration of the GLOBE Data and Information System to Utilize Cloud Computing Service

The Global Learning and Observation to Benefit the Environment (GLOBE) Data and Information System supports an international science and education program with capabilities to accept local environment observations, archive, display and visualize them along with global satellite observations. Since its inception twenty years ago, the Web and database system has been upgraded periodically to accommodate the changes in technology and the steady growth of GLOBEs education community and collection of observations. Recently, near the end-of-life of the system hardware, new commercial computer platform options were explored and a decision made to utilize Cloud services. Now the GLOBE DIS has been fully deployed and maintained using Amazon Cloud services for over two years now. This paper reviews the early risks, actual challenges, and some unexpected findings as a result of the GLOBE DIS migration. We describe the plans, cost drivers and estimates, highlight adjustments that were made and suggest improvements. We present the trade studies for provisioning, for load balancing, networks, processing, storage, as well as production, staging and backup systems. We outline the migration teams skills and required level of effort for transition, and resulting changes in the overall maintenance and operations activities. Examples include incremental adjustments to processing capacity and frequency of backups, and efforts previously expended on hardware maintenance that were refocused onto application-specific enhancements.

GLOBE Data↗

NASA NTRS2016

Technical Challenges and Lessons from the Migration of the GLOBE Data and Information System to Utilize Cloud Computing Service

The Global Learning and Observation to Benefit the Environment (GLOBE) Data and Information System supports an international science and education program with capabilities to accept local environment observations, archive, display and visualize them along with global satellite observations. Since its inception twenty years ago, the Web and database system has been upgraded periodically to accommodate the changes in technology and the steady growth of GLOBEs education community and collection of observations. Recently, near the end-of-life of the system hardware, new commercial computer platform options were explored and a decision made to utilize Cloud services. Now the GLOBE DIS has been fully deployed and maintained using Amazon Cloud services for over two years now. This paper reviews the early risks, actual challenges, and some unexpected findings as a result of the GLOBE DIS migration. We describe the plans, cost drivers and estimates, highlight adjustments that were made and suggest improvements. We present the trade studies for provisioning, for load balancing, networks, processing, storage, as well as production, staging and backup systems. We outline the migration teams skills and required level of effort for transition, and resulting changes in the overall maintenance and operations activities. Examples include incremental adjustments to processing capacity and frequency of backups, and efforts previously expended on hardware maintenance that were refocused onto application-specific enhancements.

GLOBE↗

NASA NTRSDate not supplied

LAADS DAAC Migrates to the Cloud: Lessons Learned from Communicating About Earth Science Data on the Cloud

The Level-1 and Atmosphere Archive Distribution System (LAADS) Distributed Active Archive Center (DAAC) is migrating data to the cloud. As one of twelve DAACS supported by NASA’s Earth Science Data and Information System (ESDIS), LAADS is using moving away from on premise data storage facilities to migrating to Amazon Web Services, where the massive archive of data from the Moderate Imaging Spectroradiometer (MODIS) and the Visible Infrared Imaging Radiometer Suite (VIIRS) will be available for download and post-processing transformations online. The migration is happening in three phases and LAADS is concluding its beta testing period. This poster shows the lessons learned from communicating with a select group of users about how to effectively educate data users on using data in the cloud.

Tassia Owen↗

NASA NTRS2017

Trade Study: Storing NASA HDF5/netCDF-4 Data in the Amazon Cloud and Retrieving Data via Hyrax Server / THREDDS Data Server

As part of the overall effort to understand implications of migrating ESDIS data and services to the cloud we are testing several common OPeNDAP and HDF use cases against three architectures for general performance and cost characteristics. The architectures include retrieving entire files, retrieving datasets using HTTP range gets, and retrieving elements of datasets (chunks) with HTTP range gets. We will describe these architectures and discuss our approach to estimating cost.

HDF↗