SEARCH · Search NASA

Results for “data,”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

All sources NASA NTRS DOE OSTI

DOE OSTI2023

Integrated Large-Scale Data Management Platform for Photovoltaic Power Conversion Equipment (PCE) Reliability Data: Preprint

To meet the demand for accuracy and real-time capability of PV system degradation evaluation, massive volume data is needed to run high-fidelity and high-efficiency simulations and perform advanced data analysis. However, PV farm operators have a series of difficulties with PV inverter data, such as data collection from multiple channels, massive data storage, data management and massive data analysis. To address these challenges, we developed an integrated data management platform capable of data acquisition, processing, storage, query, and performing big data analysis utilizing AI algorithms. The platform can also achieve data correctness verification and provide an effective distributed data management solution to retrieve massive data and establish a connection to distributed computational frameworks.

data management↗

DOE OSTI2024

Super Resolution for Renewable Energy Resource Data With Wind From Reanalysis Data (Sup3rWind) and Application to Ukraine [Slides]

In this work we present a novel deep learning-based downscaling method, using generative adversarial networks (GANs), for generating high-resolution wind resource data from ECMWF Reanalysis v5 data (ERA5). We show that by training a GAN model on ERA5, as opposed to coarsened high-resolution data, we achieve results that are competitive with conventional dynamical downscaling. This GAN-based downscaling method additionally reduces computational costs over dynamical downscaling by two orders of magnitude. All GANs are trained on data sampled from CONUS, selected to provide a diverse sampling of terrain conditions, and validated on observational data along with data held out from training. This cross-validation shows low error and high correlations with observations and excellent agreement with hold out data across physical distributions. Our approach is finally used to downscale 30km hourly ERA5 to 2-km 5-minute wind data, for January 2000 through December 2023, at multiple hub heights, over Ukraine, Moldova, and part of Romania. Comparisons against observational data from Meteorological Assimilation Data Ingest System (MADIS) and multiple wind farms show the same level of performance as for CONUS validation. This 24 year data record is the first member of the "super resolution for renewable energy resource data with wind from reanalysis data" dataset (Sup3rWind).

17 WIND ENERGY↗

NASA NTRS1992

Remote Sensing Data from CLARET: A Prototype Cart Data Set

A data set containing radiation, meteorological, and cloud sensor observations is documented. It was prepared for use by the Department of Energy's Atmospheric Radiation Measurement (ARM) program and other interested scientists. These data are a precursor of the types of data that ARM Cloud And Radiation Testbed (CART) sites will provide. The data are from the Cloud Lidar And Radar Exploratory Test (CLARET) conducted by the Wave Propagation Laboratory during autumn 1989 in the Denver-Boulder area of Colorado primarily for the purpose of developing new cloud-sensing techniques on cirrus. After becoming aware of this experiment, ARM scientists requested archival of subsets or the data to assist in the developing ARM program. Five CLARET cases were selected: two with cirrus, one with stratus, one with mixed-phase clouds, and one with clear skies. The cases range from 2 to 9.5 h in length. A pyranometer, pyrgeometer, pyrheliometer, and an infrared radiometer constituted the ensemble of instruments that provided surface radiation data. A lidar, radar, and ceilometer observed the cloud geometrical structure, and visual reports and all-sky camera observations were assimilated to provide cloud cover data. Radiosondes, wind profiler, RASS (profiling virtual temperature), microwave radiometers (observing column integrated liquid water and water vapor), and standard surface measurements provided meteorological data. Satellite data from the stratus case and one cirrus case were analyzed for statistics on cloud cover and top height. The main body of the selected data are available on diskette from the Wave Propagation Laboratory or Los Alamos National Laboratory. In addition to documenting the data set, this report describes CLARET and gives a bibliography of publications associated with the project. Some preliminary results of CLARET' research are also summarized. Simultaneous CO 2 lidar and radar backscatter measurements were shown to provide estimates of the effective radius of ice particles. Simultaneous radar and infrared radiometer data appear useful for estimating column-integrated numbers and average sizes of ice cloud particles. Ice water content obtained with this method compared favorably with values from another empirical technique using radar data alone. Depolarization of the CO 2 lidar signal from ice clouds was surprisingly small, suggesting that calculation of backscatter from nonspherical particles for this lidar is a tractable problem. Examples are also cited of CO 2 lidar measurements of the effective radius of water cloud drop size distributions and of inference of the size of pristine ice crystals that assume a particular orientation in the air. These parameters are all important to radiative transfer through clouds.

Clouds (Meteorology)↗

NASA NTRS2019

Modelling above-ground biomass stock over Norway using national forest inventory data with ArcticDEM and Sentinel-2 data

Boreal forests constitute a large portion of the global forest area, yet they are undersampled through field surveys, and only a few remotely sensed data sources provide structural information wall-to-wall throughout the boreal domain. ArcticDEM is a collection of high-resolution (2 m) space-borne stereogrammetric digital surface models (DSM) covering the entire land area north of 60° of latitude. The free-availability of ArcticDEM data offers new possibilities for aboveground biomass mapping (AGB) across boreal forests, and thus it is necessary to evaluate the potential for these data to map AGB over alternative open-data sources (i.e., Sentinel-2). This study was performed over the entire land area of Norway north of 60° of latitude, and the Norwegian national forest inventory (NFI) was used as a source of field data composed of accurately geolocated field plots (n=7710) systematically distributed across the study area. Separate random forest models were fitted using NFI data, and corresponding remotely sensed data consisting of either: i) a canopy height model (ArcticCHM) obtained by subtracting a high-quality digital terrain model (DTM) from the ArcticDEM DSM height values, ii) Sentinel-2 (S2), or iii) a combination of the two (ArcticCHM+S2). Furthermore, we assessed the effect of the forest- and terrain-specific factors on the models’ predictive accuracy. The best model (,i.e., ArcticCHM+S2) explained nearly 60% of the variance of the training set, which translated in the largest accuracy in terms of root mean square error (RMSE=41.4 t/ha). This result highlights the synergy between 3D and multispectral data in AGB modelling. Furthermore, this study showed that despite the importance of ArcticCHM variables, the S2 model performed slightly better than ArcticCHM model. This finding highlights some of the limitations of ArcticDEM, which, despite the unprecedented spatial resolution, is highly heterogeneous due to the blending of multiple acquisitions across different years and seasons. We found that both forest- and terrain-specific characteristics affected the uncertainty of the ArcticCHM+S2 model and concluded that the combined use of ArcticCHM and Sentinel-2 represents a viable solution for AGB mapping across boreal forests. The synergy between the two data sources allowed for a reduction of the saturation effects typical of multispectral data while ensuring the spatial consistency in the output predictions due to the removal of artifacts and data voids present in ArcticCHM data. While the main contribution of this study is to provide the first evidence of the best-case-scenario (i.e., availability of accurate terrain models) that ArcticDEM data can provide for large-scale AGB modelling, it remains critically important for other studies to investigate how ArcticDEM may be used in areas where no DTMs are available as is the case for large portions of the boreal zone.

space-borne imagery↗

NASA NTRSDate not supplied

The NASA Open Science Data Repository: Biomedical Fair Data, Analysis Tools, User Communities, Publications, and Discoveries for Deep Space Missions

Increased biomedical risks and challenges associated with deep space missions require new knowledge discovery, new health countermeasures, and development of novel ecosystems, life support, crop production, and biomedical support capabilities. To meet NASA’s Moon to Mars strategic program goals for Human and Biological Sciences, findable, accessible, interoperable, reusable (FAIR), and maximally open-access data is going to be required to enable humanity to thrive in deep space. Indeed, this cornerstone perspective on FAIR and maximally open access data was also recommended in the recent 2023-2032 Decadal Survey from the National Academies of Sciences, Engineering, and Medicine. The NASA Open Science Data Repository (OSDR) is a maximally open access and FAIR database, and meets various scientific, technical, and operational spaceflight needs. It offers public users and submitters the ability to upload, download, search, share, analyze, and visualize data across ‘omics, physiological, phenotypic, behavioral, bioimaging, video, and environmental monitoring telemetry datasets. OSDR includes NASA GeneLab, NASA Ames Life Sciences Data Archive, and the NASA Biological Institutional Scientific Collection. OSDR has >455 studies with datasets from model organisms and non-NASA human astronauts. There are ~12 datasets from the Inspiration 4 (I4) mission, spanning metagenomics, comprehensive metabolic panels, clonal hematopoiesis, spatial transcriptomics, proteomics, and cytokine panels. In the interest of data privacy, two I4 datasets have raw FASTQ and FASTA files relating to the epitranscriptome, and a new request feature is live in OSDR (with a backend review process established) which was developed based on industry norms. OSDR also recently began a collaboration with the European Space Agency (ESA) to scientifically curate and make available >200 terabytes of human and model organism space-relevant data. The OSDR submission portal is designed to ingest and curate ~25 ‘omics assay data types, and ~50 physiological-phenotypic-imaging assay data types, spanning ultrasonography, micro-computed tomography, histology, morphometric photography, rebound tonometry, gait analysis, optical coherence tomography, novel object recognition, flow cytometry, and immunohistochemistry. A suite of analysis tools are available for OSDR users including: 1) an Environmental Data Application to compare radiation, CO2, relative humidity, temperature, and other telemetry across missions and subjects, 2) the RadLab database, a collaboration between NASA, ESA, the German and Italian Space Agencies, and the Bulgarian Academy of Sciences, which compiles radiation measurements relevant to human spaceflight and provides tools for accessing and manipulating the data, and 3) a Multi-study visualization tool which enables users to look across and combine GeneLab’s omics datasets across different experiments and missions. There are ~600 volunteer OSDR Analysis Working Group (AWG) members who: 1) provide feedback on scientific standards for reuse (subject and assay metadata; processing pipelines; dataset formats and uniformed structures for machine-readability), and 2) collaborate to mine-reuse OSDR data conducting scientific analysis. OSDR has enabled 60 publications as of September 2023, many directly from AWG collaborations most notably the Cell Press package in 2020. Lastly, there are at least 15 articles which mine OSDR data part of a package of ~50 articles across Nature Portfolio with research stemming from I4, the Japan Aerospace Exploration Agency, NASA Space Biology, and the NASA Human Research Program.

space biology↗

NASA NTRSDate not supplied

NASA Open Science Data Repository: Biomedical FAIR Data, Analysis Tools, User Communities, and Discoveries for Deep Space Missions

open access↗

DOE OSTI2024

Real‐time XFEL data analysis at SLAC and NERSC: A trial run of nascent exascale experimental data analysis

X‐ray scattering experiments using free electron lasers (XFELs) are a powerful tool to determine the molecular structure and function of unknown samples (such as COVID‐19 viral proteins). XFEL experiments are a challenge to computing in two ways: (i) due to the high cost of running XFELs, a fast turnaround time from data acquisition to data analysis is essential to make informed decisions on experimental protocols; (ii) data‐collection rates are growing exponentially, requiring new scalable algorithms. Here we report our experiences analyzing data from two experiments at the Linac Coherent Light Source (LCLS) during September 2020. Raw data were analyzed on NERSC's Cori XC40 system, using the Superfacility paradigm: our workflow automatically moves raw data between LCLS and NERSC, where it is analyzed using the software package CCTBX. We achieved real time data analysis with a turnaround time from data acquisition to full molecular reconstruction in as little as 10 min—sufficient time for the experiment's operators to make informed decisions. By hosting the data analysis on Cori, and by automating LCLS‐NERSC interoperability, we achieved a data analysis rate which matches the data acquisition rate. Completing data analysis within 10 min is a first for XFEL experiments and an important milestone if we are to keep up with data‐collection trends.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2024

DICER: Data Intensive Computing Environment and Runtime for Evaluating Unprecedented Scale of Geospatial-Temporal Human Mobility Data

With the significant increase in sources and volume of human mobility data through commercial data vendors as well as microsimulation of cities, the scale of geospatial-temporal data to analyze and assess for mobility characterization has grown to the level of Big Data. There are mobility related commercial organizations deploying scalable computing, but often the system architecture, workflow, and intermediate processing components are not fully disclosed in relevant scope. Current research literature has a notable lack of studies demonstrating architectures and workflows for human mobility analytics that are implemented on a TeraByte scale of geospatial-temporal data. In this context, this paper presents a hyperscale-level system solution named DICER (Data Intensive Computing Environment and Runtime) for processing and analytics of geospatial-temporal data at big data scale. Although the cluster computing architecture of DICER with Apache Spark job running on Kubernetes cluster is not new, there are innovations in the workflow, hierarchical processing logic, and a wide range of intermediate preprocessing and mobility metrics calculation. We have performed case studies to validate the effectiveness of DICER system solution by performing detailed analytics and assessment of human mobility microsimulation output at three different scopes and scale, including a usecase with 16.97 TeraByte and 259.2 Billion rows of data. In addition, we have presented another case study of utilizing DICER to perform the same mobility processing and comparative analytics on large-scale commercially available geospatial-temporal data. All these case studies validate the efficiency and usefulness of DICER in computing population mobility characteristics from geospatial-temporal trajectory data at an unprecedented scale (not only just data volume, but also combination of: number of user entities, temporal frequency, spatial resolution, data duration).

De, Debraj↗

DOE OSTI2020

Validation of finite-element models using full-field experimental data: Levelling finite-element analysis data through a digital image correlation engine

Full-field data from digital image correlation (DIC) provide rich information for finite-element analysis (FEA) validation. However, there are several inherent inconsistencies between FEA and DIC data that must be rectified before meaningful, quantitative comparisons can be made, including strain formulations, coordinate systems, data locations, strain calculation algorithms, spatial resolutions and data filtering. As such, in this paper, we investigate two full-field validation approaches: (1) the direct interpolation approach, which addresses the first three inconsistencies by interpolating the quantity of interest from one mesh to the other, and (2) the proposed DIC-levelling approach, which addresses all six inconsistencies simultaneously by processing the FEA data through a stereo-DIC simulator to ‘level’ the FEA data to the DIC data in a regularisation sense. Synthetic ‘experimental’ DIC data were generated based on a reference FEA of an exemplar test specimen. The direct interpolation approach was applied, and significant strain errorswere computed, even though therewas no model form error, because the filtering effect of theDIC enginewas neglected. In contrast, the levelling approach provided accurate validation results, with no strain error when no model form error was present. Next, model form error was purposefully introduced via a mismatch of boundary conditions. With the direct interpolation approach, the mismatch in boundary conditions was completely obfuscated, while with the levelling approach, it was clearly observed. Finally, the ‘experimental’ DIC datawere purposefully misaligned slightly fromthe FEA data. Both validation techniques suffered from the misalignment, thus motivating continued efforts to develop a robust alignment process. In conclusion, direct interpolation is insufficient, and the proposed levelling approach is required to ensure that the FEA and the DIC data have the same spatial resolution and data filtering. Only after the FEA data have been ‘levelled’ to the DIC data can meaningful, quantitative error maps be computed.

42 ENGINEERING↗

DOE OSTI2024

Building a FAIR data ecosystem for incorporating single-cell transcriptomics data into agricultural genome to phenome research

Introduction The agriculture genomics community has numerous data submission standards available, but the standards for describing and storing single-cell (SC, e.g., scRNA- seq) data are comparatively underdeveloped. Methods To bridge this gap, we leveraged recent advancements in human genomics infrastructure, such as the integration of the Human Cell Atlas Data Portal with Terra, a secure, scalable, open-source platform for biomedical researchers to access data, run analysis tools, and collaborate. In parallel, the Single Cell Expression Atlas at EMBL-EBI offers a comprehensive data ingestion portal for high-throughput sequencing datasets, including plants, protists, and animals (including humans). Developing data tools connecting these resources would offer significant advantages to the agricultural genomics community. The FAANG data portal at EMBL-EBI emphasizes delivering rich metadata and highly accurate and reliable annotation of farmed animals but is not computationally linked to either of these resources. Results Herein, we describe a pilot-scale project that determines whether the current FAANG metadata standards for livestock can be used to ingest scRNA-seq datasets into Terra in a manner consistent with HCA Data Portal standards. Importantly, rich scRNA-seq metadata can now be brokered through the FAANG data portal using a semi-automated process, thereby avoiding the need for substantial expert curation. We have further extended the functionality of this tool so that validated and ingested SC files within the HCA Data Portal are transferred to Terra for further analysis. In addition, we verified data ingestion into Terra, hosted on Azure, and demonstrated the use of a workflow to analyze the first ingested porcine scRNA-seq dataset. Additionally, we have also developed prototype tools to visualize the output of scRNA-seq analyses on genome browsers to compare gene expression patterns across tissues and cell populations. This JBrowse tool now features distinct tracks, showcasing PBMC scRNA-seq alongside two bulk RNA-seq experiments. Discussion We intend to further build upon these existing tools to construct a scientist-friendly data resource and analytical ecosystem based on Findable, Accessible, Interoperable, and Reusable (FAIR) SC principles to facilitate SC-level genomic analysis through data ingestion, storage, retrieval, re-use, visualization, and comparative annotation across agricultural species.

Genetics & Heredity↗

NASA NTRS1975

An interdisciplinary analysis of multispectral satellite data for selected cover types in the Colorado Mountains, using automatic data processing techniques

The author has reported the following significant results. A data set containing SKYLAB, LANDSAT, and topographic data has been overlayed, registered, and geometrically corrected to a scale of 1:24,000. After geometrically correcting both sets of data, the SKYLAB data were overlayed on the LANDSAT data. Digital topographic data were then obtained, reformatted, and a data channel containing elevation information was then digitally overlayed onto the LANDSAT and SKYLAB spectral data. The 14,039 square kilometers involving 2,113, 776 LANDSAT pixels represents a relatively large data set available for digital analysis. The overlayed data set enables investigators to numerically analyze and compare two sources of spectral data and topographic data from any point in the scene. This capability is new and it will permit a numerical comparison of spectral response with elevation, slope, and aspect. Utilization of the spectral and topographic data together to obtain more accurate classifications of the various cover types present is feasible.

Hoffer, R. M.↗

NASA NTRS1991

Data compression for data archival, browse or quick-look

Soon after space and Earth science data is collected, it is stored in one or more archival facilities for later retrieval and analysis. Since the purpose of the archival process is to keep an accurate and complete record of data, any data compression used in an archival system must be lossless, and protect against propagation of error in the storage media. A browse capability for space and Earth science data is needed to enable scientists to check the appropriateness and quality of particular data sets before obtaining the full data set(s) for detailed analysis. Browse data produced for these purposes could be used to facilitate the retrieval of data from an archival facility. Quick-look data is data obtained directly from the sensor for either previewing the data or for an application that requires very timely analysis of the space or Earth science data. Two main differences between data compression techniques appropriate to browse and quick-look cases, are that quick-look can be more specifically tailored, and it must be limited in complexity by the relatively limited computational power available on space platforms.

Dozier, Jeff↗

NASA NTRS1994

Use of data description languages in the interchange of data

The Consultative Committee for Space Data Systems (CCSDS) is developing Standards for the interchange of information between systems, including those operating under different environments. The objective is to perform the interchange automatically, i.e. in a computer interpretable manner. One aspect of the concept developed by CCSDS is the use of a separate data description to specify the data being transferred. Using the description, data can then be automatically parsed by the receiving computer. With a suitably expressive Data Description Language (DDL), data formats of arbitrary complexity can be handled. The advantages of this approach are: (1) that the description need only be written and distributed once to all users, and (2) new software does not need to be written for each new format, provided generic tools are available to support writing and interpretation of descriptions and the associated data instances. Consequently, the effort of 'hard coding' each new format is avoided and problems of integrating multiple implementations of a given format by different users are avoided. The approach is applicable in any context where computer parsable description of data could enhance efficiency (e.g. within a spacecraft control system, a data delivery system or an archive). The CCSDS have identified several candidate DDL's: EAST (Extended Ada Subset), TSDN (Transfer Syntax Data Notation) and MADEL (Modified ASN.1 as a Data Description Language -- a DDL based on the Abstract Syntax Notation One - ASN.1 - specified in the ISO/IEC 8824). This paper concentrates on ESA's development of MADEL. ESA have also developed a 'proof of concept' prototype of the required support tools, implemented on a PC under MS-DOS, which has successfully demonstrated the feasibility of the approach, including the capability within an application of retrieving and displaying particular data elements, given its MADEL description (i.e. a data description written in MADEL). This paper outlines the work done to date and assesses the applicability of this modified ASN.1 as a DDL. The feasibility of the approach is illustrated with several examples.

Pignede, M.↗

NASA NTRS1998

Technology Assessment of High Capacity Data Storage Systems: Can We Avoid a Data Survivability Crisis?

This technology assessment of long-term high capacity data storage systems identifies an emerging crisis of severe proportions related to preserving important historical data in science, healthcare, manufacturing, finance and other fields. For the last 50 years, the information revolution, which has engulfed all major institutions of modem society, centered itself on data-their collection, storage, retrieval, transmission, analysis and presentation. The transformation of long term historical data records into information concepts, according to Drucker, is the next stage in this revolution towards building the new information based scientific and business foundations. For this to occur, data survivability, reliability and evolvability of long term storage media and systems pose formidable technological challenges. Unlike the Y2K problem, where the clock is ticking and a crisis is set to go off at a specific time, large capacity data storage repositories face a crisis similar to the social security system in that the seriousness of the problem emerges after a decade or two. The essence of the storage crisis is as follows: since it could take a decade to migrate a peta-byte of data to a new media for preservation, and the life expectancy of the storage media itself is only a decade, then it may not be possible to complete the transfer before an irrecoverable data loss occurs. Over the last two decades, a number of anecdotal crises have occurred where vital scientific and business data were lost or would have been lost if not for major expenditures of resources and funds to save this data, much like what is happening today to solve the Y2K problem. A pr-ime example was the joint NASA/NSF/NOAA effort to rescue eight years worth of TOVS/AVHRR data from an obsolete system, which otherwise would have not resulted in the valuable 20-year long satellite record of global warming. Current storage systems solutions to long-term data survivability rest on scalable architectures having parallel paths for data migration.

Halem, M.↗

NASA NTRS1998

Prototype Local Data Integration System and Central Florida Data Deficiency

This report describes the Applied Meteorology Unit's (AMU) task on the Local Data Integration System (LDIS) and central Florida data deficiency. The objectives of the task are to identify all existing meteorological data sources within 250 km of the Kennedy Space Center (KSC) and the Eastern Range at Cape Canaveral Air Station (CCAS), identify and configure an appropriate LDIS to integrate these data, and implement a working prototype to be used for limited case studies and data non-incorporation (DNI) experiments. The ultimate goal for running LDIS is to generate products that may enhance weather nowcasts and short-range (less than 6 h) forecasts issued in support of the 45th Weather Squadron (45 WS), Spaceflight Meteorology Group (SMG), and the Melbourne National Weather Service (NWS MLB) operational requirements. The LDIS has the potential to provide added value for nowcasts and short term forecasts for two reasons. First, it incorporates all data operationally available in east central Florida. Second, it is run at finer spatial and temporal resolutions than current national-scale operational models. In combination with a suitable visualization tool, LDIS may provide users with a more complete and comprehensive understanding of evolving fine-scale weather features than could be developed by individually examining the disparate data sets over the same area and time. The utility of LDIS depends largely on the reliability and availability of observational data. Therefore, it is important to document all existing meteorological data sources around central Florida that can be incorporated by it. Several factors contribute to the data density and coverage over east central Florida including the level in the atmosphere, distance from KSC/CCAS, time, and prevailing weather. The central Florida mesonet consists of existing surface meteorological and hydrological data available from the Tampa NWS and data servers at Miami and Jacksonville. However the utility of these data for operational use is limited, mainly because there are relatively few additional meteorological observations within 50 km of KSC/CCAS to supplement existing METAR and KSC/CCAS tower reports.

Manobianco, John↗

NASA NTRS1999

Technology Assessment of High Capacity Data Storage Systems: Can We Avoid a Data Survivability Crisis?

In a recent address at the California Science Center in Los Angeles, Vice President Al Gore articulated a Digital Earth Vision. That vision spoke to developing a multi-resolution, three-dimensional visual representation of the planet into which we can roam and zoom into vast quantities of embedded geo-referenced data. The vision was not limited to moving through space, but also allowing travel over a time-line, which can be set for days, years, centuries, or even geological epochs. A working group of Federal Agencies, developing a coordinated program to implement the Vice President's vision, developed the definition of the Digital Earth as a visual representation of our planet that enables a person to explore and interact with the vast amounts of natural and cultural geo-referenced information gathered about the Earth. One of the challenges identified by the agencies was whether the technology existed that would be available to permanently store and deliver all the digital data that enterprises might want to save for decades and centuries. Satellite digital data is growing by Moore's Law as is the growth of computer generated data. Similarly, the density of digital storage media in our information-intensive society is also increasing by a factor of four every three years. The technological bottleneck is that the bandwidth for transferring data is only growing at a factor of four every nine years. This implies that the migration of data to viable long-term storage is growing more slowly. The implication is that older data stored on increasingly obsolete media are at considerable risk if they cannot be continuously migrated to media with longer life times. Another problem occurs when the software and hardware systems for which the media were designed are no longer serviced by their manufacturers. Many instances exist where support for these systems are phased out after mergers or even in going out of business. In addition, survivability of older media can suffer from physical breakdown of components (e.g. tapes simply lose their magnetic properties after a long time in storage). As a result, a potential data survivability crisis is emerging. The scale of the crisis is comparable to that facing the Social Security System. Sometime in one or two decades, the exponential growth of data will become so great that many enterprises will not be able to migrate through their data to more permanent media during the lifetime of the media on which it resides. This will result in significant losses of data and their resultant impacts. To avoid this crisis, we need to plan and devote greater financial and intellectual resources are needed for the development and refinement of new storage media and migration technologies in order to preserve all data any organization determines worth saving permanently. This talk will explore technological solutions and suggested recommendations to address this technological data crisis.

Halem, Milton↗

NASA NTRS2008

The User Community and a Multi-Mission Data Project: Services, Experiences and Directions of the Space Physics Data Facility

From a user's perspective, the multi-mission data and orbit services of NASA's Space Physics Data Facility (SPDF) project offer a unique range of important data and services highly complementary to other services presently available or now evolving in the international heliophysics data environment. The VSP (Virtual Space Physics Observatory) service is an active portal to a wide range of distributed data sources. CDAWeb (Coordinate Data Analysis Web) enables plots, listings and file downloads for current data cross the boundaries of missions and instrument types (and now including data from THEMIS and STEREO). SSCWeb, Helioweb and our 3D Animated Orbit Viewer (TIPSOD) provide position data and query logic for most missions currently important to heliophysics science. OMNIWeb with its new extension to 1- and 5-minute resolution provides interplanetary parameters at the Earth's bow shock as a unique value-added data product. SPDF also maintains NASA's CDF (common Data Format) standard and a range of associated tools including translation services. These capabilities are all now available through webservices-based APIs as well as through our direct user interfaces. In this paper, we will demonstrate the latest data and capabilities now supported in these multi-mission services, review the lessons we continue to learn in what science users need and value in this class of services, and discuss out current thinking to the future role and appropriate focus of the SPDF effort in the evolving and increasingly distributed heliophysics data environment.

Fung, Shing F.↗

NASA NTRS2007

Processing Raw HST Data With Up-to-Date Calibration Data

On-the-Fly Reprocessing (OTFR) is a collection of data-processing routines that work within the context of the Hubble Space Telescope (HST) pipeline data-flow system. The purpose served by OTFR is to generate, on demand, scientifically useful data products from raw HST data stored in an archive. First, on the basis of the requested final data products, OTFR retrieves the corresponding sets of raw data from the archives. Next, OTFR processes the raw data sets to remove artifacts and to establish proper header and other template information. Finally, the calibration routines appropriate to the specific data sets are invoked to produce the requested data products, and the data products are released to an archive distribution system for transmission to the requesting party. OTFR offers two notable advantages: (1) Inasmuch as calibrated data occupy about 8 times as much storage space as do raw data, by obviating storage of calibrated data, OTFR reduces the storage capacity needed by the archive; and (2) the calibration routines can be updated to give requesters the benefit of the most recent calibrations.

Miller, Warren↗