Search NASA⌕ Search

SEARCH · Search NASA

Results for “data representation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,315 records · Page 73
All sourcesNASA NTRSDOE OSTI
NASA NTRS2005

New Directions in Space Operations Services in Support of Interplanetary Exploration

To gain access to the necessary operational processes and data in support of NASA's Lunar/Mars Exploration Initiative, new services, adequate levels of computing cycles and access to myriad forms of data must be provided to onboard spacecraft and ground based personnel/systems (earth, lunar and Martian) to enable interplanetary exploration by humans. These systems, cycles and access to vast amounts of development, test and operational data will be required to provide a new level of services not currently available to existing spacecraft, on board crews and other operational personnel. Although current voice, video and data systems in support of current space based operations has been adequate, new highly reliable and autonomous processes and services will be necessary for future space exploration activities. These services will range from the more mundane voice in LEO to voice in interplanetary travel which because of the high latencies will require new voice processes and standards. New services, like component failure predictions based on data mining of significant quantities of data, located at disparate locations, will be required. 3D or holographic representation of onboard components, systems or family members will greatly improve maintenance, operations and service restoration not to mention crew morale. Current operational systems and standards, like the Internet Protocol, will not able to provide the level of service required end to end from an end point on the Martian surface like a scientific instrument to a researcher at a university. Ground operations whether earth, lunar or Martian and in flight operations to the moon and especially to Mars will require significant autonomy that will require access to highly reliable processing capabilities, data storage based on network storage technologies. Significant processing cycles will be needed onboard but could be borrowed from other locations either ground based or onboard other spacecraft. Reliability will be a key factor with onboard and distributed backup processing an absolutely necessary requirement. Current cluster processing/Grid technologies may provide the basis for providing these services. An overview of existing services, future services that will be required and the technologies and standards required to be developed will be presented. The purpose of this paper will be to initiate a technological roadmap, albeit at a high level, of current voice, video, data and network technologies and standards (which show promise for adaptation or evolution) to what technologies and standards need to be redefined, adjusted or areas where new ones require development. The roadmap should begin the differentiation between non manned and manned processes/services where applicable. The paper will be based in part on the activities of the CCSDS Monitor and Control working group which is beginning the process of standardization of the these processes. Another element of the paper will be based on an analysis of current technologies supporting space flight processes and services at JSC, MSFC, GSFC and to a lesser extent at KSC. Work being accomplished in areas such as Grid computing, data mining and network storage at ARC, IBM and the University of Alabama at Huntsville will be researched and analyzed.

Bradford, Robert N.↗
DOE OSTI2020

Multichannel spectral estimation in acoustics: A state-space approach

Spectral estimation is a necessary methodology to analyze the frequency content of noisy data sets especially in acoustic applications. Many spectral techniques have evolved starting with the classical Fourier transform methods based on the well-known Wiener-Khintchine relationship relating the covariance-to-spectral density as a transform pair culminating with more elegant model-based parametric techniques that apply prior knowledge of the data to produce a high-resolution spectral estimate. Multichannel spectral representations are a class of both nonparametric, as well as parametric, estimators that provide improved spectral estimates. In any case, classical nonparametric multichannel techniques can provide reasonable estimates when coupled with peak-peaking methods as long as the signal levels are reasonably high. Furthermore, parametric multichannel methods can perform quite well in low signal level environments even when applying simple peak-picking techniques. In this paper, the performance of both nonparametric (periodogram) and parametric (state-space) multichannel spectral estimation methods are investigated when applied to both synthesized noisy structural vibration data as well as data obtained from a sounding rocket flight. It is demonstrated that for the multichannel problem, state-space techniques provide improved performance, offering a parametric alternative compared to classical methods.

42 ENGINEERING↗
NASA NTRS2006

Simulation of Urban Heat Island Mitigation Strategies in Atlanta, GA Using High-Resolution Land Use/Land Cover Data Set to Enhance Meteorological Modeling

The specification of land use/land cover (LULC) and associated land surface parameters in meteorological models at all scales has a major influence on modeled surface energy fluxes and boundary layer states. In urban areas, accurate representation of the land surface may be even more important than in undeveloped regions due to the large heterogeneity within the urban area. Deficiencies in the characterization of the land surface related to the spatial or temporal resolution of the data, the number of LULC classes defined, the accuracy with which they are defined, or the degree of heterogeneity of the land surface properties within each class may degrade the performance of the models. In this study, an experiment was conducted to test a new high-resolution LULC data set for meteorological simulations for the Atlanta, Georgia metropolitan area using a mesoscale meteorological model and to evaluate the effects of urban heat island (UHI) mitigation strategies on modeled meteorology for 2030. Simulation results showed that use of the new LULC data set reduced a major deficiency of the land use data used previously, specifically the poor representation of urban and suburban land use. Performance of the meteorological model improved substantially, with the overall daytime cold bias reduced by over 30%. UHI mitigation strategies were projected to offset much of a predicted urban warming between 2000 and 2030. In fact, for the urban core, the cooling due to UHI mitigation strategies was slightly greater than the warming associated with urbanization over this period. For the larger metropolitan area, cooling only partially offset the projected warming trend.

Crosson, William L.↗
DOE OSTI2022

Dynamic responses of submerged components in advanced reactors: experimental and numerical studies

The seismic design of an advanced nuclear reactor must consider the interaction of vessel internal components with the surrounding coolant: fluid–structure interaction (FSI). Available analytical solutions for FSI of submerged components do not accommodate multiple-component, intense seismic inputs and complex reactor and internal geometries. Physical testing of reactor vessels and internals to inform seismic design is impractical and cost-prohibitive, leaving the use of verified and validated, robust numerical models as the only plausible option for analysis and design. Physical data that could be used for validating such numerical models for multi-component shaking are not available. This article describes an experimental program performed on a 6-degree-of-freedom earthquake simulator to generate data that could support validation of seismic FSI numerical models for submerged components in commercial finite element codes. A scaled model of a base-supported reactor vessel with simplified representations of submerged internals was tested to generate submerged-component response histories for a range of seismic inputs. The generated data were used to validate numerical models in the finite element code LS-DYNA. Numerical models were validated for calculation of hydrodynamic pressure responses on internals, in-water frequencies of internals, and acceleration responses of internals. The generated data and the analysis recommendations could aid engineering analysts designing submerged components and systems for seismic effects.

Engineering↗
NASA NTRS2023

Characterization of Errors in Satellite-Based HCHO/NO 2 Tropospheric Column Ratios With Respect to Chemistry, Column-to-PBL Translation, Spatial Representation, and Retrieval Uncertainties

The availability of formaldehyde (HCHO) (a proxy for volatile organic compound reactivity) and nitrogen dioxide (NO 2 ) (a proxy for nitrogen oxides) tropospheric columns from ultraviolet–visible (UV–Vis) satellites has motivated many to use their ratios to gain some insights into the near-surface ozone sensitivity. Strong emphasis has been placed on the challenges that come with transforming what is being observed in the tropospheric column to what is actually in the planetary boundary layer (PBL) and near the surface; however, little attention has been paid to other sources of error such as chemistry, spatial representation, and retrieval uncertainties. Here we leverage a wide spectrum of tools and data to quantify those errors carefully. Concerning the chemistry error, a well-characterized box model constrained by more than 500 h of aircraft data from NASA's air quality campaigns is used to simulate the ratio of the chemical loss of HO 2 + RO 2 (LRO x ) to the chemical loss of NO x (LNO x ). Subsequently, we challenge the predictive power of HCHO/NO 2 ratios (FNRs), which are commonly applied in current research, in detecting the underlying ozone regimes by comparing them to LRO x /LNO x . FNRs show a strongly linear (R 2 =0.94) relationship with LRO x /LNO x , but only on the logarithmic scale. Following the baseline (i.e., ln(LRO x /LNO x ) = −1.0 ± 0.2) with the model and mechanism (CB06, r2) used for segregating NO x -sensitive from VOC-sensitive regimes, we observe a broad range of FNR thresholds ranging from 1 to 4. The transitioning ratios strictly follow a Gaussian distribution with a mean and standard deviation of 1.8 and 0.4, respectively. This implies that the FNR has an inherent 20 % standard error (1σ) resulting from not accurately describing the RO x –HO x cycle. We calculate high ozone production rates (PO 3 ) dominated by large HCHO × NO 2 concentration levels, a new proxy for the abundance of ozone precursors. The relationship between PO 3 and HCHO × NO 2 becomes more pronounced when moving towards NO x -sensitive regions due to nonlinear chemistry; our results indicate that there is fruitful information in the HCHO × NO 2 metric that has not been utilized in ozone studies. The vast amount of vertical information on HCHO and NO 2 concentrations from the air quality campaigns enables us to parameterize the vertical shapes of FNRs using a second-order rational function permitting an analytical solution for an altitude adjustment factor to partition the tropospheric columns into the PBL region. We propose a mathematical solution to the spatial representation error based on modeling isotropic semivariograms. Based on summertime-averaged data, the Ozone Monitoring Instrument (OMI) loses 12 % of its spatial information at its native resolution with respect to a high-resolution sensor like the TROPOspheric Monitoring Instrument (TROPOMI) (> 5.5 × 3.5 km 2 ). A pixel with a grid size of 216 km 2 fails at capturing ∼ 65 % of the spatial information in FNRs at a 50 km length scale comparable to the size of a large urban center (e.g., Los Angeles). We ultimately leverage a large suite of in situ and ground-based remote sensing measurements to draw the error distributions of daily TROPOMI and OMI tropospheric NO 2 and HCHO columns. At a 68 % confidence interval (1σ), errors pertaining to daily TROPOMI observations, either HCHO or tropospheric NO 2 columns, should be above 1.2–1.5 × 10 16 molec. cm −2 to attain a 20 %–30 % standard error in the ratio. This level of error is almost non-achievable with the OMI given its large error in HCHO. The satellite column retrieval error is the largest contributor to the total error (40 %–90 %) in the FNRs. Due to a stronger signal in cities, the total relative error (< 50 %) tends to be mild, whereas areas with low vegetation and anthropogenic sources (e.g., the Rocky Mountains) are markedly uncertain (> 100 %). Our study suggests that continuing development in the retrieval algorithm and sensor design and calibration is essential to be able to advance the application of FNRs beyond a qualitative metric.

Amir H. Souri↗
NASA NTRS1983

Standard ozone profiles from balloon and satellite data sets

Standard profiles based on upper level averaged profiles from BUV and lower level averaged profiles from balloon measurements are presented in a parametric representation as a function of time of year and latitude. The representation is a simple 4-parameter function representing the ozone amount (m-atm-cm) in each of 12 atmospheric layers defined following the standard Umkehr convention. The same parameterization is applied to the Nimbus-7 SBUV data and is compared to the BUV/balloon parameterization. The ozone variance unaccounted for by the representation is presented and discussed. The season-latitude representation reduces considerably the ozone variance at all levels and explains much of the correlation between layers. This simple representation and corresponding covariance matrix have been used as a priori information in the ozone vertical profile inversion of the Nimbus-7 SBUV experimental measurements.

Klenk, K. F.↗
NASA NTRS2021

Deep Domain Adaptation based Cloud Type Detection using Active and Passive Satellite Data

Domain adaptation techniques have been developed to handle data from multiple sources or domains. Most existing domain adaptation models assume that source and target domains are homogeneous, i.e., they have the same feature space. Nevertheless, many real world applications often deal with data from heterogeneous domains that come from completely different feature spaces. In our remote sensing application, data in source domain (from an active spaceborne Lidar sensor CALIOP onboard CALIPSO satellite) contain 25 attributes, while data in target domain (from a passive spectroradiometer sensor VIIRS onboard Suomi-NPP satellite) contain 20 different attributes. CALIOP has better representation capability and sensitivity to aerosol types and cloud phase, while VIIRS has wide swaths and better spatial coverage but has inherent weakness in differentiating atmospheric objects on different vertical levels. To address this mismatch of features across the domains/sensors, we propose a novel end-to-end deep domain adaptation with domain mapping and correlation alignment (DAMA) to align the heterogeneous source and target domains in active and passive satellite remote sensing data. It can learn domain invariant representation from source and target domains by transferring knowledge across these domains, and achieve additional performance improvement by incorporating weak label information into the model (DAMA-WL). Our experiments on a collocated CALIOP and VIIRS dataset show that DAMA and DAMA-WL can achieve higher classification accuracy in predicting cloud types.

domain adaptation↗
DOE OSTI2026

Data-Driven Modeling and Correction of Vehicle Dynamics

We develop a data-driven framework for learning and correcting nonautonomous vehicle dynamics. Physics-based vehicle models are often simplified for tractability and therefore exhibit inherent model-form uncertainty, motivating the need for data-driven correction. Moreover, nonautonomous dynamics are governed by time-dependent control inputs, which pose challenges in learning predictive models directly from temporal snapshot data. To address these, we reformulate the vehicle dynamics via a local parameterization of the time-dependent inputs, yielding a modified system composed ofa sequence of local parametric dynamical systems. Here, we approximate these parametric systems using two complementary approaches. First, we employ the dimension reduction and interpolation in parameter space (DRIPS) methodology to construct efficient linear surrogate models, equipped with lifted observable spaces and manifold-based operator interpolation. This enables data-efficient learning of vehicle models whose dynamics admit accurate linear representations in the lifted spaces. Second, for more strongly nonlinear systems, we employ flow map learning (FML), a deep neural network (DNN) approach that approximates the parametric evolution map without requiring special treatment of nonlinearities. We further extend FML with a transfer-learning-based model correction procedure, enabling the correction of misspecified prior models using only a sparse set of high-fidelity or experimental measurements, without assuming a prescribed form for the correction term. Through a suite of numerical experiments on unicycle, simplified bicycle, and slip-based bicycle models, we demonstrate that DRIPS offers robust and highly data-efficient learning of nonautonomous vehicle dynamics, while FML provides expressive nonlinear modeling and effective correction of model-form errors under severe data scarcity.

data-driven modeling↗
NASA NTRS1994

F2 peak electron density at Millstone Hill and Hobart: Comparsion of theory and measurement at solar maximum

This paper compares the observed behavior of the (F2) layer of the ionosphere at Millstone Hill and Hobart with calculations from the field line interhemispheric plasma (FLIP) model for solar maximum, solstice conditions in 1990. During the study period the daily F(sub 10.7) index varied by more than a factor of 2 (123 to 280), but the 81-day mean F(sub 10.7) (F(sub 10.7 A)) was almost constant near 190. Calculations were performed with and without the effects of vibrationally excited N2 (N(sup *)(sub 2) which affects the loss rate of atomic oxygen ions. In the case without N(sup *)(sub 2) there is generally good agreement between the model and measurement for the daytime, peak density of the F region (NmF2). Both the model and the measurement show a strong seasonal anomaly with the winter noon densities a factor of 3 to 4 greater than the summer noon densities at Millstone Hill and a factor of 2 greater at Hobart. The seasonal anomaly in the model is caused by changes in the neutral composition as given by the mass spectrometer and incoherent scatter (MSIS) 86 neutral density model. There is generally little or no increase in the observed noon NmF2 as a function of daily F(sub 10.7) except at Millstone Hill in winter. In contrast to the generally good agreement between model and data at noon, the model badly underestimates the density at night at Millstone Hill at all seasons. At Hobart the model reproduces the nighttime density variations well in both winter and summer. The international reference ionosphere (IRI) model generally provides a good representation of the average behavior of noon NmF2 and hmF2 but because the data show a lot of day-to-day variability, there are often large differences. The FLIP model is able to reproduce this variability when hmF2 is specified. The IRI model peak densities are better than the FLIP densities at night, but the IRI model does not represent the Millstone Hill summer data very well at night in 1990.

Richards, P. G.↗
DOE OSTI2022

Daylight simulation workflows incorporating measured bidirectional scattering distribution functions

Daylight predictions of architectural spaces depend on good estimates of light transfer through skylights, windows and other fenestration systems. For clear glazing and painted surfaces, parametric transmission and reflection models have proven adequate, but there are many cases where light-scattering, semi-specular shading and daylighting materials defy simple characterization. Something as commonplace as fabric roller shades and venetian blinds may turn daylight prediction into guesswork, and numerous advanced systems on the market tuned specifically to enhance daylight are not sufficiently characterized to distinguish their performance. In this paper, we describe new tools available to handle novel and specialized fabrics, materials, and devices using data-driven modelling of bi-directional scattering distribution functions (BSDFs). These representations are usually tabulated at constant or adjustable angular resolution for efficient point-in-time and annual daylight simulations. We describe a variety of BSDF simulation workflows, including some of the tools and methods that make advanced analysis possible, and highlight some of the current challenges. We conclude with a discussion of future work and how such data might be created and shared worldwide.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS1999

Adaptive Error Estimation in Linearized Ocean General Circulation Models

Data assimilation methods are routinely used in oceanography. The statistics of the model and measurement errors need to be specified a priori. This study addresses the problem of estimating model and measurement error statistics from observations. We start by testing innovation based methods of adaptive error estimation with low-dimensional models in the North Pacific (5-60 deg N, 132-252 deg E) to TOPEX/POSEIDON (TIP) sea level anomaly data, acoustic tomography data from the ATOC project, and the MIT General Circulation Model (GCM). A reduced state linear model that describes large scale internal (baroclinic) error dynamics is used. The methods are shown to be sensitive to the initial guess for the error statistics and the type of observations. A new off-line approach is developed, the covariance matching approach (CMA), where covariance matrices of model-data residuals are "matched" to their theoretical expectations using familiar least squares methods. This method uses observations directly instead of the innovations sequence and is shown to be related to the MT method and the method of Fu et al. (1993). Twin experiments using the same linearized MIT GCM suggest that altimetric data are ill-suited to the estimation of internal GCM errors, but that such estimates can in theory be obtained using acoustic data. The CMA is then applied to T/P sea level anomaly data and a linearization of a global GFDL GCM which uses two vertical modes. We show that the CMA method can be used with a global model and a global data set, and that the estimates of the error statistics are robust. We show that the fraction of the GCM-T/P residual variance explained by the model error is larger than that derived in Fukumori et al.(1999) with the method of Fu et al.(1993). Most of the model error is explained by the barotropic mode. However, we find that impact of the change in the error statistics on the data assimilation estimates is very small. This is explained by the large representation error, i.e. the dominance of the mesoscale eddies in the T/P signal, which are not part of the 21 by 1" GCM. Therefore, the impact of the observations on the assimilation is very small even after the adjustment of the error statistics. This work demonstrates that simult&neous estimation of the model and measurement error statistics for data assimilation with global ocean data sets and linearized GCMs is possible. However, the error covariance estimation problem is in general highly underdetermined, much more so than the state estimation problem. In other words there exist a very large number of statistical models that can be made consistent with the available data. Therefore, methods for obtaining quantitative error estimates, powerful though they may be, cannot replace physical insight. Used in the right context, as a tool for guiding the choice of a small number of model error parameters, covariance matching can be a useful addition to the repertory of tools available to oceanographers.

Chechelnitsky, Michael Y.↗
DOE OSTI2022

Grassmannian Diffusion Maps--Based Dimension Reduction and Classification for High-Dimensional Data

This work introduces the Grassmannian diffusion maps (GDMaps), a novel nonlinear dimensionality reduction technique that defines the affinity between points through their representation as low-dimensional subspaces corresponding to points on the Grassmann manifold. Here, the method is designed for applications, such as image recognition and data-based classification of constrained high-dimensional data where each data point itself is a high-dimensional object (i.e., a large matrix) that can be compactly represented in a lower-dimensional subspace. The GDMaps is composed of two stages. The first is a pointwise linear dimensionality reduction wherein each high-dimensional object is mapped onto the Grassmann manifold representing the low-dimensional subspace on which it resides. The second stage is a multipoint nonlinear kernel-based dimension reduction using diffusion maps to identify the subspace structure of the points on the Grassmann manifold. To this end, an appropriate Grassmannian kernel is used to construct the transition matrix of a random walk on a graph connecting points on the Grassmann manifold. Spectral analysis of the transition matrix yields low-dimensional Grassmannian diffusion coordinates embedding the data into a low-dimensional reproducing kernel Hilbert space. Further, a novel data classification/recognition technique is developed based on the construction of an overcomplete dictionary of reduced dimension whose atoms are given by the Grassmannian diffusion coordinates. Three examples are considered. First, a "toy" example shows that the GDMaps can identify an appropriate parametrization of structured points on the unit sphere. The second example demonstrates the ability of the GDMaps to revealing the intrinsic subspace structure of high-dimensional random field data. In the last ex- ample, a face recognition problem is solved considering face images subject to varying illumination conditions, changes in face expressions, and occurrence of occlusions. The technique presented high recognition rates (i.e., 95% in the best case) using a fraction of the data required by conventional methods.

42 ENGINEERING↗
DOE OSTI2024

ComStock™ 2024 Release 1 [SWR-19-33 and SWR-20-32]

ComStock™ is an NREL model of the U.S. commercial building stock. The model takes some building characteristics from the U.S. Department of Energy's (DOE's) Commercial Prototype Building Models and Commercial Reference Building. However, unlike many other building stock models, ComStock also combines these with a variety of additional public- and private-sector data sets. Collectively, this information provides high-fidelity building stock representation with a realistic diversity of building characteristics. This repository contains the source code used to build and execute ComStock models, including upgrade scenarios. In addition, the sampling of buildings characteristics used for the initial ComStock (V1.0) release is provided. The ComStock model is under active calibration and development, which is publicly visible on this repository. Execution of the ComStock workflow is managed through the buildstockbatch repository, a shared asset of ResStock™ and ComStock™ , specifically developed to scale to execution of tens of millions of simulations through multiple infrastructure providers. The dataset output from the initial ComStock (V1.0) release can be found at the accompanying ComStock data viewer website and additional information about ComStock found on the NREL Buildings Website. For more details about ongoing model development please consult the End Use Load Profiles website. ComStock is a direct result of the NREL residential stock modeling tool ResStock™ (recipient of a R&D100 award) and was inspired by the high-fidelity solar & storage adoption model dGen™. Additionally, this tool would not be possible without the decades of work undertaken by the OpenStudio® and EnergyPlus® visionaries and contributors, significant funding, feedback and support from the Los Angeles Department of Water and Power, and the Department of Energy's Building Technology Office ongoing support of and investment in building energy modeling software. is an analytic methodology for modeling the energy usage of the commercial building stock within the United States of America. The commercial building stock is represented through a sampling of complex probabilistic distributions of various features of interest for modeling energy usage within commercial buildings. Each sample from these distributions is converted into a building energy model based on the features of that specific sample. Each building energy model can be simulated as is, but additional changes can be made to the model through addition of energy conservation measures, component faults, or other desired alterations. The results of the simulations are then processed to provide insights for various stakeholders, including but not limited to policy makers, engineers, and marketers.

Horsey, Henry↗
NASA NTRSDate not supplied

CFD Validation Study of a Hypersonic Cone-Slice-Flap Configuration

Model validation is the process of determining the degree of accuracy between the real world and the model. The result of model validation can either be used to improve the model through calibration or quantify the model-form uncertainty. This work focuses on providing the model-form uncertainty through an area metric for a hypersonic cone-slice-flap geometry configuration given uncertainty in both the simulation and experimental data. A procedure will be presented that can give an accurate representation of the model-form uncertainty using a small number of high-fidelity runs. The work will also assess the impact between using different high-fidelity solvers and different turbulence models.The goal of performing this comparison is to provide a quantifiable measurement for the accuracy of each solver and turbulence model for this type of design. A low-fidelity analysis will also be performed to get a model-form uncertainty for the lower fidelity analysis. The two high-fidelty CFD solvers used are VULCAN-CFD and FUN3D and the low-fidelity results will come from Cart3D. The experimental data came from the 20-inch Mach 6Tunnel located at NASA Langley Research Center.

Laura M. White↗
NASA NTRS1999

Technology Assessment of High Capacity Data Storage Systems: Can We Avoid a Data Survivability Crisis?

In a recent address at the California Science Center in Los Angeles, Vice President Al Gore articulated a Digital Earth Vision. That vision spoke to developing a multi-resolution, three-dimensional visual representation of the planet into which we can roam and zoom into vast quantities of embedded geo-referenced data. The vision was not limited to moving through space, but also allowing travel over a time-line, which can be set for days, years, centuries, or even geological epochs. A working group of Federal Agencies, developing a coordinated program to implement the Vice President's vision, developed the definition of the Digital Earth as a visual representation of our planet that enables a person to explore and interact with the vast amounts of natural and cultural geo-referenced information gathered about the Earth. One of the challenges identified by the agencies was whether the technology existed that would be available to permanently store and deliver all the digital data that enterprises might want to save for decades and centuries. Satellite digital data is growing by Moore's Law as is the growth of computer generated data. Similarly, the density of digital storage media in our information-intensive society is also increasing by a factor of four every three years. The technological bottleneck is that the bandwidth for transferring data is only growing at a factor of four every nine years. This implies that the migration of data to viable long-term storage is growing more slowly. The implication is that older data stored on increasingly obsolete media are at considerable risk if they cannot be continuously migrated to media with longer life times. Another problem occurs when the software and hardware systems for which the media were designed are no longer serviced by their manufacturers. Many instances exist where support for these systems are phased out after mergers or even in going out of business. In addition, survivability of older media can suffer from physical breakdown of components (e.g. tapes simply lose their magnetic properties after a long time in storage). As a result, a potential data survivability crisis is emerging. The scale of the crisis is comparable to that facing the Social Security System. Sometime in one or two decades, the exponential growth of data will become so great that many enterprises will not be able to migrate through their data to more permanent media during the lifetime of the media on which it resides. This will result in significant losses of data and their resultant impacts. To avoid this crisis, we need to plan and devote greater financial and intellectual resources are needed for the development and refinement of new storage media and migration technologies in order to preserve all data any organization determines worth saving permanently. This talk will explore technological solutions and suggested recommendations to address this technological data crisis.

Halem, Milton↗
DOE OSTI2023

Optimizing training trajectories in variational autoencoders via latent Bayesian optimization approach *

Unsupervised and semi-supervised ML methods such as variational autoencoders (VAE) have become widely adopted across multiple areas of physics, chemistry, and materials sciences due to their capability in disentangling representations and ability to find latent manifolds for classification and/or regression of complex experimental data. Like other ML problems, VAEs require hyperparameter tuning, e.g. balancing the Kullback–Leibler and reconstruction terms. However, the training process and resulting manifold topology and connectivity depend not only on hyperparameters, but also their evolution during training. Because of the inefficiency of exhaustive search in a high-dimensional hyperparameter space for the expensive-to-train models, here we have explored a latent Bayesian optimization (zBO) approach for the hyperparameter trajectory optimization for the unsupervised and semi-supervised ML and demonstrated for joint-VAE with rotational invariances. We have demonstrated an application of this method for finding joint discrete and continuous rotationally invariant representations for modified national institute of standards and technology database (MNIST) and experimental data of a plasmonic nanoparticles material system. The performance of the proposed approach has been discussed extensively, where it allows for any high dimensional hyperparameter trajectory optimization of other ML models.

42 ENGINEERING↗
DOE OSTI2018

Multivariate network traffic analysis using clustered patterns

Traffic analysis is a core element in network operations and management for various purposes including change detection, traffic prediction, and anomaly detection. In this paper, we introduce a new approach to online traffic analysis based on a pattern-based representation for high-level summarization of the traffic measurement data. Unlike the past online analysis techniques limited to a single variable to summarize (e.g., sketch), the focus of this study is on capturing the network state from the multivariate attributes under consideration. To this end, we employ clustering with its benefit of the aggregation of multidimensional variables. The clustered result represents the state of the network with regard to the monitored variables, which can also be compared with the observed patterns from previous time windows enabling intuitive analysis. Finally, we demonstrate the proposed method with two popular use cases, one for estimating state changes and the other for identifying anomalous states, to confirm its feasibility. Our extensive experimental results with public traces and collected monitoring measurements from ESnet traffic traces show that our pattern-based approach is effective for multivariate analysis of online network traffic with visual and quantitative tools.

97 MATHEMATICS AND COMPUTING↗