Spectral analysis of data from the ''SIRIO'' fluxgate magnetometer
Analyzing effects on spectral analysis of uniformly sampled data streams containing missing value
Ness, N. F.↗
SEARCH · Search NASA
Results for “missing and streaming data”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Random Forest Regressor-Based Approach for Detecting Fault Location and Duration in Power Systems
Power system failures or outages due to short-circuits or “faults” can result in long service interruptions leading to significant socio-economic consequences. It is critical for electrical utilities to quickly ascertain fault characteristics, including location, type, and duration, to reduce the service time of an outage. Existing fault detection mechanisms (relays and digital fault recorders) are slow to communicate the fault characteristics upstream to the substations and control centers for action to be taken quickly. Fortunately, due to availability of high-resolution phasor measurement units (PMUs), more event-driven solutions can be captured in real time. In this paper, we propose a data-driven approach for determining fault characteristics using samples of fault trajectories. A random forest regressor (RFR)-based model is used to detect real-time fault location and its duration simultaneously. This model is based on combining multiple uncorrelated trees with state-of-the-art boosting and aggregating techniques in order to obtain robust generalizations and greater accuracy without overfitting or underfitting. Four cases were studied to evaluate the performance of RFR: 1. Detecting fault location (case 1), 2. Predicting fault duration (case 2), 3. Handling missing data (case 3), and 4. Identifying fault location and length in a real-time streaming environment (case 4). A comparative analysis was conducted between the RFR algorithm and state-of-the-art models, including deep neural network, Hoeffding tree, neural network, support vector machine, decision tree, naive Bayesian, and K-nearest neighborhood. Experiments revealed that RFR consistently outperformed the other models in detection accuracy, prediction error, and processing time.
24 POWER TRANSMISSION AND DISTRIBUTION↗
A Visual Analytics Framework for Reviewing Streaming Performance Data
Understanding and tuning the performance of extreme-scale parallel computing systems demands a streaming approach due to the computational cost of applying offline algorithms to vast amounts of performance log data. Analyzing large streaming data is challenging because the rate of receiving data and limited time to comprehend data make it difficult for the analysts to sufficiently examine the data without missing important changes or patterns. To support streaming data analysis, we introduce a visual analytic framework comprising of three modules: data management, analysis, and interactive visualization. The data management module collects various computing and communication performance metrics from the monitored system using streaming data processing techniques and feeds the data to the other two modules. The analysis module automatically identifies important changes and patterns at the required latency. In particular, we introduce a set of online and progressive analysis methods for not only controlling the computational costs but also helping analysts better follow the critical aspects of the analysis results. Finally, the interactive visualization module provides the analysts with a coherent view of the changes and patterns in the continuously captured performance data. Through a multi-faceted case study on performance analysis of parallel discrete-event simulation, we demonstrate the effectiveness of our framework for identifying bottlenecks and locating outliers.
Human-centered computing↗
Atmospheric Disturbance Environment Definition
Traditionally, the application of atmospheric disturbance data to airplane design problems has been the domain of the structures engineer. The primary concern in this case is the design of structural components sufficient to handle transient loads induced by the most severe atmospheric "gusts" that might be encountered. The concern has resulted in a considerable body of high altitude gust acceleration data obtained with VGH recorders (airplane velocity, V, vertical acceleration, G, altitude, H) on high-flying airplanes like the U-2 (Ehernberger and Love, 1975). However, the propulsion system designer is less concerned with the accelerations of the airplane than he is with the airflow entering the system's inlet. When the airplane encounters atmospheric turbulence it responds with transient fluctuations in pitch, yaw, and roll angles. These transients, together with fluctuations in the free-stream temperature and pressure will disrupt the total pressure, temperature, Mach number and angularity of the inlet flow. For the mixed compression inlet, the result is a disturbed throat Mach number and/or shock position, and in extreme cases an inlet unstart can occur (cf. Section 2.1). Interest in the effects of inlet unstart on the vehicle dynamics of large, supersonic airplanes is not new. Results published by NASA in 1962 of wind tunnel studies of the problem were used in support of the United States Supersonic Transport program (SST) (White, at aI, 1963). Such studies continued into the late 1970's. However, in spite of such interest, there never was developed an atmospheric disturbance database for inlet unstart analysis to compare with that available for the structures load analysis. Missing were data for the free-stream temperature and pressure disturbances that also contribute to the unStart problem.
Tank, William G.↗
Near-continuum, hypersonic oxygen flow over a double cone simulated by direct simulation Monte Carlo informed from quantum chemistry
A large-scale, fully resolved direct simulation Monte Carlo (DSMC) computation of a non-equilibrium, reactive flow of pure oxygen over a double cone is presented. Under the simulated near-continuum conditions, the computational demands are shown to be significant because of the wide range of length scales that must be resolved. Therefore, robust grid adaption capabilities and efficient parallelization of the Stochastic PArallel Rarefied-gas Time-accurate Analyzer (SPARTA) code that is utilized in this work are essential. The thermochemical and transport collision models were selected for efficiency and simplicity. First-principles data, obtained from the highly accurate direct molecular simulation method, were used to inform the collision models’ parameters. Importantly, because SPARTA implements molecular collision models using collision-specific energies, the resulting macroscopic relaxation rates were evaluated a posteriori via zero-dimensional heat bath simulations. The comparisons of surface properties, namely heat flux and pressure, show very close agreement with previous computational fluid dynamics (CFD) results. Differences with the measurements were found to be similar to the CFD simulations. The unresolved discrepancy with the measurements could be due to inconsistent free stream conditions with the actual experimental data or missing physical phenomena altogether, for example atomic and molecular oxygen electronically excited states, three-dimensional effects, or more complex gas–surface interactions. As shown in this work, the advantages of obtaining a DSMC particle solution for these flows reside in the method's ability to be directly informed from first principles and to seamlessly describe internal energy non-equilibrium for all modes. With the advent of exascale computing and beyond, particle methods will be an increasingly important tool to verify the validity of physical assumptions in reduced-order models via fully resolved, experimental-scale simulations, down to the level of molecular-level distributions.
Mechanics↗
Constituent Data Replacement Tool
The purpose of this tool is to estimate key parameters that may be missing in public wastewater composition datasets. The tool can be applied to develop complete treatment and critical mineral extraction profiles for leachate, produced water and other aqueous waste streams. The tool applies machine learning algorithms to replace missing data in a user’s water data set that are adjusted based on user preferences for options including algorithm type, number of features, and classification variables. The tool can use the user’s data alone or combine user data with the NEWTS USGS Produced Water Database for more robust training. This research was funded by the U.S. Department of Energy’s Office Fossil Energy and Carbon Management (FECM) through National Energy Technology Laboratory’s ongoing research under the Water Management for Power System Field Work Proposal, DE-FECM 1022428 and Critical Minerals Field Work Proposal, DE-FECM 1022420.
Aqueous Chemistry↗
Optimal Codes for the Burst Erasure Channel
Deep space communications over noisy channels lead to certain packets that are not decodable. These packets leave gaps, or bursts of erasures, in the data stream. Burst erasure correcting codes overcome this problem. These are forward erasure correcting codes that allow one to recover the missing gaps of data. Much of the recent work on this topic concentrated on Low-Density Parity-Check (LDPC) codes. These are more complicated to encode and decode than Single Parity Check (SPC) codes or Reed-Solomon (RS) codes, and so far have not been able to achieve the theoretical limit for burst erasure protection. A block interleaved maximum distance separable (MDS) code (e.g., an SPC or RS code) offers near-optimal burst erasure protection, in the sense that no other scheme of equal total transmission length and code rate could improve the guaranteed correctible burst erasure length by more than one symbol. The optimality does not depend on the length of the code, i.e., a short MDS code block interleaved to a given length would perform as well as a longer MDS code interleaved to the same overall length. As a result, this approach offers lower decoding complexity with better burst erasure protection compared to other recent designs for the burst erasure channel (e.g., LDPC codes). A limitation of the design is its lack of robustness to channels that have impairments other than burst erasures (e.g., additive white Gaussian noise), making its application best suited for correcting data erasures in layers above the physical layer. The efficiency of a burst erasure code is the length of its burst erasure correction capability divided by the theoretical upper limit on this length. The inefficiency is one minus the efficiency. The illustration compares the inefficiency of interleaved RS codes to Quasi-Cyclic (QC) LDPC codes, Euclidean Geometry (EG) LDPC codes, extended Irregular Repeat Accumulate (eIRA) codes, array codes, and random LDPC codes previously proposed for burst erasure protection. As can be seen, the simple interleaved RS codes have substantially lower inefficiency over a wide range of transmission lengths.
Hamkins, Jon↗
Imputation of urban environmental sensor data using gated attention bidirectional long short-term memory (GA-BiLSTM): methods, performance, and implications
Urban environmental monitoring networks frequently encounter significant data gaps due to sensor malfunctions, environmental disturbances, and communication failures. Reliable approaches to address these gaps are essential for ensuring the continuity and quality of environmental data streams. In this study, we developed a gated attention bidirectional long short-term memory (GA-BiLSTM) model to impute missing data in a dense urban monitoring network. Using observations from the CROCUS network in Chicago, we evaluated GA-BiLSTM against widely used approaches (XGBoost and K-nearest neighbors) under scenarios of both short-term intermittent gaps and prolonged outages. GA-BiLSTM consistently outperformed comparative methods, particularly during extended outages of up to ten days, demonstrating its ability to capture spatiotemporal dependencies across sensor nodes. Beyond performance metrics, feature importance and spatial network analyses highlighted the unexpected but critical predictive role of peripheral rural nodes, underlining their strategic value for maintaining robust urban monitoring systems. These results emphasize that advanced imputation methods can substantially improve the reliability of environmental monitoring networks and support more resilient data infrastructures for urban sustainability.
Data imputation↗
Faint and Fading Tails: The Fate of Stripped H i Gas in Virgo Cluster Galaxies
Although many galaxies in the Virgo cluster are known to have lost significant amounts of H i gas, only about a dozen features are known where the H i extends significantly outside its parent galaxy. Previous numerical simulations have predicted that H i removed by ram pressure stripping should have column densities far in excess of the sensitivity limits of observational surveys. We construct a simple model to try and quantify how many streams we might expect to detect. This accounts for the expected random orientation of the streams in position and velocity space as well as the expected stream length and mass of stripped H i. Using archival data from the Arecibo Galaxy Environment Survey, we search for any streams that might previously have been missed in earlier analyses. We report the confident detection of 10 streams as well as 16 other less-certain detections. We show that these well match our analytic predictions for which galaxies should be actively losing gas; however, the mass of the streams is typically far below the amount of missing H i in their parent galaxies, implying that a phase change and/or dispersal renders the gas undetectable. By estimating the orbital timescales, we estimate that dissolution rates of 1–10 M {sub ⊙} yr{sup −1} are able to explain both the presence of a few long, massive streams and the greater number of shorter, less-massive features.
79 ASTRONOMY AND ASTROPHYSICS↗
Flood Analysis of the Thomas Flood in California Using NASA Data in a GIS
In December 2017, the Thomas Fire burned areas in Ventura and Santa Barbara Counties in California, consuming 281,893 acres (440 square miles), and becoming the largest wildfire in California history. On January 8, 2018, a winter storm made landfall in Santa Barbara County, releasing more than 5 inches of rain onto the same scarred area, causing San Ysidro and Montecito Creeks to overflow and leading to devastating flash flooding and debris flows by January 9, 2018. Several areas that were not typically prone to flooding were at risk due to the higher than usual flows in the nearby streams and creeks. The unprecedented flooding killed at least 20 people and left several others missing. NASA's Earth Observing System Data Information System (EOSDIS) manages Earth Observation satellites and the Distributed Active Archive Centers (DAACs), where the data is stored and processed. This poster presents the use of satellite data from NASA's inventory that have the potential for use in identification and analysis of forest fire risk and subsequent after-effects within a geographic information system (GIS).
science↗
Time series methods for the analysis of soundscapes and other cyclical ecological data
Biodiversity monitoring has entered an era of ‘big data’, exemplified by a near-continuous collection of sounds, images, chemical and other signals from organisms in diverse ecosystems. Such data streams have the potential to help identify new threats, assess the effectiveness of conservation interventions, as well as generate new ecological insights. However, appropriate analytical methods are often still missing, particularly with respect to characterizing cyclical temporal patterns. Here, we present a framework for characterizing and analysing ecological responses that represent nonstationary, complex temporal patterns and demonstrate the value of using Fourier transforms to decorrelate continuous data points. In our example, we use a framework based on three approaches (spectral analysis, magnitude squared coherence, and principal component analysis) to characterize differences in tropical forest soundscapes within and across sites and seasons in Gabon. By reconstructing the underlying, cyclic behaviour of the soundscape for each site, we show how one can identify circadian patterns in acoustic activity. Soundscapes in the dry season had a complex diel cycle, requiring multiple harmonics to represent daily variation, while in the wet season there was less variance attributable to the daily cyclic patterns. Our framework can be applied to most continuous, or near-continuous ecological data collected at a fine temporal resolution, allowing ecologists to explore patterns of temporal autocorrelation at multiple levels for biologically meaningful trends. Such methods will become indispensable as biological big data are used to understand the impact of anthropogenic pressures on biodiversity and to inform efforts to mitigate them.
54 ENVIRONMENTAL SCIENCES↗
System monitoring and diagnosis with qualitative models
A substantial foundation of tools for model-based reasoning with incomplete knowledge was developed: QSIM (a qualitative simulation program) and its extensions for qualitative simulation; Q2, Q3 and their successors for quantitative reasoning on a qualitative framework; and the CC (component-connection) and QPC (Qualitative Process Theory) model compilers for building QSIM QDE (qualitative differential equation) models starting from different ontological assumptions. Other model-compilers for QDE's, e.g., using bond graphs or compartmental models, have been developed elsewhere. These model-building tools will support automatic construction of qualitative models from physical specifications, and further research into selection of appropriate modeling viewpoints. For monitoring and diagnosis, plausible hypotheses are unified against observations to strengthen or refute the predicted behaviors. In MIMIC (Model Integration via Mesh Interpolation Coefficients), multiple hypothesized models of the system are tracked in parallel in order to reduce the 'missing model' problem. Each model begins as a qualitative model, and is unified with a priori quantitative knowledge and with the stream of incoming observational data. When the model/data unification yields a contradiction, the model is refuted. When there is no contradiction, the predictions of the model are progressively strengthened, for use in procedure planning and differential diagnosis. Only under a qualitative level of description can a finite set of models guarantee the complete coverage necessary for this performance. The results of this research are presented in several publications. Abstracts of these published papers are presented along with abtracts of papers representing work that was synergistic with the NASA grant but funded otherwise. These 28 papers include but are not limited to: 'Combined qualitative and numerical simulation with Q3'; 'Comparative analysis and qualitative integral representations'; 'Model-based monitoring of dynamic systems'; 'Numerical behavior envelopes for qualitative models'; 'Higher-order derivative constraints in qualitative simulation'; and 'Non-intersection of trajectories in qualitative phase space: a global constraint for qualitative simulation.'
Kuipers, Benjamin↗
CME Evolution in the Structured Heliosphere and Effects at Earth and Mars During Solar Minimum
The activity of the Sun alternates between a solar minimum and a solar maximum, the former corresponding to a period of “quieter” status of the heliosphere. During solar minimum, it is in principle more straightforward to follow eruptive events and solar wind structures from their birth at the Sun throughout their interplanetary journey. In this paper, we report analysis of the origin, evolution, and heliospheric impact of a series of solar transient events that took place during the second half of August 2018, that is, in the midst of the late declining phase of Solar Cycle 24. In particular, we focus on two successive coronal mass ejections (CMEs) and a following high-speed stream (HSS) on their way toward Earth and Mars. We find that the first CME impacted both planets, whilst the second caused a strong magnetic storm at Earth and went on to miss Mars, which nevertheless experienced space weather effects from the stream interacting region preceding the HSS. Analysis of remote-sensing and in-situ data supported by heliospheric modeling suggests that CME–HSS interaction resulted in the second CME rotating and deflecting in interplanetary space, highlighting that accurately reproducing the ambient solar wind is crucial even during “simpler” solar minimum periods. Lastly, we discuss the upstream solar wind conditions and transient structures responsible for driving space weather effects at Earth and Mars.
Space weather↗
Bayesian Analysis of the Cosmic Microwave Background
There is a wealth of cosmological information encoded in the spatial power spectrum of temperature anisotropies of the cosmic microwave background! Experiments designed to map the microwave sky are returning a flood of data (time streams of instrument response as a beam is swept over the sky) at several different frequencies (from 30 to 900 GHz), all with different resolutions and noise properties. The resulting analysis challenge is to estimate, and quantify our uncertainty in, the spatial power spectrum of the cosmic microwave background given the complexities of "missing data", foreground emission, and complicated instrumental noise. Bayesian formulation of this problem allows consistent treatment of many complexities including complicated instrumental noise and foregrounds, and can be numerically implemented with Gibbs sampling. Gibbs sampling has now been validated as an efficient, statistically exact, and practically useful method for low-resolution (as demonstrated on WMAP 1 and 3 year temperature and polarization data). Continuing development for Planck - the goal is to exploit the unique capabilities of Gibbs sampling to directly propagate uncertainties in both foreground and instrument models to total uncertainty in cosmological parameters.
methods - statistical↗
Evaluation of New Additions to OLI Software in Predicting Mercuric and Mercurous Species in Liquid Waste Operations
Speciation of mercury during the pretreatment steps of tank waste processing is critical to successful mercury removal prior to vitrification during Liquid Waste Operations (LWO) at SRS. OLI software has been used to predict mercury speciation and activity throughout LWO. The OLI software operates based on a thermodynamic framework called the Mixed Solvent Electrolyte (MSE) framework. The MSE framework allows prediction in theoretically infinitely dilute to concentrated mixtures (e.g., purely solute solutions). Before modification to the MSE framework databanks, certain critical mercury species were missing in the MSE databank, and some thermodynamic data needed to be updated for the OLI software to accurately predict mercury chemical species in SRS waste tanks. To better reflect streams across LWO, new mercury species were integrated into the MSE database. To evaluate the changes to the OLI MSE framework per the Technical Task Request (TTR) and the Task Technical and Quality Assurance Plan (TTQAP), waste stream compositions from Tanks 38, 43, and Tank 50 decontaminated salt solution (DSS) were used as model inputs. Models were developed and executed using both the old and new databases. Compositional analyses from caustic Tank 50 DSS and caustic Tanks 38 and 43 were used as the input streams. These streams represent the most comprehensive chemical data sets where both mercury and tank constituents were measured together. Results for Tank 50 DSS predict HgO as the predominant species in both databases. Both methyl and dimethyl Hg species are present when the new database is ‘on’ and are not predicted with the new database turned ‘off’. The new database predicts a greater amount of HgO and a greater fraction of it in the solid phase. Pourbaix diagrams (potential vs. pH) generated for each Tank 50 DSS were identical regardless of which database was used. Elemental Hg and HgO were predicted in the water stable region under basic conditions. Tanks 38 and 43 follow similar trends as the Tank 50 DSS models. Unlike Tanks 38 and 50 DSS, the Tank 43 Pourbaix plot shows a region of stability for an aqueous HgOHCO3 - species between approximately pH 7-11. In all streams, when MeHg+ is included in the inputs, the new database predicts aqueous MeHgOH as the dominant species. If elemental or dimethyl mercury is in the waste stream, the new database model predicts they are unchanged and remain in those states and quantities. Additionally, the total mercury values are reported for both the measured input data and the OLI output data for all considered tanks. The summary indicates that the percentage error between the measured and calculated values is less than 1% in all cases The reconciliations and generation of the Pourbaix diagrams for Tank 50 DSS took approximately ten times longer with the new database ‘on’. In addition, over the course of that time, models with the new database ‘on’ were more likely to crash or display an error. Some modest performance improvements were noted when modeling with an i7 processor versus an i5. An example error is found in Appendix A. Furthermore, Appendix B provides V&V for two chemical systems analyzed with the OLI software, results were satisfactory. It is recommended to utilize the new databases (i.e., HCO.ddb and SR-Hg.ddb) in future Savannah River Mission Completion applications of OLI to represent pseudo steady-state. Furthermore, the integration and utilization of the new databases (i.e., HCO.ddb and SR-Hg.ddb) in modeling applications (e.g., Aspen) is also recommended.
12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
NASA Tech Briefs, March 2007
Topics include: Advanced Systems for Monitoring Underwater Sounds; Wireless Data-Acquisition System for Testing Rocket Engines; Processing Raw HST Data With Up-to-Date Calibration Data; Mobile Collection and Automated Interpretation of EEG Data; System for Secure Integration of Aviation Data; Servomotor and Controller Having Large Dynamic Range; Digital Multicasting of Multiple Audio Streams; Translator for Optimizing Fluid-Handling Components; AIRSAR Web-Based Data Processing; Pattern Matcher for Trees Constructed From Lists; Reducing a Knowledge-Base Search Space When Data Are Missing; Ground-Based Correction of Remote-Sensing Spectral Imagery; State-Chart Autocoder; Pointing History Engine for the Spitzer Space Telescope; Low-Friction, High-Stiffness Joint for Uniaxial Load Cell; Magnet-Based System for Docking of Miniature Spacecraft; Electromechanically Actuated Valve for Controlling Flow Rate; Plumbing Fixture for a Microfluidic Cartridge; Camera Mount for a Head-Up Display; Core-Cutoff Tool; Recirculation of Laser Power in an Atomic Fountain; Simplified Generation of High-Angular-Momentum Light Beams; Imaging Spectrometer on a Chip; Interferometric Quantum-Nondemolition Single-Photon Detectors; Ring-Down Spectroscopy for Characterizing a CW Raman Laser; Complex Type-II Interband Cascade MQW Photodetectors; Single-Point Access to Data Distributed on Many Processors; Estimating Dust and Water Ice Content of the Martian Atmosphere From THEMIS Data; Computing a Stability Spectrum by Use of the HHT; Theoretical Studies of Routes to Synthesis of Tetrahedral N4; Estimation Filter for Alignment of the Spitzer Space Telescope; Antenna for Measuring Electric Fields Within the Inner Heliosphere; Improved High-Voltage Gas Isolator for Ion Thruster; and Hybrid Mobile Communication Networks for Planetary Exploration.
Source record↗
Examples of mesoscale structures and short-term wind variations detected by VHF Doppler radar
The first of three wind profilers planned for operation in central and western Pennsylvania began full-time, high-quality operation during July 1985. It is located about 20 km south-southeast of University Park and operates at 50 MHz. Another 50-MHz radar and a 400-MHz radar are to be installed over the next few months, to complete a mesoscale triangle with sides of 120 to 160 km. During the period since early July, a number of weather systems have passed over the wind profiler. Those accompanied by thunderstorms caused data losses either because the Department computer system lost power or because power went out at the profiler site. A backup power supply and an automatic re-start program will be added to the profiler system to minimize such future losses. Data have normally been averaged over a one-hour period, although there have been some investigations of shorter-period averaging. In each case, preliminary examinations reveal that the profiler winds are indicative of meteorological phenomena. The only occasions of bad or missing data are obtained when airplane noise is occasionally experienced and when the returned power is nearly at the noise level, at the upper few gates, where a consensus wind cannot be determined. Jets streams, clouds, and diurnal variations of winds are discussed.
Forbes, G. S.↗