Search NASA⌕ Search

SEARCH · Search NASA

Results for “DISTRIBUTED DATA PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 703 records · Page 39

Acquisition, Sharing and Processing of Large Datasets for Strain Imaging: an Example of an Indented Ni3Al/Mo Composite

The local effects of stress from a mechanical indentation have been studied on a Ni3 Al single crystal containing submicron inclusions of molybdenum fibers. Polychromatic X ray Microscopy (PXM) was used to measure elastic and plastic deformations near the indents. Analysis of freshly acquired massive sets of PXM data has been carried out over the Science Studio network using parallel processing software FOXMAS. This network and the FOXMAS software have greatly improved the efficiency of the data processing task.. The analysis was successfully applied to study lattice orientation distribution and strain tensor components for both the Ni3 Al and the Mo phases, particularly around 8 indents patterned at the longitudinal section of the alloy.

data processing↗

Multi-Sensor Distributive On-Line Processing, Visualization, and Analysis Infrastructure for an Agricultural Information System at the NASA Goddard Earth Sciences DAAC

The Goddard Space Flight Center Earth Sciences Data and Information Services Center (GES DISC) Distributed Active Center (DAAC) is developing an Agricultural Information System (AIS), evolved from an existing TRMM On-line Visualization and Analysis System precipitation and other satellite data products and services. AIS outputs will be ,integrated into existing operational decision support system for global crop monitoring, such as that of the U.N. World Food Program. The ability to use the raw data stored in the GES DAAC archives is highly dependent on having a detailed understanding of the data's internal structure and physical implementation. To gain this understanding is a time-consuming process and not a productive investment of the user's time. This is an especially difficult challenge when users need to deal with multi-sensor data that usually are of different structures and resolutions. The AIS has taken a major step towards meeting this challenge by incorporating an underlying infrastructure, called the GES-DISC Interactive Online Visualization and Analysis Infrastructure or "Giovanni," that integrates various components to support web interfaces that ,allow users to perform interactive analysis on-line without downloading any data. Several instances of the Giovanni-based interface have been or are being created to serve users of TRMM precipitation, MODIS aerosol, and SeaWiFS ocean color data, as well as agricultural applications users. Giovanni-based interfaces are simple to use but powerful. The user selects geophysical ,parameters, area of interest, and time period; and the system generates an output ,on screen in a matter of seconds.

Teng, William↗

Foundations of Feature Selection and Classification for Non-Gaussian Distributed Targets

This report describes the author’s research toward creating algorithms for selecting useful statistical features in pattern/target classification problems in which the features are non-Gaussian distributed. In engineering practice, it is common to either (1) Not perform any feature selection procedure, or (2) Use a feature selection algorithm that assumes the features are Gaussian distributed. This process can be far from optimal if the data are actually non-Gaussian, as they frequently are in practice. This research has the goal of mitigating that problem by creating a useful algorithm that can be used in practice. The approach is to focus on the performance indices used in common feature selection algorithms, such as Branch and Bound algorithms and the Sequential Forward Selection Algorithm. Ordinarily, the performance index used to measure the class separation in feature space involves assuming the data are Gaussian and deriving tractable performance indices that use closed form equations that represent the probability density functions of the class data. The advantage of this approach is that it produces feature selection algorithms that have low computational complexity and do not require knowledge of the actual data densities. The disadvantage is that these algorithms may not perform reasonably when the data are non-Gaussian, as they commonly are in practice. This research examines the use of information-theoretic class separability measures that can deal with the non-Gaussian case. In particular, this work shows that the Hellinger Distance (a divergence measure), has very desirable mathematical properties and can be useful for feature selection. The work in this paper employs a multivariate kernel density estimator along with the Hellinger Distance and an optimal subset selection algorithm to do feature selection. This report (1) Defines the classification and feature selection problems, (2) Describes the continuous variable Hellinger distance in one dimension and discusses its mathematical properties in comparison with other distance measures, (3) Develops the discrete variable Hellinger distance expression for one and two dimensions, and shows that the discrete Hellinger distance for multiple dimensions is a straightforward extension, and (4) Presents experimental density estimation results using a new MATLAB implementation of a multivariate kernel density estimator. The draft of this report served as a starting point for Ensign Matthew Wilder’s MSEE thesis work at the Naval Postgraduate School (NPS) in 2010 and 2011. Ensign Wilder completed his thesis and graduated from NPS in June 2011 (M. J. Wilder, “Automatic Target Recognition: Statistical Feature Selection of Non-Gaussian Distributed Target Classes,” MSEE Thesis, Naval Postgraduate School, Thesis Advisor: Grace A. Clark, Second Reader: Monique P. Fargues, June 2011). In the thesis, Ensign Wilder created implementations of the Branch and Bound and Sequential Feature Selection algorithms, extended the multivariate kernel density estimator code, and evaluated the performance of Hellinger Distance-based feature selection algorithms using both simulated data and data from a real-world benchmark data set. The new algorithm was shown to be very effective for both Gaussian and non-Gaussian data sets. The advantages gained for non-Gaussian data come at the cost of increased computational complexity. Using a laptop computer for practical problems, we generally use initial feature vectors containing no more than about ten or twelve features. Proposed future research lies in creating faster kernel density estimator algorithms that would allow for the use of higher dimensional data.

42 ENGINEERING↗

The SSABLE system - Automated archive, catalog, browse and distribution of satellite data in near-real time

Historically, locating and browsing satellite data has been a cumbersome and expensive process. This has impeded the efficient and effective use of satellite data in the geosciences. SSABLE is a new interactive tool for the archive, browse, order, and distribution of satellite date based upon X Window, high bandwidth networks, and digital image rendering techniques. SSABLE provides for automatically constructing relational database queries to archived image datasets based on time, data, geographical location, and other selection criteria. SSABLE also provides a visual representation of the selected archived data for viewing on the user's X terminal. SSABLE is a near real-time system; for example, data are added to SSABLE's database within 10 min after capture. SSABLE is network and machine independent; it will run identically on any machine which satisfies the following three requirements: 1) has a bitmapped display (monochrome or greater); 2) is running the X Window system; and 3) is on a network directly reachable by the SSABLE system. SSABLE has been evaluated at over 100 international sites. Network response time in the United States and Canada varies between 4 and 7 s for browse image updates; reported transmission times to Europe and Australia typically are 20-25 s.

Simpson, James J.↗

Dynamic Data Repartitioning for Load-Balanced Parallel Particle Tracing

We present a novel dynamic load-balancing algorithm based on data repartitioning for parallel particle tracing in flow visualization. Instead of static data assignment, we dynamically repartition the data into blocks and reassign the blocks to processes to balance the workload distribution among the processes. Block repartitioning is performed based on a dynamic workload estimation method that predicts the workload in the flow field on the fly as the input. In our approach, we allow data duplication in the repartitioning, enabling the same data blocks to be assigned to multiple processes. Load balance is achieved by regularly exchanging the blocks (together with the particles in the blocks) among processes according to the output of the data repartitioning. Compared with other load-balancing algorithms, our approach does not need any preprocessing on the raw data and does not require any dedicated process for work scheduling, while it has the capability to balance uneven workload efficiently. Results show improved load balance and high efficiency of our method on tracing particles in both steady and unsteady flow.

data repartitioning↗

ThunderSecure: deploying real-time intrusion detection for 100G research networks by leveraging stream-based features and one-class classification network

Nowadays, data generated by large-scale scientific experiments are on the scale of petabytes per month. These data are transferred through dedicated high-bandwidth networks (40/100G) across distributed sites for processing, storage, and analysis. Like general purpose networks, research networks experience intrusions. However, monitoring anomalies in such high-speed network traffics is challenging given current cyber-infrastructure. Moreover, traditional network intrusion detection systems (NIDS) are signature based. However, anomaly patterns are difficult to define and that rulesets are often not updated frequently enough to reflect the changes of attack behaviors. We present ThunderSecure, a high-throughput, unsupervised learning-based intrusions detection system for 100G research networks. ThunderSecure implements an efficient packet processing and detection pipeline using multi-cores and GPUs. It extracts statistical and temporal features from real-time network data streams and feeds them to a one-class anomaly detection network. A baseline of normal distribution will be created based on the training observation. Testing traffic deviated from the learned profile will be marked as anomalies. We trained ThunderSecure on hundreds of billions of science data packets mirrored from two 100G network connections at Fermi National Accelerator Laboratory. The detection performance was evaluated on traffic captured from the same research network days and weeks after the training with different types of attack flows injected. Results show that ThunderSecure can recognize science data traffic captured long after the training and made nearly certain detection on the segment of the streams where anomalous flows were injected.

100G research network↗

A Scientific Data Provenance Harvester for Distributed Applications

Data provenance provides a way for scientists to observe how experimental data originates, conveys process history, and explains influential factors such as experimental rationale and associated environmental factors from system metrics measured at runtime. The US Department of Energy Office of Science Integrated end-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows (IPPD) project has developed a provenance harvester that is capable of collecting observations from file based evidence typically produced by distributed applications. To achieve this, file based evidence is extracted and transformed into an intermediate data format inspired in part by W3C CSV on the Web recommendations, called the Harvester Provenance Application Interface (HAPI) syntax. This syntax provides a general means to pre-stage provenance into messages that are both human readable and capable of being written to a provenance store, Provenance Environment (ProvEn). HAPI is being applied to harvest provenance from climate ensemble runs for Accelerated Climate Modeling for Energy (ACME) project funded under the U.S. Department of Energy’s Office of Biological and Environmental Research (BER) Earth System Modeling (ESM) program. ACME informally provides provenance in a native form through configuration files, directory structures, and log files that contain success/failure indicators, code traces, and performance measurements. Because of its generic format, HAPI is also being applied to harvest tabular job management provenance from Belle II DIRAC scheduler relational database tables as well as other scientific applications that log provenance related information.

Stephan, Eric G.↗

Building near-real-time processing pipelines with the spark-MPI platform

Advances in detectors and computational technologies provide new opportunities for applied research and the fundamental sciences. Concurrently, dramatic increases in the three V's (Volume, Velocity, and Variety) of experimental data and the scale of computational tasks produced the demand for new real-time processing systems at experimental facilities. Recently, this demand was addressed by the Spark-MPI approach connecting the Spark data-intensive platform with the MPI high-performance framework. In contrast with existing data management and analytics systems, Spark introduced a new middleware based on resilient distributed datasets (RDDs), which decoupled various data sources from high-level processing algorithms. The RDD middleware significantly advanced the scope of data-intensive applications, spreading from SQL queries to machine learning to graph processing. Spark-MPI further extended the Spark ecosystem with the MPI applications using the Process Management Interface. The paper explores this integrated platform within the context of online ptychographic and tomographic reconstruction pipelines.

97 MATHEMATICS AND COMPUTING↗

Planning of distributed data production for High Energy and Nuclear Physics

Modern experiments in High Energy and Nuclear Physics heavily rely on distributed computations using multiple computational facilities across the world. One of the essential types of the computations is a distributed data production where petabytes of raw files from a single source has to be processed once (per production campaign) using thousands of CPUs at distant locations and the output has to be transferred back to that source. The data distribution over a large system does not necessary match the distribution of storage, network and CPU capacity. Therefore, bottlenecks may appear and lead to increased latency and degraded performance. In this paper we propose a new scheduling approach for distributed data production which is based on the network flow maximization model. In our approach a central planner defines how much input and output data should be transferred over each network link in order to maximize the computational throughput. Such plans are created periodically for a fixed planning time interval using up-to-date information on network, storage and CPU resources. The centrally created plans are executed in a distributed manner by dedicated services running at participating sites. In conclusion, our simulations based on the log records from the data production framework of the experiment STAR (Solenoid Tracker at RHIC) have shown that the proposed model systematically provides a better performance compared to the simulated traditional techniques.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Analysis of cache for streaming tape drive

A tape subsystem consists of a controller and a tape drive. Tapes are used for backup, data interchange, and software distribution. The backup operation is addressed. During a backup operation, data is read from disk, processed in CPU, and then sent to tape. The processing speeds of a disk subsystem, CPU, and a tape subsystem are likely to be different. A powerful CPU can read data from a fast disk, process it, and supply the data to the tape subsystem at a faster rate than the tape subsystem can handle. On the other hand, a slow disk drive and a slow CPU may not be able to supply data fast enough to keep a tape drive busy all the time. The backup process may supply data to tape drive in bursts. Each burst may be followed by an idle period. Depending on the nature of the file distribution in the disk, the input stream to the tape subsystem may vary significantly during backup. To compensate for these differences and optimize the utilization of a tape subsystem, a cache or buffer is introduced in the tape controller. Most of the tape drives today are streaming tape drives. A streaming tape drive goes into reposition when there is no data from the controller. Once the drive goes into reposition, the controller can receive data, but it cannot supply data to the tape drive until the drive completes its reposition. A controller can also receive data from the host and send data to the tape drive at the same time. The relationship of cache size, host transfer rate, drive transfer rate, reposition, and ramp up times for optimal performance of the tape subsystem are investigated. Formulas developed will also show the advantages of cache watermarks to increase the streaming time of the tape drive, maximum loss due to insufficient cache, tradeoffs between cache and reposition times and the effectiveness of cache on a streaming tape drive due to idle times or interruptions due in host transfers. Several mathematical formulas are developed to predict the performance of the tape drive. Some examples are given illustrating the usefulness of these formulas. Finally, a summary and some conclusions are provided.

Chinnaswamy, V.↗

Automation of environmental compliance data acquisition and management - 15267

Regulatory sampling for environmental compliance at the Savannah River Site (SRS), Aiken, South Carolina employs several different types of sampling techniques at various sampling frequencies. These sampling events result in the collection of large quantities of data and generate substantial data-related documents. The use of paper documents resulted in the need for manual input of data collected in the field and distribution of documents from one group to the next. To efficiently and effectively manage the extensive amounts of data and documents, elements of the sampling process were automated using hand-held field tablets and an electronic application. The tablets allow for easier distribution of sampling documents, and the application allows for digital transfer of sampling information and collected field data. The transition to automated data acquisition and management has resulted in a more efficient and reliable manner for conducting business at SRS. This system benefits numerous organizations at SRS by utilizing a centralized database for efficient management of all environmental data collected. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

A Metric to Quantify Shared Visual Attention in Two-Person Teams

Introduction: Critical tasks in high-risk environments are often performed by teams, the members of which must work together efficiently. In some situations, the team members may have to work together to solve a particular problem, while in others it may be better for them to divide the work into separate tasks that can be completed in parallel. We hypothesize that these two team strategies can be differentiated on the basis of shared visual attention, measured by gaze tracking. 2) Methods: Gaze recordings were obtained for two-person flight crews flying a high-fidelity simulator (Gontar, Hoermann, 2014). Gaze was categorized with respect to 12 areas of interest (AOIs). We used these data to construct time series of 12 dimensional vectors, with each vector component representing one of the AOIs. At each time step, each vector component was set to 0, except for the one corresponding to the currently fixated AOI, which was set to 1. This time series could then be averaged in time, with the averaging window time (t) as a variable parameter. For example, when we average with a t of one minute, each vector component represents the proportion of time that the corresponding AOI was fixated within the corresponding one minute interval. We then computed the Pearson product-moment correlation coefficient between the gaze proportion vectors for each of the two crew members, at each point in time, resulting in a signal representing the time-varying correlation between gaze behaviors. We determined criteria for concluding correlated gaze behavior using two methods: first, a permutation test was applied to the subjects' data. When one crew member's gaze proportion vector is correlated with a random time sample from the other crewmember's data, a distribution of correlation values is obtained that differs markedly from the distribution obtained from temporally aligned samples. In addition to validating that the gaze tracker was functioning reasonably well, this also allows us to compute probabilities of coordinated behavior for each value of the correlation. As an alternative, we also tabulated distributions of correlation coefficients for synthetic data sets, in which the behavior was modeled as a first-order Markov process, and compared correlation distributions for identical processes with those for disparate processes, allowing us to choose criteria and estimate error rates. 3) Discussion: Our method of gaze correlation is able to measure shared visual attention, and can distinguish between activities involving different instruments. We plan to analyze whether pilots strategies of sharing visual attention can predict performance. Possible measurements of performance include expert ratings from instructors, fuel consumption, total task time, and failure rate. While developed for two-person crews, our approach can be applied to larger groups, using intra-class correlation coefficients instead of the Pearson product-moment correlation.

Gontar, Patrick↗

NSA Site Science: Use of ARM Observations from Northern Alaska to Evaluate and Improve Prediction Capabilities

The Arctic is warming at a rate nearly double that of the rest of the planet, leading to profound changes in atmospheric, oceanic, and ice processes. The U.S. Department of Energy's (DOE) Atmospheric Radiation Measurement (ARM) user facility has played a significant role in Arctic research, operating observatories in Alaska's North Slope for over 25 years. These observatories provide a rich, and wide-reaching dataset that offers insight into atmospheric processes in northern Alaska. This report details the results of a nine-year research project (2015–2024) supported by the DOE Atmospheric Systems Research (ASR) program, that leverages data from ARM’s deployment of observing facilities at Utqiaġvik (known as the North Slope of Alaska, or NSA, site) and Oliktok Point, Alaska. The project was conducted in two phases: - Phase 1 (2015–2019): Focused on understanding key atmospheric processes at Oliktok Point, including cloud formation, high-latitude precipitation, aerosol-cloud interactions, and cloud properties. - Phase 2 (2019–2024): Extended the research to the broader North Slope region, using data from both Oliktok Point and Utqiaġvik. Topics explored included surface energy budgets, atmospheric stability, ice nucleation processes, and microphysics in Arctic clouds. The project resulted in numerous research products, including 50 peer-reviewed publications and dissertations, 169 presentations, and 10 data products. These products cover a variety of topics, including: - Cloud Macro- and Microphysical Properties: Arctic clouds play a crucial role in energy transfer, and accurate representation in models is critical. The study explored cloud transitions, ice crystal shapes, and dual-wavelength radar data to understand ice crystal habits and size distributions. - Aerosol Properties and Processes: The team examined aerosol sources in the Arctic, including industrial emissions and natural sources. Observations showed significant spatial gradients in aerosol concentrations due to human activities and wildfire smoke. The influence of aerosols on cloud formation and the surface energy budget was also assessed. - Aerosol-Cloud Interactions: Research revealed that aerosols might suppress cloud ice production, affecting cloud radiative forcing and precipitation. The impact of local industrial emissions on cloud properties was also investigated. - Contextualizing the North Slope of Alaska in the context of the broader Arctic: To understand broader trends, the project evaluated large-scale circulation patterns and the influence of weather systems on the Arctic. Studies indicated that large-scale processes play a significant role in temperature patterns and the timing of snowmelt. - Advancing ARM Observational and Modeling Capabilities: The project developed new radar data products and advanced measurement techniques, including clutter mitigation and drizzle detection. Uncrewed aerial systems (UAS) and tethered balloon systems (TBS) were deployed to gather detailed atmospheric data. Additionally, the project supported 10 early career scientists, providing training and mentorship to undergraduate interns, graduate students, postdoctoral researchers, and early career researchers. These efforts contributed to the advancement of ARM research capabilities and fostered a new generation of scientists skilled in Arctic atmospheric research. Ultimately, this ASR-supported project has provided valuable insights into Arctic atmospheric processes and their broader climate implications. Recommendations for future work include continuing support for long-term observing at Arctic locations to foster additional research, further exploration of aerosol-cloud interactions and the potential impacts of enhanced industrialization of the Arctic, and expanded use of uncrewed systems to gather data in this remote and harsh environment. Additionally, the data products developed by this work, and the data products developed through the ARM infrastructure, leave a treasure-trove of additional information that should be explored for many years to come to gain additional insight into physical processes in the Arctic atmosphere that drive the rapid changes occurring in at high latitudes and their global impact.

58 GEOSCIENCES↗

Characterization of a dimer preparation method for nanoscale organic aerosol

Nanoscale dimers have application in studies of aerosol physicochemical properties such as aerosol viscosity. These particle dimers can be synthesized using the dual tandem differential mobility analyzer (DTDMA) technique, wherein oppositely charged particle streams coagulate to form dimers that can be isolated using electrostatic filtration. Although some characterization of the technique has been published, a detailed thesis on the modes and theory of operation has remained outside the scope of prior work. In this paper we present new experimental data characterizing the output DTDMA size distribution and the physical processes underlying its apparent modes. Key experimental limitations for both general applications and for viscosity measurements are identified and quantified in six distinct types of DTDMA experiments. The primary consideration is the production of an adequate number of dimers, which typically requires high mobility-selected number concentration in the range 25,000-100,000 cm -3 . The requisite concentration threshold depends upon the rate of spontaneous monomer decharging, which arises predominately from interactions of the aerosol with ionizing radiation within the coagulation chamber and is instrument location dependent. Lead shielding of the coagulation chamber reduced the first-order decharging constant from ~2.0 x 10 -5 s -1 to ~0.8 x 10 -5 s -1 in our laboratory. Dimer production at monomer diameters less than 40 nm is hindered by low bipolar charging efficiency. Results from the characterization experiments shed light on design considerations for general applications and for characterization of viscous aerosol phase transitions.

54 ENVIRONMENTAL SCIENCES↗

Evaluating the Impact of Solar Generation on Balancing Requirements in Southern Nevada System

Abstract—In this paper, the impacts of solar photovoltaic (PV) generation on balancing requirements including regulation and load following in the Southern Nevada balancing area are analyzed. The methodology is based on the “swinging door” algorithm and a probability box method developed by PNNL. The regulation and load following signals are mimicking the system’s scheduling and real-time dispatch processes. Load, solar PV generation and distributed PV generation (DG) data are used in the simulation. Different levels of solar PV generation and DG penetration profiles are used in the study. Sensitivity of the regulation requirements with respect to real-time solar PV generation forecast errors is analyzed.

Ancillary services, balancing requirements, load f↗

1988/1989 Maricopa Household Travel Study

This study was commissioned by the Maricopa Association of Governments (MAG) Transportation and Planning Office. The primary objectives of this study were to update the trip generation rates used in the MAG travel demand forecasting process and to provide data to validate the MAG trip distribution model. Demographic, socioeconomic, and travel data was collected for 2,992 households residing within the MAG Urban Planning Area. Respondents were asked to record their travel and activities for a 24-hour period. A total of 26,733 trips were recorded by 6,463 people.

1Hz data↗

Documentation for the machine-readable character coded version of the SKYMAP catalogue

The SKYMAP catalogue is a compilation of astronomical data prepared primarily for purposes of attitude guidance for satellites. In addition to the SKYMAP Master Catalogue data base, a software package of data base management and utility programs is available. The tape version of the SKYMAP Catalogue, as received by the Astronomical Data Center (ADC), contains logical records consisting of a combination of binary and EBCDIC data. Certain character coded data in each record are redundant in that the same data are present in binary form. In order to facilitate wider use of all SKYMAP data by the astronomical community, a formatted (character) version was prepared by eliminating all redundant character data and converting all binary data to character form. The character version of the catalogue is described. The document is intended to fully describe the formatted tape so that users can process the data problems and guess work; it should be distributed with any character version of the catalogue.

Warren, W. H., Jr.↗