Search NASA⌕ Search

SEARCH · Search NASA

Results for “data reduction pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Validation and Calibration of Energy Models with Real Vehicle Data from Chassis Dynamometer Experiments

Accurate estimation of vehicle fuel consumption typically requires detailed modeling of complex internal powertrain dynamics, often resulting in computationally intensive simulations. However, many transportation applications-such as traffic flow modeling, optimization, and control-require simplified models that are fast, interpretable, and easy to implement, while still maintaining fidelity to physical energy behavior. This work builds upon a recently developed model reduction pipeline that derives physics-like energy models from high-fidelity Autonomie vehicle simulations. These reduced models preserve essential vehicle dynamics, enabling realistic fuel consumption estimation with minimal computational overhead. While the reduced models have demonstrated strong agreement with their Autonomie counterparts, previous validation efforts have been confined to simulation environments. This study extends the validation by comparing the reduced energy model's outputs against real-world vehicle data. Focusing on the MidSUV category, we tune the baseline Autonomie model to closely replicate the characteristics of a Toyota RAV4. We then assess the accuracy of the resulting reduced model in estimating fuel consumption under actual drive conditions. Our findings suggest that, when the reference Autonomie model is properly calibrated, the simplified model produced by the reduction pipeline can provide reliable, semi-principled fuel rate estimates suitable for large-scale transportation applications.

42 ENGINEERING↗

Full-Depth Coadds of the WISE and First-Year NEOWISE-Reactivation Images

The Near Earth Object Wide-field Infrared Survey Explorer (NEOWISE) Reactivation mission released data from its first full year of observations in 2015. This data set includes ~2.5 million exposures in each of W1 and W2, effectively doubling the amount of WISE imaging available at 3.4 μm and 4.6 μm relative to the AllWISE release. In this paper, we have created the first ever full-sky set of coadds combining all publicly available W1 and W2 exposures from both the AllWISE and NEOWISE-Reactivation (NEOWISER) mission phases. We employ an adaptation of the unWISE image coaddition framework, which preserves the native WISE angular resolution and is optimized for forced photometry. By incorporating two additional scans of the entire sky, we not only improve the W1/W2 depths, but also largely eliminate time-dependent artifacts such as off-axis scattered moonlight. We anticipate that our new coadds will have a broad range of applications, including target selection for upcoming spectroscopic cosmology surveys, identification of distant/massive galaxy clusters, and discovery of high-redshift quasars. In particular, our full-depth AllWISE+NEOWISER coadds will be an important input for the Dark Energy Spectroscopic Instrument selection of luminous red galaxy and quasar targets. Our full-depth W1/W2 coadds are already in use within the DECam Legacy Survey (DECaLS) and Mayall z-band Legacy Survey (MzLS) reduction pipelines. Finally, much more work still remains in order to fully leverage NEOWISER imaging for astrophysical applications beyond the solar system.

79 ASTRONOMY AND ASTROPHYSICS↗

Full-sky beam convolution for cosmic microwave background applications

We introduce a publicly available full-sky beam convolution code library intended to inform the design of future cosmic microwave background (CMB) instruments and help current experiments probe potential systematic effects. The code can be used to assess the impact of optical systematics on all stages of data reduction for a realistic experiment, including analyses beyond power spectrum estimation, by generating signal timelines that may serve as input to full analysis pipelines. The design and mathematical framework of the Python code is discussed along with a few simple benchmarking results. We present a simple two-lens refracting telescope design and use it together with the code to simulate a year-long dataset for 400 detectors scanning the sky on a satellite instrument. The simulation results identify a number of sub-leading optical non-idealities and demonstrate significant B-mode residuals caused by extended sidelobes that are sensitive to polarized radiation from the Galaxy. For the proposed design and satellite scanning strategy, we show that a full physical optics beam model generates B-mode systematics that differ significantly from the simpler elliptical Gaussian model. The code is available at https://github.com/adrijd/beamconv

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

SMASH: Survey of the MAgellanic Stellar History

The Large and Small Magellanic Clouds are unique local laboratories for studying the formation and evolution of small galaxies in exquisite detail. The Survey of the MAgellanic Stellar History (SMASH) is an NOAO community Dark Energy Camera (DECam) survey of the Clouds mapping 480 deg{sup 2} (distributed over ∼2400 square degrees at ∼20% filling factor) to ∼24th mag in ugriz . The primary goals of SMASH are to identify low surface brightness stellar populations associated with the stellar halos and tidal debris of the Clouds, and to derive spatially resolved star formation histories. Here, we present a summary of the survey, its data reduction, and a description of the first public Data Release (DR1). The SMASH DECam data have been reduced with a combination of the NOAO Community Pipeline, the PHOTRED automated point-spread-function photometry pipeline, and custom calibration software. The astrometric precision is ∼15 mas and the accuracy is ∼2 mas with respect to the Gaia reference frame. The photometric precision is ∼0.5%–0.7% in griz and ∼1% in u with a calibration accuracy of ∼1.3% in all bands. The median 5 σ point source depths in ugriz are 23.9, 24.8, 24.5, 24.2, and 23.5 mag. The SMASH data have already been used to discover the Hydra II Milky Way satellite, the SMASH 1 old globular cluster likely associated with the LMC, and extended stellar populations around the LMC out to R ∼ 18.4 kpc. SMASH DR1 contains measurements of ∼100 million objects distributed in 61 fields. A prototype version of the NOAO Data Lab provides data access and exploration tools.

79 ASTRONOMY AND ASTROPHYSICS↗

THE HERSCHEL AND JCMT GOULD BELT SURVEYS: CONSTRAINING DUST PROPERTIES IN THE PERSEUS B1 CLUMP WITH PACS, SPIRE, AND SCUBA-2

We present Herschel observations from the Herschel Gould Belt Survey and SCUBA-2 science verification observations from the JCMT Gould Belt Survey of the B1 clump in the Perseus molecular cloud. We determined the dust emissivity index using four different techniques to combine the Herschel PACS+SPIRE data at 160-500 {mu}m with the SCUBA-2 data at 450 {mu}m and 850 {mu}m. Of our four techniques, we found that the most robust method was filtering out the large-scale emission in the Herschel bands to match the spatial scales recovered by the SCUBA-2 reduction pipeline. Using this method, we find {beta} Almost-Equal-To 2 toward the filament region and moderately dense material and lower {beta} values ({beta} {approx}> 1.6) toward the dense protostellar cores, possibly due to dust grain growth. We find that {beta} and temperature are more robust with the inclusion of the SCUBA-2 data, improving estimates from Herschel data alone by factors of {approx}2 for {beta} and by {approx}40% for temperature. Furthermore, we find core mass differences of {approx}< 30% compared to Herschel-only estimates with an adopted {beta} = 2, highlighting the necessity of long-wavelength submillimeter data for deriving accurate masses of prestellar and protostellar cores.

79 ASTRONOMY AND ASTROPHYSICS↗

Data handling for the geometric correction of large images

Several geometric distortions are present in remotely sensed images depending on the type of sensors and the object being observed. It is often desirable to compensate for these distortions and store the images in reference to a standard coordinate system. Digital techniques for correction are versatile and introduce a minimum of radiometric errors. The main problems to be considered in this area are the determination of the corrective transformation, resampling, and the management of the large quantities of data. It is shown that, by a judicious rearrangement of the input data, considerable reductions in the required memory capacity can be achieved. The rearrangement can be accomplished in several stages. The method presented here is amenable to pipeline implementation for processing a continuous stream of images.

Ramapriyan, H. K.↗

A 32-bit Ultrafast Parallel Correlator using Resonant Tunneling Devices

An ultrafast 32-bit pipeline correlator has been implemented using resonant tunneling diodes (RTD) and hetero-junction bipolar transistors (HBT). The negative differential resistance (NDR) characteristics of RTD's is the basis of logic gates with the self-latching property that eliminates pipeline area and delay overheads which limit throughput in conventional technologies. The circuit topology also allows threshold logic functions such as minority/majority to be implemented in a compact manner resulting in reduction of the overall complexity and delay of arbitrary logic circuits. The parallel correlator is an essential component in code division multi-access (CDMA) transceivers used for the continuous calculation of correlation between an incoming data stream and a PN sequence. Simulation results show that a nano-pipelined correlator can provide and effective throughput of one 32-bit correlation every 100 picoseconds, using minimal hardware, with a power dissipation of 1.5 watts. RTD plus HBT based logic gates have been fabricated and the RTD plus HBT based correlator is compared with state of the art complementary metal oxide semiconductor (CMOS) implementations.

Kulkarni, Shriram↗

Disturbed, diffuse, or just missing? A global study of the H I content of Hickson compact groups

Hickson compact groups (HCGs) are dense configurations of four to ten galaxies, whose H I morphology appears to follow an evolutionary sequence of three phases, with gas initially confined to galaxies, then significant amounts spread throughout the intra-group medium, and finally with almost no gas remaining in the galaxies themselves. It has also been suggested that several groups may harbour a diffuse H I component that is resolved out by interferometric observations. The H I deficiency of HCGs is expected to increase as the H I morphological phase progresses along the evolutionary sequence. If this is the case, H I deficiency would be a rough proxy for the age and evolutionary state of a HCG. We aim to test this hypothesis for the first time using a large sample of HCGs and to investigate the evidence for diffuse H I in HCGs. We performed a uniform reduction of all publicly available VLA H I observations (38 HCGs) with a purpose-built pipeline that also maximises the reproducibility of this study. The resulting H I data cubes were then analysed with the latest software tools to perform a manual separation of emission features into those belonging to galaxies and those extending into the intra-group medium. We thereby classified the H I morphological phase of each group as well as quantified their H I deficiency compared to galaxies in isolation. We find little evidence that H I deficiency can be used as a proxy for the evolutionary phase of a compact group in either of the first two phases, with the distribution of H I deficiency being consistent in both. However, for the final phase, the distribution clearly shifts to high H I deficiencies, with more than 90% of the expected H I content typically missing. Across all HCGs studied, we identify a few cases where there is strong evidence for a diffuse gas component in the intra-group medium, which might be detectable with improved observations. We also classify a new sub-phase where groups contain a lone H I -bearing galaxy, but are otherwise devoid of gas. The new morphological phase we have identified is likely the result of an evolved, gas-poor group acquiring a new, gas-rich member. The large spread of H I deficiencies in the first two morphological phases suggests that there is a broad range of initial H I content in HCGs, which is perhaps influenced by large-scale environment, and that the timescale for morphological changes is, in general, considerably shorter than the timescale for the destruction or consumption of neutral gas in these systems.

79 ASTRONOMY AND ASTROPHYSICS↗

MEASURING TRANSIT SIGNAL RECOVERY IN THE KEPLER PIPELINE. I. INDIVIDUAL EVENTS

The Kepler mission was designed to measure the frequency of Earth-size planets in the habitable zone of Sun-like stars. A crucial component for recovering the underlying planet population from a sample of detected planets is understanding the completeness of that sample-the fraction of the planets that could have been discovered in a given data set that actually were detected. Here, we outline the information required to determine the sample completeness, and describe an experiment to address a specific aspect of that question, i.e., the issue of transit signal recovery. We investigate the extent to which the Kepler pipeline preserves individual transit signals by injecting simulated transits into the pixel-level data, processing the modified pixels through the pipeline, and comparing the measured transit signal-to-noise ratio (S/N) to that expected without perturbation by the pipeline. We inject simulated transit signals across the full focal plane for a set of observations for a duration of 89 days. On average, we find that the S/N of the injected signal is recovered at MS = 0.9973({+-} 0.0012) Multiplication-Sign BS - 0.0151({+-} 0.0049), where MS is the measured S/N and BS is the baseline, or expected, S/N. The 1{sigma} width of the distribution around this correlation is {+-}2.64%. This indicates an extremely high fidelity in reproducing the expected detection statistics for single transit events, and provides teams performing their own periodic transit searches the confidence that there is no systematic reduction in transit signal strength introduced by the pipeline. We discuss the pipeline processes that cause the measured S/N to deviate significantly from the baseline S/N for a small fraction of targets; these are primarily the handling of data adjacent to spacecraft re-pointings and the removal of harmonics prior to the measurement of the S/N. Finally, we outline the further work required to characterize the completeness of the Kepler pipeline.

79 ASTRONOMY AND ASTROPHYSICS↗

Prediction of epigenetically regulated genes in breast cancer cell lines

Methylation of CpG islands within the DNA promoter regions is one mechanism that leads to aberrant gene expression in cancer. In particular, the abnormal methylation of CpG islands may silence associated genes. Therefore, using high-throughput microarrays to measure CpG island methylation will lead to better understanding of tumor pathobiology and progression, while revealing potentially new biomarkers. We have examined a recently developed high-throughput technology for measuring genome-wide methylation patterns called mTACL. Here, we propose a computational pipeline for integrating gene expression and CpG island methylation profles to identify epigenetically regulated genes for a panel of 45 breast cancer cell lines, which is widely used in the Integrative Cancer Biology Program (ICBP). The pipeline (i) reduces the dimensionality of the methylation data, (ii) associates the reduced methylation data with gene expression data, and (iii) ranks methylation-expression associations according to their epigenetic regulation. Dimensionality reduction is performed in two steps: (i) methylation sites are grouped across the genome to identify regions of interest, and (ii) methylation profles are clustered within each region. Associations between the clustered methylation and the gene expression data sets generate candidate matches within a fxed neighborhood around each gene. Finally, the methylation-expression associations are ranked through a logistic regression, and their significance is quantified through permutation analysis. Our two-step dimensionality reduction compressed 90% of the original data, reducing 137,688 methylation sites to 14,505 clusters. Methylation-expression associations produced 18,312 correspondences, which were used to further analyze epigenetic regulation. Logistic regression was used to identify 58 genes from these correspondences that showed a statistically signifcant negative correlation between methylation profles and gene expression in the panel of breast cancer cell lines. Subnetwork enrichment of these genes has identifed 35 common regulators with 6 or more predicted markers. In addition to identifying epigenetically regulated genes, we show evidence of differentially expressed methylation patterns between the basal and luminal subtypes. Our results indicate that the proposed computational protocol is a viable platform for identifying epigenetically regulated genes. Our protocol has generated a list of predictors including COL1A2, TOP2A, TFF1, and VAV3, genes whose key roles in epigenetic regulation is documented in the literature. Subnetwork enrichment of these predicted markers further suggests that epigenetic regulation of individual genes occurs in a coordinated fashion and through common regulators.

59 BASIC BIOLOGICAL SCIENCES↗

Early-type galaxies: Automated reduction and analysis of ROSAT PSPC data

Preliminary results of early-type galaxies that will be part of a galaxy catalog to be derived from the complete Rosat data base are presented. The stored data were reduced and analyzed by an automatic pipeline. This pipeline is based on a command language scrip. The important features of the pipeline include new data time screening in order to maximize the signal to noise ratio of faint point-like sources, source detection via a wavelet algorithm, and the identification of sources with objects from existing catalogs. The pipeline outputs include reduced images, contour maps, surface brightness profiles, spectra, color and hardness ratios.

Mackie, G.↗

Central Stars of Planetary Nebulae in the SMC

In FUSE cycle 3's program C056 we studied four Central Stars of Planetary Nebulae (CSPN) in the Small Magellanic Could. All FUSE observations have been successfully completed and have been reduced and analyzed. The observation of one object (SMP SMC 5) appeared to be off-target and no useful stellar flux was gathered. For another observation (SMP SMC 1) the voltage problems resulted in the loss of data from one of the SiC detectors, but we were still able to analyze the remaining data. The analysis and the results are summarized below. The FUSE data were reduced using the latest available version of the FUSE calibration pipeline (CALFUSE v2.4). The flux of these SMC post-AGB objects is at the threshold of FUSE S sensitivity, and the targets required many orbit-long exposures, each of which typically had low (target) count-rates. The background subtraction required special care during the reduction, and was done in a similar manner to our FUSE cycle 2 BOO1 objects. The resulting calibrated data from the different channels were compared in the overlapping regions for consistency. The final combined, extracted spectra of each target was then modeled to determine the stellar and nebular parameters. The FUSE spectra, combined with archival HST spectra, have been analyzed using stellar atmospheres codes such as TLUSTY and CMFGEN to derive photospheric and wind parameters of the central stars, and with ISM models to determine the amount and temperature of the surrounding atomic and molecular hydrogen. We have combined these results with those of our cycle 4 (D034) program (CSPN of the LMC) in Herald & Bianchi 2004a (paper in preparation, will be submitted to ApJ in June 2004). Two of the three SMC objects analyzed were found to have significantly lower stellar temperatures than had been predicted using nebular photoionization models, indicating either a hotter ionizing companion or the existence of strong shocks in the nebular environment. The analysis also revealed that some objects are surrounded by significant quantities of hot (e.g., 1000- 2000 K) molecular H2, similar to what we found for some LMC and Galactic CSPN (Herald & Bianchi 2002, 2003a, b, 2004b, c, d).

Bianchi, Luciana↗

The Hobby–Eberly Telescope Dark Energy Experiment (HETDEX) Survey Design, Reductions, and Detections*

In this work, we describe the survey design, calibration, commissioning, and emission-line detection algorithms for the Hobby–Eberly Telescope Dark Energy Experiment (HETDEX). The goal of HETDEX is to measure the redshifts of over a million Lyα emitting galaxies between 1.88 < z < 3.52, in a 540 deg 2 area encompassing a comoving volume of 10.9 Gpc 3 . No preselection of targets is involved; instead the HETDEX measurements are accomplished via a spectroscopic survey using a suite of wide-field integral field units distributed over the focal plane of the telescope. This survey measures the Hubble expansion parameter and angular diameter distance, with a final expected accuracy of better than 1%. Additionally, we detail the project's observational strategy, reduction pipeline, source detection, and catalog generation, and present initial results for science verification in the Cosmological Evolution Survey, Extended Groth Strip, and Great Observatories Origins Deep Survey North fields. We demonstrate that our data reach the required specifications in throughput, astrometric accuracy, flux limit, and object detection, with the end products being a catalog of emission-line sources, their object classifications, and flux-calibrated spectra.

79 ASTRONOMY AND ASTROPHYSICS↗

The European Southern Observatory-MIDAS table file system

The new and substantially upgraded version of the Table File System in MIDAS is presented as a scientific database system. MIDAS applications for performing database operations on tables are discussed, for instance, the exchange of the data to and from the TFS, the selection of objects, the uncertainty joins across tables, and the graphical representation of data. This upgraded version of the TFS is a full implementation of the binary table extension of the FITS format; in addition, it also supports arrays of strings. Different storage strategies for optimal access of very large data sets are implemented and are addressed in detail. As a simple relational database, the TFS may be used for the management of personal data files. This opens the way to intelligent pipeline processing of large amounts of data. One of the key features of the Table File System is to provide also an extensive set of tools for the analysis of the final results of a reduction process. Column operations using standard and special mathematical functions as well as statistical distributions can be carried out; commands for linear regression and model fitting using nonlinear least square methods and user-defined functions are available. Finally, statistical tests of hypothesis and multivariate methods can also operate on tables.

Peron, M.↗

Automated detection and analysis of particle beams in laser-plasma accelerator simulations

Numerical simulations of laser-plasma wakefield (particle) accelerators model the acceleration of electrons trapped in plasma oscillations (wakes) left behind when an intense laser pulse propagates through the plasma. The goal of these simulations is to better understand the process involved in plasma wake generation and how electrons are trapped and accelerated by the wake. Understanding of such accelerators, and their development, offer high accelerating gradients, potentially reducing size and cost of new accelerators. One operating regime of interest is where a trapped subset of electrons loads the wake and forms an isolated group of accelerated particles with low spread in momentum and position, desirable characteristics for many applications. The electrons trapped in the wake may be accelerated to high energies, the plasma gradient in the wake reaching up to a gigaelectronvolt per centimeter. High-energy electron accelerators power intense X-ray radiation to terahertz sources, and are used in many applications including medical radiotherapy and imaging. To extract information from the simulation about the quality of the beam, a typical approach is to examine plots of the entire dataset, visually determining the adequate parameters necessary to select a subset of particles, which is then further analyzed. This procedure requires laborious examination of massive data sets over many time steps using several plots, a routine that is unfeasible for large data collections. Demand for automated analysis is growing along with the volume and size of simulations. Current 2D LWFA simulation datasets are typically between 1GB and 100GB in size, but simulations in 3D are of the order of TBs. The increase in the number of datasets and dataset sizes leads to a need for automatic routines to recognize particle patterns as particle bunches (beam of electrons) for subsequent analysis. Because of the growth in dataset size, the application of machine learning techniques for scientific data mining is increasingly considered. In plasma simulations, Bagherjeiran et al. presented a comprehensive report on applying graph-based techniques for orbit classification. They used the KAM classifier to label points and components in single and multiple orbits. Love et al. conducted an image space analysis of coherent structures in plasma simulations. They used a number of segmentation and region-growing techniques to isolate regions of interest in orbit plots. Both approaches analyzed particle accelerator data, targeting the system dynamics in terms of particle orbits. However, they did not address particle dynamics as a function of time or inspected the behavior of bunches of particles. Ruebel et al. addressed the visual analysis of massive laser wakefield acceleration (LWFA) simulation data using interactive procedures to query the data. Sophisticated visualization tools were provided to inspect the data manually. Ruebel et al. have integrated these tools to the visualization and analysis system VisIt, in addition to utilizing efficient data management based on HDF5, H5Part, and the index/query tool FastBit. In Ruebel et al. proposed automatic beam path analysis using a suite of methods to classify particles in simulation data and to analyze their temporal evolution. To enable researchers to accurately define particle beams, the method computes a set of measures based on the path of particles relative to the distance of the particles to a beam. To achieve good performance, this framework uses an analysis pipeline designed to quickly reduce the amount of data that needs to be considered in the actual path distance computation. As part of this process, region-growing methods are utilized to detect particle bunches at single time steps. Efficient data reduction is essential to enable automated analysis of large data sets as described in the next section, where data reduction methods are steered to the particular requirements of our clustering analysis. Previously, we have described the application of a set of algorithms to automate the data analysis and classification of particle beams in the LWFA simulation data, identifying locations with high density of high energy particles. These algorithms detected high density locations (nodes) in each time step, i.e. maximum points on the particle distribution for only one spatial variable. Each node was correlated to a node in previous or later time steps by linking these nodes according to a pruned minimum spanning tree (PMST). We call the PMST representation 'a lifetime diagram', which is a graphical tool to show temporal information of high dense groups of particles in the longitudinal direction for the time series. Electron bunch compactness was described by another step of the processing, designed to partition each time step, using fuzzy clustering, into a fixed number of clusters.

43 PARTICLE ACCELERATORS↗

Xi-cam : a versatile interface for data visualization and analysis

Xi-cam is an extensible platform for data management, analysis and visualization. Xi-cam aims to provide a flexible and extensible approach to synchrotron data treatment as a solution to rising demands for high-volume/high-throughput processing pipelines. The core of Xi-cam is an extensible plugin-based graphical user interface platform which provides users with an interactive interface to processing algorithms. Plugins are available for SAXS/WAXS/GISAXS/GIWAXS, tomography and NEXAFS data. With Xi-cam 's `advanced' mode, data processing steps are designed as a graph-based workflow, which can be executed live, locally or remotely. Remote execution utilizes high-performance computing or de-localized resources, allowing for the effective reduction of high-throughput data. Xi-cam 's plugin-based architecture targets cross-facility and cross-technique collaborative development, in support of multi-modal analysis. Xi-cam is open-source and cross-platform, and available for download on GitHub.

Pandolfi, Ronald J. (ORCID:0000000308248548)↗

Embedded FPGA developments in 130 nm and 28 nm CMOS for machine learning in particle detector readout

Embedded field programmable gate array (eFPGA) technology allows the implementation of reconfigurable logic within the design of an application-specific integrated circuit (ASIC). This approach offers the low power and efficiency of an ASIC along with the ease of FPGA configuration, particularly beneficial for the use case of machine learning in the data pipeline of next-generation collider experiments. An open-source framework called "FABulous" was used to design eFPGAs using 130 nm and 28 nm CMOS technology nodes, which were subsequently fabricated and verified through testing. The capability of an eFPGA to act as a front-end readout chip was assessed using simulation of high energy particles passing through a silicon pixel sensor. A machine learning-based classifier, designed for reduction of sensor data at the source, was synthesized and configured onto the eFPGA. A successful proof-of-concept was demonstrated through reproduction of the expected algorithm result on the eFPGA with perfect accuracy. Finally, further development of the eFPGA technology and its application to collider detector readout is discussed.

47 OTHER INSTRUMENTATION↗

LCLS Big Data Handling – How I Learned to Stop Worrying and Love the Data Deluge

Advanced data and computing systems are vital to Linac Coherent Light Source (LCLS) operations, data interpretation and overall scientific productivity. The transition to MHz-era operation marks a fundamental change in scale that requires new infrastructure and architectures to link LCLS to the required scale of computing needed for scientific interpretation. The LCLS-II Data System meets big data challenges by implementing configurable data reduction that can adapt to multiple science areas, real-time analysis frameworks to provide visualization and fast feedback, and the ability to transfer data to local and remote computational facilities for near real time analysis at the appropriate scale. Feature extracted information generated in the data analysis pipeline - at the edge, local compute, or remote High-Performance Computing (HPC) resources - can be used to steer experiments and inform user decisions during beam time. Artificial Intelligence and Machine Learning (AI/ML) techniques present new opportunities to rapidly analyse large datasets and direct experiments, but create new challenges in scaling, adaptability, complexity, and trustworthiness. We describe how the LCLS-II Data System architecture addresses its data-driven challenges in the areas of data acquisition, data processing, data management, and workflow orchestration to decrease the overall time-to-science and provide a vision for future developments.

artificial intelligence↗