Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Intensive Science”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2014

FIFE-Jobsub: a grid submission system for intensity frontier experiments at Fermilab

The Fermilab Intensity Frontier Experiments use an integrated submission system known as FIFE-jobsub, part of the FIFE (Fabric for Frontier Experiments) initiative, to submit batch jobs to the Open Science Grid. FIFE-jobsub eases the burden on experimenters by integrating data transfer and site selection details in an easy to use and well-documented format. FIFE-jobsub automates tedious details of maintaining grid proxies for the lifetime of the grid job. Data transfer is handled using the Intensity Frontier Data Handling Client (IFDHC) [1] tool suite, which facilitates selecting the appropriate data transfer method from many possibilities while protecting shared resources from overload. Chaining of job dependencies into Directed Acyclic Graphs (Condor DAGS) is well supported and made easier through the use of input flags and parameters.

Box, Dennis↗
DOE OSTI2026

New approaches to Bayesian uncertainty quantification for Nuclear Science (Final Technical Report)

Inverse problems play a central role in experimentation and theory/data comparisons for many areas of modern Nuclear Physics (NP) and High-Energy Physics (HEP). Bayes’s Theorem is a powerful tool for solving Inverse Problems, providing conceptually transparent and unbiased constraints on theoretical parameters and their uncertainties (“Bayesian Inference”) and enabling the quantification of agreement or tension between models and data. However, analyses based on Bayesian Inference are often challenging for NP and HEP applications, either because of the large number of parameters in the problem, the high computational cost, or both. We propose a multi-institutional collaboration to develop and deploy novel Bayesian analysis tools that advance the scientific scope of a broad range of current and future NP experiments. This project brings together NP domain scientists working on several high-profile NP projects for which new, high-performance Bayesian Uncertainty Quantification (“Bayesian UQ”) methods are essential to carry out the science, and data scientists who are developing state-of-the-art methods applicable to these problems. The NP projects in this proposal comprise measurements of the mass and fundamental nature of the neutrino; study of the Quark-Gluon Plasma that filled the early universe; and mapping of natural and anthropogenic radiation environments. While these NP projects have very different scientific goals, with datasets and analysis approaches that differ significantly, they share common requirements for improving computationally intensive Bayesian analyses using advanced Machine Learning algorithms and will benefit strongly from a coherent effort to develop general solutions. This proposal brings together these projects and forefront ML-based data science algorithms to develop such general solutions. The methods developed in this project will also be more widely applicable, thereby advancing science in the larger Nuclear Physics portfolio.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2015

TRIO: Burst Buffer Based I/O Orchestration, In: 2015 IEEE International Conference on Cluster Computing

The growing computing power on leadership HPC systems is often accompanied by ever-escalating failure rates. Checkpointing is a common defensive mechanism used by scientific applications for failure recovery. However, directly writing the large and bursty checkpointing dataset to parallel file systems can incur significant I/O contention on storage servers. Such contention in turn degrades bandwidth utilization of storage servers and prolongs the average job I/O time of concurrent applications. Recently burst buffers have been proposed as an intermediate layer to absorb the bursty I/O traffic from compute nodes to storage backend. But an I/O orchestration mechanism is still desirable to efficiently move checkpointing data from burst buffers to storage backend. In this paper, we propose a burst buffer based I/O orchestration framework, named TRIO, to intercept and reshape the bursty writes for better sequential write traffic to storage servers. Meanwhile, TRIO coordinates the flushing orders among concurrent burst buffers to alleviate the contention on storage server. Our experimental results demonstrated that TRIO could efficiently utilize storage bandwidth and reduce the average job I/O time by 37% on average for data-intensive applications in typical checkpointing scenarios.

Computer Science↗
DOE OSTI2025

Ultimate brightness of a medium-energy synchrotron light source at operational beam intensity

Synchrotron light sources are key instruments of modern science, providing unique opportunities for groundbreaking studies in diverse scientific disciplines and driving innovation in numerous scientific and technological fields. Fourth-generation light sources provide unprecedented capabilities in imaging, spectroscopy and diffraction techniques. Ultimate brightness is the key to advancing to a smaller scale, faster response, and higher data measurement and processing rate. The brightness is primarily determined by the electron beam emittance and energy spread at operational intensity. A common feature of fourth-generation synchrotrons is the short length of the electron bunches combined with a very small transverse beam size. Consequently, the high particle density leads to strong collective effects that significantly increase the emittance and limit the achievable brightness at operational beam intensity. In this article, we summarize our studies of the emittance and brightness scaled with the beam energy and intensity, taking into account the effects of intrabeam scattering, beam-impedance interaction and bunch lengthening provided by higher-harmonic RF systems to identify optimal combinations of machine and beam parameters.

36 MATERIALS SCIENCE↗
DOE OSTI2015

A new development on measurement and control software of SANS BATAN spectrometer (SMARTer) in Serpong, Indonesia

In 2005, the main computer for data acquisition and control system of Small-angle Neutron Scattering (SANS) BATAN Spectrometer (SMARTer) was replaced since it halted to operate the spectrometer. According to this replacement, the new software for data acquisition and control system has been developed in-house. Visual Basic programming language is used in developing the software. In the last two years, many developments have been made both in the hardware and also the software to conduct the experiment is more effective and efficient. Lately, the previous motor controller card (ISA Card) was replaced with the programmable motor controller card (PCI Card) for driving one motor of position sensitive detector (PSD), eight motors of four collimators, and six motors of six pinhole discs. This new control system software makes all motors can be moved simultaneously, then it reduces significantly the consuming time of setting up the instrument before running the experiment. Along with that development, the new data acquisition software under MS Windows operating system is also developed to drive a beam stopper in X-Y directions as well as to read the equipment status such as position of the collimators and PSD, to acquire neutron counts on monitor and PSD detectors, and also to manage 12 samples position automatically. A timer object which is set in one second to read the equipment status via serial port of the computer (RS232C), and general purpose interface board (GPIB) for reading the total counts of each pixel of the PSD from histogram memory was used in this new software. The experiment result displayed in real time on the main window, and the data is saved in the special format for further data reduction and analysis. The new software has been implemented and performed for experiment using a preset count or preset time mode for absolute scattering intensity method.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2026

RTN-124: Photometric Redshifts for the Vera C. Rubin Observatory Data Preview

We present the photometric redshifts (photo-z) inferred using algorithms implemented in the Redshift Assessment Infrastructure Layers (RAIL) for the NSF-DOE Vera C. Rubin Observatory Data Preview 2 (DP2). We produce a compilation of reference redshift catalog using spectroscopic, grism and many band photometric redshift dataset hosted on the LIneA Photo-z Server. We curate training and testing set for assessing the scientific and technical performance of Rubin photo-z. The algorithm applied to the object catalog are FlexZBoost, BPZ, kNN, GPz, DNF and TPz; with a combination of 6-band and 4-band photo-z depending on availability of u and y photometry. The redshift point estimates and uncertainty estimation in tabular format through the Large Survey DataBase (LSDB).

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

ARM Aerosol Measurement Science Group 2019 Strategic Planning Workshop Report

This report summarizes the results of a U.S. Department of Energy Atmospheric Radiation Measurement (ARM) user facility workshop held in November 2019 to advance a science-based strategy for ARM’s aerosol measurement program. This was the second such workshop since the Aerosol Measurement Science Group (AMSG) was chartered in 2015 to enhance coordination of ARM observations of aerosols and atmospheric trace gases with the needs of ARM users. The results presented here reflect the AMSG’s focus in recent years on science-based strategies that will contribute to the increased use of ARM data to fulfill its mission of improving process representations and predictability in climate models. Sessions held during the workshop range from interfacing with models through aerosol sampling strategies to calibration protocols and data products. The AMSG workshops have been designed to recommend actions that will enable ARM to evolve and continue to meet its mission. To that end, the AMSG will develop an actionable plan from the recommendations outlined here. Some are well defined and can reasonably be accomplished in the short term. Others are less definite or of a larger scope that calls for a longer-term implementation. Further discussion will be required to develop and prioritize actionable items related to such areas. Task teams comprising the appropriate expertise and perspective from the AMSG and other members of the community will be formed to achieve this outcome. Some particular topics are recognized as high priority, so plans are underway to develop task teams and to hold follow-on discussions to address them. Four areas currently being considered for short, focused discussion are 1) aerosol measurements on the North Slope of Alaska, 2) improving data usability for modeling, 3) strategies for advancing remote sensing, vertical profiling, and distributed measurements of aerosols, and 4) aerosol sampling strategies at existing ARM sites to provide intensive modes of operation to promote data usage for process and modeling studies.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Seismology in the Cloud: A New Streaming Workflow

Data-intensive research in seismology is experiencing a recent boom, driven in part by large volumes of available data and advances in the growing field of data science. However, there are significant barriers to processing large data volumes, such as long retrieval times from data repositories, complex data management, and limited computational resources. New tools and platforms have reduced the barriers to entry for scientific cluster computing, including the maturation of the commercial cloud as an accessible instrument for research. Here, we build a customized research cluster in the cloud to test a new workflow for large-scale seismic analysis, in which data are processed as a stream (retrieved on-the-fly and acted upon without storing), with data from the Incorporated Research Institutions for Seismology Data Management Center. We use this workflow to deploy a spectral peak detection algorithm over 5.6 TB of compressed continuous seismic data from 2074 stations of the USArray Transportable Array EarthScope network. Using a 50-node cluster in the cloud, we completed the noise survey in 80 hr, with an average data throughput of 1.7 GB per minute. By varying cluster sizes, we find the scaling of our analysis to be sublinear, due to a combination of algorithmic limitations and data center response times. The cloud-based streaming workflow represents an order-of-magnitude increase in acquisition and processing speed compared to a traditional download-store-process workflow, and offers the additional benefits of employing a flexible, accessible, and widely used computing architecture. It is limited, however, due to its reliance on Internet transfer speeds and data center service capacity, and may not work well for repeated analyses or those for which even higher data throughputs are needed. These research applications will require a new class of cloud-native approaches in which both data and analysis are in the cloud.

58 GEOSCIENCES↗
DOE OSTI2022

Examining the Ice-Nucleating Particles from the Eastern North Atlantic (ExINP-ENA) (Field Campaign Report)

The Examining the Ice-Nucleating Particles from the Eastern North Atlantic (ExINP-ENA) field campaign was conducted at the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) user facility’s Eastern North Atlantic (ENA) observatory on Graciosa Island, Azores (39.0916° N, 28.0257° W). The ExINP-ENA campaign began on October 1, 2020, and collected data until March 28, 2021, for a total of 179 days of data generated, with a 45-day intensive operational period (IOP) from October 10 to November 24. This campaign was funded by the DOE Office of Science Early Career Research Program through grant DE-SC001879. This grant contains funding for three field campaigns. The previous field campaign at the Southern Great Plains (SGP) site in Oklahoma, ExINP-SGP, was completed in 2020 (36.6073° N, 97.4876° W), and the upcoming field campaign at the North Slope of Alaska (NSA), ExINP-NSA, will take place from October 2021 at the National Oceanic and Atmospheric Administration (NOAA) Barrow Atmospheric Baseline Observatory (71.3230° N, 156.6114° W).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

ExaFEL: extreme-scale real-time data processing for X-ray free electron laser science

ExaFEL is an HPC-capable X-ray Free Electron Laser (XFEL) data analysis software suite for both Serial Femtosecond Crystallography (SFX) and Single Particle Imaging (SPI) developed in collaboration with the Linac Coherent Lightsource (LCLS), Lawrence Berkeley National Laboratory (LBNL) and Los Alamos National Laboratory. ExaFEL supports real-time data analysis via a cross-facility workflow spanning LCLS and HPC centers such as NERSC and OLCF. Our work therefore constitutes initial path-finding for the US Department of Energy's (DOE) Integrated Research Infrastructure (IRI) program. We present the ExaFEL team's 7 years of experience in developing real-time XFEL data analysis software for the DOE's exascale supercomputers. We present our experiences and lessons learned with the Perlmutter and Frontier supercomputers. Furthermore we outline essential data center services (and the implications for institutional policy) required for real-time data analysis. Finally we summarize our software and performance engineering approaches and our experiences with NERSC's Perlmutter and OLCF's Frontier systems. This work is intended to be a practical blueprint for similar efforts in integrating exascale compute resources into other cross-facility workflows.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

ARM Aerosol Measurement Science Group 2019 Strategic Planning Workshop Report

This report summarizes the results of a U.S. Department of Energy Atmospheric Radiation Measurement (ARM) user facility workshop held in November 2019 to advance a science-based strategy for ARM’s aerosol measurement program. This was the second such workshop since the Aerosol Measurement Science Group (AMSG) was chartered in 2015 to enhance coordination of ARM observations of aerosols and atmospheric trace gases with the needs of ARM users. The results presented here reflect the AMSG’s focus in recent years on science-based strategies that will contribute to the increased use of ARM data to fulfill its mission of improving process representations and predictability in climate models. Sessions held during the workshop range from interfacing with models through aerosol sampling strategies to calibration protocols and data products. The strategies set forth here were also developed to be directly relevant to ARM’s updated Decadal Vision. The AMSG workshops have been designed to recommend actions that will enable ARM to evolve and continue to meet its mission. To that end, the AMSG will develop an actionable plan from the recommendations outlined here. Some are well defined and can reasonably be accomplished in the short term. Others are less definite or of a larger scope that calls for a longer-term implementation. Further discussion will be required to develop and prioritize actionable items related to such areas. Task teams comprising the appropriate expertise and perspective from the AMSG and other members of the community will be formed to achieve this outcome. Some particular topics are recognized as high priority, so plans are underway to develop task teams and to hold follow-on discussions to address them. Four areas currently being considered for short, focused discussion are 1) aerosol measurements on the North Slope of Alaska, 2) improving data usability for modeling, 3) strategies for advancing remote sensing, vertical profiling, and distributed measurements of aerosols, and 4) aerosol sampling strategies at existing ARM sites to provide intensive modes of operation to promote data usage for process and modeling studies.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2015

Comparison of algorithms for determination of rotation measure and Faraday structure. I. 1100–1400 MHz

Faraday rotation measures (RMs) and more general Faraday structures are key parameters for studying cosmic magnetism and are also sensitive probes of faint ionized thermal gas. A definition of which derived quantities are required for various scientific studies is needed, as well as addressing the challenges in determining Faraday structures. A wide variety of algorithms has been proposed to reconstruct these structures. In preparation for the Polarization Sky Survey of the Universe's Magnetism (POSSUM) to be conducted with the Australian Square Kilometre Array Pathfinder and the ongoing Galactic Arecibo L-band Feeds Array Continuum Transit Survey (GALFACTS), we run a Faraday structure determination data challenge to benchmark the currently available algorithms, including Faraday synthesis (previously called RM synthesis in the literature), wavelet, compressive sampling, and QU-fitting. The input models include sources with one Faraday thin component, two Faraday thin components, and one Faraday thick component. The frequency set is similar to POSSUM/GALFACTS with a 300 MHz bandwidth from 1.1 to 1.4 GHz. We define three figures of merit motivated by the underlying science: (1) an average RM weighted by polarized intensity, RM{sub wtd}, (2) the separation Δϕ of two Faraday components, and (3) the reduced chi-squared χ{sub r}{sup 2}. Based on the current test data with a signal-to-noise ratio of about 32, we find the following. (1) When only one Faraday thin component is present, most methods perform as expected, with occasional failures where two components are incorrectly found. (2) For two Faraday thin components, QU-fitting routines perform the best, with errors close to the theoretical ones for RM{sub wtd} but with significantly higher errors for Δϕ. All other methods, including standard Faraday synthesis, frequently identify only one component when Δϕ is below or near the width of the Faraday point-spread function. (3) No methods as currently implemented work well for Faraday thick components due to the narrow bandwidth. (4) There exist combinations of two Faraday components that produce a large range of acceptable fits and hence large uncertainties in the derived single RMs; in these cases, different RMs lead to the same Q, U behavior, so no method can recover a unique input model. Further exploration of all these issues is required before upcoming surveys will be able to provide reliable results on Faraday structures.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Automated Identification of Molecular Crystals’ Packing Motifs

Packing motifs—patterns in how molecules orient relative to one another in a crystal structure—are an important concept in many subdisciplines of materials science because of correlations observed between specific packing motifs and properties of interest. That said, packing motif data sets have remained small and noisy due to intensive manual labeling processes and insufficient labeling schemes. The most prominent labeling algorithms calculate relative interplanar angles of nearest neighbor molecules to determine the packing motif of a molecular crystal, but this simple approach can fail when neighbors are naively sampled isotropically around the crystal structure. To remedy this issue, here we propose an optimization algorithm, which rotates the molecular crystal structure to find representative molecules that inform the packing motif. We package this algorithm into an automated framework—Autopack—which both optimally rotates the crystal structure and labels the packing motif based on the appropriate neighboring molecules. In this work, we detail the Autopack framework and its performance, which shows improvements compared to previous state-of-the-art labeling methods, providing the first quantitative point of comparison for packing motif labeling algorithms. Furthermore, using Autopack (available at https://ipo.llnl.gov/technologies/software/autopack), we perform the first large-scale study of potential relationships between chemicals’ compositions and packing motifs, which shows that these relationships are more complex than previously hypothesized from studies that used only tens of polycyclic aromatic hydrocarbon molecules. Autopack’s capabilities help pose next steps for crystal engineering research focusing not only on a molecule’s adoption of a specific packing motif but also on new structure–property relationships.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Resolution of a discrepancy in the γ-ray emission probability from the β decay of 137 Ce g

We have deduced the emission probability of the 447-keV γ ray from the ε + β + decay of Ce g 137 (9.0 h) relative to that of the 254-keV γ ray from the Ce m 137 (34.4 h) decay in transient equilibrium. The time-dependent factor in transient equilibrium was applied following the Bateman equation for a radioactive decay chain. The isotope was produced via the La 139 ( p , 3 n ) Ce m , g 137 reaction by bombarding nat La with a proton beam from the 88-in. cyclotron at Lawrence Berkeley National Laboratory. γ -ray intensities were measured using an HPGe detector. The emission probability for the 447-keV γ ray deduced in this work is 1.21(3) (that is 1.21 ± 0.03) per hundred parent decays, which differs significantly from an earlier published value of 2.24(10). We identify the source of this discrepancy to be an incorrect use of the time-dependent factor. Additionally, we have deduced the emission probability of the 504-keV γ ray from the decay of Y g 85 (2.68 h) relative to that of the 232-keV γ ray from the Sr m 85 (1.127 h) decay in transient equilibrium. The isotope was produced via the Sr 86 ( p , 2 n ) Y g 85 reaction by bombarding SrCO 3 86 with a proton beam at the same facility. Overall, the study confirms the assumption of the time-dependent correction for recommending the emission probability of the 504-keV γ ray in the literature. Our work highlights the importance of explicit description by authors of any time-dependent correction they have made when reporting γ -ray intensities for nuclides in transient equilibrium. The need and significance of accurate and precise decay data of Ce g 137 and Y g 85 in basic science and medicine is briefly outlined.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

RTN-045: Guidelines for User Tutorials

This document defines the guidelines, principles, and formats for user-facing tutorials that demonstrate how to use the Rubin Science Platform (RSP) to analyze data from the Legacy Survey of Space and Time (LSST). All Rubin staff and the broader science community should use these guidelines when contributing to the sets of Jupyter Notebook or documentation-based tutorials maintained by the Rubin Community Science team (CST).

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

How an Autonomous Offshore Power System Can Transform the Ocean Economy - A Hypothetical Case Study Utilizing an Autonomous Offshore Power System in Northern Lights Carbon Capture and Storage Project

An Autonomous Offshore Power System (AOPS) provides in-situ power, energy storage, real-time data and communications support, asset management, and other capabilities at sea. It has applications for all offshore industries: energy, defense and security, aquaculture, science and research, and communications. Furthermore, this paper highlights how an AOPS can reduce cost, complexity, and carbon-intensity for existing offshore operations and enable new capabilities for offshore industry leaders. This new AOPS technology has two primary advantages. First, it unlocks the autonomous, electric future of the ocean economy via ‘local’ power generation and energy storage, in addition to real-time connection to the data cloud. Second, it helps enable a material change in the global energy mix through cost-effective, reliable generation and storage technology for use cases including mobile/static data-gathering and reporting systems, operating equipment, and charging networks for uncrewed surface vessels. AOPS technology will help transform the ocean economy, and thus has implications for offshore industry leaders as they push to reduce costs today and make an autonomous and decarbonized future possible.

16 TIDAL AND WAVE POWER↗
DOE OSTI2022

Do Machine Learning Approaches Offer Skill Improvement for Short-Term Forecasting of Wind Gust Occurrence and Magnitude?

Abstract Wind gusts, and in particular intense gusts, are societally relevant but extremely challenging to forecast. This study systematically assesses the skill enhancement that can be achieved using artificial neural networks (ANNs) for forecasting of wind gust occurrence and magnitude. Geophysical predictors from the ERA5 reanalysis are used in conjunction with an autoregressive term in regression and ANN models with different predictors, and varying model complexity. Models are derived and assessed for the warm (April–September) and cold (October–March) seasons for three high passenger volume airports in the United States. Model uncertainty is assessed by deriving models for 1000 different randomly selected training (70%) and testing (30%) subsets. Gust prediction fidelity in independent test samples is critically dependent on inclusion of an autoregressive term. Gust occurrence probabilities derived using five-layer ANNs exhibit consistently higher fidelity than those from regression models and shallower ANNs. Inclusion of the autoregressive term and increasing the number of hidden layers in ANNs from 1 to 5 also improve the model performance for gust magnitudes (lower RMSE, increased correlation, and model standard deviations that more closely approximate observed values). Deeper ANNs (e.g., 20 hidden layers) exhibit higher skill in forecasting strong (17–25.7 m s −1 ) and damaging (≥25.7 m s −1 ) wind gusts. However, such deep networks exhibit evidence of overfitting and still substantially underestimate (by 50%) the frequency of strong and damaging wind gusts at the three airports considered herein. Significance Statement Improved short-term forecasting of wind gusts will enhance aviation safety and logistics and may offer other societal benefits. Here we present a rigorous investigation of the relative skill of models of wind gust occurrence and magnitude that employ different statistical methods. It is shown that artificial neural networks (ANNs) offer considerable skill enhancement over regression methods, particularly for strong and damaging wind gusts. For wind gust magnitudes in particular, application of deeper learning networks (e.g., five or more hidden layers) offers tangible improvements in forecast accuracy. However, deeper networks are vulnerable to overfitting and exhibit substantial variability with the specific training and testing data subset used. Also, even deep ANNs reproduce only half of strong and damaging wind gusts. These results indicate the need for future work to elucidate the dynamical mechanisms of intense wind gusts and advance solutions to their prediction.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Real-time data processing for serial crystallography experiments

We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames ('hits'). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data, with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the `data deluge' problem from recent and future photon-science experiments, in particular on calibration requirements, computing access patterns and the need for the preservation of raw data.

47 OTHER INSTRUMENTATION↗