Search NASA⌕ Search

SEARCH · Search NASA

Results for “Clustering algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 703 records · Page 39
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

A Hydro-particle-mesh Code for Efficient and Rapid Simulations of the Intracluster Medium

We introduce the cosmological HYPER code based on an innovative hydro-particle-mesh (HPM) algorithm for efficient and rapid simulations of gas and dark matter. For the HPM algorithm, we update the approach of Gnedin & Hui to expand the scope of its application from the lower-density intergalactic medium (IGM) to the higher-density intracluster medium (ICM). While the original algorithm tracks only one effective particle species, the updated version separately tracks the gas and dark matter particles, as they do not exactly trace each other on small scales. For the approximate hydrodynamics solver, the pressure term in the gas equations of motion is calculated using robust physical models. In particular, we use a dark matter halo model, ICM pressure profile, and IGM temperature–density relation, all of which can be systematically varied for parameter-space studies. We show that the HYPER simulation results are in good agreement with the halo model expectations for the density, temperature, and pressure radial profiles. Simulated galaxy cluster scaling relations for Sunyaev–Zel’dovich (SZ) and X-ray observables are also in good agreement with mean predictions, with scatter comparable to that found in hydrodynamic simulations. HYPER also produces lightcone catalogs of dark matter halos and full-sky tomographic maps of the lensing convergence, SZ effect, and X-ray emission. These simulation products are useful for testing data analysis pipelines, generating training data for machine learning, understanding selection and systematic effects, and for interpreting astrophysical and cosmological constraints.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Accelerating and stabilizing the convergence of vibrational self-consistent field calculations via the direct inversion of the iterative subspace (vDIIS) algorithm

The vibrational self-consistent field (VSCF) method yields anharmonic states and spectra for molecular vibrations, and it serves as the starting point for more sophisticated correlated-vibration methods. Convergence of the iterative, non-linear optimization in VSCF calculations can be erratic or altogether unsuccessful, particularly for chemical systems involving low-frequency motions. In this work, a vibrational formulation of the Direct Inversion of the Iterative Subspace method of Pulay is presented and investigated. This formulation accounts for distinct attributes of the vibrational and electronic cases, including the expansion of each single-mode vibrational wavefunction in its own basis set. The resulting Direct Inversion of the Iterative Subspace method is shown to substantially accelerate VSCF convergence in all convergent cases as well as rectify many cases where Roothaan-based methods fail. Performance across systems ranging from small, rigid molecules to weakly bound molecular clusters is investigated in this analysis.

Chemistry↗
DOE OSTI2023

Nonequilibrium design strategies for functional colloidal assemblies

Here, we use a nonequilibrium variational principle to optimize the steady-state, shear-induced interconversion of self-assembled nanoclusters of DNA-coated colloids. Employing this principle within a stochastic optimization algorithm allows us to identify design strategies for functional materials. We find that far-from-equilibrium shear flow can significantly enhance the flux between specific colloidal states by decoupling trade-offs between stability and reactivity required by systems in equilibrium. For isolated nanoclusters, we find nonequilibrium strategies for amplifying transition rates by coupling a given reaction coordinate to the background shear flow. We also find that shear flow can be made to selectively break detailed balance and maximize probability currents by coupling orientational degrees of freedom to conformational transitions. For a microphase consisting of many nanoclusters, we study the flux of colloids hopping between clusters. We find that a shear flow can amplify the flux without a proportional compromise on the microphase structure. This approach provides a general means of uncovering design principles for nanoscale, autonomous, functional materials driven far from equilibrium.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Genarris 2.0: A Random Structure Generator for Molecular Crystals

Genarris is an open source Python package for generating random molecular crystal structures with physical constraints for seeding crystal structure prediction algorithms and training machine learning models. Here we present a new version of the code, containing several major improvements. A MPI-based parallelization scheme has been implemented, which facilitates the seamless sequential execution of user-defined workflows. A new method for estimating the unit cell volume based on the single molecule structure has been developed using a machine-learned model trained on experimental structures. A new algorithm has been implemented for generating crystal structures with molecules occupying special Wyckoff positions. A new hierarchical structure check procedure has been developed to detect unphysical close contacts efficiently and accurately. New intermolecular distance settings have been implemented for strong hydrogen bonds. To demonstrate these new features, we study two specific cases: benzene and glycine. Genarris finds the experimental structures of the two polymorphs of benzene and the three polymorphs of glycine. Program summary Program Title: Genarris 2.0 Program Files doi: http://dx.doi.org/10.17632/grx6mz4pjn.1 Licensing provisions: BSD-3 Clause Programming language: Python, C External routines/libraries: Spglib, ASE, pymatgen, SciPy, mpi4py, scikit-learn, PyTorch, FHI-aims. Nature of problem: Molecular crystal structure prediction. Solution method: Genarris 2.0 generates molecular crystal structures over the 230 space groups, on general and special Wyckoff positions, using physical constraints. Down-sampling of the generated structures may be performed subsequently, based on molecular crystal packing descriptors and an unsupervised machine learning algorithm. Lastly, ab initio structure relaxation may be performed for the final pool. Depending on the user-defined workflow implemented, Genarris may be used to generate diverse molecular crystal datasets to seed evolutionary algorithms or to train machine learning algorithms or as a standalone crystal structure prediction method. Restrictions: For crystal structure generation, the molecule of interest must be semi-rigid with no bond rotational degrees of freedom. Unusual features: Genarris 2.0 is a highly distributed program, making use of MPI for Python parallelization. The user has the ability to design and implement workflows by executing a user-defined list of procedures. Genarris 2.0 offers new features including a machine learning model for estimating the molecular volume in the solid state from the single molecule structure, structure generation in special Wyckoff positions of space groups, hierarchical structure checks including rigorous treatment of non-orthogonal structures, and clustering and down-selection workflows combining first principles simulations with machine learning. (C) 2020 Elsevier B.V. All rights reserved.

Crystal structure prediction↗
DOE OSTI2022

The Pristine survey – XVI. The metallicity of 26 stellar streams around the Milky Way detected with the STREAMFINDER in Gaia EDR3

ABSTRACT We use the photometric metallicities provided by the panoramic Pristine survey to study the veracity and derive the metallicities of the numerous stellar streams found by the application of the STREAMFINDER algorithm to the Gaia Early Data Release 3 data. All 26 streams present in Pristine show a clear metallicity distribution function, which provides an independent check of the reality of these structures, supporting the reliability of STREAMFINDER in finding streams and the power of Pristine to measure precise metallicities. We further present six candidate structures with coherent phase-space and metallicity signals that are very likely streams. The majority of studied streams are very metal-poor (14 structures with [Fe/H] < −2.0) and include three systems with [Fe/H] < −2.9 (C-11, C-19, and C-20). These streams could be the closest debris of low-luminosity dwarf galaxies or may have originated from globular clusters of significantly lower metallicity than any known current Milky Way globular cluster. Our study shows that the promise of the Gaia data for Galactic Archeology studies can be substantially strengthened by quality photometric metallicities, allowing us to peer back into the earliest epochs of the formation of our Galaxy and its stellar halo constituents.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

DELVE Milky Way Satellite Galaxy Census. I. Satellite Population and Survey Selection Function in DES, DELVE, and Pan-STARRS

The properties of Milky Way satellite galaxies have important implications for galaxy formation, reionization, and the fundamental physics of dark matter. However, the population of Milky Way satellites includes the faintest known galaxies, and current observations are incomplete. To understand the impact of observational selection effects on the known satellite population, we perform rigorous, quantitative estimates of the Milky Way satellite galaxy detection efficiency in three wide-field survey datasets: the Dark Energy Survey Year 6, the DECam Local Volume Exploration Data Release 3, and the Pan-STARRS1 Data Release 1. Together, these surveys cover ∼13,600 deg 2 to g ∼ 24.0 and ∼27,700 deg 2 to g ∼ 22.5, spanning ∼91% of the high-Galactic-latitude sky (∣b∣ ≥ 15°). We apply multiple detection algorithms over the combined footprint and recover 49 known satellites above a strict census detection threshold. To characterize the sensitivity of our census, we run our detection algorithms on a large set of simulated galaxies injected into the survey data, which allows us to develop models that predict the detectability of satellites as a function of their properties. We then fit an empirical model to our data and infer the luminosity function, radial distribution, and size–luminosity relation of Milky Way satellite galaxies. Our empirical model predicts a total of $265^{+79}_{-47}$ satellite galaxies with −20 ≤ M V ≤ 0, half-light radii of 15 ≤ r 1/2 , (pc) ≤ 3000, and galactocentric distances of 10 ≤ D GC (kpc) ≤ 300. We also identify a mild anisotropy in the angular distribution of the observed galaxies, at a significance of ∼2σ, which can be attributed to the clustering of satellites associated with the LMC.

Tan, Chin Yi [Univ. of Chicago, IL (United States)↗
DOE OSTI2022

Characterizing Drought Behavior in the Colorado River Basin Using Unsupervised Machine Learning

Drought is a pressing issue for the Colorado River Basin (CRB) due to the social and economic value of water resources in the region and the significant uncertainty of future drought under climate change. Here, we use climate simulations from various Earth System Models (ESMs) to force the Variable Infiltration Capacity hydrologic model and project multiple drought indicators for the sub-watersheds within the CRB. We apply an unsupervised machine learning (ML) based on Non-Negative Matrix Factorization using K-means clustering (NMFk) to synthesize the simulated historical, future, and change in drought indicators. The unsupervised ML approach can identify sub-watersheds where key changes to drought indicator behavior occur, including shifts in snowpack, snowmelt timing, precipitation, and evapotranspiration. While changes in future precipitation vary across ESMs, the results indicate that the Upper CRB will experience increasing evaporative demand and surface-water scarcity, with some locations experiencing a shift from a radiation-limited to a water-limited evaporation regime in the summer. Large shifts in peak runoff are observed in snowmelt-dominant sub-watersheds, with complete disappearance of the snowmelt signal for some sub-watersheds. The work demonstrates the utility of the NMFk algorithm to efficiently identify behavioral changes of drought indicators across space and time and to quickly analyze and interpret hydro climate model results.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Removal of spurious data in Bragg coherent diffraction imaging: an algorithm for automated data preprocessing

Bragg coherent diffraction imaging (BCDI) provides a powerful tool for obtaining high-resolution structural information from nanocrystalline materials. Here a BCDI sample consisting of a large number of randomly oriented nanoscale crystals is considered. Ideally, only one crystal is oriented to produce a Bragg peak on the detector. However, diffraction from other crystals often produces additional signals on the detector. Before the measured diffraction patterns can be processed into structural images, scientists routinely need to manually identify and remove the `alien' intensities from sources other than the intended crystal. With the development of modern high-coherence storage rings, such as the upgraded Advanced Photon Source (APS), the already slow process of manual preprocessing will be untenable for the large volumes of data that will be produced. An automated method of identifying and deleting alien intensities is proposed. This method exploits the fact that BCDI of a perfect crystal produces diffraction data with inversion symmetry around the Bragg peak. This approach uses the machine learning clustering method DBSCAN to distinguish between diffraction from multiple sources, and then calculates cluster size and inversion symmetry to assess whether clusters of intensity belong to desired data or alien signals. This approach can dramatically reduce the amount of time spent manually processing data, allowing BCDI data processing capabilities to keep pace with the technological advances of fourth-generation synchrotron light sources.

36 MATERIALS SCIENCE↗
DOE OSTI2021

SV-Sim: Scalable PGAS-based State Vector Simulation of Quantum Circuits

High-performance quantum circuit simulation in a classic HPC is still imperative in the NISQ era. Observing that the major obstacle of scalable state-vector quantum simulation arises from the massively fine-grained irregular data-exchange with remote nodes, in this paper we present SV-Sim to apply the emerging PGAS-based communication models (i.e., direct peer access for intra-node CPUs/GPUs and SHMEM for inter-node CPU/GPU clusters) for efficient scalable quantum circuit simulation. Through an orchestrated device functional pointer design, SV-Sim is able to abstract the quantum gate sets across various heterogeneous backends, including IBM/Intel/AMD CPUs, NVIDIA /AMD GPUs, and Intel MIC, in a unified framework, but still asserting outstanding performance and tractable interface to higher-level quantum programming environments, such as IBM Qiskit, Microsoft Q\# and Google Cirq. Circumventing the disability of polymorphism in GPUs and leveraging the device-initiated one-sided communication, SV-Sim can process dynamically synthesized quantum circuit in a single GPU/CPU kernel without the need of expensive JIT or runtime branching, significantly improving the performance and simplifying the programming complexity for the emerging variational quantum algorithms. Evaluations on NVIDIA A100-DGX-1, V100-DGX-2, AMD MI100, ALCF Theta, and OLCF Summit HPCs show that SV-Sim can delivery scalable performance on various state-of-the-art HPC platforms, offering a useful tool for quantum algorithm validation and verification.

Li, Ang↗
DOE OSTI2020

FLEET: Flexible Efficient Ensemble Training for Heterogeneous Deep Neural Networks

Parallel training of an ensemble of Deep Neural Networks (DNN) on a cluster of nodes is an effective approach to shorten the process of neural network architecture search and hyper-parameter tuning for a given learning task. Prior efforts have shown that data sharing, where the common preprocessing operation is shared across the DNN training pipelines, saves computational resources and improves pipeline efficiency. Data sharing strategy, however, performs poorly for a heterogeneous set of DNNs where each DNN has varying computational needs and thus different training rate and convergence speed. This paper proposes FLEET, a flexible ensemble DNN training framework for efficiently training a heterogeneous set of DNNs. We build FLEET via several technical innovations. We theoretically prove that an optimal resource allocation is NP-hard and propose a greedy algorithm to efficiently allocate resources for training each DNN with data sharing. We integrate data-parallel DNN training into ensemble training to mitigate the differences in training rates and introduce checkpointing into this context to address the issue of different convergence speeds. Experiments show that FLEET significantly improves the training efficiency of DNN ensembles without compromising the quality of the result.

Guan, Hui↗
DOE OSTI2020

State-of-the-art on research and applications of machine learning in the building life cycle

Fueled by big data, powerful and affordable computing resources, and advanced algorithms, machine learning has been explored and applied to buildings research for the past decades and has demonstrated its potential to enhance building performance. This study systematically surveyed how machine learning has been applied at different stages of building life cycle. By conducting a literature search on the Web of Knowledge platform, we found 9579 papers in this field and selected 153 papers for an in-depth review. The number of published papers is increasing year by year, with a focus on building design, operation, and control. However, no study was found using machine learning in building commissioning. There are successful pilot studies on fault detection and diagnosis of HVAC equipment and systems, load prediction, energy baseline estimate, load shape clustering, occupancy prediction, and learning occupant behaviors and energy use patterns. None of the existing studies were adopted broadly by the building industry, due to common challenges including (1) lack of large scale labeled data to train and validate the model, (2) lack of model transferability, which limits a model trained with one data-rich building to be used in another building with limited data, (3) lack of strong justification of costs and benefits of deploying machine learning, and (4) the performance might not be reliable and robust for the stated goals, as the method might work for some buildings but could not be generalized to others. Finally, findings from the study can inform future machine learning research to improve occupant comfort, energy efficiency, demand flexibility, and resilience of buildings, as well as to inspire young researchers in the field to explore multidisciplinary approaches that integrate building science, computing science, data science, and social science.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

Dark Energy Survey Year 6 Results: Synthetic-source Injection Across the Full Survey Using Balrog

Synthetic source injection (SSI), the insertion of sources into pixel-level on-sky images, is a powerful method for characterizing object detection and measurement in wide-field, astronomical imaging surveys. Within the Dark Energy Survey (DES), SSI plays a critical role in characterizing all necessary algorithms used in converting images to catalogs, and in deriving quantities needed for the cosmology analysis, such as object detection rates, galaxy redshift estimation, galaxy magnification, star-galaxy classification, and photometric performance. We present here a source injection catalog of 146 million injections spanning the entire 5000 deg 2 DES footprint, generated using the Balrog SSI pipeline. Through this SSI sample, we demonstrate that the DES Year 6 (Y6) image processing pipeline provides accurate estimates of the object properties, for both galaxies and stars, at the percent-level, and we highlight specific regimes where the accuracy is reduced. We then show the consistency between SSI and data catalogs, for all galaxy samples developed within the weak lensing and galaxy clustering analyses of DES Y6. The consistency between the two catalogs also extends to their correlations with survey observing properties (seeing, airmass, depth, extinction, etc.). Lastly, we highlight a number of applications of this catalog to the DES Y6 cosmology analysis, such as estimates of the redshift distribution and lens magnification. This dataset is the largest SSI catalog produced at this fidelity and will serve as a key testing ground for exploring the utility of SSI catalogs in upcoming surveys such as the Vera C. Rubin Observatory Legacy Survey of Space and Time.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

TAMM: Tensor algebra for many-body methods

Tensor algebra operations such as contractions in computational chemistry consume a significant fraction of the computing time on large-scale computing platforms. The widespread use of tensor contractions between large multi-dimensional tensors in describing electronic structure theory has motivated the development of multiple tensor algebra frameworks targeting heterogeneous computing platforms. In this paper, we present Tensor Algebra for Many-body Methods (TAMM), a framework for productive and performance-portable development of scalable computational chemistry methods. TAMM decouples the specification of the computation from the execution of these operations on available high-performance computing systems. With this design choice, the scientific application developers (domain scientists) can focus on the algorithmic requirements using the tensor algebra interface provided by TAMM, whereas high-performance computing developers can direct their attention to various optimizations on the underlying constructs, such as efficient data distribution, optimized scheduling algorithms, and efficient use of intra-node resources (e.g., graphics processing units). The modular structure of TAMM allows it to support different hardware architectures and incorporate new algorithmic advances. We describe the TAMM framework and our approach to the sustainable development of scalable ground- and excited-state electronic structure methods. We present case studies highlighting the ease of use, including the performance and productivity gains compared to other frameworks.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

An interlaboratory comparison of mid-infrared spectra acquisition: Instruments and procedures matter

Diffuse reflectance spectroscopy has been extensively employed to deliver timely and cost-effective predictions of a number of soil properties. However, although several soil spectral laboratories have been established worldwide, the distinct characteristics of instruments and operations still hamper further integration and interoperability across mid-infrared (MIR) soil spectral libraries. In this study, we conducted a large-scale ring trial experiment to understand the lab-to-lab variability of multiple MIR instruments. By developing a systematic evaluation of different mathematical treatments with modeling algorithms, including regular preprocessing and spectral standardization, we quantified and evaluated instruments' dissimilarity and how this impacts internal and shared model performance. We found that all instruments delivered good predictions when calibrated internally using the same instruments' characteristics and standard operating procedures by solely relying on regular spectral preprocessing that accounts for light scattering and multiplicative/additive effects, e.g., using standard normal variate (SNV). When performing model transfer from a large public library (the USDA NSSCKSSL MIR library) to secondary instruments, good performance was also achieved by regular preprocessing (e. g., SNV) if both instruments shared the same manufacturer. However, significant differences between the KSSL MIR library and contrasting ring trial instruments responses were evident and confirmed by a semi-unsupervised spectral clustering. For heavily contrasting setups, spectral standardization was necessary before transferring prediction models. Non-linear model types like Cubist and memory-based learning delivered more precise estimates because they seemed to be less sensitive to spectral variations than global partial least square regression. In summary, the results from this study can assist new laboratories in building spectroscopy capacity utilizing existing MIR spectral libraries and support the recent global efforts to make soil spectroscopy universally accessible with centralized or shared operating procedures.

58 GEOSCIENCES↗
DOE OSTI2022

Discovery of peculiar radio morphologies with ASKAP using unsupervised machine learning

Abstract We present a set of peculiar radio sources detected using an unsupervised machine learning method. We use data from the Australian Square Kilometre Array Pathfinder (ASKAP) telescope to train a self-organizing map (SOM). The radio maps from three ASKAP surveys, Evolutionary Map of Universe pilot survey (EMU-PS), Deep Investigation of Neutral Gas Origins pilot survey (DINGO), and Survey With ASKAP of GAMA-09 + X-ray (SWAG-X), are used to search for the rarest or unknown radio morphologies. We use an extension of the SOM algorithm that implements rotation and flipping invariance on astronomical sources. The SOM is trained using the images of all ‘complex’ radio sources in the EMU-PS which we define as all sources catalogued as ‘multi-component’. The trained SOM is then used to estimate a similarity score for complex sources in all surveys. We select 0.5% of the sources that are most complex according to the similarity metric and visually examine them to find the rarest radio morphologies. Among these, we find two new odd radio circle (ORC) candidates and five other peculiar morphologies. We discuss multiwavelength properties and the optical/infrared counterparts of selected peculiar sources. In addition, we present examples of conventional radio morphologies including: diffuse emission from galaxy clusters, and resolved, bent-tailed, and FR-I and FR-II type radio galaxies. We discuss the overdense environment that may be the reason behind the circular shape of ORC candidates.

Astronomy & Astrophysics↗
DOE OSTI2026

Toward scalable quantum computations of atomic nuclei

We solve the nuclear two-body and three-body bound states via quantum simulations of pionless effective field theory on a lattice in position space. While the employed lattice remains small, the usage of local Hamiltonians including two- and three-body forces ensures that the number of Pauli terms scales linearly with increasing numbers of lattice sites. We use an adaptive ansatz grown from unitary coupled cluster theory to parametrize the ground states of the deuteron and 3 He, compute their corresponding energies, and analyze the scaling of the required computational resources. Our quantum simulations reproduce exact benchmarks for 2 H and 3 He within 100 keV, requiring at most 30 layers in the ansatz and thus resulting in modest circuit depths. Additionally, we find the number of shots required to reach a given precision scales linearly in the lattice size and more mildly in the system size. Furthermore, based on the agreement with exact benchmarks and mild scaling, we conclude that this can be an efficient, scalable approach for quantum computations of nuclear ground states, particularly to prepare initial states for quantum phase estimation or other filtering algorithms.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Smart Pixels: In-pixel AI for on-sensor data filtering

We present a smart pixel prototype readout integrated circuit (ROIC) designed in CMOS 28 nm bulk process, with in-pixel implementation of an artificial intelligence (AI) / machine learning (ML) based data filtering algorithm designed as proof-of-principle for a Phase III upgrade at the Large Hadron Collider (LHC) pixel detector. The first version of the ROIC consists of two matrices of 256 smart pixels, each 25$\times$25 $\mu$m$^2$ in size. Each pixel consists of a charge-sensitive preamplifier with leakage current compensation and three auto-zero comparators for a 2-bit flash-type ADC. The frontend is capable of synchronously digitizing the sensor charge within 25 ns. Measurement results show an equivalent noise charge (ENC) of $\sim$30e$^-$ and a total dispersion of $\sim$100e$^-$ The second version of the ROIC uses a fully connected two-layer neural network (NN) to process information from a cluster of 256 pixels to determine if the pattern corresponds to highly desirable high-momentum particle tracks for selection and readout. The digital NN is embedded in-between analog signal processing regions of the 256 pixels without increasing the pixel size and is implemented as fully combinatorial digital logic to minimize power consumption and eliminate clock distribution, and is active only in the presence of an input signal. The total power consumption of the neural network is $\sim$ 300 $\mu$W. The NN performs momentum classification based on the generated cluster patterns and even with a modest momentum threshold, it is capable of 54.4% - 75.4% total data rejection, opening the possibility of using the pixel information at 40MHz for the trigger. The total power consumption of analog and digital functions per pixel is $\sim$ 6 $\mu$W per pixel, which corresponds to $\sim$ 1 W/cm$^2$ staying within the experimental constraints.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

Revisiting single inclusive jet production: timelike factorization and reciprocity

Factorization theorems for single inclusive jet production play a crucial role in the study of jets and their substructure. In the case of small radius jets, the dynamics of the jet clustering can be factorized from both the hard production dynamics, and the dynamics of the low scale jet substructure measurement, and is described by a matching coefficient that can be computed in perturbative Quantum Chromodynamics (QCD). A proposed factorization formula describing this process has been previously presented in the literature, and is referred to as the semi-inclusive, or fragmenting jets formalism. By performing an explicit two-loop calculation, we show the inconsistency of this factorization formula, in agreement with another recent result in the literature. Building on recent progress in the factorization of single logarithmic observables, and the understanding of reciprocity, we then derive a new all-order factorization theorem for inclusive jet production. The use of a jet algorithm, being only a modification of the infrared structure of the measurement, modifies the structure of convolutions in the factorization theorem, as compared to inclusive fragmentation, but maintains the universality of the inclusive hard function and its associated Dokshitzer-Gribov-Lipatov-Altarelli-Parisi (DGLAP) evolution, which are ultraviolet properties. However, the non-trivial structure of convolutions in the factorization theorem implies that the jet functions exhibit a modified evolution. We perform an explicit two-loop calculation of the jet function in both N = 4 super Yang-Mills (SYM), and for all color channels in QCD, finding exact agreement with the structure derived from our renormalization group equations. In addition, we derive several new results, including an extension of our factorization formula to jet substructure observables, a jet algorithm definition of a generating function for the energy correlators, and new results for exclusive jet functions. Our results are a key ingredient for achieving precision jet substructure at colliders.

Effective Field Theories↗