C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models
Explore the source record for details and available documents.
97 MATHEMATICS AND COMPUTING↗
SEARCH · Search NASA
Results for “LoRA”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
GeoLoRA: Geometric integration for parameter efficient fine-tuning
Low-Rank Adaptation (LoRA) has become a widely used method for parameter-efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face several challenges, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process. We introduce GeoLoRA, a novel approach that addresses these limitations by leveraging dynamical low-rank approximation theory. GeoLoRA requires only a single backpropagation pass over the small-rank adapters, significantly reducing computational cost as compared to similar dynamical low-rank training methods and making it faster than popular baselines such as AdaLoRA. This allows GeoLoRA to efficiently adapt the allocated parameter budget across the model, achieving smaller low-rank adapters compared to heuristic methods like AdaLoRA and LoRA, while maintaining critical convergence, descent, and error-bound theoretical guarantees. The resulting method is not only more efficient but also more robust to varying hyperparameter settings. We demonstrate the effectiveness of GeoLoRA on several state-of-the-art benchmarks, showing that it outperforms existing methods in both accuracy and computational efficiency.
Schotthoefer, Steffen [ORNL] (ORCID:00000002156965↗
CAFE AU LAIT: Compute-Aware Federated Augmented Low-Rank AI Training
Federated finetuning is crucial for unlocking the knowledge embedded in pretrained Large Language Models (LLMs) when data are geographically distributed across clients. Unlike finetuning with data from a single institution, federated finetuning allows collaboration across multiple institutions, enabling the utilization of diverse and decentralized datasets while preserving data privacy. Given the high computing costs of LLM training and the emphasis on energy efficiency in Federated Learning (FL), Low-Rank Adaptation (LoRA) has emerged as a widely adopted algorithm due to its significantly reduced number of trainable parameters. However, this assumes that all data silos have the necessary computing resources to compute local updates of LLMs. Nevertheless, in practice, the computing resources across clients are highly heterogeneous: while some may have access to hundreds of GPUs, others might have limited or no GPU access. Recently, federated finetuning using synthetic data has been proposed, allowing clients to participate in a collaborative training run without training LLMs locally. However, our experimental results reveal a performance gap between models trained using synthetic data and those trained using local updates. Motivated by the observed heterogeneity in computing resources and the performance gap, we propose a novel two-stage algorithm that leverages the storage and computing capabilities of a strong server. In the first stage, under the coordination of the strong server, clients with limited computing resources collaborate to generate synthetic data, which is transferred to and stored on the strong server. In the second stage, the strong server uses this synthetic data on behalf of the resource-constrained clients to perform federated LoRA finetuning alongside clients with sufficient computing resources. This approach ensures that all clients can participate in the finetuning process. Experimental results demonstrate that incorporating local updates from even a small fraction of clients improves performance compared to using synthetic data for all clients. Furthermore, we incorporate the Gaussian mechanism in both stages to guarantee client-level differential privacy.
Wang, Jiayi [ORNL]↗
Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimization
Deploying geospatial foundation models on resource-constrained edge devices demands compact architectures that maintain high downstream performance. However, their large parameter counts and the accuracy loss often induced by compression limit practical adoption.In this work, we leverage manifold-constrained optimization framework DLRT to compress large vision transformer–based geospatial foundation models during transfer learning. By enforcing structured low-dimensional parameterizations aligned with downstream objectives, this approach achieves strong compression while preserving task-specific accuracy. We show that the method outperforms of-the-shelf low-rank methods as LoRA. Experiments on diverse geospatial benchmarks confirm substantial parameter reduction with minimal accuracy loss, enabling high-performing, on-device geospatial models.
Snyder, Thomas [Yale University]↗
Atmospheric River Detection Under Changing Seasonality and Mean-State Climate: ARTMIP Tier 2 Paleoclimate Experiments
Atmospheric rivers (ARs) are filamentary structures within the atmosphere that account for a substantial portion of poleward moisture transport and play an important role in Earth's hydroclimate. However, there is no one quantitative definition for what constitutes an atmospheric river, leading to uncertainty in quantifying how these systems respond to global change. This study seeks to better understand how different AR detection tools (ARDTs) respond to changes in climate states utilizing single-forcing climate model experiments under the aegis of the Atmospheric River Tracking Method Intercomparison Project (ARTMIP). We compare a simulation with an early Holocene orbital configuration and another with CO2 levels of the Last Glacial Maximum to a preindustrial control simulation to test how the ARDTs respond to changes in seasonality and mean climate state, respectively. We find good agreement among the algorithms in the AR response to the changing orbital configuration, with a poleward shift in AR frequency that tracks seasonal poleward shifts in atmospheric water vapor and zonal winds. In the low CO2 simulation, the algorithms generally agree on the sign of AR changes, but there is substantial spread in their magnitude, indicating that mean-state changes lead to larger uncertainty. This disagreement likely arises primarily from differences between algorithms in their thresholds for water vapor and its transport used for identifying ARs. These findings warrant caution in ARDT selection for paleoclimate and climate change studies in which there is a change to the mean climate state, as ARDT selection contributes substantial uncertainty in such cases.
Atmospheric river, paleoclimate↗
Atmospheric and oceanic energy transport during North Atlantic freshening events: influences of moisture transport and hydrologic cycle feedbacks
Analogs of present-day rapid ice melt can be found in episodic discharges of icebergs that occurred during glacial periods called Heinrich events. This introduces excess meltwater into the North Atlantic and weakens the Atlantic thermohaline circulation (AMOC), triggering a hydrologic cycle–AMOC collapse feedback as the atmospheric energy transport compensates for reduced northward heat transport. Here we employ a novel series of 100-year North Atlantic “hosing” simulations to investigate atmospheric and oceanic energy transport response from freshwater forcing, focusing in particular on the role of atmospheric rivers (ARs) within atmospheric energy transport. Importantly, we use an “overwriting” methodology that allow us to attribute AMOC weakening to added North Atlantic meltwater and subsequent hydrologic cycle responses, respectively. In contrast to far-reaching response of transient eddies, our results show a substantial increase in moisture convergence from ARs that is geographically constrained to the North Atlantic midlatitudes. Such AR changes nevertheless comprise an important component of net precipitation changes over the Euro-Atlantic sector, with the amount being comparable to that from transient eddies over the subpolar Atlantic. Over the course of the century-long simulations, we demonstrate that hydrologic cycle responses to North Atlantic freshening and subsequent feedbacks, including those from ARs, account for approximately half of the simulated AMOC collapse. Furthermore, our work highlights the dynamics of atmospheric moisture transport response to North Atlantic freshening events and elucidates how intensifying moisture transport may accelerate AMOC collapse in the future.
Atmospheric Science↗
Structural Distortions and Uniaxial Negative Thermal Expansion in the Polar Dion–Jacobson Oxide RbNdTa 2 O 7
We provide deeper insight into the crystal structures, sequential structural phase transitions (I2cm → Cmce → I4/mcm → P4/mmm), thermal expansion, and electronic properties of the n = 2 Dion–Jacobson polar oxide RbNdTa 2 O 7 , through X-ray powder diffraction, neutron powder diffraction, Raman studies, and density functional theory calculations. We observed a uniaxial negative thermal expansion (NTE) across the first-order transition, I2cm → Cmce, where the unit cell contracts along the c-axis, which is driven by a contraction of the NdTa 2 O 6 layer. Here, this NTE occurs within the temperature range of the first-order phase transition and contrasts with the corkscrew mechanism typically observed in Ruddlesden–Popper phases. In RbNdTa 2 O 7 , the I2cm (hybrid improper ferroelectric) → Cmce (antipolar) transition involves crucial changes in the bond lengths of Nd and Ta polyhedra, coupled with polar to antipolar displacement of the Nd ions, leading to a net contraction in the NdTa 2 O 6 layer along the c-axis, while preserving the overall octahedral tilting magnitude. This transition highlights the intricate interplay between the Nd and Ta coordination and the associated TaO 6 distortions. Temperature-dependent Raman spectra analysis further confirms the first-order structural transition and associated NTE, providing evidence for increased bond stiffness across this transition. Additionally, using neutron powder diffraction, we have determined that the transition I4/mcm → P4/mmm occurs at approximately 1150 K. Finally, we have calculated from DFT + U, the partial density of states, the energy bandgaps, and effective masses of the charge carriers of the polar ground structure.
Chemical structure↗
Propagation and Periodicity of Mars's Northern Annular Mode Modulates the Dust Cycle
Abstract We document the propagation of annular modes—zonally symmetric patterns of variability—in Mars's atmosphere using a reanalysis dataset. Mars's Northern Annular Mode (MNAM) sees anomalies of zonal‐mean zonal wind emerge near the subtropics and migrate poleward with a period of 150 days, similarly to Earth's Southern Annular Mode. The mechanism of propagation involves the interaction of the two leading empirical orthogonal functions that define the MNAM. Moreover, the propagation encourages alternating bands of surface wind stress to migrate polewards with a 150‐day period. In addition, a 150‐day periodicity in anomalous column dust optical depth most likely emerges in response to extrema of the MNAM. The combination of the impact of the MNAM's internally forced periodicity on the surface wind stress and the seasonal cycle may contribute to the inter‐annual variability of global dust events, as suggested by a Monte Carlo estimate that correctly approximates the observed incidence of global dust events.
54 ENVIRONMENTAL SCIENCES↗
In-Conduit Hydropower for Public Water Supply Systems: Development Challenges and Costs
Conduit hydropower leverages existing water infrastructure to provide energy solutions, though deployment has been slow. A 2022 US Department of Energy (DOE)–funded report led by Oak Ridge National Laboratory (ORNL) summarizes a reconnaissance-level study of hydropower potential at national conduits (Kao et al., 2022). The report includes power and energy estimates across three major sectors of conduit hydropower potential: municipal, agricultural, and industrial. Water resources include public water supply systems (municipal and industrial), irrigation canal systems (agricultural), thermoelectric cooling systems (industrial), and wastewater systems (municipal and industrial). In total, 1.41 GW of power potential was estimated nationwide. Subsequently, DOE has funded ORNL through the Conduit Hydropower Engineering, Evaluation, and Technology Acceleration project (CHEETA) to improve deployment potential for conduit hydropower. Work thus far has focused on “in-conduit” hydropower (ICH)—or “in-pipe” hydropower—for public water supply systems, though the project is planning to cover conduit hydropower more broadly. The timeline of ORNL’s conduit hydropower related research and development (R&D) is shown in Figure 1. The results of CHEETA should benefit the hydropower industry by providing insights into costs, barriers, and other considerations that impact project development. The hydropower industry and municipal/industrial conduit stakeholders will use the output to inform development decisions and advance low-impact hydropower growth in the US.
13 HYDRO ENERGY↗
Generative AI Training Tools for Particle Physics Simulation
Explore the source record for details and available documents.
43 PARTICLE ACCELERATORS↗