Search NASA⌕ Search

Engineering topics

Schotthoefer, Steffen [ORNL] (ORCID:0000000215696560)

Publications and source records attributed to Schotthoefer, Steffen [ORNL] (ORCID:0000000215696560).

9 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Exploring Black-box Adversarial Attacks on Low-rank Constrained Neural Networks

Low-rank compression has been shown as an effective tool to reduce parameter counts of convolutional and vision transformer architectures; however, low-rank training often reduces model robustness to adversarial perturbations. In this work, we explore the effects of low-rank training on black-box attacks, where attacked images are generated without knowledge of the low-rank parameters. We find that low-rank training is not sufficient as a black-box defense and can sometimes produce worse than expected as compared to baseline models. Influencing the spectrum of the low-rank models during training, which is known to increase model robustness against white-box attacks, improves black-box performance as well.

Schnake, Stefan [ORNL] (ORCID:0000000215183538)↗
DOE OSTI2025

Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimization

Deploying geospatial foundation models on resource-constrained edge devices demands compact architectures that maintain high downstream performance. However, their large parameter counts and the accuracy loss often induced by compression limit practical adoption.In this work, we leverage manifold-constrained optimization framework DLRT to compress large vision transformer–based geospatial foundation models during transfer learning. By enforcing structured low-dimensional parameterizations aligned with downstream objectives, this approach achieves strong compression while preserving task-specific accuracy. We show that the method outperforms of-the-shelf low-rank methods as LoRA. Experiments on diverse geospatial benchmarks confirm substantial parameter reduction with minimal accuracy loss, enabling high-performing, on-device geospatial models.

Snyder, Thomas [Yale University]↗
DOE OSTI2025

A geometric framework for momentum-based optimizers for low-rank training

Low-rank pre-training and fine-tuning have recently emerged as promising techniques for reducing the computational and storage costs of large neural networks. Training low-rank parameterizations typically relies on conventional optimizers such as heavy ball momentum methods or Adam. In this work, we identify and analyze potential difficulties that these training methods encounter when used to train low-rank parameterizations of weights. In particular, we show that classical momentum methods can struggle to converge to a local optimum due to the geometry of the underlying optimization landscape. To address this, we introduce novel training strategies derived from dynamical low-rank approximation, which explicitly account for the underlying geometric structure. Our approach leverages and combines tools from dynamical low-rank approximation and momentum-based optimization to design optimizers that respect the intrinsic geometry of the parameter space. We validate our methods through numerical experiments, demonstrating faster convergence, and stronger validation metrics at given parameter budgets.

Schotthoefer, Steffen [ORNL] (ORCID:00000002156965↗
DOE OSTI2025

Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks

Deployment of neural networks on resource-constrained devices demands models that are both compact and robust to adversarial inputs. However, compression and adversarial robustness often conflict. In this work, we introduce a dynamical low-rank training scheme enhanced with a novel spectral regularizer that controls the condition number of the low-rank core in each layer. This approach mitigates the sensitivity of compressed models to adversarial perturbations without sacrificing clean accuracy. The method is model- and data-agnostic, computationally efficient, and supports rank adaptivity to automatically compress the network at hand. Extensive experiments across standard architectures, datasets, and adversarial attacks show the regularized networks can achieve over 94 compression while recovering or improving adversarial accuracy relative to uncompressed baselines.

Schotthoefer, Steffen [ORNL] (ORCID:00000002156965↗
DOE OSTI2025

Dynamic Low-Rank Training with Spectral Regularization: Achieving Robustness in Compressed Representations

Deployment of neural networks on resource-constrained devices demands models that are both compact and robust to adversarial inputs. However, compression and adversarial robustness often conflict. In this work, we introduce a dynamical low-rank training scheme enhanced with a novel spectral regularizer that controls the condition number of the low-rank core in each layer. This approach mitigates the sensitivity of compressed models to adversarial perturbations without sacrificing clean accuracy. The method is model- and data-agnostic, computationally efficient, and supports rank adaptivity to automatically compress the network at hand. Extensive experiments across standard architectures, datasets, and adversarial attacks show the regularized networks can achieve over 94\% compression while recovering or improving adversarial accuracy relative to uncompressed baselines.

Schotthoefer, Steffen [ORNL] (ORCID:00000002156965↗
DOE OSTI2025

GeoLoRA: Geometric integration for parameter efficient fine-tuning

Low-Rank Adaptation (LoRA) has become a widely used method for parameter-efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face several challenges, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process. We introduce GeoLoRA, a novel approach that addresses these limitations by leveraging dynamical low-rank approximation theory. GeoLoRA requires only a single backpropagation pass over the small-rank adapters, significantly reducing computational cost as compared to similar dynamical low-rank training methods and making it faster than popular baselines such as AdaLoRA. This allows GeoLoRA to efficiently adapt the allocated parameter budget across the model, achieving smaller low-rank adapters compared to heuristic methods like AdaLoRA and LoRA, while maintaining critical convergence, descent, and error-bound theoretical guarantees. The resulting method is not only more efficient but also more robust to varying hyperparameter settings. We demonstrate the effectiveness of GeoLoRA on several state-of-the-art benchmarks, showing that it outperforms existing methods in both accuracy and computational efficiency.

Schotthoefer, Steffen [ORNL] (ORCID:00000002156965↗
DOE OSTI2024

Geometry-aware training of factorized layers in tensor Tucker format

Reducing parameter redundancies in neural network architectures is crucial for achieving feasible computational and memory requirements during train and inference of large networks. Given its easy implementation and flexibility, one promising approach is layer factorization, which reshapes weight tensors into a matrix format and parameterizes it as the product of two rank-r matrices. However, this family of approaches often requires an initial full-model warm-up phase, prior knowledge of a feasible rank, and it is sensitive to parameter initialization.In this work, we introduce a novel approach to train the factors of a Tucker decomposition of the weight tensors. Our training proposal proves to be optimal in locally approximating the original unfactorized dynamics and stable for the initialization. Furthermore, the rank of each mode is dynamically updated during training.We provide a theoretical analysis of the algorithm, showing convergence, approximation and local descent guarantees. The method's performance is further illustrated through a variety of experiments, showing remarkable training compression rates and comparable or even better performance than the full baseline and alternative layer factorization strategies.

Zangrando, Emanuele [Gran Sasso Science Institute ↗