Search NASA⌕ Search

SEARCH · Search NASA

Results for “dataset shift”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

Solar, Wind, and Load Forecasting Dataset for MISO, NYISO, and SPP Balancing Areas

The Performance-based Energy Resource Feedback, Optimization, and Risk Management (PERFORM) program is an initiative intended to foster "a fundamental shift in grid management rooted in an understanding of asset risk and system risk" (ARPA-E 2020). Launched by the Advanced Research Projects Agency-Energy (ARPA-E), the program supports efforts to incorporate uncertainty in electric power decision making. In support of PERFORM, the National Renewable Energy Laboratory (NREL) has produced a set of time-coincident forecasts of solar, wind, and load profiles. As part of Phase I of the PERFORM effort, NREL created a dataset that consists of one year of time-coincident load, wind, and solar actuals and probabilistic forecasts based on data from the Electric Reliability Council of Texas (ERCOT) (Bryce et al. 2023). In Phase II, NREL developed similar datasets for three other U.S. Independent System Operators (ISO): the Midcontinent Independent System Operator (MISO), the New York Independent System Operator (NYISO), and the Southwest Power Pool (SPP).

24 POWER TRANSMISSION AND DISTRIBUTION↗

Time-temperature-superposition analysis of diverse datasets by the minimum-arclength method: long-term prediction with uncertainty margins

In a recent publication, we carried out an extensive analysis of an unsupervised method of determining optimum shift factors in time-temperature-superposition of accelerated-aging data that involves minimizing the vertical arclength to obtain the master curve. For synthetic Arrhenius data with a variety of noise distributions, the work showed that, in conjunction with bootstrap-resampling, the method can produce reliable estimates of the mean activation energy along with uncertainty quantification. Here, we apply the above method to six different datasets taken from the published literature and demonstrate accurate prediction of mean activation energy from the data as-is without the need for any pre-processing or fitting. We also compare uncertainty margins computed by second-order bootstrap with that by linear regression theory and show that the former appears to provide consistent margins in the presence of common noise types in real data, including intra-isotherm measurement-errors, sample-to-sample variations, and intrinsic deviation from perfect Arrhenius behavior.

36 MATERIALS SCIENCE↗

Resource-Adaptive Federated Text Generation with Differential Privacy

In cross-silo federated learning (FL), sensitive text datasets remain confined to local organizations due to privacy regulations, making repeated training for each downstream task both communication-intensive and privacy-demanding. A promising alternative is to generate differentially private (DP) synthetic datasets that approximate the global distribution and can be reused across tasks. However, pretrained large language models (LLMs) often fail under domain shift, and federated finetuning is hindered by computational heterogeneity: only resource-rich clients can update the model, while weaker clients are excluded, amplifying data skew and the adverse effects of DP noise. We propose a flexible participation framework that adapts to client capacities. Strong clients perform DP federated finetuning, while weak clients contribute through a lightweight DP voting mechanism that refines synthetic text. To ensure the synthetic data mirrors the global dataset, we apply control codes (e.g., labels, topics, metadata) that represent each client’s data proportions and constrain voting to semantically coherent subsets. This two-phase approach requires only a single round of communication for weak clients and integrates contributions from all participants. Experiments show that our framework improves distribution alignment and downstream robustness under DP and heterogeneity.

Wang, Jiayi [ORNL]↗

Alaska Ecological Conservation II: Using NASA Earth Observations to Identify Recent Changes in Vegetation Phenology and Its Impacts on Caribou Calving and Migration

Caribou are known for their long-distance migrations from wintering grounds to specified calving zones. Notably, the Western Arctic Herd (WAH) is the largest caribou herd in Northwestern Alaska that exhibits this behavior. These calving zones are highly predictable, and research suggests that calving and the availability of high-quality vegetation are correlated. We partnered with the U.S. National Park Service in Northwestern Alaska to use Terra and Aqua Moderate Resolution Imaging Spectroradiometer(MODIS)and Harmonized Landsat and Sentinel-2 (HLS) imagery to derive Normalized Difference Vegetated Index (NDVI) values and assess vegetation phenology in calving zones from 2000 –2024. Our partners seek to understand if observed changes in caribou migration patterns and calving events correlate with shifts in the availability of nutrient-rich vegetation. Our results showed spatial variations of highly vegetated zones and annual shifts in the timing and length of growing seasons. While limitations existed in the MODIS and HLS datasets, we demonstrated feasibility using National Aeronautics and Space Administration (NASA)Earth observations for NDVI analysis over the study region. Using location data, our results may further research into caribou calving behaviors.

NDVI↗

Land-use harmonization datasets for annual global carbon budgets

Abstract. Land-use change has been the dominant source of anthropogenic carbon emissions for most of the historical period and is currently one of the largest and most uncertain components of the global carbon cycle. Advancing the scientific understanding on this topic requires that the best data be used as input to state-of-the-art models in well-organized scientific assessments. The Land-Use Harmonization 2 dataset (LUH2), previously developed and used as input for simulations of the 6th Coupled Model Intercomparison Project (CMIP6), has been updated annually to provide required input to land models in the annual Global Carbon Budget (GCB) assessments. Here we discuss the methodology for producing these annual LUH2-GCB updates and extensions which incorporate annual wood harvest data updates from the Food and Agriculture Organization (FAO) of the United Nations for dataset years after 2015 and the History Database of the Global Environment (HYDE) gridded cropland and grazing area data updates (based on annual FAO cropland and grazing area data updates) for dataset years after 2012, along with extrapolations to the current year due to a lag of 1 or more years in the FAO data releases. The resulting updated LUH2-GCB datasets have provided global, annual gridded land-use and land-use-change data relating to agricultural expansion, deforestation, wood harvesting, shifting cultivation, regrowth and afforestation, crop rotations, and pasture management and are used by both bookkeeping models and dynamic global vegetation models (DGVMs) for the GCB. For GCB 2019, a more significant update to LUH2 was produced, LUH2-GCB2019 (https://doi.org/10.3334/ORNLDAAC/1851, Chini et al., 2020b), to take advantage of new data inputs that corrected cropland and grazing areas in the globally important region of Brazil as far back as 1950. From 1951 to 2012 the LUH2-GCB2019 dataset begins to diverge from the version of LUH2 used for the World Climate Research Programme's CMIP6, with peak differences in Brazil in the year 2000 for grazing land (difference of 100 000 km2) and in the year 2009 for cropland (difference of 77 000 km2), along with significant sub-national reorganization of agricultural land-use patterns within Brazil. The LUH2-GCB2019 dataset provides the base for future LUH2-GCB updates, including the recent LUH2-GCB2020 dataset, and presents a starting point for operationalizing the creation of these datasets to reduce time lags due to the multiple input dataset and model latencies.

Chini, Louise (ORCID:0000000290703505)↗

Bridging the Gap Between Astronomical Datasets: From Proof-of-Concept to AI Model Deployment with Domain Adaptation

Artificial Intelligence is transforming astrophysics, from studying stars and galaxies to analyzing cosmic large-scale structures. However, a critical challenge arises when AI models trained on simulations or past observational data are applied to new observation— leading to domain shifts, reduced robustness, and increased uncertainty of model predictions. This talk will explore these issues, highlighting examples such as galaxy morphology classification and cosmological parameter inference, where AI struggles to adapt across different datasets. We will discuss domain adaptation as a strategy to improve model generalization and mitigate biases—essential for making AI-driven discoveries reliable. Notably, these challenges extend beyond astrophysics, affecting AI applications across physics and other scientific domains. Addressing them is essential for maximizing AI’s impact in advancing scientific research.

Ćiprijanović, Aleksandra [Fermilab]↗

Datasets used in manuscript, 'High-Throughput Chromosomal Confirmation Capture (Hi-C) Metagenome Sequencing Reveals Moisture Impact on Soil Phage-Host Interactions'

Soil moisture shifts have largely unknown impacts on soil virus-host interactions. Here, we applied high-throughput chromosomal confirmation capture (Hi-C) metagenomics to link phage with their hosts in soils under wet and dry conditions. Bulk metagenomes and metatranscriptomes were analyzed from the same soil incubations. Host-associated phage diversity and the number of viruses per host increased following soil desiccation. Under wet conditions, the viral enrichment and host abundances were significantly negatively correlated, but the transcriptional activities of the phage were higher. Together, these results suggest that there was a general transition of phage from lytic to lysogenic during drying and that viral infection was higher under dry conditions. Soil desiccation also caused shifts in phage hosts and some of these were central in microbial co-occurrence networks, highlighting the impact of soil phages on microbiome structure. This study provides the first empirical evidence of phage-mediated bacterial dynamics during soil desiccation.

59 BASIC BIOLOGICAL SCIENCES↗

Class imbalance in out-of-distribution datasets: Improving the robustness of the TextCNN for the classification of rare cancer types

In the last decade, the widespread adoption of electronic health record documentation has created huge opportunities for information mining. Natural language processing (NLP) techniques using machine and deep learning are becoming increasingly widespread for information extraction tasks from unstructured clinical notes. Disparities in performance when deploying machine learning models in the real world have recently received considerable attention. In the clinical NLP domain, the robustness of convolutional neural networks (CNNs) for classifying cancer pathology reports under natural distribution shifts remains understudied. In this research, we aim to quantify and improve the performance of the CNN for text classification on out-of-distribution (OOD) datasets resulting from the natural evolution of clinical text in pathology reports. We identified class imbalance due to different prevalence of cancer types as one of the sources of performance drop and analyzed the impact of previous methods for addressing class imbalance when deploying models in real-world domains. Our results show that our novel class-specialized ensemble technique outperforms other methods for the classification of rare cancer types in terms of macro F1 scores. We also found that traditional ensemble methods perform better in top classes, leading to higher micro F1 scores. Based on our findings, we formulate a series of recommendations for other ML practitioners on how to build robust models with extremely imbalanced datasets in biomedical NLP applications.

60 APPLIED LIFE SCIENCES↗

Low-Temperature Geothermal Geospatial Datasets: An Example from Alaska: Preprint

This project is part of a broader effort focused on geothermal heating and cooling with the purpose of demonstrating the multi-faceted value of integrating GHC/GHX into national decarbonization plans and community energy plans. Currently, there is a need to better define and evaluate low-temperature geothermal resource potential to provide the basis for supporting private sector investment. For shallow GHC/GHX fields, there is no formal compilation of subsurface characteristics shared among industry practitioners that can improve system design and operations. Alaska is specifically noted in this work, because heretofore, it has not received a similar focus in geothermal potential evaluations as the contiguous United States. The methodology consists of leveraging relevant data generate a baseline low-temperature resources (< 150 degrees C) geospatial dataset to compare and analyze information accessible to anyone trying to understand the potential of GHC/GHX and small-scale low-temperature geothermal power in Alaska (e.g., energy modelers, communities, planners, and policymakers). Importantly, this project is identifying data related to (1) the evaluation of GHC/GHX in the shallow subsurface, and (2) the evaluation of low-temperature geothermal resource availability. Additionally, data is being compiled to assess repurposing of oil and gas wells to contribute co-produced fluids toward the geothermal direct use and heating and cooling resource potential. The goal of this project is facilitating future deployment of GHC/GHX analysis and community-led programs and update the low-temperature geothermal resources assessment of Alaska. A better understanding of shallow potential for geothermal heat exchange (GHX) will improve design and operations of highly efficient GHC systems. The deployment and impact that can be achieved for low temperature geothermal resources will contribute to decarbonization goals and facilitate widespread electrification by shaving and shifting grid loads.

bottom-hole temperature↗

Multiscale Effects Masked the Impact of the COVID-19 Pandemic on Electricity Demand in the United States

Shelter-in-place orders and business closures related to COVID-19 changed the hourly profile of electricity demand and created an unprecedented source of uncertainty for the grid. The potential for continued shifts in electricity profiles has implications for electricity sector investment and operating decisions that maintain reserve margins and provide grid reliability. This study reveals that understanding this uncertainty requires an understanding of the underlying drivers at the customer-class scale. This paper utilizes three datasets to compare the impacts of COVID-19 on electricity consumption across a range of spatiotemporal and customer scales. At the utility/customer-class scale, COVID-19-induced shutdowns in the spring of 2020 shifted weekday residential load profiles to resemble weekend profiles from previous years. Total commercial loads declined, but the commercial diurnal load profile was unchanged. With only total loads available at the balancing authority scale, the apparent impact of COVID-19 was smaller during the summer due in part to phased re-opening and spatial variability in re-opening, but there were still clear variations once total loads were broken down zonally. Monthly data at the state scale showed an increase in state-level residential electricity sales, a decrease in commercial sales, and a small net decrease in total sales in most states from April-August 2020. Analyses that focus on total load or a single scale may miss important changes that become apparent when the load is broken down regionally or by customer class.

COVID-19, electricity demand, multiscale, Commonwe↗

The effect of price-based demand response on carbon emissions in European electricity markets: The importance of adequate carbon prices

Price-based demand response (PBDR) has recently been attributed great economic but also environmental potential. However, the determination of its short-term effects on carbon emissions requires the knowledge of marginal emission factors (MEFs), which compared to grid mix emission factors (XEFs), are cumbersome to calculate due to the complex characteristics of national electricity markets. This study, therefore, proposes two merit order-based methods to approximate hourly MEFs and applies them to readily available datasets from 20 European countries for the years 2017–2019. Based on the calculated electricity prices, standardized daily load shifts were simulated which indicated that carbon emissions increased for 8 of the 20 countries and by 2.1% on average. Thus, under specific circumstances, PBDR leads to carbon emissions increases, mainly due to the economic advantage fuel sources such as lignite and coal have in the merit order. MEF-based load shifts reduced the mean resulting carbon emissions by 35%, albeit with 56% lower monetary cost savings compared to price-based load shifts. Finally, by repeating the load shift simulations for different carbon price levels, the impact of the carbon price on the resulting carbon emissions was analyzed. The Spearman correlation coefficient between carbon intensity and marginal cost along the German merit order substantially increased with increasing carbon price. The coefficients were -0.13 for the 2019 carbon price of 24.9 €/t, 0 for 42.6 €/t, and 0.4 for 100.0 €/t. Therefore, with adequate carbon prices, PBDR can be an effective tool for both economical and environmental improvement.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Reanalysis of Water Withdrawal for Irrigation, Electric Power, and Public Supply Sectors in the Conterminous United States, 1950–2016

Accurately measuring water use by the economy is essential for developing reliable models of water resource availability. Indeed, these models rely on retrospective analyses that provide insights into shifting human population demands and adaptions to water shortages. However, accurate, methodologically consistent, empirically authentic, and spatiotemporally comprehensive historical datasets for water withdrawals are scarce. Herein, we present a reanalysis of annual resolution (1950–2016) historical data set on irrigation, electric power, and public supply water withdrawal within the conterminous United States (US) at the county-level, and, for power plants, at the site-level. To estimate electric power water use, we synthesized a historically comprehensive list of generators and historic patterns in generation across fuels, prime movers, and cooling technologies. Irrigation water use estimation required building a crop-demand model that utilized historical information on irrigated acreage for crops and golf courses, stage-specific crop water demand, and climate information. To estimate public water supply use, we developed a random forest model constructed from information on population, infrastructure, climate, and land cover. These estimates generally agree with total county and state water use information provided by the US Geological Survey (USGS) water use circular and estimates generated from independent studies for specific years. However, we also observed discrepancies between our estimates and USGS data that appear to be caused by inconsistencies in the methods used by the USGS's primary data sources at the state level over decades of data collection, highlighting the importance of reanalysis to yield spatiotemporally consistent and intercomparable estimates of water use.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

The Impacts of Fossil Fuel Emission Uncertainties and Accounting for 3-D Chemical CO2 Production on Inverse Natural Carbon Flux Estimates from Satellite and In Situ Data

Atmospheric carbon dioxide (CO2) inversions for estimating natural carbon fluxes typically do not allow for adjustment of fossil fuel CO2 emissions, despite significant uncertainties in emission inventories and inadequacies in the specification of international bunker emissions in inversions. Also, most inversions place CO2 release from fossil fuel combustion and biospheric sources entirely at the surface. However, a non-negligible portion of the emissions actually occurs in the form of reduced carbon species, which are eventually oxidized to CO2 downwind. Omission of this ‘chemical pump’ can result in a significant redistribution of the inferred total carbon fluxes among regions. We assess the impacts of different prescriptions of fossil fuel emissions and accounting for the chemical pump on flux estimation, with a novel aspect of conducting both satellite CO2 observation-based and surface in situ-based inversions. We apply 3-D carbon monoxide (CO) loss rates archived from a state-of-the-art GEOS chemistry and climate model simulation in a forward transport model run to simulate the distribution of CO2 originating from oxidation of carbon species. We also subtract amounts from the prior surface CO2 fluxes that are actually emitted in the form of fossil and biospheric CO, methane, and nonmethane volatile organic compounds (VOCs). We find that the posterior large-scale fluxes are generally insensitive to the finer-scale spatial differences between the ODIAC and CDIAC fossil fuel CO2 gridded datasets and assumptions about international bunker emissions. However, accounting for 3-D chemical CO2 production and the surface correction shifts the global carbon sink, e.g., from land to ocean and from the tropics to the north, with a magnitude and even direction that depend on assumptions about the surface correction. A GOSAT satellite-based inversion is more sensitive to the chemical pump than one using in situ observations, exhibiting substantial flux impacts of 0.28, 0.53, and -0.47 Pg C yr(exp -1) over tropical land, global land, and oceans, due to differences in the horizontal and vertical sampling of the wo observation types. Overall, the biases from neglecting the chemical pump appear to be minor relative to the flux estimate uncertainties and the differences between the in situ and GOSAT inversions, but their relative importance will grow in the future as observational coverage further increases and satellite retrieval biases decrease.

James S Wang↗

SpaceNet 9—Cross-Sensor Alignment of Optical and SAR Imagery

Precise registration of high-resolution synthetic aperture radar (SAR) and optical imagery is necessary for realizing the full potential and benefits of multimodal image analysis. However, two significant challenges presently exist. First, there is a lack of annotated datasets and benchmarks available for high-resolution SAR–optical image registration. Second, an assessment of efficient and reliable image registration methods that can precisely align these modalities is lacking. Here, we present a holistic description of the SpaceNet 9 Challenge and its results. We present a description of the dataset and baseline algorithm along with the results of the challenge, including a description of the winning algorithms. We release the SpaceNet 9 dataset along with open-sourcing the winning algorithms and baseline. The objective of SpaceNet 9 was to compute a dense displacement map that indicates the shift needed to align pixels in an optical image to the pixels in a SAR image. The challenge launched in April 2025 and was active for approximately two months. The top five solutions reduced image alignment error from approximately 34 m to under 13 m for public and private test data, with the best results obtaining a registration error of only 8.5 and 6.7 m on the public testing and private testing dataset, respectively. Usage of pretrained image matching models, robust outlier rejection with RANSAC, and estimating local displacement were common among the top solutions. The results of this challenge provide insight into high-resolution SAR–optical image registration and offer opportunities for future benchmarking in this domain. The baseline algorithm, winning solutions, and datasets are available at https://spacenet.ai/sn9-challenge/.

benchmark datasets↗

Warming of the Global Ocean: Spatial Structure and Water-Mass Trends

This study investigates the multidecadal warming and interannual-to-decadal heat content changes in the upper ocean (0-700 m), focusing on vertical and horizontal patterns of variability. These results support a nearly monotonic warming over much of the World Ocean, with a shift toward Southern Hemisphere warming during the well-observed past decade. This is based on objectively analyzed gridded observational datasets and on a modeled state estimate. Besides the surface warming, a warming climate also has a subsurface effect manifesting as a strong deepening of the midthermocline isopycnals, which can be diagnosed directly from hydrographic data. This deepening appears to be a result of heat entering via subduction and spreading laterally from the high-latitude ventilation regions of subtropical mode waters. The basin-average multidecadal warming mainly expands the subtropical mode water volume, with weak changes in the temperature-salinity (u-S) relationship (known as ''spice'' variability). However, the spice contribution to the heat content can be locally large, for example in Southern Hemisphere. Multidecadal isopycnal sinking has been strongest over the southern basins and weaker elsewhere with the exception of the Gulf Stream/North Atlantic Current/subtropical recirculation gyre. At interannual to decadal time scales, wind-driven sinking and shoaling of density surfaces still dominate ocean heat content changes, while the contribution from temperature changes along density surfaces tends to decrease as time scales shorten.

Global Ocean↗

The DECADE cosmic shear project III: validation of analysis pipeline using spatially inhomogeneous data

We present the pipeline for the cosmic shear analysis of the Dark Energy Camera All Data Everywhere (DECADE) weak lensing dataset: a catalog consisting of 107 million galaxies observed by the Dark Energy Camera (DECam) in the northern Galactic cap. The catalog derives from a large number of disparate observing programs and is therefore more inhomogeneous across the sky compared to existing lensing surveys. First, we use simulated data-vectors to show the sensitivity of our constraints to different analysis choices in our inference pipeline, including sensitivity to residual systematics. Next we use simulations to validate our covariance modeling for inhomogeneous datasets. Finally, we show that our choices in the end-to-end cosmic shear pipeline are robust against inhomogeneities in the survey, by extracting relative shifts in the cosmology constraints across different subsets of the footprint/catalog and showing they are all consistent within 1σ to 2σ. This is done for forty-six subsets of the data and is carried out in a fully consistent manner: for each subset of the data, we re-derive the photometric redshift estimates, shear calibrations, survey transfer functions, the data vector, measurement covariance, and finally, the cosmological constraints. Our results show that existing analysis methods for weak lensing cosmology can be fairly resilient towards inhomogeneous datasets. This also motivates exploring a wider range of image data for pursuing such cosmological constraints.

79 ASTRONOMY AND ASTROPHYSICS↗

China’s carbon sinks from land-use change underestimated

The size and attribution of the regional net carbon flux from land-use change (LUC) activities (E LUC ) are often highly debated, especially in regions such as China, which has experienced decades-long extensive reforestation activities. Here, in this work, using a LUC dataset incorporating remote-sensing and national forest inventory data with two modelling approaches, we show that E LUC in China shifted from a carbon source to a sink in the 1990s, contributing to a net cumulative CO 2 removal of 2.0 Pg C during 1981–2020. From 2001 to 2020, the average E LUC was -0.14 Pg C yr -1 , accounting for over one-third of the national land carbon sinks. Forest-related LUC activities contributed greatly to national carbon fluxes, while non-forest-related activities played a dominant role in certain areas. Our findings suggest that the carbon sinks from LUC activities in China may be largely underestimated in global assessments, underscoring the need to develop region-specific modelling for evaluation and potential regulation.

54 ENVIRONMENTAL SCIENCES↗

Synthesizing land use and demographic change in Southeast Asia’s smaller urbanized areas from 2000–2015

The majority of the human population now reside in urban areas today. The United Nations estimates that nearly half of all urban dwellers currently live in cities smaller than 500 000 persons and the majority of future urban growth will take place in Asia and Africa, likely in these smaller urban areas, not mega cities. Thus, understanding the factors that influence urban demographic trajectories in small urban areas is critical to address sustainable and equitable policy initiatives related to food security, changing climate hazard exposure, and economic opportunities. Here we focus on Southeast Asia—a region historically characterized by lower urban population proportions, yet with a rapidly shifting dynamic demographic—to examine correlates of demographic change among smaller cities. We combine two open-source satellite-informed datasets: GHS urban center database (2015) and age-sex gridded data from WorldPop to calculate socio-demographic characteristics to model drivers of change in annualized urban population growth from 2000–2015 for 505 urbanized places. We find a general pattern of decreasing dependency ratios as city-size increases for most urban areas in Southeast Asia. Higher rates of growth and more variation is observed for smaller cities—those with fewer than 300 000 persons, the lowest population limit for UN data on urbanization. When examining covariates of urban population growth, we find significant statistical associations of population change in smaller urbanized areas with climatic, economic, and land cover/land use variables, but with country-specific variations. Characterizing a continuum of urban population development in the context of changing environmental, economic and climate conditions has been an important sustainable development and equity issue for decades, but newer analysis of city-level drivers allows for systematic inquiry thus moving beyond total population counts for policy-relevant insight.

Southeast Asia synthesis↗