SEARCH · Search NASA
Results for “Google Engine”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Dataset: "Widespread Drought-driven Declines in Streamflows and Water quality in the Upper Colorado River Basin (1998-2022)"
This data package contains the associated data and scripts for Nagamoto, E., Ombadi, M., Ciulla, F. et al. Widespread drought-driven declines in streamflows and water quality in the Upper Colorado River Basin during 1998-2022. Commun Earth Environ 7, 734 (2026). https://doi.org/10.1038/s43247-026-03890-5. This purpose of this study was to investigate the impact of the 21st century drought on water quantity and quality at catchments throughout the Upper Colorado River Basin (UCRB). We used stream flow, water temperature, specific conductance, air temperature, precipitation, and catchment attribute data for over 200 sites in the UCRB, collected from the National Water Information System using Basin3D (Varadharajan, 2023), GAGESII (Falcone, 2010), and the Google Earth Engine. We identified years of severe drought between 1998 and 2022 using the Standardized Precipitation Evaporation Index (SPEI), then calculated the relative change percentage of the stream flow, water temperature, and specific conductance from drought versus non-drought years. We used the attribute information from GAGESII to investigate what physical traits of catchments are associated streamflow vulnerability (greater relative change) or resilience to drought. We used land cover data from the National Land Cover Database (USGS, 2024) to assess any changes to physical attributes that may not be represented in the static attributes information in GAGESII. To increase data availability, we modeled stream temperature using methods from Willard, 2023. While the study period is water years 1998 to 2022, the raw water quantity and quality data extends to 1950 and the meteorological data extends to 1980. The data and code can be downloaded via the UCRB_drought.zip. Within the zip, the files are organized as follows: - INPUTS: Contains all input data used in UCRB_Drought_Workflow.ipynb - OUTPUTS: Contains all intermediate data created from UCRB_Drought_Workflow.ipynb as well as final products including the calculated Standardized Evapotranspiration Index (SPEI) - climatic_variables: The code used to collect meteorologic data from Google Earth Engine - feature_importance: The code used for the catchment attributes analysis - preprocessing: Code used in UCRB_Drought_Workflow_Preprocessing.ipynb - pyeto: Code used in UCRB_Drought_Workflow_Preprocessing.ipynb - calculations: Code used in UCRB_Drought_Workflow_Impacts.ipynb - plotting: Code used in UCRB_Drought_Workflow_Impacts.ipynb - README.md - UCRB_Drought_Workflow_Preprocessing.ipynb: The code used to prep raw data for the analysis - UCRB_Drought_Workflow_Impact.ipynb: The code which uses the prepped raw data for analysis, and plots all figures - requirements_ucrb-drought_v2.yml: The requirements file to create a virtual environment and Jupyter Lab kernel to run the code The INPUTS folder is organized into the following major directories and sub-directories. The "RDC_WT_SC_RAW" folder contains raw data for streamflow, water temperature, and specific conductance in a ".h5" file. The "NLCD_RAW" folder contains ".csv" files with annual land cover percentages for counties within the UCRB. The "MET_RAW" folder contains a ".csv" file with monthly meteorological data (air temperature and precipitation) for the sites in the UCRB which was obtained from code in the climatic_variables folder. The "GAGESII" folder contains ".csv" files with physical catchment attribute variables for catchments across the country. The "WT_LSTM_data" folder contains ".csv" files with calculated WT (Willard, 2023) and the associated RMSEs. The "Upper_Colorado_River_Basin_Boundary" folder contains geographic data including a shapefile for plotting in the UCRB_Drought_Workflow.ipynb. The "RESERVOIRS_RAW" folder contains ".csv" files for each reservoir in the UCRB with daily reservoir storage. There are also two files in the INPUTS folder that have combined reservoir storage data and reservoir metadata. The OUTPUTS folder is organized into the following major directories and sub-directories. The "RDC_WT_SC_data" folder contains a folder "Water_year" with the associated cleaned data, metadata, and data availability information in ".csv" files, a folder "Median_Relchange" with the relative change comparing drought to non-drought years in ".csv" files, and a folder "Peak95_Min5_Relchange" that has ".csv" files for the relative change in peak (95th %) and minimum (5th %) variables. The "NLCD_data" folder contains the difference in land cover from the beginning to end of the study period and the percentage of the county that is within UCRB bounds can be found in Nagamoto et al (2025)). The "MET_data" folder contains separated monthly air temperature and precipitation data and the calculated PET in ".csv" files. The "SPEI_data" folder contains ".csv" files with calculated SPEI values (one restricted to the study period and the other with information from the entire MET data period). The "Paper_Tables" folder contains two ".csv" files containing site information and data availability and information about the GAGESII trait aggregated categories. The base directory includes the file “flmd.csv” for a list and description of all files and the file “dd.csv” for data dictionaries. Scripts for preprocessing, analysis, and figure generation are located in the associated GitHub repository found at [https://github.com/iNAIADS/drought-impacts/tree/develop/UCRB-drought]. UPDATE 1: Title and code file updated to match submitted manuscript 10-15-2025. UPDATE 2: Code and data files updated to match revised manuscript 3-4-2026. UPDATE 3: Code and data files updated to match revised manuscript 6-7-2026. ** NOTE: DD and FLMD have not been updated yet. UPDATE 4: Added associated Manuscript information and DD and FLMD have been updated. To cite this code, please use the following BibTeX: @misc{nagamoto2025drought, author = {Emily Nagamoto and Fabio Ciulla and Mohammad Ombadi and Jared Willard and Rosemary Carroll and Charuleka Varadharajan}, title = {Dataset: "Widespread Drought-driven Declines in Streamflows and Water quality in the Upper Colorado River Basin (1998-2022)"}, year = {2025}, doi = {10.15485/2551894}, publisher = {ESS-DIVE Repository}, url = {https://data.ess-dive.lbl.gov/datasets/doi:10.15485/2551894} }
2024 Workshop - Remote Sensing and Fluxes Upscaling for Real-world Impact - Tutorial v1
The software-tutorial was developed within the 2024 Remote Sensing and Fluxes Upscaling for Real-world Impact workshop as part of the hands-on session. The workshop was supported by AmeriFlux, National Ecological Observatory Network (NEON) and CarbonDew. The software provides basic tools to perform the following tasks: - gather remote sensing images using Google Earth Engine API; - gather flux data; - perform basic functions, such as plotting time-series, perform QA of the data, compute vegetation indices; - perform correlation analysis between flux data and remote sensing data; - perform flux predictions based on remote sensing data integrated in different modalities.
Daily, 30 m Resolution NDSI Data for the East River Watershed, CO for 2000-2020
This dataset contains daily Normalized Difference Snow Index (NDSI) values at 30 m spatial resolution for the East River watershed in Colorado, USA. The temporal range of these data includes water years 2001-2020. These data were created using the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM). This model fuses low spatial and high temporal resolution data from MODIS (500 m, daily) with high spatial and low temporal resolution data from Landsat (30 m, 16 days) to create a 30m synthetic daily snow product. This product allows for the analysis of historical snow covered area trends in the East River Watershed at fine spatiotemporal resolutions where it was not available previously. This research was performed as a part of the Department of Energy’s Subsurface Biogeochemical Research Program with the primary intent of better understanding the timing and spatial patterns of water delivery to the Critical Zone in mountain watersheds. Each .zip file contains one "water year" of data (October 1 - September 30; i.e., water year 2010 starts October 1, 2010 and ends September 30, 2011). Each zip file contains the following: STARFM daily Normalized Difference Snow Index (NDSI) fusion data files in GeoTiff format with one layer for each day between Landsat data acquisition dates (i.e., for dates of Landsat acquisition, the Landsat image is included for that date). The study area is located in an area of Landsat path overlap, so Landsat dates acquisitions are every 7-9 days. Landsat NDSI files containing the high spatial (30m), low temporal (7-9 days due to Landsat path overlap) resolution data used as input to STARFM in GeoTiff format with one layer for each day. Dates for which no Landsat data were obtained are included as NoData layers. MODIS NDSI files containing the high temporal (daily), low spatial (500m) resolution data used as input to STARFM in GeoTiff format with one layer for each day. Please note the MODIS data were resampled to 30m pixels for input into the STARFM model. The data have a scale factor of 10,000 and a no data value of -32767. The projection of all datasets is WGS 84 (EPSG: 4326), which has a latitude/longitude based degree resolution of 0.0002694946 X 0.0002694946, and approximates to the 30 m spatial resolution mentioned above. The Layer Index files in .csv format. They contain information for each layer in the above GeoTiff files regarding the corresponding date for each layer, the fraction of pixels in the image that contain valid data (missing data is due to either cloud cover or poor data quality; these values are not percent snow cover). Dates of Landsat overpass are indicated in these files. If no Landsat data were able to be obtained due to cloud cover or lack of Landsat Tier 1 data available on Google Earth Engine, this is also noted.
Satellite Embedding-Based Population Imputation for Areas with Missing Building Footprint Data: A Computer Vision-Based Approach
High-resolution population modeling is important for supporting effective decision-making across diverse sectors. LandScan Mosaic generates population estimates at the level of individual buildings and aggregates them to 3 arc-second grids, and this approach performs well in regions where building footprint data are comprehensive and reliable. However, large portions of the globe still suffer from incomplete, sparse, or entirely missing building stock datasets, creating a structural limitation for strictly building-based population models. To address this research gap, this study proposes a computer vision-based framework that employs Google Earth Engine satellite embeddings and UNet, which allows us to directly impute grid-level population estimates in building-data-deficient areas. Applied to Taiwan as a case study, the framework achieved strong predictive performance with R$^{2}$ of 0.89, RMSE of 18.70, and MAE of 8.41, outperforming traditional machine learning approaches. Notably, the proposed framework effectively addressed building false-positive errors inherent in Global Human Settlement Layer (GHSL) data, correctly identifying uninhabited areas that were erroneously classified as populated. The framework also offers significant advantages for global population mapping, particularly in terms of scalability and temporal consistency, thereby extending the coverage and accuracy of high-resolution population products in data-scarce regions worldwide. Urban planners, decision makers, and related stakeholders can obtain granular population distributions to support more accurate and targeted infrastructure investment, service delivery, resource allocation, and risk assessment decisions.
Flood Susceptibility Mapping Using Machine Learning and Geospatial-Sentinel-1 SAR Integration for Enhanced Early Warning Systems
This study presents a comprehensive framework for flood susceptibility mapping by integrating geospatial factors with both statistical and machine learning models. Thirteen Flood-related factors, including DEM, slope, TWI, NDVI, etc., are extracted as features of models, and historical flood data derived from Sentinel-1 SAR from 2018 to 2023 are used as the target variables of the models. These datasets are analyzed using a frequency-based statistical model and three machine learning models, including Random Forest, XGBoost, and CNN, to generate flood susceptibility maps. The performance of each model is evaluated through AUC; and SHAP scores are separately generated for Machine learning (ML) models to explain each feature contribution in the ML model. The generated susceptibility maps are validated by high-flood-risk locations monitored by flood sensors, BLE inundation models, and flood-prone areas suggested by the Local Community Task Force. The results indicate that the XGBoost model outperforms all other models, with an AUC of 0.92 and demonstrates the highest alignment with recommended high-flood-risk locations, while the frequency-based statistical model showed the weakest performance with an AUC of 0.65. SHAP value graphs highlight the elevation, slope, and TWI as the most influential features across all models. The susceptibility maps generated by the machine learning model show strong agreement with the BLE map and high-flood-risk areas identified by the local Community Task Force.
Interplay of Topography, Fire History, and Climate on Interior Alaska Boreal Forest Vegetation Dynamics in the 21st Century: A Landsat Time-Series Analysis
This study investigates vegetation dynamics in boreal forests of Interior Alaska, focusing on topography, fire history, and climate influences. The study area includes Bonanza Creek Experimental Forest (BCEF) and surrounding region, categorized by topography (upland, floodplain, lowland) and fire history. Using Mann–Kendall trend and Theil–Sen slope analyses on Landsat-derived spectral metrics: Normalized Difference Vegetation Index (NDVI) and Normalized Burn Ratio (NBR), we observed a shift from browning to greening trends, particularly in historically burned areas. The photosynthetic activity in burned upland converged with unburned areas ~30 years post-fire, coincident with a shift towards deciduous dominance during post-fire succession. Normalized Difference Moisture Index (NDMI) trends revealed a significant increase in vegetation moisture content across all topographies. We introduce Effective Seasonal Precipitation Index (ESPI), which combines prior-year annual precipitation with current-year spring snow depth. Its positive correlation with NDMI highlights its potential for monitoring vegetation moisture dynamics at the landscape scale. Furthermore, by correlating dendrochronology-based climate indices, we found strong correlation between NDMI and normalized Supplemental Precipitation Index (nSPI), across topographies. Overall, this research provides critical insights into how climate and fire influence interior boreal vegetation, highlighting the effects of increased precipitation, and topography on shaping differential vegetation responses across the landscape.
Spatially resolved polarization swings in the supermassive binary black hole candidate OJ 287 with first Event Horizon Telescope observations
We present the first Event Horizon Telescope 1.3 mm observations of the supermassive binary black hole candidate OJ 287. The observations achieved an unprecedented angular resolution of 18 μas and reveal significant structural and polarization variability over just five days, marking the shortest timescale on which such changes have been directly imaged in this source. The inner jet exhibits a twisted ridgeline structure, with features displaying apparent superluminal motions up to about 22 c. The linear polarization maps reveal three main polarized features whose electric-vector position angles (EVPAs) change substantially over the time span of our observations, including a component with a radial polarization consistent with being produced by a recollimation shock. Most notably, we directly resolved two innermost jet components whose EVPAs rotate in opposite directions. The faster component, moving at 2.4 ± 0.9 μas/day (17.4 ± 6.5 c), exhibits counterclockwise EVPA swings of roughly 3.7° per day, while the slower component, with a proper motion of 1.4 ± 0.3 μas/day (10.2 ± 2.2 c), rotates clockwise at approximately 2.5° per day. Previous studies inferred helical magnetic fields in AGN jets from time-resolved or integrated polarization variability but lacked the angular resolution to directly image this effect. Our results provide spatially resolved evidence that a helical magnetic field threads the jet’s collimation and acceleration zone, ruling out models based on the superposition of unresolved components. Our analysis suggests that propagating shocks interact with a Kelvin–Helmholtz plasma instability, illuminating different phases of the helical magnetic field and producing the observed polarization spatial and temporal variability. Moreover, our model naturally accounts for the more rapid polarization rotation observed in the faster moving component. Our model predicts even more rapid swings in polarization, which could be tested with future observations featuring a more densely sampled time coverage.
Locating the missing large-scale emission in the jet of M87* with short EHT baselines
In very-long baseline interferometric arrays, nearly co-located stations probe the largest scales and typically cannot resolve the observed source. In the absence of a large-scale structure, closure phases constructed with these stations are zero and, since they are independent of station-based errors, they can be used to probe data issues. Here, we show how these trivial closure phases become nonzero with a brightness distribution on smaller scales than their short baseline would suggest. When applied to sources that are made up of a bright compact and large-scale diffuse component, the trivial closure phases directly measure the centroid relative to the compact source and higher-order image moments. We present a technique to measure these image moments with minimal model assumptions and validate it on synthetic Event Horizon Telescope (EHT) data. We then apply this technique to 2017 and 2018 EHT observations of M87* and find a weak preference for extended emission in the direction of the large-scale jet. We also apply it to 2021 EHT data and measure the source centroid about 1 mas northwest of the compact ring, which is consistent with the jet observed at lower frequencies.
Probing jet base emission of M87* with the 2021 Event Horizon Telescope observations
We investigate the presence and spatial characteristics of the jet base emission in M87* at 230 GHz, enabled by the significantly enhanced (u,v) coverage in the 2021 Event Horizon Telescope (EHT) observations. The integration of the 12−m Kitt Peak Telescope (USA) and NOEMA (France) stations into the array introduces two critical intermediate-length baselines to SMT (USA) and IRAM 30−m (Spain), providing sensitivity to emission structures at spatial scales of ∼250 μas and ∼2500 μas (∼ 0.02 pc and ∼ 0.02 pc). Without these new baselines, previous EHT observations of the source in 2017 and 2018 lacked the capability to constrain emission on large scales, where a “missing flux” of order ∼1 Jy is expected to reside. To probe these scales, we analyzed closure phases–robust against station-based gain calibration errors–and model the jet base emission using a simple Gaussian component offset from the compact ring emission at spatial separations > 100 μas. Our analysis revealed a Gaussian feature centered at (ΔRA ≈ 320 μ as, ΔDec. ≈ 60 μ as), projected separation of ≈ 5500 AU, with an estimated flux density of only ∼60 mJy, implying that most of the missing flux identified in previous EHT studies had to originate from different, larger scales. Brighter emission at the relevant spatial scales is firmly ruled out, and the data do not favor more complex models. This component aligns with the inferred position of the large-scale jet and is therefore physically consistent with the emission of the jet base. While our findings point to detectable jet base emission at 230 GHz, the limited coverage provided by only two intermediate baselines limits our ability to robustly reconstruct its morphology. Consequently, we treated the recovered Gaussian as an upper limit on the jet base flux density. Future EHT observations with expanded intermediate baseline coverage will be essential to constrain the structure and nature of this component with higher precision.
Full-polarization millimeter wavelength variability of Sagittarius A * during the 2018 EHT campaign
Context. Sagittarius A* (Sgr A*), the supermassive black hole at the center of the Milky Way, provides a unique laboratory to study accretion dynamics and plasma processes near the event horizon. Aims. We investigated the variability and polarization properties of Sgr A* using ALMA observations during the 2018 Event Horizon Telescope campaign. Methods. We analyzed high-cadence full-polarization light curves from ALMA at millimeter wavelengths, performed time-series analysis, and investigated the temporal behavior during an X-ray flare observed by Chandra on 2018 April 24. The variability characteristics are compared with expectations from standard accretion flow models. Results. We find low variability in total intensity (σ/μ < 10%), but significantly higher variability in linear and circular polarization (∼30% and ∼50%, respectively). A time-series analysis reveals red-noise variability, with power spectral densities between −2 and −3 across all Stokes parameters. Polarized intensity shows stable intra-day timescales, while total intensity exhibits more variable timescales, suggesting distinct emission regions, with polarization likely arising from a coherent structure. On April 24, a statistically significant inter-band delay in polarized intensity coincides with a near-simultaneous X-ray and millimeter peak that deviates from the typical delayed flare scenario. This event also features enhanced millimeter variability and coherent polarization loop evolution. The observed simultaneity challenges standard models of transient synchrotron emission with cooling delays, favoring instead a scenario of continuous energy injection in an optically thin region. Conclusions. Our results offer new constraints on the physical mechanisms driving variability in Sgr A*, and provide key observational input for refining theoretical models of accretion and plasma behavior in the vicinity of supermassive black holes.
Dynamic in-context learning with conversational models for data extraction and materials property prediction
The advent of natural language processing and large language models (LLMs) has revolutionized the extraction of data from unstructured scholarly papers. However, ensuring data trustworthiness remains a significant challenge. In this paper, we introduce PropertyExtractor, an open-source tool that leverages advanced conversational LLMs such as Google gemini-pro and OpenAI gpt-4, blends zero-shot with few-shot in-context learning, and employs engineered prompts for the dynamic refinement of structured information hierarchies—enabling autonomous, efficient, scalable, and accurate identification, extraction, and verification of material property data. Our tests on material data demonstrate precision and recall that exceed 95% with an error rate of ∼9%, highlighting the effectiveness and versatility of the toolkit. Finally, databases for 2D material thicknesses, a critical parameter for device integration, and energy bandgap values are developed using PropertyExtractor. In particular, for the thickness database, the rapid evolution of the field has outpaced both experimental measurements and computational methods, creating a significant data gap. Our work addresses this gap and showcases the potential of PropertyExtractor as a reliable and efficient tool for the autonomous generation of various material property databases, advancing the field.
Traffic Control via Connected and Automated Vehicles (CAVs): An Open-Road Field Experiment with 100 CAVs
The CIRCLES project aims to reduce instabilities in traffic flow, which are naturally occurring phenomena due to human driving behavior. Also called “phantom jams” or “stop-and-go waves,” these instabilities are a significant source of wasted energy. Toward this goal, the CIRCLES project designed a control system, referred to as the MegaController by the CIRCLES team, that could be deployed in real traffic. Our field experiment, the MegaVanderTest (MVT), leveraged a heterogeneous fleet of 100 longitudinally controlled vehicles as Lagrangian traffic actuators, each of which ran a controller with the architecture described in this article. The MegaController is a hierarchical control architecture that consists of two main layers. The upper layer is called the Speed Planner and is a centralized optimal control algorithm. It assigns speed targets to the vehicles, conveyed through the LTE cellular network. The lower layer is a control layer, running on each vehicle. It performs local actuation by overriding the stock adaptive cruise controller, using the stock onboard sensors. The Speed Planner ingests live data feeds provided by third parties as well as data from our own control vehicles and uses both to perform the speed assignment. The architecture of the Speed Planner allows for the modular use of standard control techniques, such as optimal control, model predictive control (MPC), kernel methods, and others. The architecture of the local controller allows for the flexible implementation of local controllers. Corresponding techniques include deep reinforcement learning (RL), MPC, and explicit controllers. Depending on the vehicle architecture, all onboard sensing data can be accessed by the local controllers or only some. Likewise, control inputs vary across different automakers, with inputs ranging from torque or acceleration requests for some cars to electronic selection of adaptive cruise control (ACC) setpoints in others. The proposed architecture technically allows for the combination of all possible settings proposed previously, that is {Speed Planner algorithms} × {local Vehicle Controller algorithms} × {full or partial sensing} × {torque or speed control}. As a result, most configurations were tested throughout the ramp up to the MegaVandertest (MVT).
Hierarchical Speed Planner for Automated Vehicles: A Framework for Lagrangian Variable Speed Limit in Mixed-Autonomy Traffic
Here, this article presents a novel hierarchical speed planning framework for variable speed limits in mixed-autonomy traffic environments, leveraging server-side macroscopic control and vehicle-side microscopic execution. The framework integrates real-time traffic state estimation (TSE) and reinforcement learning (RL)-based control to mitigate congestion and improve traffic flow. A TSE enhancement module combines macroscopic data from sources like INRIX with high-resolution observations from connected autonomous vehicles (CAVs), enabling predictive modeling to address latency and noise. The target speed design module employs kernel smoothing and a buffer zone strategy to optimize traffic density and flow around bottlenecks. The proposed system was validated in the largest open-road test to date with 100 CAVs, demonstrating an overall 8% traffic density decrease, with a specific decrease of 7% upstream, 10% downstream, and a 52% decrease during the congestion formation phase at bottlenecks.
Introduction to Digital Image Correlation (DIC) with annotated bibliography
Digital Image Correlation (DIC) is “a non-contact means of measuring motion and deformation using digital images of the object of interest”, capable of full-field measurements over large areas. Getting started in DIC can be difficult, since there are many different techniques, and the body of literature on DIC is extensive. IEEE Xplore alone lists nearly 11,000 references, including magazines, conference proceedings, journal articles, books, etc. A Google search returns around 145 million results. The intent of this annotated bibliography is to provide an entry point for beginners and to collect references that may be useful to more advanced practitioners.
Machine Learning Aided Modeling of Granular Materials: A Review
Artificial intelligence (AI) has become a buzzy word since Google’s AlphaGo beat a world champion in 2017. In the past five years, machine learning as a subset of the broader category of AI has obtained considerable attention in the research community of granular materials. This work offers a detailed review of the recent advances in machine learning-aided studies of granular materials from the particle-particle interaction at the grain level to the macroscopic simulations of granular flow. This work will start with the application of machine learning in the microscopic particle-particle interaction and associated contact models. Then, different neural networks for learning the constitutive behaviour of granular materials will be reviewed and compared. Finally, the macroscopic simulations of practical engineering or boundary value problems based on the combination of neural networks and numerical methods are discussed. We hope readers will have a clear idea of the development of machine learning-aided modelling of granular materials via this comprehensive review work.
Survey of Deep Learning and Physics-Based Approaches in Computational Wave Imaging
Computational wave imaging (CWI) extracts hidden structure and physical properties of a volume of material by analyzing wave signals that traverse that volume. Applications include seismic exploration of the Earth’s subsurface, acoustic imaging and nondestructive testing (NDT) in material science, and ultrasound computed tomography (USCT) in medicine. Current approaches for solving CWI problems can be divided into two categories: those rooted in traditional physics and those based on deep learning. Physics-based methods stand out for their ability to provide high-resolution and quantitatively accurate estimates of acoustic properties within the medium. However, they can be computationally intensive and are susceptible to ill-posedness and nonconvexity typical of CWI problems. Machine learning (ML)-based computational methods have recently emerged, offering a different perspective to address these challenges. Diverse scientific communities have independently pursued the integration of deep learning in CWI. This review discusses how contemporary scientific ML techniques, and deep neural networks in particular, have been developed to enhance and integrate with traditional physics-based methods for solving CWI problems. We present a structured framework that consolidates existing research spanning multiple domains, including computational imaging, wave physics, and data science. This study concludes with important lessons learned from existing ML-based methods and identifies technical hurdles and emerging trends through a systematic analysis of the extensive literature on this topic.
Supporting ARPA-E Power Grid Optimization (Final Report)
Pacific Northwest National Laboratory (PNNL), Arizona State University (ASU), Georgia Institute of Technology (Georgia Tech), Los Alamos National Laboratory (LANL), National Renewable Energy Laboratory (NREL), Texas A&M University (TAMU), The University of Texas at Austin (UT), and the University of Wisconsin-Madison (UW-M) supported the ARPA-E Grid Optimization (GO) Competition by providing a common problem formulation, data format, datasets, evaluation mechanism, scoring, rules, and results that resulted in the awarding of $\$9.24$ million dollars to teams from academia, industry, and national labs for solving three sets of increasingly difficult non-linear, security- constrained AC Optimal Powerflow (AC-OPF) optimization problems in order to increase the efficiency of the US Electric Grid. It is estimated that a 1% increase in efficiency can save $\$1$ billion. Current industry practices typically use a linear DC model (DC-OPF) in order solve the OPF problem within the time constraints of the operation schedule. The GO Competition challenges the best power engineers, mathematicians, and computer scientists to make possible operational decisions based on accurate physical models. To accomplish this, the GO Competition created a series of Challenges and funded teams to produce the best solver. Challenge 1 was to solve the security constrained Alternating Current Optimal Power Flow (ACOPF) problem. Challenge 2 extended that to by adding adjustable transformer tap ratios, phase shifting transformers, switchable shunts, price-responsive demand, ramp rate constrained generators and loads, and fast-start unit commitment (UC). Furthermore, Challenge 2 was a maximization problem while Challenge 1 was a minimization problem. While Challenge 3 was being developed, the entrants were invited to find better solutions to the Challenge 2 synthetic datasets with no restrictions on time, hardware, or algorithms. The Challenge 2 solutions turned out to be very good. Challenge 3 expanded the Challenge 2 problem further by using multiperiod dynamic markets, including advisory models for extreme weather events, day-ahead markets, and the real-time markets with an extended look-ahead. These problems included active bid-in demand and topology optimization. Together the Challenges used nearly 30 million CPU hours. Since each team was working on the same problem, using the same data, and running on the same hardware, fair comparisons could be drawn as to the best solver. The datasets were varied enough, however, that the best solver for one dataset was not necessarily the best at another, so cumulative scores were used. The process was managed by the PNNL maintained website https://GOCompetition.energy.gov, where Entrants could find information about the problem, the data, the rules, submit their solver for evaluation, and see the scores of all the competing teams on a Leaderboard. Interest was world-wide but only American teams were eligible for prizes. The Competition has produced 34 journal articles 115 papers and been cited over 500 times in the literature, including 12 dissertations (4 from foreign countries; Columbia (2), Germany, and Italy) and 3 from the DOE ExaScale project. Software developed by Pearl Street Technologies for Challenges 1 and 2 is now deployed by Southwest Power Pool (SPP) and Midcontinent Independent Service Operator (MISO). Other teams have received inquiries from venture capitalists. Google DeepMind has thanked the Competition for making the datasets developed for the Competition public. They are using it to train machine learning models. The larger datasets have billions of unknowns to be solved for, but only a small percent matter in the final solution. Knowing what unknowns are important can dramatically speedup the solution.