Eliciting Experts' Judgments about Uncertain Probabilities: A Brief Primer to Using Expert Elicitation Well.
Abstract not provided.
Butler, Karin↗
SEARCH · Search NASA
Results for “expert”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding
Decoding visual experiences from fMRI offers a powerful avenue to understand human perception and develop advanced brain-computer interfaces. However, current progress often prioritizes maximizing reconstruction fidelity while overlooking interpretability, an essential aspect for deriving neuroscientific insight. To address this gap, we propose MoRE-Brain, a neuro-inspired framework designed for high-fidelity, adaptable, and interpretable visual reconstruction. MoRE-Brain uniquely employs a hierarchical Mixture-of-Experts architecture where distinct experts process fMRI signals from functionally related voxel groups, mimicking specialized brain networks. The experts are first trained to encode fMRI into the frozen CLIP space. A finetuned diffusion model then synthesizes images, guided by expert outputs through a novel dual-stage routing mechanism that dynamically weighs expert contributions across the diffusion process. MoRE-Brain offers three main advancements: First, it introduces a novel Mixture-of-Experts architecture grounded in brain network principles for neuro-decoding. Second, it achieves efficient cross-subject generalization by sharing core expert networks while adapting only subject-specific routers. Third, it provides enhanced mechanistic insight, as the explicit routing reveals precisely how different modeled brain regions shape the semantic and spatial attributes of the reconstructed image. Extensive experiments validate MoRE-Brain’s high reconstruction fidelity, with bottleneck analyses further demonstrating its effective utilization of fMRI signals, distinguishing genuine neural decoding from over-reliance on generative priors. Consequently, MoRE-Brain marks a substantial advance towards more generalizable and interpretable fMRI-based visual decoding.
Wei, Yuxiang [Georgia Institute of Technology]↗
Operational experience with the new CMS DAQ-Expert
The data acquisition (DAQ) system of the Compact Muon Solenoid (CMS) at CERN reads out the detector at the level-1 trigger accept rate of 100 kHz, assembles events with a bandwidth of 200 GB/s, provides these events to the high level-trigger running on a farm of about 30k cores and records the accepted events. Comprising custom-built and cutting edge commercial hardware and several 1000 instances of software applications, the DAQ system is complex in itself and failures cannot be completely excluded. Moreover, problems in the readout of the detectors,in the first level trigger system or in the high level trigger may provoke anomalous behaviour of the DAQ systemwhich sometimes cannot easily be differentiated from a problem in the DAQ system itself. In order to achieve high data taking efficiency with operators from the entire collaboration and without relying too heavily on the on-call experts, an expert system, the DAQ-Expert, has been developed that can pinpoint the source of most failures and give advice to the shift crew on how to recover in the quickest way. The DAQ-Expert constantly analyzes monitoring data from the DAQ system and the high level trigger by making use of logic modules written in Java that encapsulate the expert knowledge about potential operational problems. The results of the reasoning are presented to the operator in a web-based dashboard, may trigger sound alerts in the control room and are archived for post-mortem analysis - presented in a web-based timeline browser. We present the design of the DAQ-Expert and report on the operational experience since 2017, when it was first put into production.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
Expert evaluation of LLM world models: A high-T c superconductivity case study
Large Language Models (LLMs) show great promise as a powerful tool for scientific literature exploration. However, their effectiveness in providing scientifically accurate and comprehensive answers to complex questions within specialized domains remains an active area of research. Using the field of high-temperature cuprates as an exemplar, we evaluate the ability of LLM systems to understand the literature at the level of an expert. We construct an expert-curated database of 1,726 scientific papers that covers the history of the field, and a set of 67 expert-formulated questions that probe deep understanding of the literature. We then evaluate six different LLM-based systems for answering these questions, including both commercially available closed models and a custom retrieval-augmented generation (RAG) system capable of retrieving images alongside text. Experts then evaluate the answers of these systems against a rubric that assesses balanced perspectives, factual comprehensiveness, succinctness, and evidentiary support. Among the six systems, two using RAG on curated literature outperformed existing closed models across key metrics, particularly in providing comprehensive and well-supported answers. We discuss promising aspects of LLM performances as well as critical short-comings of all the models. The set of expert-formulated questions and the rubric will be valuable for assessing expert level performance of LLM based reasoning systems.
36 MATERIALS SCIENCE↗
Reading Between the Lines: Measuring the Effects of Linguistic-Based Indicators of Deception on Experts’ Identification and Categorization of Disinformation
There is currently very limited research into how experts analyze and assess potentially fraudulent content in their expertise areas, and most research within the disinformation space involves very limited text samples (e.g., news headlines). The overarching goal of the present study was to explore how an individual’s psychological profile and the linguistic features in text might influence an expert’s ability to discern disinformation/fraudulent content in academic journal articles. At a high level, the current design tasked experts with reading journal articles from their area of expertise and indicating if they thought an article was deceptive or not. Half the articles they read were journal papers that had been retracted due to academic fraud. Demographic and psychological inventory data collected on the participants was combined with performance data to generate insights about individual expert susceptibility to deception. Our data show that our population of experts were unable to reliably detect deception in formal technical writing. Several psychological dimensions such as comfort with uncertainty and intellectual humility may provide some protection against deception. This work informs our understanding of expert susceptibility to potentially fraudulent content within official, technical information and can be used to inform future mitigative efforts and provide a building block for future disinformation work.
99 GENERAL AND MISCELLANEOUS↗
Expert perspectives on the wind plant of the future
Abstract Wind power technology has changed rapidly in recent years. Technology innovation, evolving power markets, and competing land and ocean uses continue to influence the design and operation of wind turbines and plants. Anticipating these trends and their impact on future facilities can inform commercial strategies and research priorities. Drawing from a recent survey of 140 of the world's foremost wind experts, we identify expectations of future wind plant design in 2035, both for onshore and offshore wind. Experts anticipate continued growth in turbine size, to 5.5 (onshore) and 17 MW (offshore), with plants located in increasingly less favorable wind and siting regimes. They expect plant sizes of 1,100 MW for fixed‐bottom and 600 MW for floating offshore wind. Experts forecast enhanced grid‐system value from wind through significant to widespread use of larger rotors, hybrid projects with batteries and hydrogen production, and more. To explain experts' perspectives on future plant design and operation, we identify five mechanisms: economies of unit, plant, and resource scale; grid‐system value economies; and production efficiencies. We characterize learning effects as a moderating influence on the strength of these mechanisms. In combination, experts predict that these design choices support levelized cost of energy reductions of 27% (onshore) and 17%–35% (floating and fixed‐bottom offshore) by 2035 compared to today, while enhancing wind energy's grid service offerings. Our findings provide a much‐needed benchmark for representing future wind technologies in power sector models and address a critical research gap by explaining the economics behind wind energy design choices.
17 WIND ENERGY↗
Efficient and assured reinforcement learning-based building HVAC control with heterogeneous expert-guided training
Abstract Building heating, ventilation, and air conditioning (HVAC) systems account for nearly half of building energy consumption and $$20\%$$ of total energy consumption in the US. Their operation is also crucial for ensuring the physical and mental health of building occupants. Compared with traditional model-based HVAC control methods, the recent model-free deep reinforcement learning (DRL) based methods have shown good performance while do not require the development of detailed and costly physical models. However, these model-free DRL approaches often suffer from long training time to reach a good performance, which is a major obstacle for their practical deployment. In this work, we present a systematic approach to accelerate online reinforcement learning for HVAC control by taking full advantage of the knowledge from domain experts in various forms . Specifically, the algorithm stages include learning expert functions from existing abstract physical models and from historical data via offline reinforcement learning, integrating the expert functions with rule-based guidelines, conducting training guided by the integrated expert function and performing policy initialization from distilled expert function. Moreover, to ensure that the learned DRL-based HVAC controller can effectively keep room temperature within the comfortable range for occupants, we design a runtime shielding framework to reduce the temperature violation rate and incorporate the learned controller into it. Experimental results demonstrate up to 8.8 X speedup in DRL training from our approach over previous methods, with low temperature violation rate.
Xu, Shichao↗
Developing expert scientific consensus on the environmental and societal effects of marine artificial structures prior to decommissioning
Thousands of artificial (‘human-made’) structures are present in the marine environment, many at or approaching end-of-life and requiring urgent decisions regarding their decommissioning. No consensus has been reached on which decommissioning option(s) result in optimal environmental and societal outcomes, in part, owing to a paucity of evidence from real-world decommissioning case studies. To address this significant challenge, we asked a worldwide panel of scientists to provide their expert opinion. They were asked to identify and characterise the ecosystem effects of artificial structures in the sea, their causes and consequences, and to identify which, if any, should be retained following decommissioning. Experts considered that most of the pressures driving ecological and societal effects from marine artificial structures (MAS) were of medium severity, occur frequently, and are dependent on spatial scale with local-scale effects of greater magnitude than regional effects. The duration of many effects following decommissioning were considered to be relatively short, in the order of days. Overall, environmental effects of structures were considered marginally undesirable, while societal effects marginally desirable. Experts therefore indicated that any decision to leave MAS in place at end-of-life to be more beneficial to society than the natural environment. However, some individual environmental effects were considered desirable and worthy of retention, especially in certain geographic locations, where structures can support improved trophic linkages, increases in tourism, habitat provision, and population size, and provide stability in population dynamics. The expert analysis consensus that the effects of MAS are both negative and positive for the environment and society, gives no strong support for policy change whether removal or retention is favoured until further empirical evidence is available to justify change to the status quo. The combination of desirable and undesirable effects associated with MAS present a significant challenge for policy- and decision-makers in their justification to implement decommissioning options. Decisions may need to be decided on a case-by-case basis accounting for the trade-off in costs and benefits at a local level.
54 ENVIRONMENTAL SCIENCES↗
Efficient bifurcation and tabulation of multi-dimensional combustion manifolds using deep mixture of experts: An a priori study
This work describes and validates an approach for autonomously bifurcating turbulent combustion man-ifolds to divide regression tasks amongst specialized artificial neural networks (ANNs). This approach relies on the mixture of experts (MoE) framework, where each neural network is trained to be specialized in a given portion of the input space. The assignment of different input regions to the experts is determined by a gating network, which is a neural network classifier. In some previous studies, it has been demonstrated that bifurcation of a complex combustion manifold and fitting different ANNs for each part leads to better fits or faster inference speeds. However, the manner of bifurcation in these studies was based on heuristic approaches or clustering techniques. In contrast, the proposed technique enables automatic bifurcation using non-linear planes in high-dimensional turbulent combustion manifolds that are often associated with complex behavior due to different dominating physics in various zones. Furthermore, the proposed concept is validated using 4-dimensional (4D) and 5D flamelet tables, showing that the errors obtained with a given network size, or conversely the network size required to achieve a given accuracy, is considerably reduced. The effect of the number of experts on inference speed is also investigated, showing that by increasing the number of experts from 1 to 8, the inference time can be approximately reduced by a factor of two. Moreover, it is shown that the MoE approach divides the input manifold in a physically intuitive manner, suggesting that the MoE framework can elucidate high-dimensional datasets in a physically meaningful way.
42 ENGINEERING↗
Expert elicitation survey predicts 37% to 49% declines in wind energy costs by 2050
Wind energy has experienced accelerated cost reduction over the past five years—far greater than predicted in a 2015 expert elicitation. Here we report results from a new survey on wind costs, compare those with previous results and discuss the accuracy of the earlier predictions. In this work, we show that experts in 2020 expect future onshore and offshore wind costs to decline 37–49% by 2050, resulting in costs 50% lower than predicted in 2015. This is due to cost reductions witnessed over the past five years and expected continued advancements. If realized, these costs might allow wind to play a larger role in energy supply than previously anticipated. Considering both surveys, we also conclude that there is considerable uncertainty about future costs. Our results illustrate the importance of considering cost uncertainty, highlight the value and limits of using experts to reveal those uncertainties, and yield possible lessons for energy modellers and expert elicitation.
17 WIND ENERGY↗
Method for Generating Expert Derived Confidence Scores
We executed a pilot demonstration of a methodology for developing a new confidence metric to help operators calibrate their trust in ML event classifiers. This confidence metric was derived from domain expert judgment and was accompanied with a qualitative description describing the reason for each confidence rating. After learning the boundaries of an ML’s performance by studying a subset of events an SME rated his confidence in the ML’s ability to classify similar events and provided an explanation for his ratings. To demonstrate this methodology, we developed our expert driven confidence scores for the ML event classifier within the ESAMS. Next, we assessed the accuracy of the human expert confidence scores relative to the ML’s uncertainty quantification scores. This report includes a description of our methodology, summary of our findings and future directions.
97 MATHEMATICS AND COMPUTING↗
A Bayesian model for multivariate discrete data using spatial and expert information with application to inferring building attributes
When modeling sparsely observed multivariate data, strong prior information elicited from experts can be used to bolster predictive accuracy and counteract sampling bias. Similarly, modeling autocorrelation in space can help make use of co-occurrence patterns present in many types of spatial data. To make use of both expert prior information and spatial structure, we propose a novel graphical model for a spatial Bayesian network developed specifically to address challenges in inferring the attributes of buildings from geographically sparse observational data. This model is implemented as the sum of a spatial multivariate Gaussian random field and a tabular conditional probability function in real-valued space prior to projection onto the probability simplex. This modeling form is especially suitable for the usage of prior information in the form of sets of atomic rules obtained from experts. To perform inference with missing data, we implement a Markov chain Monte Carlo scheme composed of alternating steps of Gibbs sampling of missing entries and Hamiltonian Monte Carlo for model parameters. A case study in building attribution is presented to highlight the advantages and limitations of this approach.
97 MATHEMATICS AND COMPUTING↗
MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
Mixture of Experts (MoE) models have enabled the scaling of Large Language Models (LLMs) and Vision Language Models (VLMs) by achieving massive parameter counts while maintaining computational efficiency. However, MoEs introduce several inference-time challenges, including load imbalance across experts and the additional routing computational overhead. To address these challenges and fully harness the benefits of MoE, a systematic evaluation of hardware acceleration techniques is essential. We present MoE-Inference-Bench, a comprehensive study to evaluate MoE performance across diverse scenarios. We analyze the impact of batch size, sequence length, and critical MoE hyperparameters such as FFN dimensions and number of experts on throughput. We evaluate several optimization techniques on Nvidia H100 GPUs, including pruning, Fused MoE operations, speculative decoding, quantization, and various parallelization strategies. Our evaluation includes MoEs from the Mixtral, DeepSeek, OLMoE and Qwen families. The results reveal performance differences across configurations and provide insights for the efficient deployment of MoEs.
Chitty-Venkata, Krishna Teja↗
Expert Elicitation on Wind Farm Control
Wind farm control is an active and growing field of research in which the control actions of individual turbines in a farm are coordinated, accounting for inter-turbine aerodynamic interaction, to improve the overall performance of the wind farm and to reduce costs. The primary objectives of wind farm control include increasing power production, reducing turbine loads, and providing electricity grid support services. Additional objectives include improving reliability or reducing external impacts to the environment and communities. In 2019, a European research project (FarmConners) was started with the main goal of providing an overview of the state-of-the-art in wind farm control, identifying consensus of research findings, data sets, and best practices, providing a summary of the main research challenges, and establishing a roadmap on how to address these challenges. Complementary to the FarmConners project, an IEA Wind Topical Expert Meeting (TEM) and two rounds of surveys among experts were performed. From these events we can clearly identify an interest in more public validation campaigns. Additionally, a deeper understanding of the mechanical loads and the uncertainties concerning the effectiveness of wind farm control are considered two major research gaps.
17 WIND ENERGY↗
Subsea permafrost carbon stocks and climate change sensitivity estimated by expert assessment
The continental shelves of the Arctic Ocean and surrounding seas contain large stocks of organic matter (OM) and methane (CH4), representing a potential ecosystem feedback to climate change not included in international climate agreements. We performed a structured expert assessment with 25 permafrost researchers to combine quantitative estimates of the stocks and sensitivity of organic carbon in the subsea permafrost domain (i.e. unglaciated portions of the continental shelves exposed during the last glacial period). Experts estimated that the subsea permafrost domain contains ~560 gigatons carbon (GtC; 170–740, 90% confidence interval) in OM and 45 GtC (10–110) in CH 4 . Current fluxes of CH 4 and carbon dioxide (CO 2 ) to the water column were estimated at 18 (2–34) and 38 (13–110) megatons C yr –1 , respectively. Under Representative Concentration Pathway (RCP) RCP8.5, the subsea permafrost domain could release 43 Gt CO 2 – equivalent (CO 2 e) by 2100 (14–110) and 190 Gt CO 2 e by 2300 (45–590), with ~30% fewer emissions under RCP2.6. The range of uncertainty demonstrates a serious knowledge gap but provides initial estimates of the magnitude and timing of the subsea permafrost climate feedback.
54 ENVIRONMENTAL SCIENCES↗
Clean Energy to Communities: Expert Match
Apply to the U.S. Department of Energy's (DOE) Clean Energy to Communities - Expert Match program to receive 40 to 60 hours of analytical support from DOE national laboratories. Expert Match researchers bring years of experience working with community stakeholders to provide customized solutions for unique energy challenges. By combining innovative technology, state-of-the-art modeling, and the ability to test clean energy plans before installing them, we can help close the gap between clean energy ambitions and real-world deployment. National lab researchers are partnering with communities across the country to help address local energy challenges and goals. Some examples are Bealsville, Florida and Cohoes, New York.
Bealsville↗
An expert-driven literature review of “negative” chemicals for developmental neurotoxicity (DNT) in vitro assay evaluation
To date, approximately 200 chemicals have been tested in US Environmental Protection Agency (EPA) or Organization for Economic Co-operation and Development (OECD) developmental neurotoxicity (DNT) guideline studies, leaving thousands of chemicals without traditional animal information on DNT hazard potential. To address this data gap, a battery of in vitro DNT new approach methodologies (NAMs) has been proposed. Evaluation of the performance of this battery will increase the confidence in its use to determine DNT chemical hazards. One approach to evaluate DNT NAM performance is to use a set of chemicals to evaluate sensitivity and specificity. Since a list of chemicals with potential evidence of in vivo DNT has been established, this study aims to develop a curated list of “negative” chemicals for inclusion in a “DNT NAM evaluation set”. A workflow, including a literature search followed by an expert-driven literature review, was used to systematically screen 39 chemicals for lack of DNT effect. Expert panel members evaluated the scientific robustness of relevant studies to inform chemical categorizations. Following review, the panel discussed each chemical and made categorical determinations of “Favorable”, “Not Favorable”, or “Indeterminate” reflecting acceptance, lack of suitability, or uncertainty given specific limitations and considerations, respectively. Further, the panel determined that 10, 22, and 7 chemicals met the criteria for “Favorable”, “Not Favorable”, and “Indeterminate”, for use as negatives in a DNT NAM evaluation set. Ultimately, this approach not only supports DNT NAM performance evaluation but also highlights challenges in identifying large numbers of negative DNT chemicals.
59 BASIC BIOLOGICAL SCIENCES↗
Evaluating Climate Models’ Cloud Feedbacks Against Expert Judgment
Abstract The persistent and growing spread in effective climate sensitivity (ECS) across global climate models necessitates rigorous evaluation of their cloud feedbacks. Here we evaluate several cloud feedback components simulated in 19 climate models against benchmark values determined via an expert synthesis of observational, theoretical, and high‐resolution modeling studies. We find that models with smallest feedback errors relative to these benchmark values generally have moderate total cloud feedbacks (0.4–0.6 W m −2 K −1 ) and ECS (3–4 K). Those with largest errors generally have total cloud feedback and ECS values that are too large or too small. Models tend to achieve large positive total cloud feedbacks by having several cloud feedback components that are systematically biased high rather than by having a single anomalously large component, and vice versa. In general, better simulation of mean‐state cloud properties leads to stronger but not necessarily better cloud feedbacks. The Python code base provided herein could be applied to developmental versions of models to assess cloud feedbacks and cloud errors and place them in the context of other models and of expert judgment in real‐time during model development.
58 GEOSCIENCES↗