ChemReasoner - Catalyst Discovery via Large Language Model-driven Reasoning( 32832-E)
ChemReasoner - Catalyst Discovery via Large Language Model-driven Reasoning
Choudhury, Sutanay [Pacific Northwest National Lab↗
Engineering topics
Ji, Heng
Publications and source records attributed to Ji, Heng.
SCIMON: Scientific Inspiration Machines Optimized for Novelty
We explore and enhance the ability of neu- ral language models to generate novel scien- tific directions grounded in literature. Work on literature-based hypothesis generation has traditionally focused on binary link prediction— severely limiting the expressivity of hypothe- ses. This line of work also does not focus on optimizing novelty. We take a dramatic depar- ture with a novel setting in which models use as input background contexts (e.g., problems, experimental settings, goals), and output natu- ral language ideas grounded in literature. We present SCIMON, a modeling framework that uses retrieval of “inspirations” from past scien- tific papers, and explicitly optimizes for novelty by iteratively comparing to prior papers and up- dating idea suggestions until sufficient novelty is achieved. Comprehensive evaluations reveal that GPT-4 tends to generate ideas with over- all low technical depth and novelty, while our methods partially mitigate this issue. Our work represents a first step toward evaluating and developing language models that generate new ideas derived from the scientific literature.
Ji, Heng↗
pnnl/chemreasoner
ChemReasoner - Catalyst Discovery via Large Language Model-driven Reasoning
Choudhury, Sutanay↗
Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Design
Discovering novel catalysts requires complex reasoning involving multiple chemical properties and resultant trade-offs, leading to a combinatorial growth in the search space. While large language models (LLM) have demonstrated novel capabilities for chemistry through complex instruction following capabilities and high quality reasoning, a goal-driven combinatorial search using LLMs has not been explored in detail. In this work, we present a Monte Carlo Tree Search-based approach that improves beyond state-of-the-art chain-of-thought prompting variants to augment scientific reasoning. We introduce two new reasoning datasets: 1) a curation of computational chemistry simulations, and 2) diverse questions written by catalysis researchers for reasoning about novel chemical conversion processes. We improve over the best baseline by 25.8\% and find that our approach can augment scientist's reasoning and discovery process with novel insights.\footnote{All resources will be publicly available upon publication.
Sprueill, Henry W.↗
KEBLM: Knowledge-Enhanced Biomedical Language Models
Pretrained language models (PLMs) have demonstrated strong performance on many natural language processing (NLP) tasks. Despite their great success, these PLMs are typically pretrained only on unstructured free texts without leveraging existing structured knowledge bases that are readily available for many domains, especially scientific domains. As a result, these PLMs may not achieve satisfactory performance on knowledge-intensive tasks such as biomedical NLP. Comprehending a complex biomedical document without domain-specific knowledge is challenging, even for humans. Inspired by this observation, we propose a general framework for incorporating various types of domain knowledge from multiple sources into biomedical PLMs. We encode domain knowledge using lightweight adapter modules, bottleneck feed-forward networks that are inserted into different locations of a backbone PLM. For each knowledge source of interest, we pretrain an adapter module to capture the knowledge in a self-supervised way. We design a wide range of self-supervised objectives to accommodate diverse types of knowledge, ranging from entity relations to description sentences. Once a set of pretrained adapters is available, we employ fusion layers to combine the knowledge encoded within these adapters for downstream tasks. Each fusion layer is a parameterized mixer of the available trained adapters that can identify and activate the most useful adapters for a given input. Our method diverges from prior work by including a knowledge consolidation phase, during which we teach the fusion layers to effectively combine knowledge from both the original PLM and newly-acquired external knowledge using a large collection of unannotated texts. After the consolidation phase, the complete knowledge-enhanced model can be fine-tuned for any downstream task of interest to achieve optimal performance. Extensive experiments on many biomedical NLP datasets show that our proposed framework consistently improves the performance of the underlying PLMs on various downstream tasks such as natural language inference, question answering, and entity linking. These results demonstrate the benefits of using multiple sources of external knowledge to enhance PLMs and the effectiveness of the framework for incorporating knowledge into PLMs. Finally, while primarily focused on the biomedical domain in this work, our framework is highly adaptable and can be easily applied to other domains, such as the bioenergy sector.
96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗