DOE OSTI · 3008947
A Performant, Scalable Processing Pipeline for High‐Quality and FAIR Environmental Sensor Data
Abstract
High-resolution environmental monitoring is necessary to record, understand, and predict biogeochemical and ecological changes particularly in coastal systems but brings significant challenges in processing and making rapidly available the resulting data. The COMPASS-FME project established a network of coastal observational sites across the Chesapeake Bay and western Lake Erie regions extensively instrumented with soil, vegetation, and weather sensors logging data every 15 min. Our data processing framework, written in R and completely open source, prioritizes rapid model-experiment iteration and makes biogeochemical data rapidly available for quality assurance/quality control, analysis, and model ingestion. This pipeline is distinguished by a standardized and modular approach to data curation, extensive metadata and documentation, and its high performance. These attributes combine to make biogeochemical data rapidly accessible across COMPASS-FME and the broader community. Flexible, powerful, and reproducible approaches to handling high-volume environmental data are crucial for accelerating biogeosciences research.
Keep this discovery
Explore connections, maps & timelines
Pennington, Stephanie C. [Pacific Northwest National Laboratory (PNNL), College Park, MD (United States). Joint Global Change Research Institute] (ORCID:0000000326851092), Bond‐Lamberty, Ben [Pacific Northwest National Laboratory (PNNL), College Park, MD (United States). Joint Global Change Research Institute] (ORCID:0000000195254633), Peixoto, Roberta Bittencourt [Univ. of Toledo, OH (United States)], Chen, Xingyuan [Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)] (ORCID:0000000319285555), Cheng, Selina L. [Smithsonian Environmental Research Center, Edgewater, MD (United States)] (ORCID:0000000191048541), Machado‐Silva, Fausto [Univ. of Toledo, OH (United States)] (ORCID:0000000221653284), Maier, Kurt [Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)] (ORCID:0000000219337135), Phillips, Evan [Smithsonian Environmental Research Center, Edgewater, MD (United States)], Regier, Peter J. [Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)] (ORCID:0000000283751314), Stearns, Alice [Smithsonian Environmental Research Center, Edgewater, MD (United States)] (ORCID:0009000576283419), Ward, Nicholas D. [Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)] (ORCID:0000000161745581), Weintraub, Michael N. [Univ. of Toledo, OH (United States); Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)] (ORCID:0000000296232855), Wilson, Stephanie J. [Smithsonian Environmental Research Center, Edgewater, MD (United States)] (ORCID:0000000254840748), Bailey, Vanessa [Univ. of Toledo, OH (United States); Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)] (ORCID:0000000222488890), Rich, Roy [Smithsonian Environmental Research Center, Edgewater, MD (United States)] (ORCID:0000000204617388). 2025-11-08. A Performant, Scalable Processing Pipeline for High‐Quality and FAIR Environmental Sensor Data. https://doi.org/10.1029/2025jg008807
Cite the original work for its findings. Save a collection to share your selection of sources.