DOE OSTI · 3009798
What to Support When You’re Compressing
Abstract
Over the last nearly 20 years, lossy compression has become an essential aspect of HPC applications’ data pipelines, allowing them to overcome limitations in storage capacity and bandwidth and, in some cases, increase computational throughput and capacity. However, with the adoption of lossy compression comes the requirement to assess and control the impact lossy compression has on scientific outcomes. In this work, we take a major step forward in describing the state of practice and by characterizing workloads. We examine applications’ needs and compressors’ capabilities across 9 different supercomputing application domains. We present 24 takeaways that provide best practices for applications, operational impacts for facilities achieving compressed data, and gaps in application needs not addressed by production compressors that point towards opportunities for future compression research.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Cappello, Franck [Argonne National Laboratory (ANL), Argonne, IL (United States); Univ. of Chicago, Lemont, IL (United States)] (ORCID:0000000278903934), Underwood, Robert Raymond [Argonne National Laboratory (ANL), Argonne, IL (United States); Univ. of Chicago, Lemont, IL (United States)] (ORCID:000000021464729X), Alexeev, Yuri Antonovich [Argonne National Laboratory (ANL), Argonne, IL (United States)] (ORCID:0000000150662254), Baker, Alison [National Center for Atmospheric Research (NCAR), Boulder, CO (United States)] (ORCID:0000000324367838), Bozdağ, Ebru [Colorado School of Mines, Golden, CO (United States)] (ORCID:0000000242693533), Burtscher, Martin [Texas State Univ., San Marcos, TX (United States)] (ORCID:0000000177173354), Chard, Kyle [Univ. of Chicago, IL (United States); Argonne National Laboratory (ANL), Chicago, IL (United States)] (ORCID:0000000273704805), Di, Sheng [Argonne National Laboratory (ANL), Argonne, IL (United States); Univ. of Chicago, Lemont, IL (United States)] (ORCID:0000000273395256), Felker, Kyle Gerard [Argonne National Laboratory (ANL), Argonne, IL (United States)] (ORCID:000000023501482X), O'Grady, Paul Christopher [SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States)] (ORCID:0009000048348389), Guo, Hanqi [The Ohio State Univ., Columbus, OH (United States)] (ORCID:0000000177761834), Huang, Yafan [Univ. of Iowa, Iowa City, IA (United States)] (ORCID:0000000173706766), Jiang, Peng [Univ. of Iowa, Iowa City, IA (United States)] (ORCID:0000000177436062), Jin, Sian [Temple Univ., Philadelphia, PA (United States)] (ORCID:0009000992500611), Johansson, Petter [KTH Royal Inst. of Technology, Stockholm (Sweden)] (ORCID:0000000191602549), Li, Shaomeng [NVIDIA Corporation, Santa Clara, CA (United States)] (ORCID:0000000182952248), Liang, Xin [Univ. of Kentucky, Lexington, KY (United States)] (ORCID:0000000206301600), Lindahl, Erik [Stockholm Univ. (Sweden)] (ORCID:0000000227342794), Lindstrom, Peter [Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)] (ORCID:0000000338174199), Lukić, Zarija [Lawrence Livermore National Laboratory (LLNL), Berkeley, CA (United States)] (ORCID:0009000757307838), Lundborg, Magnus [KTH Royal Inst. of Technology, Solna (Sweden)] (ORCID:0000000208737854), Lykov, Danylo [NVIDIA Corporation, Santa Clara, CA (United States)] (ORCID:0000000229792633), Nagaso, Masaru [RIKEN Center for Computational Science, Kobe (Japan)] (ORCID:0000000235666174), Sato, Kento [RIKEN Center for Computational Science, Kobe (Japan)] (ORCID:0000000178502121), Singh, Amarjit [RIKEN Center for Computational Science, Kobe (Japan)] (ORCID:0000000270817329), Son, Seung Woo [Univ. of Massachusetts, Lowell, MA (United States)] (ORCID:000000018922418X), Song, Shihui [Univ. of Iowa, Iowa City, IA (United States)] (ORCID:0009000794268591), Tang, William [Princeton Plasma Physics Laboratory (PPPL), Princeton, NJ (United States)] (ORCID:000000034183520X), Tao, Dingwen [Indiana Univ., Bloomington, IN (United States)] (ORCID:0000000154224497), Tian, Jiannan [Univ. of Kentucky, Lexington, KY (United States)] (ORCID:0000000311019148), Yoshii, Kazutomo [Argonne National Laboratory (ANL), Argonne, IL (United States)] (ORCID:0000000319045383), Zhao, Kai [Florida State Univ., Tallahassee, FL (United States)] (ORCID:0000000153283962). 2025-11-15. What to Support When You’re Compressing. https://doi.org/10.1145/3712285.3759856
Cite the original work for its findings. Save a collection to share your selection of sources.