Search NASA⌕ Search

Engineering topics

Illingworth, Robert

Publications and source records attributed to Illingworth, Robert.

A Ceph S3 Object Data Store for HEP

We present a novel data format design that obviates the need for data tiers by storing individual event data products in column objects. The objects are stored and retrieved through Ceph S3 technology, with a layout designed to minimize metadata volume and maximize data processing parallelism. Performance benchmarks of data storage and retrieval are presented.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

ROBIN (RuciO/BIgData Express/SENSE) A Next-Generation High-Performance Data Service Platform

Increasingly, scientific research discoverydepends on the analysis of extremely large amounts of data. Thisdata is gathered through very large scientific instruments, verylarge numbers of small devices, and mid-scale instruments.These devices generate exponentially increasing volumes of data.Such data is often obtained, analyzed, stored, visualized, andtransferred at and among sites world-wide by largecollaborations of science communities. Managing and movingextremely large volumes of data across today’s networks is aspecial multidimensional challenge. Also, existing datamanagement and movement services and tools often areinadequate to address all specific requirements. This paperdescribes the various components of this challenge and howthose issues can be addressed by ROBIN, a uniquecomprehensive set of integrated services designed specificallyfor managing and moving extremely large amounts of data overlong distances, e.g., thousands of miles around the globe. Thispaper also describes the results of initial experiments using thatset of integrated services.

Wu, Wenji↗

Simons Foundation Data Management

The Simons Foundation is a non-profit organization that funds scientific research. Their Autism Research is generating significant amounts of genomic data, which they need to both archive and make available for researchers around the globe. The Foundation already stores data at Fermilab, as per agreement FRA-2015-0006. With this CRADA, the Fermilab Scientific Computing Division has developed a prototype solution for the data management needs of the Foundation, modifying services developed for High Energy Physics. The tools allow researchers to select datasets from a metadata catalogue of Foundation data, transfer selected files or datasets from Fermilab to remote storage, and enable the Foundation to manage access and authorization of the files. The prototypical solutions developed through this CRADA is considered the Phase I of a body of work that can be expanded and transitioned to production quality in a subsequent phase II, to be determined with the Foundation.

97 MATHEMATICS AND COMPUTING↗