PROCESS Data Infrastructure and Data Services

Authors

  • Reginald Cushing Institute of Informatics, University of Amsterdam, Amsterdam, Netherlands
  • Onno Valkering Institute of Informatics, University of Amsterdam, Amsterdam, Netherlands
  • Adam Belloum Institute of Informatics, University of Amsterdam, Amsterdam & Netherlands eScience Center, 1098 XG Amsterdam, Netherlands
  • Souley Madougou Netherlands eScience Center, 1098 XG Amsterdam, Netherlands
  • Martin Bobák Institute of Informatics, Slovak Academy of Sciences, 845 07 Bratislava, Slovakia
  • Ondrej Habala Institute of Informatics, Slovak Academy of Sciences, 845 07 Bratislava, Slovakia
  • Viet Tran Institute of Informatics, Slovak Academy of Sciences, 845 07 Bratislava, Slovakia
  • Jan Meizner UCC Cyfronet AGH, AGH University of Science and Technology Krakow, Poland
  • Piotr Nowakowski UCC Cyfronet AGH, AGH University of Science and Technology Krakow, Poland
  • Mara Graziani University of Applied Sciences of Western Switzerland, 3960 Sierre, Switzerland
  • Henning Müller University of Applied Sciences of Western Switzerland, 3960 Sierre, Switzerland

DOI:

https://doi.org/10.31577/cai_2020_4_724

Keywords:

Exascale data management, distributed file systems, microservice architecture

Abstract

Due to energy limitation and high operational costs, it is likely that exascale computing will not be achieved by one or two datacentres but will require many more. A simple calculation, which aggregates the computation power of the 2017 Top500 supercomputers, can only reach 418 petaflops. Companies like Rescale, which claims 1.4 exaflops of peak computing power, describes its infrastructure as composed of 8 million servers spread across 30 datacentres. Any proposed solution to address exascale computing challenges has to take into consideration these facts and by design should aim to support the use of geographically distributed and likely independent datacentres. It should also consider, whenever possible, the co-allocation of the storage with the computation as it would take 3 years to transfer 1 exabyte on a dedicated 100 Gb Ethernet connection. This means we have to be smart about managing data more and more geographically dispersed and spread across different administrative domains. As the natural settings of the PROCESS project is to operate within the European Research Infrastructure and serve the European research communities facing exascale challenges, it is important that PROCESS architecture and solutions are well positioned within the European computing and data management landscape namely PRACE, EGI, and EUDAT. In this paper we propose a scalable and programmable data infrastructure that is easy to deploy and can be tuned to support various data-intensive scientific applications.

Downloads

Download data is not yet available.

Downloads

Published

2021-01-12

How to Cite

Cushing, R., Valkering, O., Belloum, A., Madougou, S., Bobák, M., Habala, O., … Müller, H. (2021). PROCESS Data Infrastructure and Data Services. COMPUTING AND INFORMATICS, 39(4), 724–756. https://doi.org/10.31577/cai_2020_4_724

Issue

Section

Special Section Articles

Most read articles by the same author(s)

1 2 > >>