Middleware

The eXtreme DataCloud (XDC) is a EU H2020 funded project under grant agreement 777367, aimed at developing scalable technologies for federating storage resources and managing data in highly distributed computing environments. The services provided will be capable of operating at the unprecedented scale required by the most demanding, data intensive, research experiments in Europe and Worldwide.
XDC will be based on existing tools, whose technical maturity is proved and that the project will enrich with new functionalities and plugins already available as prototypes (TRL6+) that will be brought at the production level (TRL8+) at end of XDC.
The targeted platforms are the current and next generation e-Infrastructures deployed in Europe, such as the European Open Science Cloud (EOSC), the European Grid Infrastructure (EGI), the Worldwide LHC Computing Grid (WLCG) and the computing infrastructures funded by other public and academic initiatives.
All the developments will be community-driven and tested against real life use cases provided by the consortium partners representing research communities belonging to a variety of scientific domains: Life Science, Astrophysics, High Energy Physics, Photon Science and Clinical Research.
The XDC project aims at opening new possibilities to scientific research communities in Europe and worldwide by supporting the evolution of e-Infrastructure services for Exascale data resources. The XDC software will be released as Open Source platforms available for general exploitation.

The EOSC-hub project brings together multiple service providers to create the Hub: a single contact point for European researchers and innovators to discover, access, use and reuse a broad spectrum of resources for advanced data-driven research. For researchers, this will mean a broader access to services supporting their scientific discovery and collaboration across disciplinary and geographical boundaries.

The project mobilises providers from the EGI Federation, EUDAT CDI, INDIGO-DataCloud and other major European research infrastructures to deliver a common catalogue of research data, services and software for research. EOSC-hub collaborates closely with GÉANT and the EOSCpilot and OpenAIRE-Advance projects to deliver a consistent service offer for research communities across Europe.

EOSC-hub is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement 777536. The generous EU funding received by the project is complemented with a contribution from the EGI Foundation and its participants, and in-kind contributions made available by service providers of the EGI Federation.

En la era de la exaescala, se espera que trabajos paralelos muy grandes se ejecuten en un número asombroso de procesadores, por lo que el tiempo medio entre fallos será de unos 30 minutos o incluso menos debido a errores silenciosos. Este hecho se ha convertido en una piedra angular en el camino hacia la próxima generación de supercomputadores. Por lo tanto, se necesitan nuevas funciones de manejo de errores, detección de errores e implementaciones de software a nivel de sistema y/o aplicación para proporcionar tolerancia a fallos contra errores silenciosos.
Este proyecto pretende así garantizar que las futuras aplicaciones de exascala se ejecuten satisfactoriamente y generen resultados correctos mientras corren en sistemas inestables, tratando de evitar la inacción y los bloqueos y pérdidas de rendimiento.
El principal objetivo de este proyecto es sobreponerse a errores silenciosos con una aproximación multidisciplinar, abordando la física de neutrones que originan muchos de estos errores y proponiendo nuevos diseños y desarrollos matemáticos y computacionales para superarlos proveyendo así un entorno tolerante a fallos.
Esta actividad se enmarca dentro de una más ambiciosa de e-Ciencia llevada a cabo junto a otras Unidades del CIEMAT y el PIC: la federación de distintas arquitecturas heterogéneas de datos y cálculo de manera transparente al usuario, así como en la eficiencia computacional y energética de los recursos.

The key concept proposed in the DEEP Hybrid DataCloud project is the need to support intensive computing techniques that require specialized HPC hardware, like GPUs or low-latency interconnects, to explore very large datasets. A Hybrid Cloud approach enables the access to such resources that are not easily reachable by the researchers at the scale needed in the current EU e-infrastructure.
We also propose to deploy under the common label of “DEEP as a Service” a set of building blocks that enable the easy development of applications requiring these techniques: deep learning using neural networks, parallel post-processing of very large data, and analysis of massive online data streams . These services will be deployed in the project testbed, offered to the research communities linked to the project through pilot applications, and integrated under the EOSC framework, where they can be further scaled up in the future.