Galaxy Distributed Infrastructure

Q3 2022
  • Implementation challenges
    • User / Resource provider environments
  • Result description

    Building a distributed network in the whole of Europe (12 distributed computing nodes in total), so that a computational job ( including VREs, like Jupyter notebooks) can be executed on national resource providers.

    Problem addressed

    1. Development of additional public (national) nodes that can be used by a large number of users.
    2. Connecting countries that do not have the resources to run their own Galaxy server, so that their computational jobs (including VREs like Jupyter notebooks) can be executed on national resource providers.
    3. Bring your own compute (BYOC). Develop a mechanism to plug-in own/institutional computing resources into the Galaxy European Science Gateway.
    4. Bring your own storage (BYOS). Develop a mechanism to plug-in own/institutional storage into the Galaxy European Science Gateway.
    5. Development of a scheduler that takes into account proximity to the data source, energy efficiency and environmental overhead when scheduling the job distribution over the European servers.

    Who can use the result

    All researchers on planet earth.

    Timeline

    Timepoint 1: Galaxy has been functional since ESG's start (and earlier), also addressing distributed resources, and can be used productively (Q3 2022). Timepoint 2: The project has reached the milestones "Bring your own compute" (BYOC) and "Bring your own Storage" (BYOS) with August 31st 2023, via principally new connectors. Timepoint 3: Astrophysics flavored Galaxy workbench with an initial set of tools and visualisations (Q1 2024) Timepoint 4: Finally, although the majority of tasks has been carried out for Result 1, the development goes on until the project's end in August 2025, majorly addressing more protocols and increasing convenience.

    How to use the result

    Type of result