B2STAGE
How to shift large amounts of data
Version 4
February 2016
This work is licensed under the CreativeCommons CC-BY 4.0 licence.
Attribution: EUDAT – www.eudat.eu
B2STAGE is…
a reliable, efficient, light-weight andeasy-to-use service to transferresearch data sets between EUDATstorage resources and high-performance computing (HPC)workspaces
2
A truly pan-European Infrastructure
3
EUDAT offers common dataservices to both researchcommunities and individualsthrough a network of 35European organisations.
EUDAT wants to enableEuropean researchers fromany discipline to preserve,find, access, and processdata in a trusted environment,as part of a CollaborativeData Infrastructure.
European infrastructures
Technology Providers
Research Communities
Community-Driven Solutions
4
PHYSICAL SCIENCES
& ENGINEERING
SOCIAL SCIENCES& HUMANITIES
MATERIALS &ANALYTICAL FACILITIES
ENVIRONMENTAL SCIENCES
MAPPER
BIOMEDICAL &MEDICAL SCIENCES
EUDAT services are designed, built andimplemented based on user communityrequirements.
The EUDAT Service Suite
5
move large amounts of databetween data stores and high-performance compute resources
re-ingest computational resultsback into EUDAT
deposit large data sets into EUDATresources for long-term preservation
Facilitating communities to:
Features:
high-speed transfer
reliable and light-weight
manages permanent PIDs
6
B2STAGE Features
Why use B2STAGE?
7
Research challenges are getting largerand more complex:
E.g. full-Earth climate simulation,coupled simulations of multiple organsin the human body, seismic analyses ofearthquakes at continental scale
High level
benefits
Researcher data and compute demands are rising fast
Efficient transfer of data to high performance computing(HPC) workspaces is essential especially in distributedcomputing, where resources are geographicallydispersed
Why use B2STAGE?
8
Facilitates transfer of large datacollections from EUDAT storageresources to HPC facilities.
Specific UserRequirements
Provides the means to re-ingest computational resultsback into the EUDAT infrastructure.
Ingests data sets into EUDAT resources for long-termpreservation.
Offers reliable, efficient, easy-to-use tools to managedata transfers.
The Data Staging Script is the only tool handlingdata transfer using PIDs.
Who can use B2STAGE?
Researchers can transfer large data collectionsfrom EUDAT storage resources to HPC facilities forprocessing.
Community Managers can replicate communitydata through a lightweight service and ingest datasets to EUDAT storage resources for long termpreservation.
9
How can you use B2STAGE?
EUDAT offers B2STAGE to all registered researchersand interested communities, enabling them tomake use of the service to stage data out of EUDAT,and ingest computational results back.
Access to remote HPC facilities should benegotiated and arranged by individual users inparallel.
To help researchers use the B2STAGE service, EUDAToffers documentation, training material and aservice helpdesk.
10
For more information please email:
How can you use B2STAGE?
11
How does B2STAGE work?
12
GridFTP server
iRODS-DSI
User desktop
GridFTP client
data
control
PIDRegistry
PID
control
HPC
GridFTP server
User desktop
How does B2STAGE work?
13
GridFTP client
File system
GridFTP server
iRODS-DSI
PIDRegistry
PID
data
control
B2STAGE User communities
VPH Community ingesting data onto EUDAT resources
Approximately 12TB will be ingested through thisservice
VPH data also replicated between RZG and PSNCsites
B2STAGE will foster the collaboration with EGI and PRACEto develop cross-infrastructure usage:
B2STAGE will be the main service to enable theinteroperability of these infrastructures.
Numerous new communities to adopt it as part of the2015 and 2016 Calls for Collaboration
14
B2STAGE summary
B2STAGE offers:
data staging functionalities to easily andefficiently transfer data from EUDAT storageresources to HPC facilities
a powerful mechanism to ingest data ontoEUDAT resources
a script to facilitate the staging, ingest andretrieval of PID information of transferred data
B2STAGE is unique in handling PIDs for the data
15
Future features
The Data Staging Script will be replaced by amodular and extensible python library which willfurnish the users with a programmable interfacetowards most of the EUDAT services.
16
17
Thank you
