PowerPoint Presentation

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

GridX1 Services Project

A. Agarwal, A. Berman, A. Charbonneau, R. Desmarais, I. Gable, R. Impey, D.Leske, G. Mateescu, A. Norton, W. Podaima, S. Popov, D. Quesnel

S. Ramage, S. Reynen, R. Sobie, C. Usher, D. Vanderster

CANARIE Inc., Ottawa, Ontario, Canada

National Research Council, Ottawa, Ontario, Canada

University of Victoria, Victoria, British Columbia, Canada

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Overview

•Motivation

•Concepts

–Web Services

–Grid Services

–Second Generation Web Services

•Project Goals

–Determine Requirements for a Grid

–Evaluate web services based grid computing

–Implement a web services based grid

–Provide users with an easy to use interface to a web services grid

•Requirements

–Meta scheduler and Registry Requirements

–Monitoring Requirements

–Grid Projects Evaluation

•Implementation

–GridX1 Services Project Test bed and Middleware

–Meta Scheduler

–Registry

–Monitoring

–Job Submission Tool

•Summary

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Motivation

The increased use of grid technology for scientific analysis has raisedseveral issues over the passed five to six years.

•Users of the grid have to be quite knowledgeable to implement clienttools that use grid resources.

•Creating extensions to add functionality to a grid are very difficult.

•Administration of grid users and resources are time consuming.

To ease these burdens, research in providing web services basedstandards to manage grid communication is being developed to easedevelopment, scalability, accessibility and usability.

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Concepts :web services

•A web service is any program which makes itselfavailable over the internet and uses XML as a wayof communicating with other programs. [1]

•Web services can be put together to form anarchitecture. For example, the Service OrientedArchitecture.

•The Service Oriented Architecture is a commonway of providing users access to a service. Thesetraditional web services are considered statelessin that they have no memory of passed events. [2]

Service Oriented Architecture [2]

Basic web service usage [4]

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Concepts : grid services

•Grid Services, such as those providedby Globus 2, provide the user with theability to access large amounts ofprocessor power and data storage.

•The Job Manager service is the mainservice provided to users. In particularglobus offers a gram service whichaccepts user job submission.

Globus Grid Services [3]

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Concepts :Second Generation Web Services

•Second Generation Web Services, such asGlobus 4, is an extension of web services. Inparticular the OGSA is an architecture that can beused to implement second generation webservices [4].

•The WSRF (Web Services ResourceFramework) implemented by Globus andspecifies Stateful web services as required by theOGSA.

•The WSRF uses web services and gives them amemory. In particular the WSRF describes a WebService resource.

•A Web Service Resource is a web service with aresource (such as memory). This leads to statefulweb services which is required by a web servicesgrid.

Globus services offered using WSRF [4]

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Project Goals :Determine Requirements for a Grid

•Typically a grid consists of a Metascheduler, Registry, Monitor andone or more grid resources.

•Before we go ahead and constructa web services based grid, it wouldbe advantageous to understandwhat is required of the grid.

We must know:

•Meta scheduler requirements

•Registry requirements

•Monitoring requirements

Typical Grid Components

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Project Goals :Evaluate web services based grid computing

•To gain insight into how grids are currently being implemented, anevaluation of several popular grids (including gridX1) was completedand the grids were assessed by a set of grid requirements whichwere elicited from many knowledgeable grid guru’s.

•Several organizations were chosen to evaluate their grid servicesimplementation, these include GridX1, Teragrid, Gridbus, EGEE,NorduGrid and the OSG

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Project Goals :Implement a web services based grid

Once the evaluation of current grids is completed, a web services gridwill be implemented on it. This will require:

•Creation of a test bed : setup networked machines to be assigned asresources or registry or meta scheduler or monitor …..

•Installation of selected web services grid middleware

•Installation/Creation of a Meta scheduler

•Installation/Creation of a Registry

•Installation/Creation of a Monitor

•Setup of Selected Sites to use the test bed

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Project Goals :Provide users with an easy to use interface to a web services grid

•To provide a better experience when using the web services based grid, a user interface to usethe grid will be provided.

•This could be in the form of a client tool or possibly with a web browser.

Client Tool [5]

Web Broswer [6]

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Requirements :Meta Scheduler Requirements

•Meta Scheduler requirementsdocumenthttp://wiki.gridx1.ca/twiki/svndocs/Software RequirementsSpecification.pdf

Primary Meta Scheduling Requirements

•Use web services to receive jobs fromclients

•Use web services to schedule jobs toresources

•***put picture here to clarify usage****

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Requirements :Registry Requirements

•Registry Requirements documenthttp://wiki.gridx1.ca/twiki/svndocs/Software RequirementsSpecification.pdf

Primary Registry Requirements

•Use web services to receive resourceupdates

•Allow web services access to registryinformation

•***put picture here to clarify****

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Requirements :Monitoring Requirements

Monitoring Requirements document

http://wiki.gridx1.ca/twiki/pub/Main/SvnDo cs/GridMonitoringRequirementsAndDe sign.pdf

Primary Monitoring Requirements

•Allow web browser access to storedmonitored information

•Use Web services to determine status ofgrid to be stored in local database

•****insert picture here to clarify***

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Requirements :Grid Projects Evaluation

• One of the project Goals was to complete anevaluation of current grid projects. This evaluationwould be used to determine how best toimplement a web services grid of our own.

•The requirements document can be foundhttp://wiki.gridx1.ca/twiki/svndocs/SoftwareRequire mentsSpecification.pdf

•The analysis document can be foundhttp://wiki.gridx1.ca/twiki/pub/Main/SvnDocs/Regis tryAndMetaschedulerReview.pdf

• From the evaluation, we concluded that Teragrid,Gridbus and EGEE grid implementations would bethe best grids to follow for the implementation ofour own web services based grid.

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Implementation : GridX1 Services Project Test bed and Middleware

•Having Chosen Teragrid as a model toimplement our own web services grid, weneed computers to install and test gridsoftware.

•To do this, we created a gridX1 servicesproject test bed.

•Currently the test bed consists of 15machines. This includes:

–2 Registries

–3 Meta schedulers

–4 Head Nodes

–4 test worker nodes

–2 monitors

GridX1 Services Project Test bed Document:http://wiki.gridx1.ca/twiki/svndocs/TestBed.pdf

•To facilitate web services communication betweenthe machines, Globus 4 was chosen.

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Implementation :Meta Scheduler

The Meta Scheduler is the most important componentof a grid along with grid middleware to handle theweb services communication.

Globus 4 was chosen to be implemented on the testbed grid. Globus 4 is web services bases so itmeets are need to use web services as acommunication standard. Globus has a condorjob manager script as well which GabrielMateescu has been re-writing to serve ourpurposes. This script can be used to turn Globussubmitted jobs into condor jobs which can then bescheduled.

The Meta Scheduler chosen to be implemented isCondorG. CondorG is Globus aware and can useGlobus to re-submit jobs to other Globusresources.

Condor Meta scheduler using Globus 4 web services

Daniel Vanderster University of Victoria

National Research Council and the University of Victoria

Implementation :Registry

The Registry is the next most importantcomponent of a grid. In this case, Globus 4comes with a built in registry service calledan index service.

The index service however, did not completelymeet are needs and we had to add twoprograms created by Andre Charbonneauto make it useful to us.

Along with the Globus index server, we added aresource extractor for each resource headnode wishing to publish to our registry, anda resource class ad extractor which isneeded by the meta scheduler to getinformation on registered resources.