Home About Software Documentation Support Outreach Ecosystem Blog Dev Awards Team & Sponsors

GridWay 5.0 Stable Release Notes (released on June 2, 2006)

The GridWay Project team is pleased to make 5.0 available for download. GridWay meta-scheduling technology gives end users, application developers and managers of Globus infrastructures a scheduling functionality similar to that found on local DRM (Distributed Resource Management) systems. The GridWay Metascheduler is a Globus ProtoProject, so it adheres to Globus philosophy and guidelines for collaborative development. Highlights of this release include:

  • Advanced scheduling capabilities on a grid consisting of Globus services: dynamic discovery & selection, opportunistic migration, performance slowdown detection, support for self-adaptive applications and support for the definition of new scheduling policies
  • Detection and recovery from remote and local failure situations
  • DRM-like commands to submit, monitor, synchronize and control single, array and interdependent jobs; monitor Globus resources and users; and extract Grid accounting information
  • Full support for C and JAVA DRMAA OGF standard for the development of distributed applications on Globus services
  • Straightforward deployment that does not require new services apart from those provided by the Globus Toolkit: MDS, GRAM, GridFTP and RFT
  • Modular architecture that allows an easy incorporation of new grid services and interoperability between different grid infrastructures (Globus WS, Globus pre-WS and EGEE)

1. Documentation

GridWay 5.0 documentation includes in HTML and PDF formats:

  • GridWay 5.0 Documentation: Installation & Configuration Guide
  • GridWay 5.0 Documentation: User Guide
  • GridWay 5.0 Documentation: Application Developer Guide (DRMAA C and JAVA bindings)
  • GridWay 5.0 Documentation: GridWay Developer Guide
  • GridWay 5.0 Documentation: Command Reference
  • GridWay 5.0 Documentation: DRMAA C Reference
  • GridWay 5.0 Documentation: DRMAA JAVA Reference

2. Contents of the Distribution

The GridWay 5.0 software distribution is made up of the following components:

  • GridWay core (dispatch, information, execution and transfer managers)
  • User commands
  • DRMAA C and JAVA libraries
  • Information MADs to interface MDS2 (MDS schema), MDS2 (Glue schema), MDS4, and static host list file
  • Execution MADs to interface pre-WS GRAM and WS GRAM
  • Transfer MADs to interface GridFTP and RFT

3. Support

Support for the GridWay Metascheduler is mainly provided through documentation and mailing lists.

4. Collaborative Development

GridWay is a community project, benefiting from code and support contributions from individuals and corporations around the world. The development discussion mailing list is the forum for suggestions and comments for code changes. We'd be very happy to hear from experienced programmers and managers of grid infrastructures who'd like to join the development team.

5. Licensing

GridWay is distributed and licensed for use under the terms of the Apache License, Version 2.0.

Copyright 2002-2006 GridWay Team, Distributed Systems Architecture Group, Universidad Complutense de Madrid.

6. Installing

GridWay 5.0 has been tested on Linux and Solaris platforms.

7. New Features about Architecture and Deployment

Along with many bug fixes, GridWay 5.0 has been rewritten from scratch to include the following new features and improve its robustness:

7.1. Improve GridWay's modular architecture

In GridWay 4.0.2, we introduced architecture for the execution manager module based on a MAD (Middleware Access Driver) to interface Grid execution services. In this release we take advantage of this architecture to implement an information manager module with a MAD that interfaces Grid information services, and a transfer manager module with a MAD that interfaces Grid data services. Moreover, we will decouple the scheduling process from the dispatch manager through the use of a external and selectable scheduler module.

We have measured considerable application performance gains with this new architecture:

  • The overhead induced by the information retrieval has been drastically reduced.
  • The new brokering system allows to overlap the preparation (prolog stage) and termination (epilog stage) of two jobs submitted to the same host.
  • With the new Middleware Drivers, Prolog and Epilog stages no longer need to be executed through the GRAM service, so we expect an important improvement in the preparation and termination times.

7.2. Easier deployment

No services are required in the client system.

7.3 No configuration needed to interface Grid Information Systems

GridWay 5.0 is shipped with drivers to interface: MDS2 (MDS schema), MDS2 (Glue schema), MDS4, and static host list file (Round-robin).

7.4 Job performance monitoring

The previous release of GridWay supports job migration when performance degradation is detected. However, this feature was not documented. In this new release you will find that.

  1. Jobs are equipped with a light-weight self monitoring system (through the wrapper module, there is no need to modify user applications). The job is migrated when it is no receiving as much CPU as you expected.
  2. Checkpoints will be periodically retrieved to the client machine or a checkpoint server defined by the user.

7.5 Credential Renewal

Some compute-intensive jobs requires long lived credentials to execute them. GridWay is able to interface MyProxy services to renew user credentials.

7.6 GridWay Daemon Fault Recovery

We have added a new mode to recover the jobs submitted through GridWay to the Grid in case of local failure.

7.7 Multiple-user support

This release includes support for multiple users interfacing to the same gwd daemon. This way the installation and configuration of GridWay will be performed by the system manager and the users could submit, control and monitor their jobs from a front-end or from submission hosts (that do not require GridWay and Globus installation).

7.8 Improved support for LCG based Grids

Staging is now performed with the Storage Element, when available.

7.9 Support for GateWays

GridWay can be now interfaced via GRAM. Please refer to Grid4Utility for more information.

8. New Features about Funcionality

8.1. Scheduler skeleton

Those interested in developing new schedulers can find a scheduler skeleton in the source tree. This program template allows the development of new scheduler modules. Also a new user round-robin scheduler is distributed with GridWay 5.0.

8.2 Resource Selection based on common resource attributes

The user no longer needs to specify an LDAP or XPath filter, but a set of Boolean expressions, function of the resource properties. Resources are ranked using a numeric expression, function of the resource properties. We will no longer support ad hoc programs to rank resources.

8.3 Improve GridWay's template and daemon configuration files

Some efforts have been made to make variable notation more consistent in GridWay configuration files:

  1. GridWay 5.0 simplifies job-template contents. Although we will support per-job configuration of scheduling, and module variables, configuration parameters will be used by default ($GW_LOCATION/etc/job_template.default).
  2. GridWay 5.0 checks template syntax.
  3. Resource properties and expressions follow intuitive semantics, e.g. REQUIREMENTS = (CPU_MHZ > 1800) & (ARCH = “i686”).; RANK = (CPU_MHZ * 2) + FREE_MEM_MB

8.4 Improved client utilities

Client tools have been modified to support the new features described above. They also include a new option (-h) that provides online help information.

8.5 A new command has been added, gwhosts, to show the resources available in the testbed

This new command accesses the underlying information system driver for monitoring grid resources, being also able to perform matchmaking of requirement and rank expressions with resource attributes.

8.6 Accounting support

This release includes support for usage accounting, using the Berkeley Data Base library. Usage statistics can be access with the gwacct command.

8.7 Hard Kill

Jobs can be now killed in any state, use gwkill -9.

8.8 Job Dependency Capability

GridWay allows job submission to be dependent on the completion of other jobs. This new funcionality provides support for the execution of workflows.

8.9 Cleaner log files

Log files are now cleaner. The logging system does not output verbose debugging information, log messages are classified: [I] information, [W] warnning or [E] error.

8.10 Full DRMAA support

GridWay 5.0 included a functional subset of the Distributed Resource Management Application API. In this release we have included full support to the current standard; some new features have been added to GridWay:

  1. New HOLD and RELEASE states. Jobs can arrive the HOLD state when submitted
  2. JAVA binding implementation
  3. GridWay executes the standard DRMAA test-suites

8.11 Thread-safety in DRMAA libraries

DRMAA library can be used in multi-threaded programs.

Admin · Login