Home About Software Documentation Support Outreach Ecosystem Blog Dev Awards Team & Sponsors

Grid[Way] Job Template Manager

About

The Grid[Way] Job Template Manager is an extension to the GridWay Metascheduler especially useful for parameter sweeping. It provides a lots of features in order to create, delete, submit, purge, kill and get information about a set of job templates with similar structure.

It is possible to define two levels of features:

  • Job Template manager: Create and Delete job templates. These two subcommands are always available. Their mission is, respectivly, to generate and remove the files containing the job description. An argument file with the description of how to span the parameter space is compulsory for the creation.
  • GridWay Job control: Submit, Purge, Kill, and Info subcommands which interact with the GridWay instance in order to perform control and monitoring over the jobs related to the job templates.

Development

The project development is hosted as a subspace of The GridWay Metascheduler. There you may find an activity log as well as the official repository and downloads.

Cites

The tool is an effort of academic research. If you used it or obtained benefit from it, it would be very nice to hear from you and get recognition through the following cite:

Alejandro Lorca, Eduardo Huedo, Ignacio M. Llorente
The Grid[Way] Job Template Manager, a tool for parameter sweeping
Computer Physics Communications, 182 (2011) pp. 1047-1060
ISSN 0010-4655, DOI: 10.1016/j.cpc.2010.12.041.

Installation

Download

Latest release: Version 1.0, (26 Feb 2010)

Older preview releases

Configuration

The GridWay Job Template Manager does not require explicit configuration. It should work out of the box. In order to modify some parameters, use the --config option.

Usage

Usage: gw_job_template_manager [OPTION] SUBCOMMAND ARG(S)

 SUBCOMMANDS:
  -c, --create            Create templates
    ARG:  <PARAMETER_FILE>
         PARAMETER_FILE   File with parameter description,
    OPTION[S]: --worker WORKER_FILE --template TEMPLATE_FILE
         WORKER_FILE      Executable file to be run remotely,
         TEMPLATE_FILE    Append extra variables from template file.

  -d, --delete            Delete templates
    ARG:  <all|[un]submitted|[un]finished|[un]successful|FROM-TO>
         all              all the templates are deleted,
         [un]submitted    only those which were [not] submmited,
         [un]finished     only those whose jobs have [not] finished,
         [un]successful   only those whose jobs finished [un]succesfully,
         FROM-TO          deletes the range [FROM,TO].

  -s, --submit            Submit the jobs from templates
    ARG:  <all|[un]submitted|[un]finished|[un]successful|FROM-TO>
         all              all the templates are submitted/resubmitted,
         [un]submitted    only those which were [not] submmited,
         [un]finished     only those whose jobs have [not] finished,
         [un]successful   only those whose jobs finished [un]succesfully,
         FROM-TO          submit the range [FROM,TO].

  -p, --purge             Purge the existing jobs from templates
    ARG:  <all|[un]finished|[un]successful|FROM-TO>
         all              all the jobs are purged (waiting if necessary),
         [un]finished     only those which have [not] finished,
         [un]successful   only those [un]succesful,
         FROM-TO          purge the range [FROM,TO].

  -k, --kill              Kill the existing jobs from templates
    ARG:  <all|[un]finished|[un]successful|FROM-TO>
         all              all the jobs are killed (potentially slow),
         [un]finished     only those which have [not] finished,
         [un]successful   only those [un]succesful,
         FROM-TO          kill the range [FROM,TO],
    OPTION: --signal SIG
         SIG              kill with signal passed to gwkill.

  -i, --info              Information about status of the submitted jobs,
    ARG:  <history|now|evolution>
         history          full historic information about each job,
         now              last status update for each job,
         evolution        timely snapshots of job statuses,

 OPTIONS:
      --debug             Debugging information
      --config key=value  Assign a key=value pair for configuration settings.

 HELP and ABOUT:
  -v, --version           Version number of the programme.
  -l, --license           Credits and license.
  -h, --help              Print this help.

Argument file syntax

Each line in the argument file correspond, in order, to an argument to be passed to the worker binary. The syntax for each line is the following:

  • LOOPTYPE=LIST VALUE=an_explicit_value [VALUE=another_explicit_value] [FUNCTION=perl_function]
  • LOOPTYPE=[EXP]RANGE START=initial_value END=final_value [POINTS=number_of_points|STEP=increment] [SKIP=exclusion_value] [FUNCTION=perl_function]

Fields in brackets are optional.

  • LOOPTYPE describes the structure, possible assigments are RANGE, EXPRANGE or LIST
  • INITIAL_VALUE and FINAL_VALUE might be characters, integers or reals.
  • POINTS is a positive integer > 1, default if absent is 2.
  • STEP is an alternatively to POINTS, with it you can provide a fixed increment
  • SKIP is an element you do not wish to appear
  • FUNCTION is a perl function or operation. It preceedes the argument of the RANGE for sampling

Examples

  1. Hello World! (1 string argument passed to the binary, 1 job template). The parameter file is
    LOOPTYPE=LIST, VALUE="Hello World!"

    Then with this command

    gw_job_template_manager --worker /bin/echo --create parameter.in
    
    you create a file '0_worker_Hello_World!.jt' with the following content

    NAME = 0_echo
    EXECUTABLE = echo
    ARGUMENTS = "Hello World!"
    STDOUT_FILE = 0_worker_Hello_World!.out
    STDERR_FILE = 0_worker_Hello_World!.err
  2. A sample of x={1,2,3,4,5} (1 argument passed to the worker, 5 job templates)
    LOOPTYPE=RANGE, START=1, END=5, POINTS=5

    Your file 3_worker_4.jt in this case look like:

    NAME = 3_worker
    EXECUTABLE = worker
    ARGUMENTS = 4
    STDOUT_FILE = 3_worker_4.out
    STDERR_FILE = 3_worker_4.err
  3. A more complex mixed sample of algorithm names and exponential sampling with output file (3 arguments, 6 job templates). The argument.in file
    LOOPTYPE=LIST, VALUE=Taylorexpansion, VALUE=Exactanalytic
    LOOPTYPE=EXPRANGE, START=1, END=10000, STEP=2
    LOOPTYPE=${JT_ID}.output

    and the template.in file for appending lines to the job templates

    OUTPUT_FILES = ${JT_ID}.output

    Then you have the following job template files:

    $ ls -1 *.jt
    0_worker_Taylorexpansion_1_0.output.jt
    1_worker_Taylorexpansion_100_1.output.jt
    2_worker_Taylorexpansion_10000_2.output.jt
    3_worker_Exactanalytic_1_3.output.jt
    4_worker_Exactanalytic_100_4.output.jt
    5_worker_Exactanalytic_10000_5.output.jt
    
    and the content of 5_worker_Exactanalytic_10000_5.output.jt

    NAME = 5_worker
    EXECUTABLE = worker
    ARGUMENTS = Exactanalytic 10000 5.output
    STDOUT_FILE = 5_worker_Exactanalytic_10000_5.output.out
    STDERR_FILE = 5_worker_Exactanalytic_10000_5.output.err
    OUTPUT_FILES = 5.output

Training Material

Presentations

Admin · Login