Batch Shipyard Quickstart

Choose your speed:

  1. Normal Quickstart
  2. Ludicrous Speed Quickstart
  3. Step-by-Step Guide
  4. Notebooks

Normal Quickstart

If you would like to use Batch Shipyard from the command line, this quickstart doc will provide step-by-step instructions. Although Batch Shipyard supports any type of Batch processing or HPC workload, for the following quickstart guide, we will select from the following Deep Learning recipes as an example workflow:

The example MNIST training sample will be used on one Azure Batch compute node regardless of which Deep Learning framework you prefer for the following. Note that this quickstart guide focuses on Docker container execution but the commands are agnostic to which containers (Docker or Singularity) you are using - only the configuration changes with respect to which ecosystem you are using.

  1. Installation of Batch Shipyard to your local machine has been completed or you are using Batch Shipyard from within Azure Cloud Shell.
  2. Create a directory to hold your configuration files. For this quickstart guide, create a directory named config.
  3. Copy the sample configuration files from the Deep Learning framework recipe of your choice to the config directory:
  4. Edit the config/credentials.yaml file and populate it with your Azure Batch and Azure Storage credentials. If you do not have an Azure Batch account, you can create one via the Azure Portal, Azure CLI 2.0, or Azure PowerShell. You can create a standard general purpose Azure Storage account using any of the aforementioned methods similar to creating an Azure Batch account.
  5. Edit the config/config.yaml file and edit the following settings:
    • storage_account_settings to link to the storage account named in step 4.
  6. In the main batch-shipyard directory (which should contain the shipyard or shipyard.cmd helper scripts if on Linux or Windows, respectively), run the following commands:
# change working directory to the config directory
cd config

# create the compute pool
# NOTE: if you are on Windows, use ..\shipyard.cmd instead of ../shipyard
../shipyard pool add

# ... wait for pool to allocate ...

# add the training job and tail the output
# if CNTK-CPU-OpenMPI, Caffe2-CPU, Chainer-CPU, Keras+Theano-CPU, MXNet-CPU, TensorFlow-CPU, or Torch-CPU
../shipyard jobs add --tail stdout.txt
# if Caffe-CPU
../shipyard jobs add --tail stderr.txt

The --tail option of the jobs add command will stream the stderr or stdout file to your local console which will provide you progress information about your job.

Once you are finished interacting with your jobs, tasks and pool, you can remove them with the following commands:

# after you are done interacting with jobs/tasks/pool
../shipyard jobs del --wait
../shipyard pool del

Ludicrous Speed Quickstart

Pre-jump checklist:

  • Linux, Mac or WSL machine with network access
  • git and Python3 is installed
  • Comfortable with Linux commandline
  • Have an active Azure subscription
  • Understand how to use the Azure Portal

Spin up FTL drives:

  1. Create Batch account if you do not have one
  2. Retrieve the Batch account key and service URL
  3. Create Storage account if you do not have one (preferably in the same region as the Batch account)
  4. Retrieve the Storage account key

Execute jump:

git clone https://github.com/Azure/batch-shipyard.git
cd batch-shipyard
./install.sh
export SHIPYARD_CONFIGDIR=recipes/TensorFlow-CPU/config
nano $SHIPYARD_CONFIGDIR/credentials.yaml
# edit required properties in file and save
./shipyard pool add
./shipyard jobs add --tail stdout.txt

Jump complete:

# clean up
./shipyard jobs del -y --wait
./shipyard pool del -y
unset SHIPYARD_CONFIGDIR

Step-by-step Tutorial

The From Scratch: Step-by-step guide will provide detailed steps on how to construct your own set of configuration files to execute on Batch Shipyard.

Notebooks

If you are interested in executing Batch Shipyard on a Jupyter environment without having to install anything, you can utilize the community contributed Jupyter Notebooks that you can instantiate on Azure Notebooks to quickly get started.

Note: Because these notebooks are community contributed and supported, they may not be up-to-date with the latest Batch Shipyard version and may not work properly.

Commandline Usage Guide

Batch Shipyard Usage contains explanations for all of the actions available with commandline interface.

In-Depth Configuration Guide

It is recommended to review the full in-depth configuration guide for explanations of all of the configuration options available for Batch Shipyard.

Graphical Interfaces

You can also use the Azure Portal or Batch Labs to view more properties of your Azure Batch accounts, pools, nodes, jobs and tasks. You can view your Azure Storage accounts on Azure Portal or with Microsoft Azure Storage Explorer.