Batch Shipyard Quickstart

If you are interested in executing Deep Learning workloads on Batch Shipyard and want to jump in without having to install anything, you can utilize the Deep Learning Jupyter Notebooks on Azure Notebooks to quickly get started.

If you would like to use Batch Shipyard from the command line, this quickstart doc will provide step-by-step instructions. Although Batch Shipyard supports various types of workloads, for the following quickstart example, we will select from the following Deep Learning recipes to quickly get started:

The example MNIST training sample will be used on one Azure Batch compute node regardless of which Deep Learning framework you prefer for the following. Note that this quickstart guide focuses on Docker container execution but the commands are agnostic to which containers (Docker or Singularity) you are using - only the configuration changes with respect to which ecosystem you are using.

  1. Installation of Batch Shipyard to your local machine has been completed or you are using Batch Shipyard from within Azure Cloud Shell.
  2. Create a directory to hold your configuration files. For this quickstart guide, create a directory named config.
  3. Copy the sample configuration files from the Deep Learning framework recipe of your choice to the config directory:
  4. Edit the config/credentials.yaml file and populate it with your Azure Batch and Azure Storage credentials. If you do not have an Azure Batch account, you can create one via the Azure Portal, Azure CLI 2.0, or Azure PowerShell. You can create a standard general purpose Azure Storage account using any of the aforementioned methods similar to creating an Azure Batch account.
  5. Edit the config/config.yaml file and edit the following settings:
    • storage_account_settings to link to the storage account named in step 4.
  6. In the main batch-shipyard directory (which should contain the shipyard or shipyard.cmd helper scripts if on Linux or Windows, respectively), run the following commands:
# NOTE: if you are on Windows, use shipyard.cmd instead of ./shipyard

# create the compute pool
./shipyard pool add --configdir config
# alternatively (if not on Windows), you can specify the configdir parameter
# as an environment variable which some may find as more convenient
SHIPYARD_CONFIGDIR=config ./shipyard pool add

# ... wait for pool to allocate ...

# add the training job and tail the output
# if CNTK-CPU-OpenMPI, Caffe2-CPU, Chainer-CPU, Keras+Theano-CPU, MXNet-CPU, TensorFlow-CPU, or Torch-CPU
./shipyard jobs add --configdir config --tail stdout.txt
# if Caffe-CPU
./shipyard jobs add --configdir config --tail stderr.txt

The --tail option of the jobs add command will stream the stderr or stdout file to your local console which will provide you progress information about your job.

Once you are finished interacting with your jobs, tasks and pool, you can remove them with the following commands:

# ... done interacting with jobs/tasks/pool
./shipyard jobs del --configdir config --wait
./shipyard pool del --configdir config

Step-by-step Tutorial

The From Scratch: Step-by-step guide will provide detailed steps on how to construct your own set of configuration files to execute on Batch Shipyard.

Commandline Usage Guide

Batch Shipyard Usage contains explanations for all of the actions available with commandline interface.

In-Depth Configuration Guide

It is recommended to review the full in-depth configuration guide for explanations of all of the configuration options available for Batch Shipyard.

Graphical Interfaces

You can also use the Azure Portal or Batch Labs to view more properties of your Azure Batch accounts, pools, nodes, jobs and tasks. You can view your Azure Storage accounts on Azure Portal or with Microsoft Azure Storage Explorer.