Batch Shipyard Quickstart
If you are interested in executing Deep Learning workloads on Batch Shipyard and want to jump in without having to install anything, you can utilize the Deep Learning Jupyter Notebooks on Azure Notebooks to quickly get started.
If you would like to use Batch Shipyard from the command line, this quickstart doc will provide step-by-step instructions. Although Batch Shipyard supports various types of workloads, for the following quickstart example, we will select from the following Deep Learning recipes to quickly get started:
- CNTK-CPU-OpenMPI
- Caffe-CPU
- Caffe2-CPU
- Chainer-CPU
- Keras+Theano-CPU
- MXNet-CPU
- TensorFlow-CPU
- Torch-CPU
The example MNIST training sample will be used on one Azure Batch compute node regardless of which Deep Learning framework you prefer for the following. Note that this quickstart guide focuses on Docker container execution but the commands are agnostic to which containers (Docker or Singularity) you are using - only the configuration changes with respect to which ecosystem you are using.
- Installation of Batch Shipyard to your local machine has been completed or you are using Batch Shipyard from within Azure Cloud Shell.
- Create a directory to hold your configuration files. For this quickstart
guide, create a directory named
config
. - Copy the sample configuration files from the Deep Learning framework recipe
of your choice to the
config
directory: - Edit the
config/credentials.yaml
file and populate it with your Azure Batch and Azure Storage credentials. If you do not have an Azure Batch account, you can create one via the Azure Portal, Azure CLI 2.0, or Azure PowerShell. You can create a standard general purpose Azure Storage account using any of the aforementioned methods similar to creating an Azure Batch account. - Edit the
config/config.yaml
file and edit the following settings:storage_account_settings
to link to the storage account named in step 4.
- In the main
batch-shipyard
directory (which should contain theshipyard
orshipyard.cmd
helper scripts if on Linux or Windows, respectively), run the following commands:
# NOTE: if you are on Windows, use shipyard.cmd instead of ./shipyard # create the compute pool ./shipyard pool add --configdir config # alternatively (if not on Windows), you can specify the configdir parameter # as an environment variable which some may find as more convenient SHIPYARD_CONFIGDIR=config ./shipyard pool add # ... wait for pool to allocate ... # add the training job and tail the output # if CNTK-CPU-OpenMPI, Caffe2-CPU, Chainer-CPU, Keras+Theano-CPU, MXNet-CPU, TensorFlow-CPU, or Torch-CPU ./shipyard jobs add --configdir config --tail stdout.txt # if Caffe-CPU ./shipyard jobs add --configdir config --tail stderr.txt
The --tail
option of the jobs add
command will stream the stderr or stdout
file to your local console which will provide you progress information about
your job.
Once you are finished interacting with your jobs, tasks and pool, you can remove them with the following commands:
# ... done interacting with jobs/tasks/pool ./shipyard jobs del --configdir config --wait ./shipyard pool del --configdir config
Step-by-step Tutorial
The From Scratch: Step-by-step guide will provide detailed steps on how to construct your own set of configuration files to execute on Batch Shipyard.
Commandline Usage Guide
Batch Shipyard Usage contains explanations for all of the actions available with commandline interface.
In-Depth Configuration Guide
It is recommended to review the full in-depth configuration guide for explanations of all of the configuration options available for Batch Shipyard.
Graphical Interfaces
You can also use the Azure Portal or Batch Labs to view more properties of your Azure Batch accounts, pools, nodes, jobs and tasks. You can view your Azure Storage accounts on Azure Portal or with Microsoft Azure Storage Explorer.