Batch Shipyard Quickstart
If you are interested in executing Deep Learning workloads on Batch Shipyard and want to jump in without having to install anything, you can utilize the Deep Learning Jupyter Notebooks on Azure Notebooks to quickly get started.
If you would like to use Batch Shipyard from the command line, this quickstart doc will provide step-by-step instructions. Although Batch Shipyard supports various types of workloads, for the following quickstart example, we will select from the following Deep Learning recipes to quickly get started:
- CNTK-CPU-OpenMPI
- Caffe-CPU
- Caffe2-CPU
- Chainer-CPU
- Keras+Theano-CPU
- MXNet-CPU
- TensorFlow-CPU
- Torch-CPU
The example MNIST training sample will be used on one Azure Batch compute node regardless of which Deep Learning framework you prefer for the following. Note that this quickstart guide focuses on Docker container execution but the commands are agnostic to which containers (Docker or Singularity) you are using - only the configuration changes with respect to which ecosystem you are using.
- Installation of Batch Shipyard to your local machine has been completed or you are using Batch Shipyard from within Azure Cloud Shell.
- Create a directory to hold your configuration files. For this quickstart
guide, create a directory named
config
. - Copy the sample configuration files from the Deep Learning framework recipe
of your choice to the
config
directory: - Edit the
config/credentials.yaml
file and populate it with your Azure Batch and Azure Storage credentials. If you do not have an Azure Batch account, you can create one via the Azure Portal, Azure CLI 2.0, or Azure PowerShell. You can create a standard general purpose Azure Storage account using any of the aforementioned methods similar to creating an Azure Batch account. - Edit the
config/config.yaml
file and edit the following settings:storage_account_settings
to link to the storage account named in step 4.
- In the main
batch-shipyard
directory (which should contain theshipyard
orshipyard.cmd
helper scripts if on Linux or Windows, respectively), run the following commands:
# change working directory to the config directory cd config # create the compute pool # NOTE: if you are on Windows, use ..\shipyard.cmd instead of ../shipyard ../shipyard pool add # ... wait for pool to allocate ... # add the training job and tail the output # if CNTK-CPU-OpenMPI, Caffe2-CPU, Chainer-CPU, Keras+Theano-CPU, MXNet-CPU, TensorFlow-CPU, or Torch-CPU ../shipyard jobs add --tail stdout.txt # if Caffe-CPU ../shipyard jobs add --tail stderr.txt
The --tail
option of the jobs add
command will stream the stderr or stdout
file to your local console which will provide you progress information about
your job.
Once you are finished interacting with your jobs, tasks and pool, you can remove them with the following commands:
# after you are done interacting with jobs/tasks/pool ../shipyard jobs del --wait ../shipyard pool del
Step-by-step Tutorial
The From Scratch: Step-by-step guide will provide detailed steps on how to construct your own set of configuration files to execute on Batch Shipyard.
Commandline Usage Guide
Batch Shipyard Usage contains explanations for all of the actions available with commandline interface.
In-Depth Configuration Guide
It is recommended to review the full in-depth configuration guide for explanations of all of the configuration options available for Batch Shipyard.
Graphical Interfaces
You can also use the Azure Portal or Batch Labs to view more properties of your Azure Batch accounts, pools, nodes, jobs and tasks. You can view your Azure Storage accounts on Azure Portal or with Microsoft Azure Storage Explorer.