Batch Shipyard Quickstart
Choose your speed:
If you would like to use Batch Shipyard from the command line, this quickstart doc will provide step-by-step instructions. Although Batch Shipyard supports any type of Batch processing or HPC workload, for the following quickstart guide, we will select from the following Deep Learning recipes as an example workflow:
The example MNIST training sample will be used on one Azure Batch compute node regardless of which Deep Learning framework you prefer for the following. Note that this quickstart guide focuses on Docker container execution but the commands are agnostic to which containers (Docker or Singularity) you are using - only the configuration changes with respect to which ecosystem you are using.
- Installation of Batch Shipyard to your local machine has been completed or you are using Batch Shipyard from within Azure Cloud Shell.
- Create a directory to hold your configuration files. For this quickstart
guide, create a directory named
- Copy the sample configuration files from the Deep Learning framework recipe
of your choice to the
- Edit the
config/credentials.yamlfile and populate it with your Azure Batch and Azure Storage credentials. If you do not have an Azure Batch account, you can create one via the Azure Portal, Azure CLI 2.0, or Azure PowerShell. You can create a standard general purpose Azure Storage account using any of the aforementioned methods similar to creating an Azure Batch account.
- Edit the
config/config.yamlfile and edit the following settings:
storage_account_settingsto link to the storage account named in step 4.
- In the main
batch-shipyarddirectory (which should contain the
shipyard.cmdhelper scripts if on Linux or Windows, respectively), run the following commands:
# change working directory to the config directory cd config # create the compute pool # NOTE: if you are on Windows, use ..\shipyard.cmd instead of ../shipyard ../shipyard pool add # ... wait for pool to allocate ... # add the training job and tail the output # if CNTK-CPU-OpenMPI, Caffe2-CPU, Chainer-CPU, Keras+Theano-CPU, MXNet-CPU, TensorFlow-CPU, or Torch-CPU ../shipyard jobs add --tail stdout.txt # if Caffe-CPU ../shipyard jobs add --tail stderr.txt
--tail option of the
jobs add command will stream the stderr or stdout
file to your local console which will provide you progress information about
Once you are finished interacting with your jobs, tasks and pool, you can remove them with the following commands:
# after you are done interacting with jobs/tasks/pool ../shipyard jobs del --wait ../shipyard pool del
- Linux, Mac or WSL machine with network access
gitand Python3 is installed
- Comfortable with Linux commandline
- Have an active Azure subscription
- Understand how to use the Azure Portal
Spin up FTL drives:
- Create Batch account if you do not have one
- Retrieve the Batch account key and service URL
- Create Storage account if you do not have one (preferably in the same region as the Batch account)
- Retrieve the Storage account key
git clone https://github.com/Azure/batch-shipyard.git cd batch-shipyard ./install.sh export SHIPYARD_CONFIGDIR=recipes/TensorFlow-CPU/config nano $SHIPYARD_CONFIGDIR/credentials.yaml # edit required properties in file and save ./shipyard pool add ./shipyard jobs add --tail stdout.txt
# clean up ./shipyard jobs del -y --wait ./shipyard pool del -y unset SHIPYARD_CONFIGDIR
The From Scratch: Step-by-step guide will provide detailed steps on how to construct your own set of configuration files to execute on Batch Shipyard.
If you are interested in executing Batch Shipyard on a Jupyter environment without having to install anything, you can utilize the community contributed Jupyter Notebooks that you can instantiate on Azure Notebooks to quickly get started.
Note: Because these notebooks are community contributed and supported, they may not be up-to-date with the latest Batch Shipyard version and may not work properly.
Commandline Usage Guide
Batch Shipyard Usage contains explanations for all of the actions available with commandline interface.
In-Depth Configuration Guide
It is recommended to review the full in-depth configuration guide for explanations of all of the configuration options available for Batch Shipyard.
You can also use the Azure Portal or Batch Labs to view more properties of your Azure Batch accounts, pools, nodes, jobs and tasks. You can view your Azure Storage accounts on Azure Portal or with Microsoft Azure Storage Explorer.