Remote Filesystems with Batch Shipyard
The focus of this article is to explain how to provision a standalone file server or storage cluster for use as a shared file system.
Notabene: Creating a standalone remote filesystem with Batch Shipyard is independent of Azure Batch and all Batch related functionality in Batch Shipyard. You may create a filesystem in Azure with Batch Shipyard and manage it with the tooling in Batch Shipyard or in the Azure Portal without having to create an Azure Batch account. However, if you do want to use this filesystem with the Azure Batch service, many convenience features are present to make using such filesystems relatively painless with your jobs and tasks.
The ability to have a shared file system that all compute nodes can access is vital to many HPC and batch processing workloads. Azure Batch provides simple mechanisms for scaling a workload, but most non-trivial compute tasks often require access to shared data, whether that is as simple as configuration files to more complicated scenarios such as shared model files or shared output for validation.
Some scenarios are natively handled by Azure Batch through resource files or through Batch Shipyard with data ingress. However, there are many scenarios where this is insufficient and only a real shared file system will suffice.
Batch Shipyard includes support for automatically provisioning an entire file server with attached disks or a GlusterFS storage cluster for both scale up and scale out scenarios.
- Support for multiple file server types: NFS or GlusterFS
- Support for SMB/CIFS on top of NFS or GlusterFS mountpoints to enable file sharing to Windows clients
- Automatic provisioning of all required resources for the storage cluster including managed disks, virtual networks, subnets, network interfaces, IP addresses and DNS labels, network security groups, availability sets, virtual machines and extensions
- Suite of commandline tooling for cluster management including zero-downtime disk array expansion and storage cluster resize (scale out), status queries tailored to file server types and hassle-free SSH for administration
- Support for cluster suspension (deallocation) and restart
- Support for definining and managing multiple clusters simultaneously
- Support for btrfs, XFS, ext4, ext3 and ext2 filesystems
- Automatic disk array construction via RAID-0 through btrfs or Linux software RAID (mdadm)
- Consistent private IP address allocation per virtual machine and virtual machine to disk mapping
- Automatic network security rule configuration based on file server type, if requested
- Automatic placement in an availability set for GlusterFS virtual machines
- Support for accelerated networking
- Automatic boot diagnostics enablement and support for serial console access
- Automatic SSH keypair provisioning and setup for all file servers in storage cluster
- Configuration-driven data ingress support via scp and rsync+ssh, including concurrent multi-node parallel transfers with GlusterFS storage clusters
Azure Batch Integration Features
- Automatic linking between Azure Batch pools (compute nodes) and Batch Shipyard provisioned remote filesystems
- Support for mounting multiple disparate Batch Shipyard provisioned remote filesystems concurrently to the same pool and compute nodes
- Automatic failover for HA GlusterFS volume file lookups (compute node client mount) through remote filesystem deployment walk to find disparate upgrade and fault domains of the GlusterFS servers
- Automatic volume mounting of remote filesystems into a Docker container executed through Batch Shipyard
A Batch Shipyard provisioned remote filesystem is built on top of different resources in Azure. These resources are from networking, storage and compute. To more readily explain the concepts that form a Batch Shipyard standalone storage cluster, let's start with a high-level conceptual layout of all of the components and possible interacting actors.
+------------------------------------------------------------+ | | | +-------------------------------+ | | | | | | | +---------------------------+ | | | | | +-------------+ | | | | | | | Data | Data | | | +--------------------+ | | | | | Disk | Disk | | | | Subnet B | | | | | Virtual | 00 | 01 | | | | 10.1.0.0/16 | | | | | Machine A +-------------+ | | | +----------------+ | | | | | | Data | Data | | | | | | | | | | | GlusterFS | Disk | Disk | | | | | Azure Batch | | | | | | Server 0 | 02 | .. | | | | | Compute Node X | | | | | | +------+------+ | | | | | | | | | | RAID-0 Array | | | +------------+ | | | +--------------+ Mount | | +-----------+ +------------+ | | | Private IP | | | | | External <--------------> Public IP | | Private IP <-------> 10.1.0.4 | | | | | Client | Brick | | | 18.104.22.168 | | 10.0.0.4 | | | +------------+---+ | | | Mount | Data | | +-----------+--+-----^------+ | | | | | (if allowed) | | | | | +--------------------+ | +------^-------+ | | | | | | | | | | +--------------------+ | | | | +-----------+--+-----v------+ | | | | +----------------------> Public IP | | Private IP | | | +------------+---+ | | Brick Data | | | 22.214.171.124 | | 10.0.0.5 <-------> Private IP | | | | | | +---------------------------+ | | | 10.2.1.4 | | | | | | | +-------------+ | | | +------------+ | | | | | | | Data | Data | | | | | | | | | | | | Disk | Disk | | | | | Azure Virtual | | | | | | Virtual | 00 | 01 | | | | | Machine Y | | | | | | Machine B +-------------+ | | | | | | | | | | | Data | Data | | | | +----------------+ | | | | | GlusterFS | Disk | Disk | | | | Subnet C | | | | | Server 1 | 02 | .. | | | | 10.2.1.0/24 | | | | | +------+------+ | | +--------------------+ | | | | RAID-0 Array | | | | | +---------------------------+ | | | | Subnet A | | | | 10.0.0.0/24 | | | +-------------------------------+ | | Virtual Network | | 10.0.0.0/8 | +------------------------------------------------------------+
The base layer for all of the resources within a standalone provisioned
filesystem is an Azure Virtual Network. This virtual network can be shared
amongst other network-level resources such as network interfaces. The virtual
network can be "partitioned" into sub-address spaces through the use of
subnets. In the example above, we have three subnets where
Subnet A 10.0.0.0/24 hosts the GlusterFS infrastructure,
Subnet B 10.1.0.0/16 contains a pool of Azure Batch compute nodes, and
Subnet C 10.2.1.0/24 contains other Azure virtual machines. No resource
Subnet B or
Subnet C is required for the Batch Shipyard provisioned
filesystem to work, it is just to illustrate that other resources can
access the filesystem within the same virtual network if configured to do
If your configuration is NFS instead, then the above illustration would be
simplified to a single virtual machine (
Virtual Machine A in
Subnet A 10.0.0.0/24) only. However, other non-GlusterFS specific
concepts still apply regarding other Azure resources.
The storage cluster depicted is a 2-node GlusterFS distributed file system with attached disks. Each node has a number of managed disks attached to it arranged in a RAID-0 disk array. The array (and ultimately the filesystem sitting on top of the disk array) holds the GlusterFS brick for the virtual machine. Because the managed disks are backed to Azure Storage LRS (locally redundant storage), there is no practical need to for mirroring or striping at this level.
For each Azure Virtual Machine hosting a brick of the GlusterFS server, two IP addresses are provisioned, in addition to a fully qualified domain name that resolves to the public IP address. The public IP address allows for external clients to SSH into the virtual machine for diagnostic, maintenance, debugging, data transfer and other tasks. The SSH inbound network security rule can be tightened according to your requirements in the configuration file. Additionally, inbound network rules can be applied to allow the filesystem to be mounted externally as well. The private IP address is an address that is only internally routable on the virtual network. Resources that are on the virtual network will/should use these IP addresses for access to the filesystem.
And finally, when provisioning GlusterFS servers, rather than NFS servers, Batch Shipyard automatically places the virtual machines in an availability set along with maximally spreading virtual machines across update and fault domains. Single instance NFS servers will not be placed in an availbility set, however, if using a premium storage virtual machine size along with all premium disks, then you may qualify for single instance SLA.
In order to create storage clusters, there are a few configuration changes that must be made to enable this feature.
Azure Active Directory Authentication Required
Azure Active Directory authentication is required to create storage clusters. Additionally, if leveraging integration features with Batch pools, then the virtual network shared between the storage cluster and the Batch pool must be the same.
Your service principal requires at least
Contributor role permission in
order to create the resources required for the storage cluster.
The following is an example for Azure Active Directory authentication in the credentials configuration.
credentials: # management settings required with aad auth management: aad: # valid aad settings (or at the global level) subscription_id: # subscription id required # ... other required settings
Please see this page for a full explanation of each remote filesystem and storage cluster configuration option.
The following will step through and explain the major configuration portions. The RemoteFS configuration file has four top-level properties:
remote_fs: resource_group: # resource group for all resources, can be overridden location: # Azure region for all storage cluster resources managed_disks: # disk settings storage_clusters: # storage cluster settings
It is important to specify a location that is appropriate for your storage cluster and if joining to a Batch pool, must be within the same region.
Managed Disks Configuration
managed_disks section describes disks to be created for use with
managed_disks: resource_group: # optional resource group just for the disks # premium disks have provisioned IOPS and can provide higher throughput # and lower latency with consistency. If selecting premium disks, # you must use a premium storage compatible vm_size. premium: true disk_size_gb: # size of the disk, please see Azure Manage Disk docs disk_names: - # list of disk names
Storage Cluster Configuration
storage_clusters section describes one or more storage clusters to
create and manage.
storage_clusters: # unique name of the storage cluster, this is the "storage cluster id" mystoragecluster: resource_group: # optional resource group just for the storage cluster hostname_prefix: # hostname prefix and prefix for all resources created ssh: # ssh settings public_ip: enabled: # true or false for enabling public ip. If public ip is not # enabled, then it is only accessible via the private network. static: # true or false if public ip should be static virtual_network: # virtual network settings. If joining to a Batch pool, ensure that # the virtual network resides in the same region and subscription # as the Batch account. It is recommended that the storage cluster # is in a different subnet than that of the Batch pool. network_security: # network security rules, only "ssh" is required. All other settings # are for external access and not needed for joining with Batch pools # as traffic remains private/internal only for that scenario. file_server: type: # nfs or glusterfs mountpoint: # the mountpoint on the storage cluster nodes mount_options: - # fstab mount options in list format server_options: glusterfs: # this section is only needed for "glusterfs" type transport: tcp # tcp is only supported for now volume_name: # name of the gluster volume volume_type: # type of volume to create. This must be compatible # with the number of bricks. # other key:value pair tuning options can be specified here nfs: # this section is only needed for "nfs" type # key:value (where value is a list) mapping of /etc/exports options samba: # optional section, if samba server setup is required vm_count: # 1 for nfs, 2+ for glusterfs vm_size: # Azure VM size to use. This must a premium storage compatible # size if using premium managed disks. fault_domains: # optional tuning for the number of fault domains accelerated_networking: # true to enable accelerated networking vm_disk_map: # cardinal mapping of VMs to their disk arrays, e.g.: '0': # note that this key must be a string disk_array: - # list of disks in this disk array filesystem: # filesystem to use, see documentation on available kinds raid_level: # this should be set to 0 if disk_array has more than 1 # disk. If disk_array has only 1 disk, then this property # should be omitted. prometheus: # optional monitoring settings
Batch Pool Integration
If you wish to use your storage cluster in conjunction with a Batch pool, then you will need to modify the credentials, global, pool, and jobs configuration files.
Azure Active Directory authentication for Batch is required for joining a storage cluster with a Batch pool.
credentials: # batch aad settings required if monitoring batch pools batch: aad: # valid aad settings (or at the global level) account_service_url: # valid batch service url resource_group: # batch account resource group management: aad: # valid aad settings (or at the global level) subscription_id: # subscription id required # ... other required settings
You must specify the storage cluster under
global_resources such that
bound Batch pools will provision the correct software to mount the storage
# ... other global configuration settings global_resources: # ... other global resources settings volumes: shared_data_volumes: mystoragecluster: # this name must match exactly with the storage cluster # id from the RemoteFS configuration that you intend # to link volume_driver: storage_cluster container_path: # that path to mount this storage cluster in # containers when jobs/tasks execute mount_options: # optional fstab mount options bind_options: # optional bind options to the container, default is "rw"
The pool configuration file must specify a valid virtual network. Because of this requirement, you must use Azure Active Directory authentication for Batch.
pool_specification: # ... other pool settings virtual_network: # virtual network settings must have the same virtual network as the # RemoteFS configuration. However, it is strongly recommended to have # the Batch pool compute nodes reside in a different subnet.
The jobs configuration must refer to the shared data volume such that it understands to mount the volume into the container for the task or all tasks under a job.
job_specifications: - id: # job id shared_data_volumes: # this name must match exactly with the global_resources # shared_data_volumes name. If specified at the job level, then all # tasks under the job will mount this volume. - mystoragecluster # ... other job settings tasks: - shared_data_volumes: - # storage cluster can be specified for fine grained control at # a per task level # ... other task settings
The workflow for creating a storage cluster is first creating the managed disks, then the storage cluster itself. Below is an example command usage.
# create managed disks shipyard fs disks add # create storage cluster shipyard fs cluster add <storage-cluster-id>
If there were provisioning errors during
fs cluster add but the provisioning
had not yet reached the VM creation phase, you can remove the orphaned
# clean up a failed provisioning that did not reach VM creation shipyard fs cluster del <storage-cluster-id> --generate-from-prefix
If any VMs were created and the provisioning failed after that, you can
delete normally (without
After there is no need for the storage cluster, you can either suspend the storage cluster or delete it. Note that suspending a glusterfs storage cluster is considered experimental.
# suspend a storage cluster shipyard fs cluster suspend <storage-cluster-id> # restart a suspended storage cluster shipyard fs cluster start <storage-cluster-id> # delete a storage cluster shipyard fs cluster del <storage-cluster-id>
Please see this page for detailed documentation
fs command usage.
Usage with Batch Pools
If joining to a Batch pool, the storage cluster must be created first.
After which, commands such as
pool add and
jobs add should work
normally with the storage cluster mounted into containers if configuration
Sample recipes for RemoteFS storage clusters of NFS and GlusterFS types can be found in the recipes area.