[ARVADOS] updated: 2.1.0-249-gc1a84bf6f
Git user
git at public.arvados.org
Thu Jan 7 18:41:37 UTC 2021
Summary of changes:
doc/admin/spot-instances.html.textile.liquid | 23 ++++++++++++++-------
lib/cloud/azure/azure.go | 30 +++++++++++-----------------
2 files changed, 28 insertions(+), 25 deletions(-)
via c1a84bf6f6c570cc632a5ba8c6406543e2206e3a (commit)
from 045e3127cb48845c7d988d01488c055f02ae2ec3 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit c1a84bf6f6c570cc632a5ba8c6406543e2206e3a
Author: Ward Vandewege <ward at curii.com>
Date: Thu Jan 7 13:40:53 2021 -0500
16106: update Azure preemptible node code after real world testing. Add
documentation.
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/admin/spot-instances.html.textile.liquid b/doc/admin/spot-instances.html.textile.liquid
index 7f49d6961..7b4789295 100644
--- a/doc/admin/spot-instances.html.textile.liquid
+++ b/doc/admin/spot-instances.html.textile.liquid
@@ -12,11 +12,11 @@ SPDX-License-Identifier: CC-BY-SA-3.0
This page describes how to enable preemptible instances. Preemptible instances typically offer lower cost computation with a tradeoff of lower service guarantees. If a compute node is preempted, Arvados will restart the computation on a new instance.
-Currently Arvados supports preemptible instances using AWS spot instances.
+Currently Arvados supports preemptible instances using AWS and Azure spot instances.
h2. Configuration
-To use preemptible instances, set @UsePreemptibleInstances: true@ and add entries to @InstanceTypes@ with @Preemptible: true@ to @config.yml at . Typically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. The @Price@ for preemptible instances is the maximum bid price, the actual price paid is dynamic and may be lower. For example:
+To use preemptible instances, set @UsePreemptibleInstances: true@ and add entries to @InstanceTypes@ with @Preemptible: true@ to @config.yml at . Typically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. The @Price@ for preemptible instances is the maximum bid price, the actual price paid is dynamic and will likely be lower. For example:
<pre>
Clusters:
@@ -42,11 +42,17 @@ Clusters:
When @UsePreemptibleInstances@ is enabled, child containers (workflow steps) will automatically be made preemptible. Note that because preempting the workflow runner would cancel the entire workflow, the workflow runner runs in a reserved (non-preemptible) instance.
-If you are using "arvados-dispatch-cloud":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html no additional configuration is required.
+No additional configuration is required, "arvados-dispatch-cloud":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html will now start preemptable instances where appropriate.
+
+h3. Cost Tracking
+
+Preemptable instances prices are declared at instance request time and defined by the maximum price that the user is willing to pay per hour. By default, this price is the same amount as the on-demand version of each instance type, and this setting is the one that @arvados-dispatch-cloud@ uses for now, as it doesn't include any pricing data to the spot instance request.
+
+For AWS, the real price that a spot instance has at any point in time is discovered at the end of each usage hour, depending on instance demand. For this reason, AWS provides a data feed subscription to get hourly logs, as described on "Amazon's User Guide":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-data-feeds.html.
h2. Preemptible instances on AWS
-For general information, see "using Amazon EC2 spot instances":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html .
+For general information, see "using Amazon EC2 spot instances":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html.
h3. Permissions
@@ -58,9 +64,12 @@ BaseHTTPError: AuthFailure.ServiceLinkedRoleCreationNotPermitted: The provided c
The account needs to have a service linked role created. This can be done by logging into the AWS account, go to _IAM Management_ → _Roles_ and create the @AWSServiceRoleForEC2Spot@ role by clicking on the @Create@ button, selecting @EC2@ service and @EC2 - Spot Instances@ use case.
-h3. Cost Tracking
+h2. Preemptible instances on Azure
+
+For general information, see "Use Spot VMs in Azure":https://docs.microsoft.com/en-us/azure/virtual-machines/spot-vms.
-Amazon's Spot instances prices are declared at instance request time and defined by the maximum price that the user is willing to pay per hour. By default, this price is the same amount as the on-demand version of each instance type, and this setting is the one that @arvados-dispatch-cloud@ uses for now, as it doesn't include any pricing data to the spot instance request.
+When starting preemptible instances on Azure, Arvados configures the eviction policy to 'delete', with max price set to '-1'. This has the effect that preemptible VMs will not be evicted for pricing reasons. The price paid for the instance will be the current spot price for the VM type, up to a maximum of the price for a standard, non-spot VM of that type.
-The real price that a spot instance has at any point in time is discovered at the end of each usage hour, depending on instance demand. For this reason, AWS provides a data feed subscription to get hourly logs, as described on "Amazon's User Guide":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-data-feeds.html.
+Please note that Azure provides no SLA for preemptible instances. Even in this configuration, preemptable instances can still be evicted for capacity reasons. If that happens and a container is aborted, Arvados will try to restart it, subject to the usual retry rules.
+Spot pricing is not available on 'B-series' VMs, those should not be defined in the configuration file with the _Preemptible_ flag set to true. Spot instances have a separate quota pool, make sure you have sufficient quota available.
diff --git a/lib/cloud/azure/azure.go b/lib/cloud/azure/azure.go
index ad1f8c532..100d87c33 100644
--- a/lib/cloud/azure/azure.go
+++ b/lib/cloud/azure/azure.go
@@ -491,21 +491,6 @@ func (az *azureInstanceSet) Create(
}
}
- priority := compute.Regular
- evictionPolicy := compute.Deallocate
- var billingProfile compute.BillingProfile
- var maxPrice float64
- if instanceType.Preemptible {
- priority = compute.Spot
- evictionPolicy = compute.Delete
- // Setting maxPrice to -1 is the equivalent of paying spot price, up to the
- // normal price. This means the node will not be pre-empted for price
- // reasons. It may still be pre-empted for capacity reasons though. And
- // Azure offers *no* SLA on spot instances.
- maxPrice = -1
- billingProfile = compute.BillingProfile{MaxPrice: &maxPrice}
- }
-
vmParameters := compute.VirtualMachine{
Location: &az.azconfig.Location,
Tags: tags,
@@ -514,9 +499,6 @@ func (az *azureInstanceSet) Create(
VMSize: compute.VirtualMachineSizeTypes(instanceType.ProviderType),
},
StorageProfile: storageProfile,
- Priority: priority,
- EvictionPolicy: evictionPolicy,
- BillingProfile: &billingProfile,
NetworkProfile: &compute.NetworkProfile{
NetworkInterfaces: &[]compute.NetworkInterfaceReference{
{
@@ -546,6 +528,18 @@ func (az *azureInstanceSet) Create(
},
}
+ var maxPrice float64
+ if instanceType.Preemptible {
+ // Setting maxPrice to -1 is the equivalent of paying spot price, up to the
+ // normal price. This means the node will not be pre-empted for price
+ // reasons. It may still be pre-empted for capacity reasons though. And
+ // Azure offers *no* SLA on spot instances.
+ maxPrice = -1
+ vmParameters.VirtualMachineProperties.Priority = compute.Spot
+ vmParameters.VirtualMachineProperties.EvictionPolicy = compute.Delete
+ vmParameters.VirtualMachineProperties.BillingProfile = &compute.BillingProfile{MaxPrice: &maxPrice}
+ }
+
vm, err := az.vmClient.createOrUpdate(az.ctx, az.azconfig.ResourceGroup, name, vmParameters)
if err != nil {
// Do some cleanup. Otherwise, an unbounded number of new unused nics and
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list