Reliability in Azure Storage Actions

Azure Storage Actions is a fully managed platform that automates data management tasks for Azure Blob Storage and Azure Data Lake Storage. You schedule storage tasks to run against your storage accounts. You can configure tasks to delete blobs, set blob tags, and change access tiers based on the age of your data.

When you use Azure, reliability is a shared responsibility. Microsoft provides a range of capabilities to support resiliency and recovery. You're responsible for understanding how those capabilities work within all of the services you use, and selecting the capabilities you need to meet your business objectives and uptime goals.

This article describes how to make Azure Storage Actions resilient to transient faults, configuration problems, availability zone failures, region-wide failures, and service maintenance. It also describes backup and restore options and key information about the Azure Storage Actions service-level agreement (SLA).

Production deployment recommendations for reliability

For production workloads, we recommend that you:

Reliability architecture overview

This section describes some of the important aspects of how the service works that are most relevant from a reliability perspective. The section introduces the logical architecture, which includes some of the resources and features that you deploy and use. It also discusses the physical architecture, which provides details on how the service works under the covers.

Logical architecture

When you use Azure Storage Actions, you create one or more storage task resources. Each storage task contains conditions that select objects and operations to perform on matching objects. To run a storage task against a storage account, you create a storage task assignment. The assignment is stored as part of the storage account resource and identifies the objects to target, when and how often the task runs, and where execution reports are stored. When a task run reaches the Completed status, the service writes an execution report to the location specified in the assignment.

You're responsible for configuring each assignment and assigning an appropriate Azure role to the storage task's managed identity.

Physical architecture

Azure Storage Actions has two planes that are relevant to reliability:

• Control plane: Requests flow through Azure Resource Manager to the Azure Storage Actions resource provider, which handles management operations and asynchronous provisioning workflows for storage tasks and assignments.

• Data plane: Includes the scheduler, workers that evaluate storage task conditions and perform the configured operations against target storage accounts, and reporting components. These components run on Azure Storage infrastructure.

Resilience to transient faults

Transient faults are short, intermittent failures in components. They occur frequently in a distributed environment like the cloud, and they're a normal part of operations. Transient faults correct themselves after a short period of time. It's important that your applications can handle transient faults, usually by retrying affected requests.

All cloud-hosted applications should follow the Azure transient fault handling guidance when they communicate with any cloud-hosted APIs, databases, and other components. For more information, see Recommendations for handling transient faults.

Transient faults can affect Azure Storage Actions in two main ways:

• Management operations: If you automate management of Storage Actions, such as deploying or modifying storage tasks, you're responsible for handling transient faults in your automation. Use bounded retries with an appropriate delay between attempts.

• Task execution: When a target storage account throttles operations, Azure Storage Actions backs off and retries after a delay. As long as the assignment continues to make progress, Storage Actions continues retrying for the lifetime of the workflow. If the assignment makes no progress listing or processing blobs for four hours, Storage Actions terminates the execution so that another assignment can be dispatched.

Azure Storage Actions processes only one assignment at a time for each target storage account. It self-regulates task execution so that application workloads on the target storage account take precedence. The execution scale depends on the transaction capacity available in the account. When less transaction capacity is available, a task run might be throttled and take longer to complete. For more information, see Scale dependence on transaction capacity available for the storage account.

Not all task execution failures are transient. For example, a task might fail because its managed identity lacks the required permissions, the number of requests exceeds a scale limit, or the target storage account blocks network access. You're responsible for monitoring task status, metrics, and dashboards and correcting these errors. An execution report is created only for a run that reaches the Completed status. Failed and skipped runs don't produce an execution report. Compare actual run times with the assignment schedule to identify runs that didn't occur. For information about identifying and resolving task errors, see Storage task troubleshooting.

Resilience to configuration problems

Storage task conditions determine which objects a task operates on, and an incorrect condition or operation can affect unintended objects. Microsoft runs each assignment according to its configuration. You're responsible for validating the assignment before you allow it to modify data, especially when it includes destructive or irreversible operations.

Use the following validation methods:

• Condition preview: While you compose task conditions, use the preview feature in the Azure portal or the Preview Actions REST API to evaluate a sample of up to 5,000 blobs. A preview returns immediate results and doesn't generate an execution report.

• Mock run: Before you enable the assignment to perform operations, create a mock run for final validation. A mock run evaluates every blob in the assignment scope without performing the configured operations. It runs asynchronously like a regular assignment and produces a CSV execution report. Review the report to verify which blobs match the conditions and which operations the task would perform.

Validation reduces the risk of unintended operations, but it doesn't replace data protection. Before you use storage tasks to delete blobs, enable soft delete for blobs on each target storage account. Soft delete allows you to recover deleted blobs during the configured retention period.

Review storage task assignments together with other data management policies that operate on the same blobs. Don't configure overlapping tasks or policies to perform conflicting operations. For example, a storage task delete operation can fail if a Blob Storage lifecycle management policy attempts to delete the same blob.

Resilience to availability zone failures

Availability zones are physically separate groups of datacenters within an Azure region. When one zone fails, services can fail over to one of the remaining zones.

The Azure Storage Actions control plane is zone-redundant. Microsoft automatically manages its distribution across multiple availability zones, and you don't select the zones. During an availability zone failure, you can continue to manage storage tasks and assignments.

The resilience of task execution depends on the redundancy configuration of each target storage account because the tasks operate on the data in those accounts. If task execution must continue during a zone failure, configure each target account with a zone-redundant storage option (ZRS, GZRS, or RA-GZRS).

The following example shows the Storage Actions control plane and a task that targets a zone-redundant account in a three-zone region. Both the control plane and the target account can continue operating if one zone becomes unavailable:

By contrast, an LRS target account isn't zone-redundant. In the following example, the control plane spans all three zones, and the target account is shown in availability zone 1 to illustrate an account that doesn't span zones. If the zone that hosts the target account becomes unavailable, the control plane remains available, but the task can't execute against the target account until the account recovers:

Requirements

• Region support: When you configure a storage task assignment, the storage task and target storage account must be in the same region. To make task execution resilient to zone failures, choose a region that supports both Azure Storage Actions and availability zones.

• Storage account configuration: Configure each target account with a zone-redundant storage option. Locally redundant storage (LRS), geo-redundant storage (GRS), and read-access geo-redundant storage (RA-GRS) don't provide zone redundancy in the primary region.

Considerations

Control-plane zone redundancy doesn't make task execution zone-resilient when a target storage account isn't available. Evaluate the redundancy configuration of every target account separately.

Cost

Zone redundancy for the Storage Actions control plane is automatic, and there's no additional Storage Actions charge to enable it. Standard usage charges for task executions, targeted objects, and invoked operations still apply. For more information, see Azure Storage Actions pricing.

The redundancy option that you select for each target storage account affects your Azure Storage charges. For more information, see Blob Storage pricing.

Configure availability zone support

Zone redundancy is automatic for the Storage Actions control plane. You don't configure zone redundancy separately for storage tasks or assignments.

To make task execution resilient to availability zone failures, select a zone-redundant storage option when you create each target storage account, or change the redundancy configuration of an existing account.

Behavior when all zones are healthy

This section describes what to expect when the Storage Actions control plane is available, target storage accounts use a zone-redundant storage option, and all availability zones are operational.

• Cross-zone operation: Microsoft distributes Storage Actions control-plane infrastructure across availability zones. Task runs operate against each target storage account, and Azure Storage transparently distributes requests across the infrastructure that hosts the account.

• Cross-zone data replication: Storage Actions doesn't store or replicate the blob data that tasks process.

  Azure Storage synchronously replicates data in accounts that use a zone-redundant storage option across availability zones in the primary region. Azure Storage doesn't complete a write operation until it replicates the data across the zones used by the account.

Behavior during a zone failure

This section describes what to expect when the Storage Actions control plane is available, target storage accounts use a zone-redundant storage option, and there's an outage in one availability zone.

• Detection and response: Microsoft detects and responds to failures that affect the Storage Actions control plane.

  For target accounts that use a zone-redundant storage option, Azure Storage automatically routes task execution and storage operations to healthy zones. You don't need to initiate failover.

• Notification: Microsoft doesn't automatically notify you when a zone is down. However, you can use Azure Service Health to understand the overall health of the service, including any zone failures, and you can set up Service Health alerts to notify you of problems.

• Active requests: Storage tasks that target available zone-redundant accounts continue to make progress. Tasks that target an unavailable account might fail.

  Pending run-once assignments might still be picked up. Runs that were interrupted or failed might require you to manually reschedule them, depending on where processing stopped.

• Expected data loss: Storage Actions doesn't store a separate copy of your blob data. Data loss expectations depend on the redundancy configuration of each target account. When an account uses a zone-redundant storage option, Azure Storage synchronously replicates its data across availability zones in the primary region, so a zone failure isn't expected to cause data loss for successfully completed writes.

• Expected downtime: The Storage Actions control plane remains available during a zone failure.

  Task execution continues as long as the target storage account remains accessible. If the account becomes unavailable, such as an LRS storage account in an affected zone, then affected runs might fail.

• Redistribution: Microsoft manages the response and redistribution of the Storage Actions control plane.

  Azure Storage automatically routes operations for accounts that use a zone-redundant storage option to healthy zones. No customer action is required.

Zone recovery

When the failed zone recovers, Microsoft restores normal control-plane distribution. Azure Storage restores normal operation for zone-redundant accounts. Scheduled task assignments continue according to their schedules.

Recurring assignments resume on their configured schedules. Pending run-once assignments might still be picked up. Check the status of runs affected by the outage. Depending on where processing stopped, you might need to reschedule interrupted or failed runs.

Test for zone failures

Microsoft manages zone distribution, failover, and recovery for the Storage Actions control plane and for zone-redundant storage accounts. You don't need to test platform zone failures. Test your operational procedures by verifying that you can identify failed task operations and safely run them again.

Resilience to region-wide failures

The resilience of storage task execution during a region-wide failure depends on the redundancy configuration of each target storage account. If a target account uses a geo-redundant storage option (GRS, GZRS, RA-GRS, and RA-GZRS), the storage account can fail over to its paired region and future storage task runs can continue in the new primary region.

Geo-redundant target storage accounts

For a geo-redundant storage account, the region that currently accepts write operations is the primary region. Azure Storage asynchronously replicates data to the account's paired region, which is the secondary region.

Storage tasks operate against the account in its primary region. After the account fails over, the previous secondary region becomes the new primary region, and future storage task runs automatically execute there. Azure Storage Actions doesn't require a separate failover action.

Requirements

• Region support: The target storage account must be in an Azure region that has a paired region.

• Storage account configuration: Configure the target account to use a geo-redundant storage option.

Cost

Geo-redundant storage configurations incur charges for replication and storage in the secondary region. For more information, see Blob Storage pricing.

Configure multiregion support

Select a geo-redundant storage option when you create the target storage account, or change the redundancy configuration of an existing account.

If you create a storage task assignment while the target account uses a non-geo-redundant option and later convert the account to GRS, GZRS, RA-GRS, or RA-GZRS, update the assignment before you initiate customer-managed account failover. You can change any assignment property. The update causes the Storage Actions control plane to detect the account's new geo-redundant configuration.

Behavior when all regions are healthy

This section describes what to expect when a target storage account uses a geo-redundant storage option and both the primary and secondary regions are operational.

• Cross-region operation: During normal operation, the storage task definition, assignment, and target storage account are in the primary region. The secondary region doesn't process storage task operations.

• Cross-region data replication: Azure Storage Actions doesn't independently replicate the blob data that tasks process.

  Azure Storage asynchronously replicates data from the primary region to the secondary region. Because replication is asynchronous, recent changes might not yet exist in the secondary region.

Behavior during a region failure

This section describes what to expect when the primary region of a target storage account that uses a geo-redundant storage option is unavailable.

• Detection and response: Storage account failover can be customer-managed or Microsoft-managed. Microsoft-managed failover is reserved for exceptional circumstances and might occur only after an extended outage, so don't rely on it for your disaster recovery plan. Follow the Blob Storage failover guidance to decide whether and when to initiate customer-managed failover.

  Azure Storage Actions doesn't require a separate failover action. However, if you created the assignment while the target account used LRS and later converted the account to GRS or GZRS, update the assignment before you initiate customer-managed account failover. For more information, see Configure multiregion support.

• Notification: Microsoft doesn't automatically notify you when a region is down. However, you can use Azure Service Health to understand the overall health of the service, including any region failures, and you can set up Service Health alerts to notify you of problems.

• Active requests: During storage account failover, the primary and secondary storage account endpoints are temporarily unavailable. Storage task runs that are in progress might fail. After failover, check the status of affected task runs and rerun failed or incomplete operations as appropriate.

  For a recurring assignment, if the previous iteration remains in progress when the next iteration is scheduled, Storage Actions skips the new iteration. The next iteration runs at its designated trigger time after the previous iteration finishes.

• Expected data loss: Azure Storage Actions doesn't maintain a separate copy of your blob data.

  During an unplanned failover, recent changes can be lost if they haven't replicated to the secondary region. A task that fails while in progress might also have completed operations on only some of its target objects. After failover, check the status of affected task runs and rerun failed or incomplete operations as appropriate. For details about storage account recovery points and data loss, see Expected data loss during a Blob Storage region failure.

• Expected downtime: Before failover, storage tasks can't make progress while the target storage account is unavailable.

  During failover, the storage account endpoints remain unavailable, so task runs can't make progress and might fail.

  After failover completes, the Storage Actions data-plane components in the storage account's new primary region process future task assignment runs. The storage task resource remains in its original region, so reporting for the task definition might remain affected while that region is unavailable. Reporting for other sections of the resource continues to work.

  The total interruption includes the time to detect the outage, decide whether to initiate failover, and complete the storage account failover.

  If no failover occurs, task execution remains unavailable until the original primary region recovers.

  Management operations remain available during a region failure.

• Redistribution: Azure Storage updates the storage account endpoints as failover completes. Future task assignment runs automatically execute against the account in the new primary region. You don't need to reconfigure the storage task or assignment.

Region recovery

The failback process depends on the storage account failover type. Follow the Blob Storage region recovery guidance to fail back the target storage account. Azure Storage Actions doesn't require additional configuration after storage account failback.

Test for region failures

To test your recovery process without the data loss associated with an unplanned failover, create representative storage task assignments against a nonproduction storage account that uses a geo-redundant storage option, and then initiate a customer-managed planned failover. Review any in-progress run and verify that a future scheduled run executes against the account in the new primary region. For failover testing guidance, see Test for region failures in Azure Blob Storage.

Custom multiregion solutions for resiliency

Built-in region failover support for Azure Storage Actions is available only when the target storage account is in a paired region and uses a geo-redundant storage option.

If your multiregion storage solution uses independent storage accounts in each region instead of storage account failover, you can deploy a storage task in the same region as each target account and create an assignment for each account. You're responsible for replicating data between the accounts, keeping task definitions and assignments synchronized, and controlling which assignments run during normal operation and failover. Ensure that assignments in different regions don't perform conflicting operations on replicated data.

Backup and restore

To recover your Azure Storage Actions configuration after accidental deletion, define your storage tasks and assignments as infrastructure as code (IaC), and store the definitions in source control. Include task conditions and operations, schedules, target storage account scopes, report container settings, managed identities, and required role assignments. You can deploy a storage task by using Bicep, and you can define storage task assignments in Bicep.

Storage tasks and assignments consist of configuration and don't contain the customer data that tasks process. Use the IaC approach described previously to recover this configuration. You're responsible for protecting and recovering the data in your target storage accounts separately. For information about Azure Storage data protection options, see Data protection overview.

Test your recovery procedure in a nonproduction environment. Before you enable recreated assignments, use condition previews and mock runs to verify that the task targets the intended blobs. For more information, see Best practices for using storage tasks.

Resilience to service maintenance

Microsoft regularly applies service updates and performs other maintenance. The Azure platform handles these activities automatically, ensuring that maintenance is seamless and transparent to you. No downtime is expected during maintenance events unless you've been advised through Azure Service Health planned maintenance.

Service-level agreement

Azure Storage Actions doesn't offer an availability service-level agreement (SLA).

Related content

• Azure Storage Actions overview
• Best practices for using storage tasks
• Monitor Azure Storage Actions
• Reliability in Azure Blob Storage
• Reliability in Azure