Configuring Databricks automated tasks

Overview

Databricks automated tasks let you configure scheduled jobs that automatically resubmit failed Databricks workflow job runs. You define filter conditions and a schedule — Turbo360 identifies matching failed runs and resubmits them based on the criteria you set.

Business value

In Databricks workflows where tasks are chained, a single run failure can cascade and disrupt the entire pipeline. Automated tasks eliminate the need to monitor failures manually and resubmit them one by one, reducing recovery time and ensuring workflow continuity across your Business Applications.

How it works

When an automated task runs, Turbo360 queries the Databricks workflow job for failed runs within the configured time window and applies your filter conditions to select which runs to resubmit. Only runs that match all specified filters are resubmitted. If no filters are configured, all failed workflow job runs within the time window are resubmitted.

Turbo360 then triggers a resubmit action for each selected run and records the outcome in the task history. You can review results, completed counts, and any errors from the task history view.

Steps

Use the following steps to create and configure a Databricks automated task. Navigate to Business Applications > Automated Tasks to get started.

Create a Databricks automated task

Creating an automated task schedules Turbo360 to automatically resubmit failed Databricks workflow job runs based on the filters and schedule you define.

1. Click New configuration.
2. Select Databricks as the resource type.
3. Select the target Databricks workflow job from your Business Application.
4. Configure the schedule for the task.
5. Under Resubmit filters, configure one or more of the following:
   • Failure Occurrence — resubmit runs that failed within the past x hours from the task run time. Set x to match your acceptable failure window.
   • Failure Reason — resubmit only runs whose failure reason matches one of the values you specify.
   • Task status — resubmit only failed runs where the task status matches one of the selected statuses.
6. Under Notifications, configure your notification channels and recipients (see Notifications).
7. Click Save.

Alert configuration

Configure notifications in the final section of the automated task configuration panel. You can set one or more notification channels and provide individual email addresses or a group distribution list.

Under Advanced settings, you can restrict when notifications are sent:

• Send alerts only when the automated task fails — you receive a notification only if the task itself fails to execute.
• Send alerts only when the completed count is greater than 0 — you receive a notification only when at least one run was resubmitted.

Enabling both options notifies you when the task fails or when the completed resubmission count exceeds zero.

Example scenario

Your nightly data ingestion Databricks workflow fails intermittently due to a transient upstream data source timeout. Rather than reviewing failures each morning, you configure an automated task to run every hour. You set Failure Occurrence to 1 hour and Task status to Failed. Each hour, Turbo360 identifies any workflow runs that failed with that status in the past hour and resubmits them automatically. You enable Send alerts only when the completed count is greater than 0 so you only receive a notification when a resubmission actually occurs.

Troubleshooting

1. Automated task runs but no workflow runs are resubmitted
   Cause: The filter conditions are too restrictive, or no Databricks workflow job runs failed within the configured time window.
   Fix: Review the Failure Occurrence window and confirm that failed runs exist in the workflow job run history within that period. Broaden or remove filters to verify.

2. Task fails with an authorization error
   Cause: The service principal or token used by Turbo360 does not have sufficient permissions on the Databricks workspace.
   Fix: Verify that the credentials configured for the Databricks resource in Turbo360 have the required permissions to resubmit workflow job runs.

3. Resubmitted runs fail again immediately
   Cause: The underlying cause of the original failure has not been resolved (for example, an upstream data source or dependency is still unavailable).
   Fix: Investigate the failure reason in the Databricks workflow job run history. Address the root cause before relying on automated resubmission.

4. Notifications are not being received
   Cause: Notification channels are not configured, or the Advanced settings filters are preventing delivery.
   Fix: Confirm that at least one notification channel and recipient is configured. If Send alerts only when the completed count is greater than 0 is enabled, verify that resubmissions are actually occurring.

5. Task history shows no records
   Cause: The task has not yet run, or runs are being filtered before reaching the resubmit stage.
   Fix: Check the task schedule and confirm it is active. Review the task's last run time in the Automated Tasks list.

Related articles

• What is an automated task?
• Scheduling an automated task
• Configuring Logic App automated tasks
• Configuring Runbook automated tasks
• Task history