mirror of
https://github.com/go-gitea/gitea.git
synced 2026-08-03 03:19:39 +00:00
Backport #38616 by @bircni Fixes jobs that get stuck in `cancelling` after Gitea is restarted while a job is running. Reproduction: 1. Run a job with `sleep 100` 2. Stop Gitea and wait 120s (> 100s) 3. Start Gitea — the job is still `running`; cancel it, and it stays in `cancelling` ## Cause A cancellation is only ever delivered to a runner as the *response* to its `UpdateTask` RPC. A runner that has already given up on the task — it crashed, or it failed to report the final state while Gitea was unreachable — never calls `UpdateTask` again, so it never learns about the cancellation. The task then sits in `cancelling` until `stop_zombie_tasks` reaps it, which needs `ZOMBIE_TASK_TIMEOUT` (10m) of silence and only runs every 5 minutes. Cancelling actually made this worse. xorm rewrites the `updated` column on every `UPDATE`, even when `Cols()` restricts the update to `status`, so persisting the `cancelling` status reset the zombie clock. Pressing cancel pushed the cleanup a full `ZOMBIE_TASK_TIMEOUT` into the future instead of bringing it forward. ## Change `StopTask` now skips the `cancelling` handshake when the task has had no state report from its runner for longer than `TaskReportTimeout` (1 minute) and cancels it directly. This joins the two existing fallbacks — runner deleted, and runner without cancelling support — so every cancel path (web UI, API, concurrency, rerun) is covered. Runners report the state of a running task every few seconds, so a minute of silence means the runner is gone. The value is a constant rather than a setting because it only decides whether the runner is still reachable, not whether a task should be killed — `ZOMBIE_TASK_TIMEOUT` still owns that. ### Tradeoff If a runner is alive but has been silent for over a minute and is cancelled in that window, it skips the graceful post-step cleanup added in #37275. No work is lost: the runner still learns the outcome on its next report, because `UpdateTask` returns the task status as the result and the runner stops there. That is the behaviour that existed before #37275. Co-authored-by: bircni <bircni@icloud.com> Co-authored-by: Zettat123 <zettat123@gmail.com> Co-authored-by: silverwind <me@silverwind.io>
This commit is contained in:
+14
-1
@@ -60,6 +60,12 @@ type ActionTask struct {
|
||||
Updated timeutil.TimeStamp `xorm:"updated index"`
|
||||
}
|
||||
|
||||
// taskReportTimeout is how long a task may go without contact from its runner before the
|
||||
// runner is assumed gone. Runners report state and stream logs every few seconds, both of
|
||||
// which refresh ActionTask.Updated. Shorter than setting.Actions.ZombieTaskTimeout because
|
||||
// it only decides whether the runner is reachable, not whether the task should be killed.
|
||||
const taskReportTimeout = time.Minute
|
||||
|
||||
var successfulTokenTaskCache *lru.Cache[string, any]
|
||||
|
||||
func init() {
|
||||
@@ -567,6 +573,10 @@ func StopTask(ctx context.Context, taskID int64, status Status) error {
|
||||
status = StatusCancelled
|
||||
} else if !runner.HasCancellingSupport {
|
||||
status = StatusCancelled
|
||||
} else if task.Updated.AddDuration(taskReportTimeout) < now {
|
||||
// A runner that stopped reporting will never acknowledge the cancellation either,
|
||||
// so skip the handshake instead of waiting for the zombie task cleanup.
|
||||
status = StatusCancelled
|
||||
}
|
||||
}
|
||||
|
||||
@@ -581,7 +591,10 @@ func StopTask(ctx context.Context, taskID int64, status Status) error {
|
||||
return err
|
||||
}
|
||||
|
||||
return UpdateTask(ctx, task, "status")
|
||||
// NoAutoTime keeps "updated" at the runner's last contact: re-cancelling an already
|
||||
// cancelling task must not defer the timeout above or the zombie task cleanup.
|
||||
_, err := e.ID(task.ID).Cols("status").NoAutoTime().Update(task)
|
||||
return err
|
||||
}
|
||||
|
||||
task.Status = status
|
||||
|
||||
Reference in New Issue
Block a user