Heartbeats to decouple job runtime with RescueStuckJobsAfter?

[akshayravikumar]

Hi there! I'm trying to build a production system leveraging River and I'm liking it so far.

My biggest issue has been that our jobs can take several hours to run, and we're often running into states where a job is "stuck" in a running state because of a Kubernetes job or something (even though we're attempting a graceful shutdown). I don't want to set RescueStuckJobsAfter to be too high, because that might unnecessarily slow down the system.

I don't fully understand why RescueStuckJobsAfter has to be an effective hard max on the runtime of a job. Would it be possible to expose some sort of "job heartbeat" that lets River know that a job is still alive and well, and use that to check whether a job needs to be rescued?

[akshayravikumar]

I implemented this myself by wrapping all my "work" functions with a Ticker that updates a metadata.health_check_at key every 30 seconds, and I have a separate "supervisor" process that periodically deletes all "running" jobs that haven't had a health check in a while. This lets me prune "running" jobs from failed deployments almost immediately.

This won't deal with jobs that have genuinely deadlocked or something, but I might just keep RescueStuckJobsAt very high, and just have some logs for jobs that are taking surprisingly long.

[bgentry]

You may have some issues with deleting those stuck jobs altogether. Typically I would recommend scheduling them for a retry, or marking them as completed/discarded/cancelled. Unless you're really sure nobody has any references to them remaining anywhere and won't try to complete them at some point.

[akshayravikumar]

Oops I meant I was marking them as "discarded", but I eventually changed to scheduling them for a retry. Are there any plans to add something like this to the core library?

[gzxu]

Hi, we have run into a similar issue where long-running jobs can be abruptly terminated during pod restarts, leaving them in the running state.

Since River’s built-in rescuer is based on RescueStuckJobsAfter, which must be greater than the configured job timeout, we are also exploring an out-of-band heartbeat mechanism with similar rescue logic based on heartbeat freshness.

@bgentry Would you consider reopening this issue? Heartbeat-based rescue seems important for long-running jobs.

[brandur]

Yeah, going to reopen this as it does seem to be useful as an extra precaution for jobs that end up stuck in less conventional ways.

This may end up coming in as a Pro feature though.

[monkey92t]

Hi，

I’ve encountered a similar issue. Some of my jobs run very quickly, taking only 3–5 minutes, while others are extremely long-running, ranging from 5 to 10 hours or even longer. For this reason, I set a custom JobTimeout for each worker individually.

However, we cannot guarantee worker reliability — system crashes, power outages, hardware failures can all occur, leaving jobs stuck in the running state. After the worker restarts, it cannot pick up or re-execute these stuck jobs. To trigger rescheduling for such jobs, we have to wait for the duration of max(RescueStuckJobsAfter, JobTimeout). For example, if my JobTimeout is set to 10 hours, I have to wait a minimum of 10 hours before the job can be rescheduled.

The only workaround would be to set a sufficiently short JobTimeout paired with a small RescueStuckJobsAfter value, but this causes long-running jobs to enter constant retries and flood the logs with context cancellation timeout errors.

The proper solution is to introduce job leases: while a worker process remains alive, it periodically refreshes the lease expiry timestamp (every 30 seconds or 1 minute) to signal that it is still operational. If the lease timestamp is not updated for an extended period, the worker is deemed crashed, and the built-in JobRescuer background scanner will reclaim the job.

Since native lease support is not available in River at present, I came up with a temporary workaround: when a job starts executing, spawn a goroutine to update the attempted_at column in the river_job table every minute. I am aware that attempted_at has its original intended semantics, and I am unsure about all potential side effects of directly modifying this field. Even so, this approach can resolve the stuck job problem because the JobRescuer logic relies on the attempted_at timestamp to identify stuck tasks.

A cleaner, more robust solution would be to add a dedicated lease expiry column. The JobRescuer would scan this lease timestamp to decide whether a job needs to be re-scheduled.

[brandur]

@monkey92t Thanks for the thoughtful writeup! Agree with what you said and we will probably implement this feature, although it may be for pro versions.