Inserting and working jobs

River’s most basic use involves defining job arguments and workers, starting a client, and inserting jobs to be worked.

Job args and workers

Each job kind has a stable string identifier, JSON-encoded arguments, and a worker that executes it. Use the same kind and JSON field names when inserting jobs from another language.

Jobs are defined in struct pairs, with an implementation of JobArgs and one of Worker[T JobArgs].

Job args contain json annotations and define how jobs are serialized to and from the database, along with a “kind”, a stable string that uniquely identifies the job.

type SortArgs struct {
    // Strings is a slice of strings to sort.
    Strings []string `json:"strings"`
}

func (SortArgs) Kind() string { return "sort" }

Workers expose a Work function that dictates how jobs run.

type SortWorker struct {
    // An embedded WorkerDefaults sets up default methods to fulfill the rest of
    // the Worker interface:
    river.WorkerDefaults[SortArgs]
}

func (w *SortWorker) Work(ctx context.Context, job *river.Job[SortArgs]) error {
    sort.Strings(job.Args.Strings)
    fmt.Printf("Sorted strings: %+v\n", job.Args.Strings)
    return nil
}

Define an args object that supplies the job kind and JSON payload, then a worker with the same kind. The worker reads the decoded arguments from job.args:

require "riverqueue-sequel"

class SortArgs
  attr_reader :strings

  def initialize(strings:)
    @strings = strings
  end

  def kind = "sort"

  def to_json = JSON.generate(strings: strings)
end

class SortWorker
  def self.kind = "sort"

  def work(job)
    puts "Sorted strings: #{job.args.fetch('strings').sort.inspect}"
  end
end

Derive JobArgs and Serde’s serialization traits for the args struct, then write an async worker that accepts Job<SortArgs>:

use riverqueue::{BoxError, Job, JobArgs, WorkContext, WorkOutcome};
use serde::{Deserialize, Serialize};

#[derive(Clone, Debug, Deserialize, JobArgs, Serialize)]
#[river(kind = "sort")]
struct SortArgs {
    strings: Vec<String>,
}

async fn sort(_context: WorkContext, mut job: Job<SortArgs>) -> Result<WorkOutcome, BoxError> {
    job.args.strings.sort();
    println!("Sorted strings: {:?}", job.args.strings);
    Ok(WorkOutcome::Complete)
}

Use defineJob to give the job a kind and an argument schema. The handler receives those arguments on job.args:

import { Client, Workers, defineJob } from "riverqueue";
import { z } from "zod";

const sort = defineJob({
  kind: "sort",
  schema: z.object({ strings: z.array(z.string()) }),
});

async function sortStrings({ job }: { job: { args: { strings: string[] } } }) {
  console.log("Sorted strings:", [...job.args.strings].sort());
}

Other job args options

Job args can also implement JobArgsWithInsertOpts to provide default InsertOpts for the job at insertion time:

func (AlwaysHighPriorityArgs) InsertOpts() river.InsertOpts {
  return river.InsertOpts{
    Queue: "high_priority",
  }
}

See the CustomInsertOpts example for complete code.

Define insert_opts on the args object to supply defaults for that job kind, such as its destination queue:

class AlwaysHighPriorityArgs
  def kind = "always_high_priority"
  def to_json = JSON.generate({})
  def insert_opts = River::InsertOpts.new(queue: :high_priority)
end

Set insertion defaults in the #[river(...)] attribute. Here, jobs of this kind default to the high_priority queue:

#[derive(Clone, Debug, serde::Deserialize, riverqueue::JobArgs, serde::Serialize)]
#[river(kind = "always_high_priority", queue = "high_priority")]
struct AlwaysHighPriorityArgs {}

Use the job definition’s defaults to configure insertion options shared by jobs of that kind:

const alwaysHighPriority = defineJob({
  kind: "always_high_priority",
  defaults: { queue: "high_priority" },
});

Other worker options

The use of WorkerDefaults provides default implementations for most functions in the Worker interface, but the defaults can be overridden as appropriate.

Timeout lets a worker provide a timeout:

func (w *LongRunningWorker) Timeout(job *Job[MyJobArgs]) time.Duration {
    return 1 * time.Hour
}

NextRetry lets a worker provide a custom retry schedule:

// NextRetry always schedules the next retry for 10 seconds from now.
func (w *ConstantRetryTimeWorker) NextRetry(job *Job[MyJobArgs]) time.Time {
    return time.Now().Add(10*time.Second)
}

Workers can override timeouts and retry times:

def timeout(_job) = 3600
def next_retry(_job, _error) = Time.now.utc + 10

Implement Worker when a job kind needs custom timeout or retry behavior. These methods belong in its implementation:

fn timeout(&self, _job: &riverqueue::Job<MyArgs>) -> riverqueue::WorkerTimeout {
    riverqueue::WorkerTimeout::After(std::time::Duration::from_secs(3600))
}

fn next_retry(
    &self,
    _job: &riverqueue::Job<MyArgs>,
    _error: &riverqueue::WorkError,
    _now: riverqueue::chrono::DateTime<riverqueue::chrono::Utc>,
) -> Option<std::time::Duration> {
    Some(std::time::Duration::from_secs(10))
}

Pass worker-specific options when registering the handler. This example allows an hour per attempt and schedules retries ten seconds later:

workers.add(myJob, handler, {
  timeout: { hours: 1 },
  retryPolicy: (_job, now) => now.add({ seconds: 10 }),
});

Registering workers

Jobs are uniquely identified by their “kind” string. Workers are registered on start up so that River knows how to assign jobs to workers:

workers := river.NewWorkers()

// AddWorker panics if the worker is already registered or invalid:
river.AddWorker(workers, &SortWorker{})

AddWorker panics in case of invalid configuration. Given its succinct syntax and that bad configuration should prevent a worker process from booting, panicking is probably a reasonable compromise for most applications. However, for those who find it distastely, AddWorkerSafely is also provided:

workers := river.NewWorkers()
if err := river.AddWorkerSafely(workers, &SortWorker{}); err != nil {
    panic("handle this error")
}

Add the worker class to a River::Workers registry so the client knows which job kinds it can work:

workers = River::Workers.new.add(SortWorker)

let mut workers = riverqueue::Workers::new();
workers.add_fn(sort)?;

const workers = new Workers().add(sort, sortStrings);

Starting a client

A client inserts jobs, runs workers, and participates in maintenance. This client works the default queue with up to 100 concurrent jobs.

A River Client provides an interface for job insertion and manages job processing and maintenance services. A client’s created with a database pool, driver, and config struct containing a Workers bundle and other settings. Here’s a client Client working one queue ("default") with up to 100 worker goroutines at a time:

riverClient, err := river.NewClient(riverpgxv5.New(dbPool), &river.Config{
    Queues: map[string]river.QueueConfig{
        river.QueueDefault: {MaxWorkers: 100},
    },
    Workers: workers,
})
if err != nil {
    panic(err)
}

// Run the client inline. All executed jobs will inherit from ctx:
if err := riverClient.Start(ctx); err != nil {
    panic(err)
}

Connect through the Sequel driver and configure the worker registry and queue concurrency before calling start:

DB = Sequel.connect(ENV.fetch("DATABASE_URL"))

client = River::Client.new(
  River::Driver::Sequel.new(DB),
  config: River::Config.new(queues: {default: 100}, workers: workers)
)

client.start

Build a client with the Postgres pool, worker registry, and queue configuration. Keep the run handle returned by start for shutdown:

use riverqueue::{Client, QueueConfig, sqlx::PgPool};

let pool = PgPool::connect(&std::env::var("DATABASE_URL")?).await?;

let client = Client::builder(pool.clone())
    .workers(workers)
    .queue("default", QueueConfig::new(100))
    .build()?;

let mut run = client.start()?;

Construct a client with a Postgres driver, registered workers, and queue concurrency, then await start to begin working jobs:

import { PgDriver } from "@riverqueue/driver-pg";
import { Pool } from "pg";

const pool = new Pool({ connectionString: process.env.DATABASE_URL });

const client = new Client(new PgDriver(pool), {
  queues: { default: { maxWorkers: 100 } },
  workers,
});

const run = await client.start();

Stopping

The client should also be stopped on program shutdown. There’s a number of ways to go about this (see graceful shutdown), but the shortest is to cancel the context send to Start when the program’s ready to stop. For example, to stop on SIGINT/SIGTERM:

riverClient, err := river.NewClient(riverpgxv5.New(dbPool), &river.Config{
    SoftStopTimeout: 10 * time.Second,
    ...
})
if err != nil {
    panic(err)
}

signalCtx, stop := signal.NotifyContext(ctx, syscall.SIGINT, syscall.SIGTERM)
defer stop()

// Stop fetching new work and wait for active jobs to finish. Cancel jobs after
// SoftStopTimeout elapses.
if err := riverClient.Start(signalCtx); err != nil {
    panic(err)
}

<-riverClient.Stopped()

Alternatively, use an explicit call to Stop:

if err := riverClient.Stop(ctx); err != nil {
    panic(err)
}

There are some complexities around ensuring clients stop cleanly, but also in a timely manner. Read Graceful shutdown for more details on River’s stop modes.

Stop the client before disconnecting the database so workers can finish and persist their results:

client.stop
DB.disconnect

Await stop on the run handle when the application is ready to shut down:

run.stop().await?;

Wait for the run to stop before closing the database pool:

await run.stop();
await pool.end();

Read Graceful shutdown for signal handling and stop modes. Insert-only clients do not need to be started or stopped.

Inserting jobs

Transactional enqueueing keeps jobs consistent with the application data they depend on. A job inserted in a transaction becomes visible to workers only after the transaction commits.

Client.InsertTx is used in conjunction with an instance of job args to insert a job to work on a transaction:

_, err = riverClient.InsertTx(ctx, tx, SortArgs{
    Strings: []string{
        "whale", "tiger", "bear",
    },
}, nil)
if err != nil {
    panic(err)
}

See the InsertAndWork example for complete code.

Client.Insert that doesn’t take a transaction is also available, although as described in Transactional enqueuing, inserting jobs in transactions is usually more appropriate to avoid bugs.

_, err = riverClient.Insert(ctx, SortArgs{
    Strings: []string{
        "whale", "tiger", "bear",
    },
}, nil)
if err != nil {
    panic(err)
}