Keep scheduled compositions cheap

Every time a composition runs, the platform records its progress: a state row for the run, for each layer and for each step that executes. That record is what gives you history, retries and Lumen visibility. It is also the cost, and it is paid per run, whether or not the run had anything to do.

A queue polled every minute is 1,440 runs a day. If the queue is empty most of the time, almost all of that is spent confirming that there is nothing to do.

The rest of this page is how to get from the first row of that table to the last two.

1. Pick what starts the run

Schedules your end users create from your app (schedule.app.create) are created paused and fire at most every 15 minutes.

2. Fixed times: cron with a timezone

schedule.create takes either interval_minutes or schedule_expression, never both. The expression is five cron fields (minute hour day-of-month month day-of-week) read in timezone, an IANA name. Without timezone it is read in UTC.

Weekdays at 08:00 in São Paulo:

{
  "target_workflow_type": "orn:orkestia:composition:<org>:lineage/<lineage_uuid>",
  "name": "Morning briefing (weekdays 08:00)",
  "schedule_expression": "0 8 * * 1-5",
  "timezone": "America/Sao_Paulo"
}

• The answer includes next_occurrences, the next instants (UTC) the schedule will really fire. Read it before you walk away.
• Daylight saving time is followed. A local time the clock skips does not fire that day; a local time it repeats fires once.
• Point the schedule at the composition's lineage ORN (returned by composition.save as orn). Every fire then runs the active version, so the schedule follows your saves. A concrete virtual.<uuid>@<version> pins that one version until you re-point it.
• To change an existing schedule, schedule.update takes the same schedule_expression and timezone. timezone on its own re-reads the stored expression in the new zone.

3. Queues: a change trigger, not a poll

A change trigger starts your composition when rows of one App Data table are inserted or updated. An idle queue then costs no runs at all, and a burst of inserts becomes one run.

Step 1. Declare events: true on the table in your data structure and apply it (see Declare structures):

databases:
  - slug: crm
    tables:
      - slug: signups
        ownership: { kind: app }
        events: true              # or { ops: [insert] } to capture inserts only
        fields:
          - { slug: email, type: text, required: true }
          - { slug: status, type: text }
          - { slug: welcomed_at, type: timestamp }

Step 2. Create the trigger with trigger.create:

{
  "identity_app_uuid": "<app uuid>",
  "table_slug": "signups",
  "target_workflow_type": "orn:orkestia:composition:<org>:lineage/<lineage_uuid>",
  "run_as": { "kind": "staff_actor", "staff_actor_uuid": "<actor uuid>" },
  "ops": ["insert"],
  "where": { "status": "pending" },
  "debounce_s": 5,
  "max_batch": 500
}

Step 3. Let the target composition receive what the trigger sends. Declare these inputs, all optional; inputs the composition does not declare are dropped:

{
  "name": "welcome_new_signups",
  "inputs": [
    { "name": "records",      "type": "list" },
    { "name": "trigger_uuid", "type": "string" },
    { "name": "gap",          "type": "boolean" },
    { "name": "mode",         "type": "string" }
  ],
  "layers": [
    {
      "name": "load",
      "steps": [
        {
          "name": "todo",
          "workflow_type": "appdata.query.run",
          "condition": { "field": "records", "op": "truthy" },
          "input_mapping": {
            "identity_app_uuid": { "source": "static", "value": "<app uuid>" },
            "statement": { "source": "static", "value": "SELECT record_uuid, email FROM signups WHERE record_uuid::text = ANY($1) AND welcomed_at IS NULL" },
            "params": { "source": "list", "items": [{ "source": "input", "field": "records" }] }
          }
        }
      ]
    }
  ]
}

records holds the ids of the matching rows, not their values. Your first layer re-reads exactly those rows, so it always sees current data.

What to know about delivery:

• At least once. A record can arrive twice. Make the work idempotent (the welcomed_at IS NULL check above, a unique field, or an idempotency key).
• One run at a time per trigger. Changes that arrive while a run is in flight are delivered together in the next run.
• No self-loops. Writes your own run makes through workflow steps do not fire the same trigger again.
• It protects itself. A trigger that fires in a storm, or whose starts keep failing, pauses itself. trigger.resume starts it again; events that arrived meanwhile are kept and delivered.
• Creating, updating and deleting a trigger needs an organization admin. Manage them with trigger.list, trigger.get, trigger.update, trigger.pause, trigger.resume and trigger.delete.

4. Skip empty work

When you do need a schedule, make the empty run as cheap as possible:

1. Layer 1 is one cheap check: a count, or a query with a LIMIT.
2. Every later step carries a condition on that check. A skipped step writes no rows, and a skipped sub-composition skips everything inside it.

{
  "name": "mark_overdue_invoices",
  "description": "Daily: mark open invoices past their due date as overdue",
  "layers": [
    {
      "name": "check",
      "steps": [
        {
          "name": "pending",
          "workflow_type": "appdata.query.run",
          "input_mapping": {
            "identity_app_uuid": { "source": "static", "value": "<app uuid>" },
            "statement": { "source": "static", "value": "SELECT json_build_object('record_uuid', record_uuid, 'patch', json_build_object('status', 'overdue')) FROM invoices WHERE status = 'open' AND due_date < current_date ORDER BY record_uuid LIMIT 500" }
          }
        }
      ]
    },
    {
      "name": "shape",
      "steps": [
        {
          "name": "patches",
          "workflow_type": "control.collection.flatten",
          "condition": { "step": "pending", "field": "row_count", "op": "gt", "value": 0 },
          "input_mapping": {
            "items": { "source": "step", "step": "pending", "field": "rows" },
            "depth": { "source": "static", "value": 1 }
          }
        }
      ]
    },
    {
      "name": "write",
      "steps": [
        {
          "name": "mark",
          "workflow_type": "appdata.update-as-app",
          "condition": { "step": "pending", "field": "row_count", "op": "gt", "value": 0 },
          "input_mapping": {
            "identity_app_uuid": { "source": "static", "value": "<app uuid>" },
            "database_slug": { "source": "static", "value": "billing" },
            "table_slug": { "source": "static", "value": "invoices" },
            "patches": { "source": "step", "step": "patches", "field": "result" },
            "source": { "source": "static", "value": "overdue-sweep" }
          }
        }
      ]
    }
  ]
}

Conditions take {"step": S, "field": F, "op": OP, "value": V}. field may be a dotted path into the step's output. Operators: eq, ne, in, not_in, truthy, falsy, gt, gte, lt, lte, contains, not_contains, regex. Combine them with {"all_of": [...]}, {"any_of": [...]} and {"not": ...}.

An interval schedule whose runs keep finding nothing to do (ten in a row) may be slowed down automatically, up to 15 minutes between fires. The next run that does real work restores the normal pace. If the work needs a prompt reaction, that is the sign it should be a change trigger.

5. Keep pure logic inline

Pure control.* steps (value, collection, predicate, math.compute, switch.select, text and time.now) are evaluated inside the run instead of starting a step of their own, so they add no state rows. Consecutive layers of pure steps are fused into one. In the example above, patches is inline.

You do not need to mark them. Check the result instead: composition.validate and composition.save return inline_steps, the steps that will run inline.

• "inline": false on a step opts it out. "inline": true on a step that is not eligible is a validation error.
• A step with fan_out or transient is never inline.
• control.flow.guard, control.flow.wait, control.poll.until, control.http.request and control.math.random always run as their own step.
• Don't split pure logic into sub-compositions just for structure. A sub-composition is a run of its own.

A version saved before inline evaluation was available keeps its old compiled form. To upgrade it, save again with recompile_inline: true. That writes a new version even if the definition is unchanged. Schedules and triggers that point at the lineage ORN pick it up; anything pinned to the old virtual.<uuid>@<version> is listed in stale_dependents and must be re-pointed.

6. Write only what changed, and carry little data

• Write the difference. Don't upsert every record on every pass. Select only the rows whose stored value differs from the value you would write, as the WHERE status = 'open' AND due_date < current_date clause does above, and write those.
• Make writes idempotent. Use unique fields, upserts keyed on an external id (appdata.upsert-as-app) or an expected_updated_at precondition, so a retry or a repeated delivery is a no-op.
• Pass ids, not documents. Each layer's state carries the outputs of the steps before it. Select only the columns you need, LIMIT and page, and pass ids to later steps that can read what they need.
• Read results, not echoes. Pure control.* steps return their result fields (result, count, value, …), not a copy of their inputs. Map from those fields.
• Let poison items leave the queue. An item that fails the same way every time (for example "this name is already taken") must be resolved, by adopting the existing resource or marking the item failed with a reason. Otherwise every run retries it forever.

7. Read the cost before you activate

Before saving, composition.validate (and composition.save) return cost_estimate. For the overdue-invoices example:

{
  "passed": true,
  "inline_steps": ["patches"],
  "cost_estimate": {
    "floor":   { "steps": 1, "rows": 6 },
    "ceiling": { "steps": 3, "rows": 9 },
    "layers": 3,
    "unbounded_fanouts": [],
    "unresolved": []
  }
}

When scheduling, schedule.create and schedule.update return estimated_rows_per_day (floor, ceiling, runs_per_day, rows_per_run) and warnings. A schedule that fires more often than every 5 minutes and writes more than 20,000 rows a day gets a cost warning. cost_ack: true accepts it; only pass it when you can say why in the schedule's name.

After a day, run data.composition.cost with the composition's lineage_uuid or the schedule_uuid (window_hours defaults to 24, up to 168). It reports runs, rows, rows_per_run, bytes and empty_runs, the runs that did nothing after layer 1. If empty_runs is more than about half of runs_with_stats, go back to sections 1 and 4.

Checklist

Before composition.activate, schedule.create or trigger.create:

• What starts the run matches the table in section 1: a trigger for a queue, cron with a timezone for fixed times.
• Schedules and triggers point at the lineage ORN, or at the version you just saved.
• Layer 1 is one cheap check, and every later step and sub-composition has a condition.
• No control.flow.guard used as an early stop.
• inline_steps is not empty when the composition has pure logic.
• Writes are diff-only and idempotent; failing items leave the queue.
• Step outputs are small: ids and the columns you need.
• You have read cost_estimate and estimated_rows_per_day, and there is no unexplained cost_ack.
• A day later, data.composition.cost shows a low share of empty runs.

Related

• Creating & Exposing Virtual Workflows
• Receive webhooks with a re-query
• Declare structures
• Platform services