Destaris
Browse the docs

Dedupe & caching

Only act on what's new — never process the same content twice.

A lot of automation polls a source — an inbox, a feed, an API — on a schedule. The hard part isn't fetching; it's only acting on what you haven't already handled. The dedupe cache makes that a built-in, deterministic step.

The pattern

Two steps work together:

  • Dedupe cache (task.cache-unseen) — given a list of items with stable keys, it passes through only the ones the workflow hasn't seen before.
  • Mark seen (task.cache-mark) — records the items you've now handled, so the next run skips them.

A typical "act on new items" workflow looks like:

  1. Fetch a list (e.g. open issues, new emails).
  2. task.cache-unseen → keep only the items not handled before.
  3. Do the work on those items (transform, AI, notify…).
  4. task.cache-mark → remember them as handled.

Why it matters

  • Idempotent by default. Run the workflow as often as you like; each item is acted on once.
  • No wasted runs. You don't re-process — or re-pay for AI on — content you've already handled, which keeps hybrid workflows cheap.
  • Reliable polling. Schedules can overlap or retry without double-sending.

Keys

Dedup works on a stable key per item — typically an id from the source system (an issue id, a message id). Choose a key that uniquely and durably identifies an item so re-fetches map to the same cache entry.

Claiming items while you work on them

Marking an item seen happens after you've handled it. That leaves a small gap: if two runs are going at once — the same workflow's next scheduled run, or a second workflow watching the same source — both can grab the same brand-new item before either has marked it, and you act on it twice.

Turn on Claim on the dedupe step (task.cache-unseen) to close that gap. When it's on, each item the step lets through is immediately reserved as in progress, so any other run in flight sees it as taken and skips it. The reservation:

  • is shared by every workflow using the same namespace, so sibling workflows cooperate instead of colliding;
  • expires on its own after a time you set (15 minutes by default), so an item is never stuck reserved if a run is interrupted before it finishes;
  • is replaced by the permanent "seen" record once task.cache-mark runs.

Claiming is opt-in — leave it off for a single, non-overlapping workflow; turn it on when the same items could be picked up by more than one run at a time.