The workflow engine said OK after a failed step

TL;DR. An RPC acknowledgement proves that a command completed at the transport boundary. It does not prove that the workflow succeeded. When embedding an asynchronous workflow engine, define success from one durable terminal record that contains the run-level outcome. If the engine acknowledges the command but never produces that record, the result is indeterminate or a platform failure, never success.

The first end-to-end test of our embedded workflow engine looked reassuring. The host submitted a flow, the engine answered OK, and the child process exited normally. Then we looked at the step output. The only action had failed exactly as the test intended.

Nothing was corrupted, and the engine had not lied. We had asked the wrong layer for the answer.

The acknowledgement described execution of an engine command. The workflow verdict arrived through a separate run record. A failed step could therefore coexist with an OK acknowledgement. Treating the latter as business success turned a clean workflow failure into a false positive.

This trap is broader than any one engine. It appears whenever a host embeds an asynchronous worker and compresses multiple protocol layers into a single boolean.

There are three kinds of success

The word "success" is overloaded in an embedded execution system:

Layer	Question it answers	What success does not prove
Admission	Did the host accept the execution request?	That the engine started or any step ran
Transport or RPC	Did the engine handle the command without a protocol-level error?	That the workflow's steps succeeded
Workflow outcome	Did the run reach a defined terminal state?	Nothing above it; this is the user-visible result

The layers often line up on a happy path, which is why the bug survives unit tests. A valid request is accepted, the engine processes it, and the flow succeeds. Every signal is positive.

A deliberately failing step breaks that accidental correlation. The request is valid, the engine correctly executes the command, and the engine correctly reports the step failure. Admission and transport can both succeed while the workflow fails.

HTTP status has the same ambiguity. A 2xx response can mean "queued," "accepted," or "here is the current execution object." None of those meanings implies a terminal workflow outcome unless the API contract explicitly says so.

Choose one authoritative terminal record

The host needs a single completion invariant:

An execution is complete only when the durable run record contains a recognized terminal outcome, and the user-visible status is derived from that outcome.

For us, that record contains the run-level state and enough per-step information to explain a failure. The engine's command acknowledgement remains useful, but only for diagnosing protocol and process failures.

This choice matches the durable model described in Activepieces' execution documentation. Its run log records the status, output, duration, and error of finished steps, and is written again at the final state. The same log supports replay after interruption. That makes the terminal record semantically richer than an RPC envelope by design.

Durable is important here. An in-memory callback can tell the host what happened, but if the host crashes after receiving it and before saving it, the system has no result to show later. The durable record should be the object that billing, notifications, retries, and the user-facing execution page all read. Multiple downstream definitions of success will drift.

Ordering is a versioned contract, not a verdict

In the Activepieces 0.86.3 path we embed, the engine writes its final run record before it acknowledges the execution command. That ordering is useful, and our host treats it as a versioned protocol obligation that must be re-tested on every upgrade.

It still does not make the acknowledgement a workflow verdict. The ordering tells the host when it may expect the authoritative record to exist. The record, not the later envelope, says whether the run succeeded or failed.

If an engine guarantees record-before-ack, an OK acknowledgement without that record is a protocol violation and therefore a platform failure. If an engine makes no ordering guarantee, the host must accept the signals independently and wait for the terminal record within a bounded window. Either contract is workable. The unsafe contract is "ACK means the workflow succeeded."

This resembles the reason Stripe advocates one stable, request-level canonical log line: downstream systems become reliable when they share one hardened record rather than reconstructing reality from whichever low-level event they happened to receive. A workflow terminal record is not merely a log line, but the design lesson is the same: make one semantic record carry the decision.

Missing terminal state is not workflow failure

Once the terminal record is authoritative, another subtle choice appears. What should the host do if the engine exits or acknowledges the command but no terminal record exists?

It should not guess.

Calling the run successful is obviously unsafe. Calling it a normal workflow failure is also wrong because that blames the user's definition and may trigger user-configured retry behavior. The step might have succeeded, failed, or produced an external side effect before the result was lost.

We classify the missing record as indeterminate or a platform failure. That distinction preserves three useful truths:

• A workflow failure means the engine reached a terminal outcome and attributed failure to the run.
• A platform failure means the host cannot prove the workflow's terminal outcome.
• A success means the authoritative record explicitly says the workflow succeeded.

This taxonomy matters for more than UI wording. Retry policy must consider whether an unobserved side effect may already have happened. Billing should not silently equate "the host lost the verdict" with completed customer work. Alerts should page the platform operator instead of telling a user to fix a valid workflow.

The contract test is a step that must fail

The smallest useful integration test is intentionally unpleasant:

1. Submit a valid workflow with one deterministic, side-effect-free step that must fail.
2. Confirm the host accepts the execution.
3. Wait for the public execution record to become terminal.
4. Assert the run is failed, the failing step is identified, and an error is present.
5. Assert that repeating the same execution identity does not run it twice.

The test should cross the real engine boundary. Mocking the engine to return "failed" only proves the host handles the answer you expected. It does not prove which message in the real protocol carries that answer.

Run this test on every engine upgrade. We pin the embedded engine version because this contract is behavioral, not merely structural. A type checker can verify that an acknowledgement has a status field; it cannot verify what that field means after a failed step.

Add two companion cases once the core test is green. A successful step proves the host does not turn every acknowledgement into failure. A process or transport interruption proves missing terminal state maps to platform failure rather than workflow failure. The three cases create a compact truth table for the protocol.

Keep the layers visible in the product

The host should expose execution states that match what it actually knows: queued or running before a terminal record, succeeded or failed when the record says so, and a separate platform-failure state when the verdict is missing. Polling APIs should return the durable execution object, not replay an ephemeral acknowledgement.

This is how plori workflows are presented: an execution has a status and per-step outcome that can be retrieved after the initial run request. The engineering benefit is consistency. The API, notifications, metering, and support tools all discuss the same execution instead of translating different low-level signals.

It also makes operations easier. The acknowledgement answers "did the protocol remain healthy?" The terminal record answers "what happened to the workflow?" Keeping both signals lets an alert distinguish engine transport regressions from ordinary user-step failures without exposing protocol details to the user.

Limitation

This finding comes from embedding a pinned Activepieces 0.86.3 engine behind our own host, not from running the stock Activepieces server. The exact messages and persistence path are version-specific. The general rule is portable, but every engine integration must probe its own failure behavior and repeat that probe on upgrade. Do not assume another engine's acknowledgement has the same semantics, and do not assume a future version preserves an undocumented ordering.

FAQ

Why is an OK acknowledgement not workflow success?

Because it can acknowledge successful handling of the execution command. A step may still reach a failed terminal state. Only the run-level terminal record answers the workflow question.

What if the acknowledgement arrives before the terminal record?

Follow the engine's versioned contract. If that order is allowed, keep the execution non-terminal and wait within a bounded window. If the engine guarantees the opposite order, classify the missing record as a platform failure. Never infer workflow success from the acknowledgement.

How should a missing terminal record be reported?

As indeterminate or a platform failure. It is neither proven success nor a normal workflow failure, and automatic retries must account for possible unobserved side effects.

What should be tested during a workflow engine upgrade?

At minimum: one successful step, one deterministic failing step, and one engine or transport interruption. Assert the final public execution state for all three, not only the initial response.