Exchanged with software we did not write.
A separate harness starts a pinned HL7 Da Vinci burden-reduction payer reference implementation in a container and performs real PAS, CRD and DTR exchanges against it. Nothing is mocked, nothing is replayed, and the result is never added to the CMS-0057-F acceptance score.
CMS-0057-F is the federal rule that requires Medicare Advantage, Medicaid, CHIP, and some exchange plans to offer FHIR APIs for patient access, provider access, payer-to-payer exchange, and prior authorization by January 1, 2027.
A test suite you wrote against fixtures you wrote proves something narrower
Cloud Health Office has a mature acceptance suite that proves it implements what its own CMS-0057-F acceptance specification requires. Everything that suite exercises is Cloud Health Office code against Cloud Health Office fixtures. This harness answers a different question, and the only way to answer it is to talk to somebody else's code.
Can Cloud Health Office exchange standards-conformant requests and responses with an independent Da Vinci implementation it does not own?
So the harness starts real HL7 Da Vinci reference implementations in containers, pinned by image digest, and performs real FHIR and CDS Hooks exchanges across a process boundary. Nothing is mocked, replayed, or reproduced inside Cloud Health Office and then called external validation.
| Internal acceptance | External interoperability | |
|---|---|---|
| Question | Does Cloud Health Office implement the required behaviour? | Can it interoperate with an independent implementation? |
| Vocabulary | PASSABLE / PARTIAL / GAP / N/A | Passed / Failed / Skipped / NotRun |
| Counterparty | Cloud Health Office fixtures | Software Cloud Health Office did not write |
| Published as | the acceptance evidence snapshot | the interoperability run snapshot |
The two are never merged into one score. An interoperability result never changes a CMS-0057-F scenario status, and a CMS-0057-F status never implies an interoperability result. The run document states that separation inside the artifact itself, so a consumer reading only the JSON still sees it.
What is actually sent, and what comes back
Three exchanges run today against the pinned HL7 Da Vinci burden-reduction payer reference implementation, with Cloud Health Office acting as the provider-side client through its own production code paths.
CRD — three codes, three different answers from the payer's own rules
Cloud Health Office fetches the payer's CDS Hooks discovery document, which advertises six CRD services,
and resolves order-sign from discovery by hook — never hard-coded and
never taken by list position. A server may reorder or rename services between releases, and a client that
indexed into the array would quietly start testing something else instead of failing honestly.
It then submits three synthetic draft orders carrying different billing codes, and the payer's own rule fixtures return three genuinely different determinations:
| Billing code | Upstream rule fixture | The payer's determination |
|---|---|---|
| L8000 | PriorAuthRequired | covered, pa-needed=auth-needed, doc-needed=no-doc, plus a DTR questionnaire canonical |
| J3490 | ExcludedServices | covered=not-covered |
| E0100 | matches no fixture | covered=conditional, info-needed=detail-code |
The third code is what makes the first two meaningful. Two differing answers could be two hard-coded branches; three distinct answers, one of which is the no-rule default, show the payer actually resolving rules per billing code. The scenario asserts all three are distinct.
A CDS Hooks request must name a FHIR server the service may dereference for anything prefetch did not supply. Rather than pointing that at a placeholder and hoping it is never called, the harness points it at a listener it actually runs and fails if any callback arrives — so the scenario is correct by construction rather than accidentally successful.
The implementation observation worth carrying forward: this payer answers order-sign
with zero cards and one system action carrying the whole determination. A client that inspected
only cards would conclude nothing happened.
DTR — chained from the payer's determination, not from a local fixture
This is the exchange that makes the set a workflow rather than three unrelated endpoint demos. Cloud Health Office does not choose the questionnaire. The payer decides which one applies when it evaluates coverage, and the scenario follows that decision into the payer's DTR surface — which is exactly what a provider system must do in production.
Selecting a questionnaire from a Cloud Health Office fixture would have tested a much weaker thing: that the payer answers for a canonical Cloud Health Office already knew. Chaining means the payer's questionnaire-selection rule is consumed rather than mirrored, and the run evidence records the linkage — the scenario it chained from, and the exact canonical it followed — so a run reads as a workflow rather than as isolated green rows.
Two CRD determinations lead to two different questionnaires, and the scenario asserts the canonicals differ. That is what shows the chain follows the payer's decision rather than returning a constant.
The request carries only what the operation requires — the coverage, and the canonical CRD returned,
verbatim. Padding it with resources the operation does not use would make it look richer while proving
less. The synthetic member will not match on the payer's side, the payer says so in an
OperationOutcome, and the scenario records that rather than suppressing it: refusing to trust
a sender-supplied reference as a lookup key is a sensible privacy property, not a defect.
PAS $submit
A synthetic PAS request bundle is posted to the payer's Claim/$submit, and the returned PAS
response bundle is parsed and validated with the same FHIR parser Cloud Health Office's own service runs
— profiles, ClaimResponse shape and X12 review-action coding. The submitted service code
matches no rule on the reference implementation, so the payer answers review action A3. That
is a deterministic, content-independent path on purpose: this scenario is a proof of
protocol interoperability, not of prior-authorization decision logic.
A pinned dependency, or the result means nothing next month
Every external artifact is an image digest, or a release tag plus the commit it resolves to. Never a floating tag, never an unpinned branch — and a test fails the build if one appears.
- The upstream commit for each image comes from its
org.opencontainers.image.revisionlabel: what the image was actually built from, not an assumption about a default branch. - The compose stack must start exactly the digests recorded in the manifest, asserted by test, and containers run with pulls disabled so an unpinned image cannot be fetched mid-scenario.
- Pins never move on their own. There is no auto-update. A nightly failure after an upstream change is a finding to triage, not a signal to bump — and an upgrade is always a reviewable diff naming the exact old and new upstream artifact.
- When a scenario fails after a pin upgrade, Cloud Health Office is fixed only if Cloud Health Office is wrong. A discrepancy may equally be an upstream bug or an IG ambiguity, and the harness does not assign blame automatically.
Version awareness, stated rather than assumed
Each result records which IG version each side was operating under. Where Cloud Health Office declares an IG family (STU 2.2.x) and the reference implementation reports a point release, the run reports a mismatch rather than claiming an agreement it cannot demonstrate. Two live examples the harness has recorded:
- The
Claim/$inquireOperationDefinitioncanonical differs between the two sides. It is recorded as a Warning and deliberately not adjudicated — the follow-up is to check the published IG and correct whichever side is wrong.Claim/$submit, which both sides name identically, is asserted. - Cloud Health Office advertises no
davinci-crd.versionextension in its own CDS Hooks discovery while the payer does, so a CRD client cannot tell which CRD version Cloud Health Office implements. Recorded, not adjudicated.
A Warning does not fail a scenario. Only assertions do, and they are limited to things that are unambiguously protocol requirements.
Security posture of the harness itself
- Opt-in. External scenarios skip unless explicitly enabled, so an ordinary test run never downloads or starts third-party code.
- No secrets. The interop stack references no repository secret, cloud credential or token, and no external container is given the Docker socket.
- Network isolation. External containers sit on their own bridge network with host ports bound to loopback.
- The trust model is untouched. There is no authentication-disable switch, and nothing here weakens the SMART middleware.
- Synthetic only. Every value that reaches an external implementation comes from one synthetic identity set — identifiers valid in format, so a format-validating implementation accepts them, and naming nobody.
Scenario inventory and the latest published run
The inventory is the source of truth for which scenarios exist; execution status comes from a harness run, never from the inventory. A scenario listed with no result in a run is reported NotRun — deliberately, because a placeholder must never look like a result.
Loading the published external interoperability evidence. It records the scenarios the harness runs, the pinned implementation each was run against, and what the last published run found.
What this evidence does not establish
Limitations
- This is not certification. Using an HL7 reference implementation does not make the result an HL7, CMS or ONC endorsement, and no independent body has assessed anything here.
- A pin is the version tested, not every implementation of the same specification, and not that implementation forever.
- Protocol interoperability is not payer rule parity. Cloud Health Office and the reference implementation are two different payers with two different rule sets; a difference in coverage decisions is not a defect, and comparing them on non-identical content would compare rule content rather than implementations. Parity on identical rule content is deferred.
- Cloud Health Office is the client in every executed scenario. The profile for running Cloud Health Office as the server under test exists but is exercised by no scenario yet.
- No CQL is executed. A returned Library's structure would be validated; its CQL is not run. This proves package exchange, not rule evaluation.
- FHIR structure is validated; DTR profile conformance is not independently validated. Resources are parsed and DTR-specific structure is checked by hand, but no IG profile validator is run against the StructureDefinitions. A declared
meta.profileis a claim by the sender, and is not treated here as validation. - No Inferno conformance suite executes. The PDex and DTR test kits are pinned and the runner seam exists; no suite has been run, and the inventory reports those scenarios as not executed rather than omitting them.
- Synthetic data only, and no exchange with a real payer's production system is claimed.
Designed to extend without redesign
The inventory, the version manifest and the evidence document are all list-shaped: adding PAS
$inquire against the same payer, an Inferno DTR payer-server suite, a PDex suite, or a
second reference implementation means adding rows, not reworking the harness. Scenarios that are defined
but not yet executed are published as such, so the surface a reader sees always distinguishes built
and run from declared.
Independently exercised, and pinned so you can reproduce it.
The scenario inventory, the pinned digests and the run command are all in the repository. One command starts the dependency, runs the exchange and tears the stack down.