Adding New Runners

This tutorial shows how to add a new prediction backend to COFOLDER without changing the shared validate, screen, or oracle workflows.

This is the authoritative contributor-facing guide for runner authoring. Other docs may summarize the runner system or preserve compatibility notes, but new runners should follow this document's contract and examples.

The key idea is:

• COFOLDER handles workflow orchestration, input mutation, repeats, result gathering, and analytics.
• A runner handles backend-specific dependency checks, option loading, optional backend-specific system preparation, execution, and normalization into COFOLDER's canonical bundle.

Acceptance Before Promotion

When runner work is ready for review, do not rely on unit tests alone.

Before promoting runner-, backend-, or shared CLI-adjacent changes toward main, run the manual backend acceptance lane documented in Backend Acceptance Tutorial. That lane verifies clean-install behavior and exercises the real validate, screen, and oracle CLI workflows in separate backend environments.

Supported runner capabilities

Runner	Components	Constraints	MSA behavior	Metric groups	Known limits
boltz1	Protein, ligand, DNA, RNA	Bond; one pocket at distance 6	Supplied or generated/reused Boltz MSA	Confidence	No contact constraints or affinity groups
boltz2	Protein, ligand, DNA, RNA	Bond, pocket, contact	Supplied or generated/reused Boltz MSA	Confidence, binding, affinity	Affinity depends on a compatible ligand/system activation
boltz-community	Protein, ligand, DNA, RNA	Bond, pocket, contact	Supplied or generated/reused Boltz MSA	Confidence, binding, affinity	Must not share an environment with conflicting PyPI boltz lines
openfold3	Protein, ligand, DNA, RNA	One pocket	Supplied MSA or OpenFold3 query setup	Confidence	No consumed bond/contact constraints and no affinity groups in the supported integration

All unsupported combinations fail during preflight or produce explicit unsupported metric states. Runner-specific confidence fields remain distinct when they are not semantically equivalent.

Where Runners Live

Runner modules are discovered from src/cofolder/modules/runners/.

Each runner file must expose a module-level RUNNER object. Discovery is handled by cofolder.modules.runners.discover_runners(), which scans the package and registers RUNNER.name.

Examples in the current codebase:

• src/cofolder/modules/runners/boltz1_runner.py
• src/cofolder/modules/runners/boltz_runner.py
• src/cofolder/modules/runners/boltz2_runner.py
• src/cofolder/modules/runners/boltz_community_runner.py

For the current contract, Boltz2Runner backed by BoltzRunner is the in-tree reference implementation. It already exercises the typed runtime surface, canonical normalized bundle layout, explicit metric_outcomes, and early boundary validation path used by the shared workflows.

Boltz1Runner is also on the supported contract path, but it is intentionally the narrower-capability example: it supports confidence metrics only, and affinity-related metric groups stay explicit unsupported outcomes rather than simulated payloads.

BoltzCommunityRunner follows the same full-capability supported path as Boltz2Runner, but it keeps a distinct coexistence rule: it must fail clearly if a PyPI boltz package-line install is present in the same environment.

Runner Lifecycle

At a high level, COFOLDER calls a runner in six steps:

1. check_availability() / ensure_available()
2. load_options(options_path)
3. validate_system(..., check_atom_names=False) on the source system
4. prepare_system(system_obj, options_obj, wrk_dir, conformers, sdf_file, logger) if the backend needs preparation
5. validate_system(..., check_atom_names=True) on the prepared system
6. run(request)

The supported contract types live in src/cofolder/modules/runners/contracts.py, and src/cofolder/modules/runners/base.py provides the default no-op preparation path for simple runners.

Import and author against RunnerExecutionRequest, RunnerExecutionResult, and RunnerPreparationResult.

Every runner must also declare input_capabilities = RunnerInputCapabilities(...). List only entity and constraint types that the adapter preserves and the backend actually consumes. BaseRunner.validate_system() applies this declaration and produces runner-specific preflight errors; override it only for additional native schema rules.

Input Provided By COFOLDER

COFOLDER gives the runner two kinds of input:

1. Preparation-time input

prepare_system(...) receives:

• system_obj: parsed system YAML object, already loaded by COFOLDER
• options_obj: runner-specific options object returned by load_options()
• wrk_dir: the shared raw working directory for the run
• conformers: optional conformer mode requested by the user
• sdf_file: optional SDF path provided by the user
• logger: workflow logger

The runner returns a RunnerPreparationResult object:

@dataclass(slots=True)
class RunnerRuntime:
    diffusion_samples: int = 1
    cache_path: str | None = None
    model_name: str | None = None


@dataclass(slots=True)
class RunnerPreparationResult:
    system_obj: Any
    options_obj: Any
    warnings: list[str] = field(default_factory=list)
    runtime: RunnerRuntime = field(default_factory=RunnerRuntime)

Use this stage for backend-specific preparation, for example:

• forcing a backend-specific option such as model: boltz2
• converting ligands into backend cache formats
• enriching the system object before COFOLDER writes the final YAML used for execution
• recording explicit runtime metadata such as cache location or diffusion sample count

If your backend does not need custom preparation, inherit BaseRunner.prepare_system(...) as-is. The default implementation is a no-op and already returns the original system, options, and empty typed runtime metadata.

2. Execution-time input

run(...) receives a RunnerExecutionRequest:

@dataclass(slots=True)
class RunnerExecutionRequest:
    runner_name: str
    system_name: str
    system_path: Path
    system_obj: Any
    options_path: Path
    options_obj: Any
    repeat: int
    seed: int
    seed_provenance: RepeatSeedProvenance
    backend_identity: RunnerBackendIdentity
    repeat_dir: Path
    raw_dir: Path
    logger: logging.Logger
    timings: Any | None = None
    label_prefix: str | None = None
    runtime: RunnerRuntime = field(default_factory=RunnerRuntime)
    identity: OutputIdentity | None = None
    chain_identities: tuple[RunnerChainIdentity, ...] = ()

Important fields:

• system_path: the YAML path COFOLDER wrote after shared input handling and runner preparation
• repeat_dir: runner-owned directory for one repeat, typically raw/repeat_<n>/
• seed: effective backend-facing seed, equal to seed_provenance.effective_seed
• seed_provenance: requested, resolved, derived, and effective seed metadata
• backend_identity: selected runner plus detected backend package/version status
• timings: optional timing collector used by the shared debug summary
• label_prefix: per-repeat label for timing metrics
• identity: the public invocation/repeat identity supplied by the recipe
• chain_identities: the validated runner-index-to-system-entity mapping; do not reconstruct chain identity from DataFrame row position

Private normalized runner boundary

The runner must normalize backend output into the private boundary structure below inside request.repeat_dir / "normalized". These files support validation and debugging; they are not the versioned public workflow output.

Required files:

• normalized/system_metrics.csv
• normalized/chain_metrics.csv
• normalized/manifest.json
• normalized/structures/ containing copied .cif, .mmcif, or .pdb files

The runner then returns a RunnerExecutionResult:

@dataclass(slots=True)
class RunnerExecutionResult:
    runner_name: str
    raw_output_dir: Path
    normalized_dir: Path
    structures_dir: Path
    system_metrics_path: Path
    chain_metrics_path: Path
    manifest_path: Path
    diffusion_samples: int = 1
    capabilities: set[str] = field(default_factory=set)
    warnings: list[str] = field(default_factory=list)
    runtime: RunnerRuntime = field(default_factory=RunnerRuntime)
    sample_records: list[dict[str, Any]] = field(default_factory=list)
    chain_identities: tuple[RunnerChainIdentity, ...] = ()
    records: list[PublicRecord] = field(default_factory=list)
    backend_identity: RunnerBackendIdentity | None = None
    seed_provenance: RepeatSeedProvenance | None = None

Built-in runners declare backend_name and backend_distribution. The inherited detect_backend_identity() uses installed distribution metadata and returns one of detected, unavailable, or unparseable; unavailable version metadata does not abort an otherwise usable run. Override resolve_effective_seed() only when a backend must transform COFOLDER's derived seed, and return an explicit backend_adjusted status and reason when doing so.

Metric Outcome States

The normalized bundle distinguishes runner metric-group outcomes with five exact states:

• computed: the required normalized payload is present and schema-valid
• unsupported: the selected runner does not support the requested metric group
• missing: the runner declares support, but the required normalized payload is absent
• failed: the runner attempted production, but the metric group did not complete successfully
• not_requested: the group belongs to the runner profile but was not requested

Unsupported requested groups warn and continue. A malformed, missing, or failed repeat becomes an output_validation failure record; other valid repeats continue. The workflow raises only when no repeat remains usable.

Shared validation no longer derives requested metric-group outcomes from CSV shape or runner capabilities. If a requested runner metric group matters to workflow behavior, the runner must emit an explicit metric_outcomes entry for it.

Boltz2Runner is the reference example of the supported authoring path: it emits explicit metric_outcomes directly from runner-owned normalization code, and shared validation only verifies that authored state instead of inventing backend-specific logic in recipes.

Boundary Validation

COFOLDER validates the normalized bundle immediately after run(...) returns and before shared gather steps begin.

That validation checks:

• required bundle files and directories exist under normalized/
• canonical CSV columns are present
• structure files and sample_records agree
• typed records and explicit chain identities satisfy the public identity contract
• explicit metric outcomes, if supplied, are consistent with runner capabilities and payload shape

If validation fails, the workflow stops after the attempted runner execution, and partial artifacts remain in raw/repeat_<n>/ for debugging.

After that, COFOLDER takes over again:

• normalized structures are gathered into results/structures/
• per-repeat private CSVs are converted into the versioned public record bundle under results/
• shared analytics such as structure metrics, reproduction metrics, bias, and robustness run on the normalized bundle

The merge step is implemented in src/cofolder/modules/analytics/aggregation.py.

manifest.json may still include runtime_context for debugging and provenance, but shared recipe logic should rely on the typed RunnerRuntime contract rather than re-reading loose runtime dictionaries.

Canonical CSV Expectations

At minimum, the normalized CSVs should include these common columns. Treat them as the shared contract floor, not as the complete set of scores your backend may expose.

system_metrics.csv

• cif_file
• model_name
• repeat
• diffusion_sample

Optional runner-native metrics can then be added, for example:

• ptm
• iptm
• confidence_score
• sample_ranking_score
• avg_plddt
• has_clash

chain_metrics.csv

• conf_chain_id
• cif_file
• model_name
• repeat
• diffusion_sample

COFOLDER later adds shared chain metadata such as:

• CHAIN_ID
• ENTITY_TYPE
• ligand_molecule_id

If your runner supports affinity outputs, add the normalized chain-level columns COFOLDER already expects:

• affinity_pred_value
• affinity_probability_binary
• pIC50
• IC50_M
• pIC50_kcal_per_mol

Backend-native chain metrics that are not part of the shared minimum contract may also be added explicitly, for example chain_pair_iptm_A_B.

Score Semantics

The normalized CSV contract is intentionally conservative:

• Canonical columns are the minimum shared interface that downstream COFOLDER code may rely on.
• Backend-native extra columns are encouraged when they preserve useful meaning that does not fit an existing canonical column.
• Alias mapping into a canonical column is only acceptable when the backend-native field is truly semantically equivalent.

Current guidance for confidence-like outputs:

• Shared validation is intentionally backend-agnostic for confidence_metrics. If your runner returns an explicit confidence_metrics outcome, shared code treats that outcome as authoritative instead of enforcing backend-specific confidence column names or counts.
• This means each x_runner.py owns its own confidence completeness rules, confidence naming, and any decision to preserve backend-native confidence fields as scalar columns or companion artifacts.
• If your runner declares required_columns or required_artifacts on confidence_metrics, keep those references aligned with the actual normalized bundle so contradictory outputs still fail fast.
• Alias mapping into a canonical confidence column is still only acceptable when the backend-native field is truly semantically equivalent. If it is not, keep the backend-native score explicit rather than inventing equivalence.
• OpenFold3 is a motivating example: fields such as sample_ranking_score, avg_plddt, gpde, chain_pair_iptm_*, and bespoke_iptm_* should remain runner-owned confidence outputs unless the runner can honestly justify a narrower alias.

This distinction matters because COFOLDER treats validation metrics, model-derived confidence metrics, and affinity-related outputs as different concepts.

Companion Artifacts

Some confidence outputs are not scalar CSV values. Per-atom or pairwise arrays such as plddt, pae, or pde should be preserved as normalized companion artifacts instead of being silently dropped or squeezed into a scalar column.

For backends such as OpenFold3, these companion artifacts may be part of the runner's confidence_metrics surface rather than optional debugging extras. Shared validation only verifies that declared companion artifacts are well-formed and exist inside normalized/; the runner decides which of them belong to confidence semantics. By contrast, runtime or provenance files such as timing.json remain runtime metadata and should not be modeled as confidence artifacts unless the backend has a separate, explicit reason to expose them that way.

Recommended pattern:

• place non-tabular artifacts under normalized/artifacts/
• reference them in manifest.json under companion_artifacts
• return the same metadata through RunnerExecutionResult(companion_artifacts=[...])

Each companion artifact entry should include:

• label: stable backend-facing name such as pae or plddt
• relative_path: path relative to normalized/
• optional kind: short shape hint such as matrix or per_atom
• optional description: concise human-readable explanation

Example manifest fragment:

{
  "companion_artifacts": [
    {
      "label": "pae",
      "relative_path": "artifacts/pae.npy",
      "kind": "matrix",
      "description": "Predicted aligned error matrix."
    }
  ]
}

Capabilities

Each runner declares the metric groups it can natively provide through capabilities.

Current shared runner metric groups are:

• confidence_metrics
• affinity_metrics
• affinity_metrics_ext

If a user requests a metric group the selected runner does not support, COFOLDER warns and continues. The corresponding public metric records use status="unsupported" and value=null.

This means you should declare only the metrics your backend can truly normalize.

Minimal Runner Skeleton

The following skeleton shows the expected shape:

from __future__ import annotations

import json
from pathlib import Path

import pandas as pd

from cofolder.modules.runners import (
    BaseRunner,
    RunnerExecutionRequest,
    RunnerExecutionResult,
    RunnerMetricOutcome,
    RunnerRuntime,
    build_runner_public_records,
)


class MyRunner(BaseRunner):
    name = "my-runner"
    capabilities = {"confidence_metrics"}

    def check_availability(self) -> tuple[bool, str | None]:
        return self.check_distribution_available(
            distribution_name="my-backend",
            missing_message=(
                "The selected 'my-runner' backend is not installed. "
                'Install it with `pip install "cofolder[my-runner]"`.'
            ),
        )

    def load_options(self, options_path: Path):
        # Return any runner-specific options object you want.
        return {"options_path": str(options_path)}

    def run(self, request: RunnerExecutionRequest) -> RunnerExecutionResult:
        raw_output_dir = request.repeat_dir / "my_backend_output"
        normalized_dir = request.repeat_dir / "normalized"
        structures_dir = normalized_dir / "structures"
        structures_dir.mkdir(parents=True, exist_ok=True)
        raw_output_dir.mkdir(parents=True, exist_ok=True)

        # 1. Run your backend here.
        # 2. Copy normalized structure files into normalized/structures/.
        # 3. Build canonical CSVs and preserve any backend-native extras.
        structure_name = f"{request.repeat}_{request.system_name}_model_0.cif"
        (structures_dir / structure_name).write_text("data_demo", encoding="utf-8")

        system_df = pd.DataFrame([
            {
                "cif_file": structure_name,
                "model_name": request.system_name,
                "repeat": request.repeat,
                "diffusion_sample": 0,
                "ptm": 0.82,
                "iptm": 0.76,
                "confidence_score": 0.91,
            }
        ])
        chain_df = pd.DataFrame([
            {
                "conf_chain_id": 0,
                "cif_file": structure_name,
                "model_name": request.system_name,
                "repeat": request.repeat,
                "diffusion_sample": 0,
                "chains_ptm": 0.88,
            }
        ])

        system_df = attach_runner_provenance(system_df, request)
        chain_df = attach_runner_provenance(chain_df, request)
        system_metrics_path = normalized_dir / "system_metrics.csv"
        chain_metrics_path = normalized_dir / "chain_metrics.csv"
        system_df.to_csv(system_metrics_path, index=False)
        chain_df.to_csv(chain_metrics_path, index=False)

        sample_records = [
            {
                "repeat": request.repeat,
                "diffusion_sample": 0,
                "cif_file": structure_name,
            }
        ]
        sample_records = attach_sample_provenance(sample_records, request)

        manifest_path = normalized_dir / "manifest.json"
        manifest = {
            "runner": self.name,
            "backend": backend_manifest_value(request.backend_identity),
            "seed": seed_manifest_value(request.seed_provenance),
            "capabilities": sorted(self.capabilities),
            "repeat": request.repeat,
            "raw_output_dir": str(raw_output_dir),
            "normalized_dir": str(normalized_dir),
            "system_metrics_path": str(system_metrics_path),
            "chain_metrics_path": str(chain_metrics_path),
            "structures_dir": str(structures_dir),
            "diffusion_samples": 1,
            "runtime_context": {"diffusion_samples": 1},
            "warnings": [],
            "sample_records": sample_records,
        }
        manifest_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8")

        return RunnerExecutionResult(
            runner_name=self.name,
            raw_output_dir=raw_output_dir,
            normalized_dir=normalized_dir,
            structures_dir=structures_dir,
            system_metrics_path=system_metrics_path,
            chain_metrics_path=chain_metrics_path,
            manifest_path=manifest_path,
            diffusion_samples=1,
            capabilities=set(self.capabilities),
            runtime=RunnerRuntime(diffusion_samples=1),
            sample_records=sample_records,
            chain_identities=request.chain_identities,
            records=build_runner_public_records(system_df, chain_df, request),
            backend_identity=request.backend_identity,
            seed_provenance=request.seed_provenance,
            metric_outcomes={
                "confidence_metrics": RunnerMetricOutcome(
                    state="computed",
                    required_columns=(
                        "system_metrics.ptm",
                        "system_metrics.iptm",
                        "system_metrics.confidence_score",
                        "chain_metrics.chains_ptm",
                    ),
                ),
            },
        )


RUNNER = MyRunner()

Recommended Development Steps

When adding a new runner, this is the safest sequence:

1. Add the runner module under src/cofolder/modules/runners/.
2. Implement check_availability() first, so CLI selection fails early with a useful install message.
3. Implement load_options() and only override prepare_system() if you truly need backend-specific mutation.
4. Implement run() to write a complete normalized bundle even before all metrics are available.
5. Declare only the metric groups you can truly support in capabilities and molecular inputs you can faithfully consume in input_capabilities.
6. Add targeted tests for:
7. runner discovery
8. availability checking
9. normalized bundle creation
10. any version- or backend-line-specific behavior
11. supported and rejected entity/constraint inputs

Testing a New Runner

At minimum, add:

• one discovery test to confirm the runner appears in list_runner_names()
• one availability test to confirm the install hint is actionable
• one bundle test to confirm system_metrics.csv, chain_metrics.csv, manifest.json, and normalized structures are produced

The existing runner tests are good templates:

• tests/modules/runners/test_boltz1_runner.py
• tests/modules/runners/test_boltz_runner.py
• tests/modules/runners/test_boltz_community_runner.py
• tests/modules/runners/test_contracts.py
• tests/modules/runners/test_validators.py
• tests/modules/runners/test_registry.py

Practical Notes

• Prefer normalizing backend-specific output inside the runner, not in validate.
• Keep raw backend artifacts inside the runner-owned repeat directory and expose them through manifest.json.
• Keep normalized companion artifacts under normalized/ and declare them consistently in both the manifest and RunnerExecutionResult.
• If you declare runner-authored required_columns or required_artifacts, make sure they point only to payload entries that actually exist in the normalized bundle.
• If your backend does not support a metric group, do not fake support. Let COFOLDER handle the warning and empty-column behavior.
• If your backend line has version-specific behavior, encode that directly in check_availability(), as the current Boltz runners do.

Summary

A new COFOLDER runner only needs to do three things well:

1. Check that the correct backend is installed.
2. Execute the backend from the standardized RunnerExecutionRequest.
3. Write the normalized bundle that the shared analytics pipeline expects.

If you keep that boundary clean, the existing validate, screen, oracle, gather, and analytics code can stay unchanged.