omds

OMDS — oh-my-datascience

One Python package (omds) that does two things over the same cores:

1. A data-science guardrail toolkit for coding agents. A set of omds-*
CLIs plus portable [Agent Skills](skills/) that let any coding agent check
Python for train/test leakage and metric misuse, trace data lineage through
a code graph, and keep a durable experiment ledger and episodic memory.
Optionally reachable as an MCP server or as a Claude Code plugin.

2. E-GDS — an offline evolutionary optimizer for ML pipelines. The omds
CLI (init/evolve/promote/inspect/models) evolves a pipeline
workspace's own code with GEPA (Genetic-Pareto reflective evolution),
targets mutations using a code graph, and admits a variant only through five
blocking gates ending in a real, sandboxed training run on holdout data.

These are not two codebases. Both halves stand on the same **ontological
grounding layer** (omds.ontology — probability-simplex bounds, proper scoring
rules, leakage rules, sparsemax caveats) and the same code graph layer
(omds.codegraph — AST dependencies + data lineage). The axiom registry that
blocks a GEPA promotion is literally the registry omds-guardrails check runs
against the file your agent just edited: both call
omds.ontology.static_check.run with omds.ontology.axioms.default_registry.

``
omds.ontology omds.codegraph
(axiom registry, static AST rules, (AST deps, def-use chains,
runtime assertion injection) data lineage, git blame)
│ │
┌─────────────┴──────────────┬───────────────────┴──────────────┐
│ │ │
omds-* CLIs + skills/ omds-mcp (9 tools) E-GDS: omds evolve
Claude Code plugin → GEPA → 5 gates
PRIMARY surface OPTIONAL adapters → internal PR
(agent-neutral) (same cores, no new logic) (offline optimizer)
`

Everything is model-agnostic and SLM-first: the LLM layer speaks only
OpenAI-compatible /v1/chat/completions, so it runs against local small models
(Qwen2.5-Coder and friends via Ollama / llama.cpp / vLLM / LM Studio) — the
design floor every prompt must work within — and swapping in a frontier API
model is a config change. The guardrail toolkit needs no model at all.

Status

Both halves are shipped and green. uv run pytest reports **730 passed,
6 skipped** (all six skips are opt-in tiers — see [Testing](#testing) — plus 2
pre-existing sklearn UndefinedMetricWarnings). The nine MCP tools, the
omds-* CLIs, the seven skills, the plugin bundle, the GEPA evolution engine,
the omds-slm local-SLM harness, and the benchmarks/tabular_multiclass/
benchmark (which passes all five promotion gates as its own seed) are all
covered. The one thing no test in the default suite exercises is a live
model: that path is smoke-tested behind an opt-in gate (see **Local SLM
support (≤7B)** under [Shared](#shared)).

One capability was deliberately removed, not deferred. v1 shipped an
omds run command driving a multi-agent runtime loop (AutoGen GraphFlow
step agents, task decomposition, a repair loop). That whole stack was deleted
in the v2 harness rework — there is no omds run today, and the
autogen-* dependencies are gone. What survives of omds.runtime is the
model-agnostic, network-free part the evolution gates reuse: task loading, the
skills catalog, and validate_workspace. Traces are now written by the
evolution evaluator (one per candidate evaluation), not by an agent loop, and
omds inspect traces reads those. Rationale:
[docs/adr/2026-07-18-mcp-pivot.md](docs/adr/2026-07-18-mcp-pivot.md).

- Architecture decision record: [docs/adr/2026-07-18-mcp-pivot.md](docs/adr/2026-07-18-mcp-pivot.md)
(read the Decision and both amendments — they reshape it)
- v1 design spec: [docs/specs/2026-07-11-egds-design.md](docs/specs/2026-07-11-egds-design.md)
- v2 harness design: [docs/specs/2026-07-12-omds-harness-design.md](docs/specs/2026-07-12-omds-harness-design.md)

Contents

New here? [Prerequisites](#prerequisites) → [Install](#step-1-install-the-python-package-required-for-every-route)
→ [Confirm it works](#step-2-confirm-it-works-about-a-minute) →
[Wire it into your agent](#step-3-wire-it-into-a-coding-agent--pick-one). That is
the whole setup, and it needs no model and no network.

| Part 1 — agent toolkit | Part 2 — E-GDS optimizer | Shared |
| ---------------------- | ------------------------ | ------ |
| [Getting started](#getting-started) | [Quickstart](#quickstart) | [Local SLM support (≤7B)](#local-slm-support-7b--omds-slm) |
| [Skills](#skills--the-primary-surface) | [How it works](#how-it-works) | [SLM-first design rules](#slm-first-design-rules) |
| [CLIs](#clis--the-primary-surface) | [The promotion gates](#the-promotion-gates-run-in-order-all-blocking) | [Repository layout](#repository-layout) |
| [MCP server](#mcp-server--an-optional-adapter) | [Config reference](#config-reference-omdsyaml) | [Testing](#testing) |
| [RLM / REPL agents](#rlm--repl-agents--the-direct-import-surface) | | |
| [In-kernel leakage guard](#in-kernel-leakage-guard--opt-in) | [Testing without a live model](#testing-without-a-live-model) | [Verifying a real install](#verifying-a-real-install) |
| [Troubleshooting](#troubleshooting) | | [Non-goals](#non-goals) · [License](#license) · [History](#history) |

---

Part 1 — the agent-facing toolkit

Getting started

Prerequisites

- Python 3.12 or newer (requires-python = ">=3.12"). Nothing older works.
3.12, 3.13 and 3.14 are each verified by a real install-and-run, not just
declared.
- git — omds-codegraph blame shells out to it, and E-GDS clones workspaces
with git worktree. The rest of the toolkit runs fine without a repo.
- [uv](https://docs.astral.sh/uv/) — the recommended installer (see Step 1),
and required for a checkout or for Part 2. Plain pip works for the toolkit
if you would rather not add another tool.
- No model, no API key, no network. Everything in Part 1 is static analysis
and local files. A model is needed only for Part 2 (E-GDS) and omds-slm.

Step 1: install the Python package (required for every route)

OMDS is installed from this repository, not from PyPI — pip install omds
will fail with "No matching distribution found". Pick the route that matches
what you want to do; they are not exclusive, and the third is a superset.

(a) Just the agent toolkit — uv tool install. OMDS is mostly ten command
line tools, which is exactly what uv tool is for: one command, all ten on your
PATH, in an isolated environment, with no venv to create or activate.

`bash
uv tool install --python 3.12 \
"git+https://github.com/spkc83/omds.git#subdirectory=python/omds"
`

> Pass --python 3.12. It is not optional padding. On uv 0.5.x, omitting it
> can build the tool environment on Python 3.11 despite the package requiring
> >=3.12, and then install 3.12-built binaries into it — omds-guardrails
> fails with a numpy C-extension ImportError that looks like a broken numpy
> rather than a wrong interpreter. Newer uv handles this; pinning costs nothing
> and works on both.

(b) Into your own project environment — when you want the in-kernel half.
Route (a) is deliberately isolated, so omds is not importable from your own
Python: from omds import kernel_guard raises ModuleNotFoundError even though
the CLIs work. The [in-kernel leakage
guard](#in-kernel-leakage-guard--opt-in) and the omds.dstools helpers the
visualization skill uses run inside your kernel, so they need the package
installed alongside your own code:

`bash
uv pip install "git+https://github.com/spkc83/omds.git#subdirectory=python/omds"


plain pip works identically:

(c) From a checkout — required for Part 2. omds init copies benchmark
files that are not shipped inside the wheel, so E-GDS needs the repo itself:

`bash
git clone https://github.com/spkc83/omds.git
cd omds
uv sync # installs omds editable + the test toolchain


or, without the uv workspace: uv pip install -e python/omds

Every route gives you the ten console scripts (omds, omds-guardrails,
omds-codegraph, omds-dstools, omds-memory-store, omds-sessionpad,
omds-skill, omds-mcp, omds-install, omds-slm) and the seven skill files
bundled inside the package. All three wiring routes below assume this step —
in particular a Claude Code plugin cannot install a Python package, so it is
never optional.

Step 2: confirm it works (about a minute)

Worth doing before wiring anything into an agent — it separates "the package is
broken" from "my agent is not calling it". Create a file with a textbook leak in
it:

`bash
mkdir /tmp/omds-demo && cd /tmp/omds-demo
cat > pipeline.py <<'EOF'
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

def prep(X, y):
scaler = StandardScaler()
scaler.fit(X) # fit on everything...
X_train, X_test, y_train, y_test = train_test_split(X, y) # ...then split
return X_train, X_test, y_train, y_test
EOF

omds-guardrails check --files pipeline.py
`

That prints one violation — the scaler saw the test rows before the split:

`json
{"violations":[{"axiom":"ax:fit_on_train_only","severity":"certain","file":"pipeline.py","line":7,
"message":"pipeline.py:7: scaler/encoder fit on full data before split - fit() argument 'X' is also the full dataset passed to train_test_split in this function",
"needs_runtime_proof":false}],"summary":{"certain":1,"warn":0}}
`

If you see {"violations":[],"summary":{"certain":0,"warn":0}}, the package
works but the pattern was not matched — note that the fit-then-split sequence has
to sit inside a function to be detected. A command not found means the
install did not put the scripts on your PATH; see
[Troubleshooting](#troubleshooting).

Two more, to see the rest of the surface:

`bash
omds-codegraph lineage --symbol prep --root .


{"lineage":[{"from":"pipeline.py::<module>","to":"pipeline.py::prep","kind":"defines"}]}

omds-skill lint skills/leakage-audit # only from a checkout


{"ok":true,"checks":[{"check":"format","passed":true,...}, ...]}

(a) omds-install — the recommended route; works from the installed package
alone, because the skill files ship inside the wheel:

`bash
cd /path/to/your/project
omds-install claude # --project defaults to the current directory
`

`
Merged omds MCP server into .mcp.json
Copied skills into .claude/skills
`

It writes exactly two things, and nothing else:

`jsonc
// .mcp.json — merged in, never overwritten
{"mcpServers": {"omds": {"type": "stdio", "command": "omds-mcp"}}}
`

`
.claude/skills/{eda,evaluation,feature-engineering,leakage-audit,
model-training,reporting,visualization}/
`

For any other agent, name the two paths yourself:

`bash
omds-install here --config <mcp-config.json> --skills-dir <that agent's skills dir>
`

claude is literally here with .mcp.json and .claude/skills pre-filled;
there is no per-agent behavior, only per-agent paths. The merge into an existing
MCP config is non-destructive and idempotent — rerunning it is safe, and it will
not disturb MCP servers you already have configured.

(b) By hand — nothing above is magic. The skills are just files and the MCP
entry is one line, so from a checkout:

`bash
cp -r skills/* .claude/skills/ # the primary surface
claude mcp add --scope project omds -- omds-mcp # the optional MCP adapter
`

(c) The Claude Code plugin — this repo is itself a plugin
(.claude-plugin/plugin.json plus a single-plugin marketplace), packaging the
same skills/, the same omds-mcp server, and a guardrail hook:

`bash
/plugin marketplace add spkc83/omds
/plugin install omds@omds
`

To try a local checkout without going through the marketplace, launch Claude
Code with claude --plugin-dir . from the repo root.

The plugin route additionally installs a PostToolUse hook that runs
omds-guardrails check on every Python file Claude writes or edits, blocking
the turn on an axiom-certain leakage violation. Skills are namespaced under
it as omds:<skill-name> (e.g. omds:leakage-audit). The hook is stdlib-only
and fails open in every abnormal state — including omds-guardrails not
being on PATH — so an uninstalled package makes it silently do nothing rather
than break your editing.

Step 4: confirm the agent can see it

In a Claude Code session, in the project you just wired up:

| Check | Expect |
| ----- | ------ |
| /mcp | omds listed and connected. (Route (a)/(b) prompt for approval on first use; the plugin route does not.) |
| ask it to list skills | leakage-audit and six others — namespaced omds:leakage-audit under the plugin route |
| "check pipeline.py for data leakage" | it runs guardrails_check / omds-guardrails check and reports ax:fit_on_train_only |

If /mcp shows omds as failed, run omds-mcp in your shell: it should start
and wait silently for stdio input (Ctrl-C to exit). A traceback instead means
the agent is reporting a real install problem — see
[Troubleshooting](#troubleshooting).

For the whole install checked mechanically in one command, see [Verifying a real
install](#verifying-a-real-install).

Skills — the primary surface

[skills/](skills/) holds seven agent-neutral SKILL.md files — eda,
feature-engineering, model-training, evaluation, visualization,
reporting, leakage-audit. Nothing in them is tied to a particular agent;
any tool that reads the SKILL.md format (Claude Code, Codex, Cursor, Gemini
CLI, Copilot, …) can use them. Each teaches one workflow in terms of the real
omds-* CLIs, so installing the package is enough to make every step actionable.
See [skills/README.md](skills/README.md) for the format and for adding one.

omds-skill lint is the contract, executable. Writing a skill by hand is
easy to get subtly wrong, and the expensive mistake is a skill that tells an
agent to run a command that does not exist:

`bash
omds-skill lint skills/leakage-audit # a directory or its SKILL.md


{"ok":true,"checks":[{"check":"format","passed":true,...}, ...]}

Six checks run — format, description_shape, no_coupling_tokens,
surface_references, size, distinctness — and **all of them are reported
even after one fails**, because an author fixing a skill wants the whole list in
one pass. The one that earns its keep is surface_references: it resolves every
omds- command, MCP tool and omds. module path a skill mentions against the
live registries (the installed console-script entry points, the Typer apps
behind them, and the MCP tool manager) rather than a hardcoded list — a
hardcoded list being exactly the drift it exists to catch. It found real drift
the first time it ran: eda's own description cited omds_dstools.df_summary,
a module name that has not existed since the monopackage merge.

test_skills_portable.py calls the same lint_skill core, so the command you
run by hand and the assertion the suite makes are the same code, not two
descriptions of it.
Distinctness is checked against the shipped set because skills/README.md's
warning is literal: a vague or duplicated description degrades routing for every
other skill, not just the new one.

CLIs — the primary surface

Every command exits 0 always: the decision lives in the JSON payload, never
in the exit status. Stdout is nothing but JSON; logs go to stderr.

check analyzes the files you name, but builds its code graph over their common
parent directory — so it prunes virtualenvs (by name and by pyvenv.cfg),
caches and node_modules rather than walking them, and a file anywhere in the
tree that cannot be parsed is skipped rather than fatal. A file **you asked
about** that could not be read is different: it is named in an unreadable
key, present only when non-empty, because a guardrail must never answer "clean"
for a file it never managed to read.

`bash


static leakage / scoring / shape checks over on-disk files…

omds-mcp (official mcp SDK, FastMCP, stdio) exposes exactly nine tools.
Each one calls the same core function its CLI calls — guardrails_check and
omds-guardrails check both land in run_static_check — so the tool result is
byte-identical to the CLI's --json stdout and there is no second copy of any
logic. That DRY property is why keeping three surfaces costs almost nothing.

The server is not required. It exists for hosts that sandbox shell access
or prefer typed tool calls. The mapping (identical to skills/README.md):

| CLI (primary) | Equivalent MCP tool |
| -------------------------- | --------------------- |
| omds-guardrails check | guardrails_check |
| omds-codegraph lineage | codegraph_lineage |
| omds-codegraph blame | codegraph_blame |
| omds-codegraph context | codegraph_context |
| omds-dstools ledger show | dstools_ledger_show |
| omds-memory-store add | memory_add |
| omds-memory-store recall | memory_recall |
| omds-sessionpad append | sessionpad_append |
| omds-sessionpad show | sessionpad_show |

Domain outcomes are normal results, never protocol errors: a leakage violation,
an unresolved symbol, a file outside a git repo, an empty ledger, or a session
with no memory DB all return a payload. Only genuine faults (an unreadable
path, unparseable input) surface as tool errors. codegraph_context takes
detail ("summary" default / "full") and max_items so a small-window
model is never blown up by an unbounded dump — counts still reports the true
totals and truncated says whether anything was elided.

The visualization skill additionally uses the in-kernel omds.dstools
helpers (df_summary, capture_plot, ledger), which have no CLI or MCP
equivalent because they run inside your own Jupyter/IPython kernel.
omds.dstools.install() publishes them onto builtins for later cells.

RLM / REPL agents — the direct-import surface

A growing class of agents replaces JSON tool calling with a **persistent Python
environment**: the model writes code, tools are ordinary functions, and big
results live in variables the model slices instead of re-reading. That is the
Recursive Language Model design ([Zhang, Kraska & Khattab,
arXiv:2512.24601](https://arxiv.org/abs/2512.24601)) and the shape of hosts
built on it, such as [Prime
Agent](https://github.com/PrimeIntellect-ai/prime-agent), whose one built-in
tool is an IPython kernel.

For such hosts, **the best OMDS surface is neither MCP nor the CLIs — it is a
plain import**. Every OMDS tool is a Python function returning a JSON-ready
dict, and the nine functions below are the same objects the CLIs, the MCP
server and omds-slm dispatch to. In a REPL-native agent, the import is
the integration:

`python
from omds.guardrails_cli import run_static_check

report = run_static_check(files=["leaky.py"], diff=None) # a plain dict
print(report["summary"])


{'certain': 1, 'warn': 0}

The nine cores, by tool name (the names omds-mcp and omds-slm also use):

| Tool name | Import |
| --------- | ------ |
| guardrails_check | from omds.guardrails_cli import run_static_check |
| codegraph_lineage | from omds.codegraph_cli import lineage_json |
| codegraph_blame | from omds.codegraph_cli import blame_json |
| codegraph_context | from omds.codegraph_cli import context_json |
| dstools_ledger_show | from omds.dstools_cli import ledger_show_json |
| memory_add | from omds.memory_store_cli import memory_add_json |
| memory_recall | from omds.memory_store_cli import memory_recall_json |
| sessionpad_append | from omds.sessionpad_cli import sessionpad_append_json |
| sessionpad_show | from omds.sessionpad_cli import sessionpad_show_json |

All nine follow the repo-wide contract: they return their result (never
print), they raise only for programmer errors (an unknown keyword, say), and a
"finding" — a leakage violation, an empty lineage — is data in the payload,
not an exception.

The 7 [skills](#skills--the-primary-surface) work in these hosts unchanged:
they follow the [Agent Skills](https://agentskills.io) format, and each
SKILL.md has been checked against the validation rules Prime Agent's skill
loader enforces (frontmatter name matching the directory, lowercase
a-z0-9- names ≤ 64 chars, non-empty description ≤ 1024 chars,
instruction-only — no stray pyproject.toml that would mark a skill
Python-backed). In-kernel extras (omds.dstools.df_summary, capture_plot)
are usable the same way, since a REPL host is a kernel.

To point a local small model at this surface instead of a frontier host,
see the experimental
[omds-slm repl](#the-experimental-repl-protocol--omds-slm-repl) protocol,
which wraps these same functions in a resource-limited worker under the C5
turn cap.

In-kernel leakage guard — opt-in

The static checks above read code. The other half is a runtime guard that
fires only on data that actually reaches fit():

`python
from omds import kernel_guard
kernel_guard.install()

X_test = kernel_guard.tag_test(X_test)
StandardScaler().fit(X_test) # raises LeakageViolation
StandardScaler().fit(X_train) # untagged data is forwarded verbatim
`

It is a tagged-object check, not a taint-tracking system: install() wraps
every sklearn estimator's fit, is idempotent, and fails open to a no-op if
sklearn/numpy cannot be imported. The violation message embeds a delimited JSON
payload (parse_violation_payload recovers it) so an out-of-process reader —
an agent reading a cell's error output, a log scraper — gets structure out of
what a kernel serializes as plain traceback text.

To arm it for every kernel in an environment without a bootstrap cell:

`bash
omds-guardrails install-kernel-hook # writes omds_autostart.pth into site-packages
omds-guardrails kernel-hook-status
omds-guardrails uninstall-kernel-hook
`

Read the warning before running that. A .pth executes at interpreter
startup for every Python process in that environment — scripts, CI jobs,
python -c, all of it — not just kernels. It is written to make that nearly
free (it imports only sys at startup, and the guard itself installs only
inside an ipykernel process), but it is still an environment-wide change.
Prefer a venv you own over a shared or system interpreter. Installing the
package deliberately does not arm it: shipping the .pth in the wheel would
make that decision on your behalf, so it is a separate, explicit, reversible
opt-in.

Troubleshooting

pip install omds → "No matching distribution found"
Expected — OMDS is installed from the repository, not PyPI. Use one of the three
routes in [Step 1](#step-1-install-the-python-package-required-for-every-route).

**ModuleNotFoundError: No module named 'omds' in your own Python or notebook,
even though the omds- commands work*
You installed with uv tool install, which is isolated by design — it puts the
CLIs on PATH without putting the package on your sys.path. That is correct
for the CLIs and wrong for the in-kernel guard and omds.dstools. Install it
into the environment your kernel runs, per
[Step 1(b)](#step-1-install-the-python-package-required-for-every-route). Having
it in both places is fine.

omds-guardrails fails with a numpy C-extension ImportError
Almost always a tool environment built on the wrong interpreter — the message
blames numpy, but the cause is a Python 3.11 environment holding 3.12-built
binaries. The traceback names the offending interpreter ("The Python version
is: Python 3.11 from …"). Rebuild the environment pinned:

`bash
uv tool install --force --python 3.12 \
"git+https://github.com/spkc83/omds.git#subdirectory=python/omds"
`

omds-guardrails: command not found
The package installed into an environment whose bin/ is not on your PATH —
almost always a virtualenv that is not activated. Check where it went:

`bash
python -c "import omds, pathlib; print(pathlib.Path(omds.__file__).parent)"
`

Then either activate that environment, or call the scripts by full path
(/path/to/venv/bin/omds-guardrails). Configure your agent with the **absolute
path** to omds-mcp if it does not inherit your shell's PATH — a common cause
of an MCP server that works in a terminal but not in the agent.

The install failed on the Python version
omds requires 3.12+. python -V to check; install into a newer
interpreter (uv venv --python 3.12).

check reports no violations on obviously leaky code
The static packs match specific shapes, not intent. ax:fit_on_train_only
needs the fit-then-split sequence inside a function — at module level it is
not matched. This is a deliberate precision-over-recall trade: a guardrail that
cries wolf gets turned off. The runtime
[in-kernel guard](#in-kernel-leakage-guard--opt-in) is the complement, firing on
data that actually reaches fit() regardless of code shape.

The output has an unreadable key
A file you asked about could not be read or parsed — a non-UTF-8 encoding, a
syntax error, a permissions problem; reason says which. Those files were
not analyzed, so treat them as unchecked rather than clean. Files elsewhere
in the directory that cannot be parsed are simply skipped and never appear here.

MCP server shows as failed in the agent
Run omds-mcp directly in a shell. It should start and sit silently waiting for
stdio input (Ctrl-C to quit) — that is a healthy server, since it speaks a
protocol, not a CLI. A traceback or command not found is the real problem;
see the PATH note above.

The plugin's guardrail hook never fires
By design it fails open in every abnormal state, including
omds-guardrails not being on PATH — a missing package makes it do nothing
rather than break your editing, which also means a broken install looks exactly
like a clean file. Confirm omds-guardrails check --files <a leaky file> works
in your shell first. Note the hook only inspects .py files that Claude
writes or edits.

/plugin marketplace add spkc83/omds cannot find the plugin
The marketplace manifest is read from the repository's default branch. Use
claude --plugin-dir . from a checkout to test the plugin without publishing.

omds: command not found / omds init cannot find benchmark files
Part 2 needs the repository, not just the installed package — omds init
copies benchmark files that are not shipped inside the wheel. Clone and
uv sync.

Still stuck? [scripts/verify_install.py](scripts/verify_install.py) checks
every mechanical part of an install in one command and names what it cannot
check — see [Verifying a real install](#verifying-a-real-install).

---

Part 2 — E-GDS, the offline evolutionary optimizer

Quickstart

`bash


1. install (from a checkout — omds init copies benchmark files that are not

workspace its own git history on main (evolve/promote clone it via

git worktree and need a repo to clone from)

omds inspect also takes traces, violations, and graph;
omds models probe -c mytask/omds.yaml records each configured model's
capabilities into $OMDS_HOME/models.json (default ~/.omds/models.json).

omds.yaml (written by init alongside task.yaml) points at
http://localhost:11434/v1 by default — edit llm.base_url /
llm.roles.*.model to point at any other OpenAI-compatible endpoint or model.
No live endpoint handy? evolve and models probe also work against the
network-free fake:// scheme (see
[Testing without a live model](#testing-without-a-live-model)).

How it works

`
+---------------------------------------------------------------------------+
| task.yaml + workspace/ (goal, data roles, metrics, bounds, eval budget) |
+---------------------------------------------------------------------------+
|
+---------------------------------------------------------------------------+
| ONTOLOGICAL GROUNDING LAYER (omds.ontology) |
| Axiom registry · builtin packs (leakage, scoring, evaluation, shapes, |
| simplex_transforms) · static AST rules · runtime assertion injection |
+---------------------------------------------------------------------------+
|
+---------------------------------------------------------------------------+
| CODE GRAPH LAYER (omds.codegraph) |
| AST dependency extraction · def-use chains · data lineage · blame |
+---------------------------------------------------------------------------+
|
+---------------------------------------------------------------------------+
| OFFLINE GEPA EVOLUTION ENGINE (omds.evolution) |
| gepa optimize_anything (single-task search, dataset=None) · |
| code-graph-targeted, ontology-briefed reflective mutation proposer · |
| Pareto objectives/frontier · the five gates · internal-PR promotion |
+---------------------------------------------------------------------------+
| |
each candidate evaluated in an isolated clone traces + frontier.json
(omds.sandbox: git worktree + uv venv, rlimits, (omds.telemetry, read by
network=False) omds inspect)
`

Two distinct git repositories are involved:

- The harness repo (this one) — the framework code. Not mutated by GEPA.
- The pipeline workspace — a per-task git repo under <task_dir>/workspace/
holding the evolving pipeline code, prompts, and configs. GEPA variants are
git worktree clones of it; promotion is a merge into its main via an
internal PR (branch → rebase → gate rerun → merge). This confines genetic
mutation to a blast radius the gates fully cover.

gepa==0.1.1's optimize_anything API in single-task search mode
(dataset=None) is the evolution core — it matches E-GDS's
one-candidate/one-evaluation model exactly, with a custom_candidate_proposer
doing code-graph-targeted, ontology-briefed reflective mutation instead of
gepa's default LLM-based reflection. The reflector is the only live LLM call in
the loop; everything else (clones, gates, training runs) is real work with no
model in it.

The promotion gates (run in order, all blocking)

1. ontology_static — AST-pattern axiom checks over the workspace; zero
error-severity violations (warnings need a runtime proof — see gate 5).
2. pytest — the pipeline workspace's own contract test suite.
3. pyright — type check (pyright missing from the clone's venv is a
documented SKIP-pass, not a hard failure).
4. ruff — correctness lint rules only (F, E9, B).
5. runtime_validation — a real instrumented pipeline run on holdout data:
zero unresolved runtime-assertion warnings, metrics within task.yaml
bounds, no regression beyond evolution.regression_tolerance.

A gate-passing, frontier-selected variant becomes an internal PR:
omds promote --dry-run renders the report (metrics delta, axioms touched,
blame targets, trace ids) without merging; omds promote --approve <id>
rebases onto main, reruns all five gates post-rebase, then merges.
promotion.require_human: true (the config default) always stops at the
report; --dry-run forces the report-only path regardless of that setting.

Config reference (omds.yaml)

`yaml
llm:
base_url: http://localhost:11434/v1 # any OpenAI-compatible endpoint; or fake://... (see below)
api_key: local
roles:
planner: {model: qwen2.5-coder:14b}
coder: {model: qwen2.5-coder:14b}
reflector: {model: qwen2.5-coder:14b, temperature: 0.7} # drives GEPA's mutation proposals
auditor: {model: qwen2.5-coder:7b} # cheap roles can run smaller models

evolution:
max_evals: 12 # gepa's max_metric_calls (budget-driven, not generation-driven)
parallel: 2 # gepa's evaluator thread pool; each call works in its own env clone
max_repair_iterations: 3
regression_tolerance: 0.0
promotion_require_human: true # false enables full promotion autonomy

data_dir: runs # traces/frontier reports, relative to this file unless absolute
`

- Roles → models. Each role binds to a model independently, so cheap roles
can run a smaller/faster model. With the agent runtime removed, reflector
is the only role the shipped loop actually calls; planner/coder/auditor
are still accepted and still validated, but nothing dispatches to them today.
- EGDS_-prefixed env vars override the file (EGDS_LLM__BASE_URL, …).
The prefix predates the egds → omds rename and was left alone
deliberately, so existing configs keep working.
- llm.base_url also accepts fake://... — see below.
- task.yaml (separate from omds.yaml) declares the task itself: goal, task
type, dataset paths/roles, metrics, and bounds/eval_budget.
- agents: and orchestration: no longer exist. They were shaped for the
AutoGen orchestration path deleted in the v2 rework and outlived it by a
release — validating cleanly while nothing read them, which reads as a working
feature. Unknown keys are rejected, so an omds.yaml still carrying them now
fails with a validation error naming them; delete the two blocks.

Testing without a live model

omds.yaml's llm.base_url accepts the fake: scheme (is_fake_base_url in
omds.cli_support) as a first-class alternative to a real http(s):// URL —
e.g. base_url: fake://local. Under it, evolve and models probe substitute
deterministic, network-free stand-ins for every LLM call (the reflector's
mutation proposals, the capability probe) while every other part of the loop
stays real: actual git worktree env clones, real uv sync, real
ontology/pytest/pyright/ruff/runtime gates, and a real sandboxed training run.
This is what the default (non-opt-in) CLI test suite runs against, and what the
fake-LLM tier of tests/e2e/test_full_loop.py drives end-to-end.

> Write it as fake://local, not the bare fake: — a plain YAML scalar ending
> in a colon is invalid YAML and would need quoting to parse at all.

---

Shared

Local SLM support (≤7B) — omds-slm

omds-slm (omds.slm) drives a local small model over the same nine tool
cores the CLIs and the MCP server call. It lives here rather than in Part 1
because it is the one place a model is pointed at the toolkit: Part 1's
guardrails, code graph, ledger and memory need no model at all, and Part 2's
GEPA loop uses a model for mutation proposals, not for tool use.

Start with the caveat, because it is the design. This is a **supervised,
turn-capped** harness, not an autonomous agent — the SLM scope guard (C5) is
no autonomous missions on a 7B, and the turn cap is what makes that
structurally true rather than merely intended:

- --max-turns is a hard ceiling on model calls, not a hint. The loop makes
at most that many calls and then stops, whatever the model asks for.
- stopped_reason: max_turns is a normal outcome meaning *unfinished, your
turn* — exit code 0. The loop deliberately does not spend an extra call
asking the model to wrap up, because a manufactured conclusion at the cap
would hide exactly the fact you need to see. Read the transcript and decide.
- MAX_TURNS_CEILING = 20 bounds what you may even ask for. A cap of 200 is an
autonomous mission wearing a cap's clothes; the remedy for a task that
genuinely needs more steps is another supervised invocation, not a bigger
number.

`bash


a local model behind an OpenAI-compatible endpoint (Ollama shown)

…and stopped_reason: max_turns (4/4 turns) is the other normal ending.

--json prints the whole LoopResult: every message sent, the tools offered that

--base-url (default http://localhost:11434) is Ollama's native root;
the chat client appends /v1 itself. Any OpenAI-compatible endpoint works —
llama.cpp's server, vLLM, LM Studio — though the context-window probe below is
Ollama-specific and degrades to a static guess elsewhere.

What it does that a plain chat loop does not, all of it in service of a
window that may be 4–8k rather than 200k:

1. It probes the model's REAL served window (omds.slm.probe) instead of
guessing from the model name. Ollama's /api/show publishes the trained
context length in GGUF metadata — and, crucially, the loop **clamps that by
the Modelfile's num_ctx**, because the window Ollama actually serves is
often much smaller than the one the model was trained for. Budgeting against
the larger number is how you overflow a context you thought you had. Every
failure here (no server, missing metadata) degrades silently to the static
table in omds.llm.registry: a capability probe must never be the reason a
run cannot start.
2. It narrows the tool set per turn (omds.slm.funnel) — hard triggers →
BM25 → synonym-expanded BM25 → optional embeddings → reciprocal-rank fusion
→ top-5 of the nine tools. The narrowing re-runs every turn against the
current request, so the offered subset follows the conversation instead of
being frozen at turn 1.
3. It budget-packs the turn (omds.slm.context) so tool schemas can never
crowd out the task. The system prompt and the current request are
priority-0 — never truncated, never dropped. Everything else is shed to fit
context_length - reserve_output, oldest history first, then skills,
then the worst-ranked tools last. If the system prompt and the task alone
do not fit, the run stops and says so (stopped_reason:
context_too_small, exit 1) rather than quietly truncating your question and
confidently answering something you did not ask.
4. It bounds what comes back from a tool, twice — once at the arguments,
using each core's own knobs (codegraph_context's max_items,
memory_recall's k), so the core never builds the oversized payload; and
again at the rendered result, with a flat character budget, because seven of
the nine tools are bounded only there.

The wire protocol is JSON-in-text, not native tool calling ({"tool": …,
"arguments": {…}} or {"answer": …}), parsed leniently through code fences and
surrounding prose — local 7B tool-call support is too uneven to depend on. A
tool the model invented, malformed arguments, or a core that raises are all
normal turn outcomes: the error is fed back and the model gets to correct
itself, within the cap.

The experimental REPL protocol — omds-slm repl

`bash
uv run omds-slm repl \
--model qwen2.5-coder \
--task "Check leaky.py for data leakage." \
--max-turns 4
`

repl is a second protocol over the same harness, borrowing the central
move of [Recursive Language Models](#rlm--repl-agents--the-direct-import-surface):
instead of emitting one JSON tool call per turn, the model writes **one fenced
Python code block per turn**, executed in a persistent worker process where the
nine tools are preloaded as functions. A full codegraph_context payload lands
in a variable the model slices on later turns — print(ctx["symbols"][:5]) —
so the context window bounds the conversation, not the data. The JSON
loop's answer to an oversized payload is truncation; the REPL loop's answer is
that the model pages through it programmatically.

What is identical to run, on purpose (C5 is a property of the loop, not
the wire format): the hard --max-turns ceiling, MAX_TURNS_CEILING = 20, the
identical-failure stall stop, prose-reply-is-an-answer, the per-turn tool
funnel and budget packing, and the exit-0/decision-in-the-payload rule. A cell
that raises is a normal turn — the traceback is fed back for the model to fix.
The model finishes with a plain-text reply (no code fence) or by calling
final_answer("…") inside a cell.

What is different, and why it is opt-in:

- The worker executes model-generated code (omds.slm.repl_worker, a
subprocess run under the [sandbox module](#repository-layout)'s posture:
CPU/memory rlimits, hermetic env, best-effort unshare -n network
isolation). Those limits bound runaways; they are **not a security
boundary** — the code runs with your filesystem permissions. Supervised runs
on trusted workspaces only.
- It asks more of the model. Writing correct Python against a namespace is
strictly harder than emitting one JSON object — and the JSON protocol was
chosen precisely because small-model tool calling is uneven. The RLM results
are reported on frontier models; whether a ≤7B model sustains this protocol
is exactly what the opt-in live smoke run exists to find out. Until it shows
that, run stays the default surface.
- A hung cell kills the worker and stops the run as stopped_reason:
worker_error (exit 1): the namespace died with the process, so continuing
would misreport what the variables still hold.

--json prints the full ReplResult — every cell, its captured
stdout/stderr, and whether the output was truncated — the same auditable-record
discipline as run.

> Maturity, honestly. The harness itself is well covered by unit tests that
> inject the transport, the capabilities and the tool caller — no test in the
> default suite touches a socket (the REPL protocol's tests do exercise its
> real worker subprocess, with a scripted model). The **live path is
> smoke-tested only behind an opt-in gate**
> (python/omds/tests/slm/test_ollama_smoke.py), which skips unless an Ollama
> is reachable and the model is pulled — and that smoke covers the JSON
> protocol only; no live model has been pointed at repl yet, which is
> why it is labelled experimental. If you are the first person to point this
> at a model that is not qwen2.5-coder, expect to tune the prompt.

SLM-first design rules

Every prompt in the system is built to work on a 7B coder model with an 8–32k
window — SLMs are the design floor, never the ceiling:

1. OpenAI-compatible transport only. One httpx-based async client
(omds.llm.client) speaking /v1/chat/completions — no provider SDKs.
2. Structured output via constrained decoding, not hope. Schema-validated
JSON (omds.llm.structured.generate), with a bounded repair loop that
re-prompts with the bad output plus the validation error; persistent failure
is a StructuredOutputError, never coerced or silently dropped.
3. A model capability registry (omds.llm.registry) records per-model
context length, max output tokens, schema-decoding mode, and native-tool-call
support, probed once via omds models probe and persisted.
4. Context budgeting (omds.llm.budget.fit) assembles prompts from
prioritized sections and truncates the lowest-priority ones first, so no call
ever exceeds the model's window. Priority-0 sections are never dropped.
5. Micro-prompts. No prompt does two jobs. With the agent runtime removed
the shipped surface is small — the reflector's mutation proposal and the
capability probe are the only live LLM calls — but both go through rules 2
and 4 rather than raw chat().
6. Bounded tool output. The MCP adapter's codegraph_context caps per-file
symbols (max_items) and drops per-symbol detail by default (detail), so
an SLM-hosted client gets a response that fits its window while still seeing
true totals.

> The v1 "capability-adaptive dispatch" rule (native tool calls vs. a structured
> NextAction fallback) is gone, along with the EgdsChatClient that
> implemented it. Nothing in the shipped tree dispatches on a model's probed
> tool_calls capability; the registry still records the flag.

Repository layout

`
python/omds/ # the omds package — a uv workspace member, the installable artifact
src/omds/
ontology/ # axiom registry + packs (leakage, scoring, evaluation, shapes,
# simplex_transforms), static AST checks, runtime assertions
codegraph/ # AST -> networkx graph, def-use chains, data lineage, blame
dstools/ # in-kernel helpers: df_summary, capture_plot, experiment ledger
memory/ # session pad (markdown) + SQLite FTS5/KV episodic store
llm/ # httpx OpenAI-compatible client, structured-output repair loop,
# capability registry, context budgeter
evolution/ # gepa evaluator, mutation proposer, Pareto objectives,
# the five gates, internal-PR promotion
sandbox/ # git-worktree + uv-venv env clones, rlimit/network-isolated exec
telemetry/ # TraceRecord/RunArtifact models, metric collectors
runtime/ # task spec loading, skills catalog, offline validation gate
slm/ # omds-slm — the supervised, turn-capped local-SLM harness:
# Ollama capability probe, tool funnel, budget-aware
# turn assembler, turn-capped JSON loop, and the
# experimental REPL protocol (repl.py + its worker
# subprocess, repl_worker.py) — see Shared, above
pipeline.py # @stage / Pipeline contract — the evolvable-code hook
cli.py # omds init / evolve / promote / inspect / models
guardrails_cli.py # omds-guardrails check + the kernel-hook install/uninstall/status
codegraph_cli.py # omds-codegraph lineage / blame / context
dstools_cli.py # omds-dstools ledger show
memory_store_cli.py # omds-memory-store add / recall
sessionpad_cli.py # omds-sessionpad append / show (the markdown session pad)
mcp_server.py, mcp_tools_*.py # omds-mcp — the 9-tool MCP adapter
install.py # omds-install the optional convenience installer
kernel_guard.py # runtime fit()-on-tagged-test-data guard
kernel_hook.py # the opt-in .pth that arms it environment-wide
tests/mcp/ # MCP tools, installer, skill portability, plugin-bundle invariants
tests/slm/ # probe/funnel/assembler/loop injected; the REPL protocol against
# its real worker subprocess; plus the opt-in live-Ollama
# smoke (skipped unless a model is actually there)

skills/ # the 7 portable, agent-neutral Agent Skills
hooks/ # PostToolUse guardrail hook (stdlib-only, fail-open) + hooks.json
.claude-plugin/ # plugin.json + marketplace.json — the Claude Code plugin manifest
.mcp.json # the project-scoped MCP entry, for this repo itself
benchmarks/tabular_multiclass/ # end-to-end benchmark: make_data.py, task.yaml, workspace_seed/
tests/ # the E-GDS suite: unit + integration, plus opt-in e2e/ and benchmarks/
docs/ # specs, plans, ADRs, research
`

The tree is Python only — there is no TypeScript, no bun, no node. The
pi-era TS algorithms were kept in an archive/ for one release while the skill
funnel was ported to Python; that port is done and better covered than the
original, so the archive was removed rather than left as a second, unbuilt
definition of the same thing. It is still readable in git history:

`bash
git show f3d221f:archive/ts-extension/skill-funnel.ts # the funnel, now omds/slm/funnel.py
git show f3d221f:archive/ts-extension/runner.ts # the workflow runner, never ported
git show f3d221f:archive/ts-extension/README.md # what each file was and why it was kept
`

Testing

There is no CI. This repo runs no GitHub Actions workflow — the three gates
below are run locally, and nothing enforces them on push. Run all three
before you commit; that is the entire safety net.

`bash
uv run pytest # default suite: 730 passed, 6 skipped — no network, no live model
uv run ruff check # correctness lint (F, E9, B); currently clean
uv run pyright # static types over python/omds/src; currently clean

EGDS_SLOW=1 uv run pytest tests/benchmarks/ tests/test_packaging.py
# + the real-clone 5-gate benchmark (uv sync per clone)
# and a real uv build of the sdist + wheel
EGDS_E2E=1 uv run pytest tests/e2e/ # + the fake-LLM end-to-end tier (spec §1.3), no network

best-effort: the real-endpoint E2E tier, against a live local model

ollama serve + ollama pull qwen2.5-coder, then it simply runs:

The six default skips are exactly those opt-in tiers (three in tests/e2e/,
one in tests/benchmarks/, one the SLM smoke, one the real uv build in
tests/test_packaging.py). The env-var names still use the EGDS_ prefix.

[tests/test_packaging.py](tests/test_packaging.py) is worth knowing about:
every other test runs against the uv workspace — an editable install resolved by
uv.lock — which is not what a user gets. Two defects lived in that gap (an
uncapped mcp that resolved to a version without mcp.server.fastmcp, and a
source distribution that could not build at all), so these assert on the
declared metadata and on real built artifacts rather than on the imported
package.

Verifying a real install

The suite proves the code works; it does not prove an installation is wired
up. [scripts/verify_install.py](scripts/verify_install.py) does that half, in
one command, against a throwaway temp directory:

`bash
uv run python scripts/verify_install.py # add --keep to inspect the temp dir
`

It checks that every console script resolves, that omds-install claude writes
the MCP entry and copies the skills, that a real omds-mcp subprocess speaks
stdio MCP and returns an ax:fit_on_train_only violation from
guardrails_check, that memory round-trips, that the plugin manifests validate
and the plugin's .mcp.json matches omds.install._OMDS_SERVER, that the
PostToolUse hook both blocks on a certain violation **and stays silent with
omds-guardrails off PATH**, and that the in-kernel guard raises on tagged
test data. The live SLM run is included when an Ollama is reachable and skipped
with a reason when it is not. It exits non-zero only on a real failure, and
touches nothing outside its temp dir.

It also prints, at the end, the things it cannot verify — chiefly that
Claude Code actually connects in-session, that a skill surfaces as
omds:leakage-audit, and that the hook fires on a real model edit. Those need a
human with a live session; the script is explicit about not covering them.

python/omds/tests/slm/test_ollama_smoke.py is the only test in the tree that
talks to a live model. Its gate is not an env var but reachability: a GET
/api/tags with a 2-second timeout at collection time, and a check that the
model is really pulled — so it skips with a specific reason (server down vs.
model missing) instead of failing in transport, and a black-holed port cannot
stall the suite. EGDS_SLM_SMOKE_BASE_URL (Ollama's native root, no /v1)
and EGDS_SLM_SMOKE_MODEL override the defaults. It asserts only what must be
true of a working harness — the run terminated inside its cap, no transport or
budget fault, guardrails_check actually dispatched — never the model's prose,
because a 7B at temperature 0.2 is not deterministic.

tests/e2e/test_full_loop.py is the single source of truth for spec §1.3's
success criteria: the benchmark evolves end-to-end and GEPA's frontier has a
feasible, gate-passing member promotable via internal PR; an injected ontology
violation (unnormalized logits into log_loss) is rejected by the ontology gate
naming the axiom and the offending code-graph node; and every
candidate-evaluation subprocess runs with network=False.

Non-goals

GPU fine-tuning/RL of the underlying models; non-tabular modalities (vision,
audio); a web UI; distributed multi-machine execution; a full OWL/RDF reasoner;
harness self-mutation (GEPA only evolves a pipeline workspace, never omds
itself). Per the SLM scope guard: no autonomous missions on a 7B — anything
model-driven here stays bounded and supervised.

License

[Apache License 2.0](LICENSE) — see [NOTICE](NOTICE) for the attribution
notices that §4(d) asks redistributors to carry along. Both files ship inside
the wheel and the sdist, so anyone who installs the package has them.

Apache-2.0 rather than a permissive-but-silent licence mainly for the **express
patent grant** in §3 and its termination clause, which MIT simply does not
address. The practical differences if you build on this: keep the LICENSE and
NOTICE with any redistribution, and state what you changed if you ship
modified files (§4(b)). There is no copyleft — Apache-2.0 does not require you
to open your own code.

No third-party source is vendored anywhere in the tree; everything external is a
normal declared dependency. The project holds no contributions from anyone other
than the copyright holder, so the relicence from MIT (2026-08-04) needed no
third-party consent.

History

v2 of this project was briefly built as an extension of a pinned, fast-moving
single-maintainer coding-agent fork (@oh-my-pi/pi-coding-agent); that layer
was decommissioned and replaced by the harness-agnostic core described above.
The one algorithm worth keeping — the skill funnel — was ported to Python as
omds.slm.funnel; the rest is in git history (see
[Repository layout](#repository-layout)). Both that decision
and the two amendments that reshaped it — Skills+CLI promoted over MCP
(2026-07-20), and the Claude Code plugin added as a third thin surface
(2026-07-31) — are recorded in
[docs/adr/2026-07-18-mcp-pivot.md](docs/adr/2026-07-18-mcp-pivot.md). The
framework evaluation that chose AutoGen for v1's since-deleted runtime is kept
for the record at
[docs/specs/2026-07-12-framework-evaluation.md`](docs/specs/2026-07-12-framework-evaluation.md).

Prerequisites

- Python 3.12 or newer (requires-python = ">=3.12"). Nothing older works.
3.12, 3.13 and 3.14 are each verified by a real install-and-run, not just
declared.
- git — omds-codegraph blame shells out to it, and E-GDS clones workspaces
with git worktree. The rest of the toolkit runs fine without a repo.
- [uv](https://docs.astral.sh/uv/) — the recommended installer (see Step 1),
and required for a checkout or for Part 2. Plain pip works for the toolkit
if you would rather not add another tool.
- No model, no API key, no network. Everything in Part 1 is static analysis
and local files. A model is needed only for Part 2 (E-GDS) and omds-slm.

- Python 3.12 or newer (requires-python = ">=3.12"). Nothing older works.
3.12, 3.13 and 3.14 are each verified by a real install-and-run, not just
declared.
- git — omds-codegraph blame shells out to it, and E-GDS clones workspaces
with git worktree. The rest of the toolkit runs fine without a repo.
- [uv](https://docs.astral.sh/uv/) — the recommended installer (see Step 1),
and required for a checkout or for Part 2. Plain pip works for the toolkit
if you would rather not add another tool.
- No model, no API key, no network. Everything in Part 1 is static analysis
and local files. A model is needed only for Part 2 (E-GDS) and omds-slm.