GitHub Checks

Run an eval suite against the MCP server built from every pull request, and read the result as a check run, a pull-request comment, and a machine-readable JSON block.

Connect a repository and MCPJam runs the eval suite bound to it against the MCP server built from each pull request’s head commit, then reports back as a check run named MCPJam Evals.

Nothing about the check reads your default branch. It builds the pull request, starts the server it produces, runs the suite against that server, and reports what it measured.

GitHub Checks is configured in the hosted app, at Settings → Integrations → GitHub. Every write on this page — connecting, installing, disconnecting, and changing per-repository settings — requires an organization owner or admin. Members can open the page to see the current setup, but all controls are disabled and a note explains why.

Connect a GitHub account starts it. You sign in to GitHub so we can confirm which accounts you administer, then pick one from the list that comes back — an organization, or your own account. Installing the app is not on its own proof that an installation is yours to connect, which is why the sign-in step is there and not skippable.

Accounts that already have the app appear in that list ready to connect.

Install on another account, in that same list, is not generally available yet. During the beta the MCPJam app can only be installed on the MCPJam organization, so there is no other account for it to reach — following it sends you to the existing installation rather than offering a choice. Even once that opens up, GitHub sends an administrator who already has the app somewhere to that installation instead of asking which account to use. So for now, connect an account that already appears in the list.

A GitHub account connects to one MCPJam organization at a time, for the whole account rather than per repository. Two organizations cannot split the repositories of one GitHub account between them. If the account is already connected elsewhere, connecting it again is refused — free it up in the organization holding it, or use a different account.

Which organization is holding it is rolling out: until it reaches your deployment the refusal names nothing, so ask an owner of the GitHub account, who can see the installation. After it reaches you the account picker marks the account before you click, and names the holding organization when you are a member of it — when you are not, it still cannot name it, because that is a fact about an organization you cannot see.

The pull-request comment described below is rolling out: it is enabled per deployment by an operator, and until it is on for yours a connected repository gets the check run and no comment. The per-repository toggle is visible before then and records your choice; nothing is posted until the rollout reaches you.

Suite credentials in approved forks

Each connected repository has Allow suite credentials in approved forks in Settings and in its suite’s connected-repository section. It is off by default; only organization admins can change it.

• Off: approved fork runs require a credential-free suite. Checks needing saved credentials report Action required.
• On: approved fork runs can use that suite’s existing credentials and connected services. Only approve code you trust. This does not inject saved secrets into the PR server.

Every new commit needs maintainer approval. Enabling the setting does not upgrade old approvals or running jobs: approve again. Changes to selected credentials also require fresh approval. Disabling blocks further credential access and stops affected runs; credentials already delivered cannot be taken back. Changing the connected suite resets the setting off. Environments with plugin credential grants that cannot be verified remain blocked.

mcpjam.yaml

MCPJam can usually work out how to build and start a server on its own. When it cannot — or when you would rather say it explicitly than let detection guess — commit an mcpjam.yaml at the root of the repository.

A committed file is authoritative: MCPJam runs what it says and never falls back to detection. A file that is present but invalid fails the check rather than being ignored, because ignoring it would run a server the author never declared.

mcpjam.yaml

version: 1

checks:
  build: npm ci && npm run build
  start: node dist/server.js
  port: 8080
  path: /mcp
  env:
    LOG_LEVEL: debug

Fields

Field	Required	What it is
version	Yes	Must be 1.
checks.build	Yes	The command that installs and compiles. An install-only ecosystem still declares one — npm ci, uv sync --frozen.
checks.start	Yes	The command that starts the server. It must keep running.
checks.port	Yes	An integer, 1–65535. The port your server listens on.
checks.path	Yes	The MCP endpoint path. Must start with /.
checks.transport	No	Only streamable-http, which is also the default. Version 1 of this file is HTTP-only.
checks.env	No	Environment variables to set before start.

Unknown keys inside checks: are rejected — inside the authoritative section a stray key is a typo that would otherwise silently change what runs. Unknown keys at the top level are ignored, so the file can grow new sections without breaking older checks. The file itself is capped at 32 KB.

env

checks.env is a flat mapping of at most 20 entries. Names must be uppercase environment-variable names (^[A-Z][A-Z0-9_]*$, up to 64 characters) and values must be strings of up to 1024 characters — quote anything that looks like a number, a boolean, or an empty value, or the file is rejected rather than coerced.

Never put a credential in checks.env. This file is committed to the repository, so it is readable by anyone who can read the repository, and MCPJam stores its values verbatim alongside the check. It is for the flags a server needs in order to boot — LOG_LEVEL, FIXTURE_MODE — and for nothing that would matter if it were printed in public.

Binding and ports

The port is what has to match; the bind address is not. MCPJam reaches your server through the sandbox’s own host bridge, from inside the box, so a server bound to loopback is reachable. What fails is listening on a port other than the one checks.port declares, or a process that exits after startup — both are reported as an unhealthy server, and the check body says so.

If your framework needs a HOST (or equivalent) variable set to bind the way you want, set it in checks.env like any other boot flag. Only add it if your framework actually needs it — it is not something MCPJam requires.

The pull-request comment

Alongside the check run, MCPJam posts one comment per pull request and updates that same comment in place on every later push. It is posted for every conclusion, including a pass — so the comment is always current, and there is never a thread of stale verdicts to scroll past.

The comment carries:

• The verdict line — what the check concluded on this head commit.
• What the result is not. A check that could not produce a verdict says so in the comment rather than leaving you to infer it from a colour.
• “What to fix” — one line per failing case, saying what to go and change. Each line names the case and the one thing to look at; it does not claim to know why the case failed.
• A collapsed JSON block, described below, for coding agents.

Comments are posted for every connected repository by default, and an admin can turn them off per repository — see Turning comments off.

The machine-readable JSON block

The last section of the comment is a collapsed, fenced JSON block. It exists so a coding agent working on the pull request can read the result without parsing prose, and it is stable enough to be worth parsing.

It is a bounded projection of MCPJam’s eval decision contract — not the whole contract, and not a dump of the run. It carries:

Key	What it says
schemaVersion	The version of this projection. Read it before anything else.
The head SHA	The commit this result describes. A result whose SHA is not the tip is out of date.
The outcome and conclusion	What MCPJam measured, and what it reported to GitHub.
Trial counts	How many trials ran, passed, and failed.
One entry per failing trial	Its first failed stage, its failure category, and the tool names the case expected versus the ones it observed.
omittedFields	The fields of the contract this projection deliberately leaves out.

Two things to know when reading it:

• It is a projection, and it says so. Anything the projection drops is named in omittedFields, so a consumer can tell “MCPJam did not include this” from “MCPJam measured nothing here”. Do not infer an absent field’s value.
• The first failed stage is a location, not a diagnosis. It says where the chain stopped — connection, discovery, selection, the tool call, the response, or the user’s request being satisfied — and nothing about why. Treating it as a cause is how a change lands in the wrong file.

Review comments on removed tool names

When a failing case asserts a tool that was never called, MCPJam looks for that tool name in the pull request’s own diff. If the diff removes a line containing the name in quotes, and the name does not come back on any added line, MCPJam leaves a single review comment on that removed line.

That comment is a location, not a cause. It says: the case expected this tool, this line is where the name stopped existing, start here. It does not claim the removal is a mistake — renaming or deleting a tool on purpose is an ordinary thing for a pull request to do, and when it is, the fix is to update the case in MCPJam rather than to put the line back.

MCPJam never requests changes. Review comments are posted as plain comments, so they never sit as a blocking review on your pull request.

What appears in a comment, and what never does

Comments and review comments are public on a public repository. What MCPJam writes there is deliberately bounded.

What can appear:

• fixed MCPJam copy — the same sentences on every repository;
• the head SHA, the check-run id, and the pull-request number;
• links back into MCPJam;
• eval case titles, as you wrote them;
• tool names;
• stage reasons and counts;
• the same clamped build and server output the check run already shows.

What never appears:

• organization, project, or installation identifiers;
• model or provider names;
• prompts;
• tool-call arguments;
• credentials — evidence strings are credential-redacted before they are rendered.

If a case title or a tool name is itself sensitive, treat it as published: those are the two fields that carry your own words into a public comment.

Server authentication

Some MCP servers require OAuth login before they will answer the MCP initialize request. When MCPJam detects a Bearer challenge during the health probe, it marks the server as requiring authorization and looks for a project-shared OAuth connection to use.

MCPJam derives the OAuth connection from the repository’s selected suite. When that suite has one authorized project-shared OAuth connection, there is nothing else connected-repository settings need you to choose.

Settings shows a small action only when attention is needed: Authorize for a suite connection with no saved project authorization, Reconnect after a check reports that authorization no longer works, or a picker when the suite contains more than one eligible OAuth connection. These actions open the project’s existing Servers page and OAuth flow.

When the required authorization is missing or rejected, the GitHub check reports Action required and skips execution. Personal and unrelated project connections are never selected automatically.

Per-repository switches

Each connected repository has three labeled switches:

Switch	What it controls
Checks	Whether the eval suite runs on pull requests to this repository.
Conformance	Whether a conformance check runs alongside the eval suite. Disabled and dimmed while Checks is off, because it is a sub-setting of checks.
Comments	Whether MCPJam posts a pull-request comment with the result. Independent of Checks — you can pause checks while still allowing MCPJam to comment, or turn comments off while checks keep running.

Turning comments off

Comments are on for every connected repository by default, including repositories connected before the feature existed. An admin opts a repository out; there is nothing to opt in to.

Go to Settings → Integrations → GitHub, find the repository in Connected repositories, and turn off the Comments switch.

This setting changes nothing about the check itself — whether it runs, what it measures, and how it reports are all decided elsewhere. Only the comment stops.