> ## Documentation Index
> Fetch the complete documentation index at: https://mcpjam-mintlify-docs-update-pr-4609-1788333745897.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# GitHub Checks

> Run an eval suite against the MCP server built from every pull request, and read the result as a check run, a pull-request comment, and a machine-readable JSON block.

Connect a repository and MCPJam runs the eval suite bound to it against the MCP
server built from each pull request's **head commit**, then reports back as a
check run named **MCPJam Evals**.

Nothing about the check reads your default branch. It builds the pull request,
starts the server it produces, runs the suite against that server, and reports
what it measured.

<Note>
  GitHub Checks is configured in the hosted app, at **Settings → Integrations →
  GitHub**. Every write on this page — connecting, installing, disconnecting, and
  changing per-repository settings — requires an organization owner or admin.
  Members can open the page to see the current setup, but all controls are
  disabled and a note explains why.

  **Connect a GitHub account** starts it. You sign in to GitHub so we can confirm
  which accounts you administer, then pick one from the list that comes back —
  an organization, or your own account. Installing the app is not on its own
  proof that an installation is yours to connect, which is why the sign-in step
  is there and not skippable.

  Accounts that already have the app appear in that list ready to connect.

  **Install on another account**, in that same list, is **not generally available
  yet.** During the beta the MCPJam app can only be installed on the MCPJam
  organization, so there is no other account for it to reach — following it sends
  you to the existing installation rather than offering a choice. Even once that
  opens up, GitHub sends an administrator who already has the app somewhere to
  that installation instead of asking which account to use. So for now, connect
  an account that already appears in the list.

  A GitHub account connects to **one MCPJam organization at a time**, for the
  whole account rather than per repository. Two organizations cannot split the
  repositories of one GitHub account between them. If the account is already
  connected elsewhere, connecting it again is refused — free it up in the
  organization holding it, or use a different account.

  Which organization is holding it is **rolling out**: until it reaches your
  deployment the refusal names nothing, so ask an owner of the GitHub account,
  who can see the installation. After it reaches you the account picker marks
  the account before you click, and names the holding organization when you are
  a member of it — when you are not, it still cannot name it, because that is a
  fact about an organization you cannot see.

  The **pull-request comment** described below is rolling out: it is enabled per
  deployment by an operator, and until it is on for yours a connected repository
  gets the check run and no comment. The per-repository toggle is visible before
  then and records your choice; nothing is posted until the rollout reaches you.
</Note>

## Suite credentials in approved forks

Each connected repository has **Allow suite credentials in approved forks** in
Settings and in its suite's connected-repository section. It is off by default;
only organization admins can change it.

* **Off:** approved fork runs require a credential-free suite. Checks needing
  saved credentials report **Action required**.
* **On:** approved fork runs can use that suite's existing credentials and
  connected services. Only approve code you trust. This does not inject saved
  secrets into the PR server.

Every new commit needs maintainer approval. Enabling the setting does not upgrade
old approvals or running jobs: approve again. Changes to selected credentials
also require fresh approval. Disabling blocks further credential access and
stops affected runs; credentials already delivered cannot be taken back.
Changing the connected suite resets the setting off. Environments with plugin
credential grants that cannot be verified remain blocked.

## `mcpjam.yaml`

MCPJam can usually work out how to build and start a server on its own. When it
cannot — or when you would rather say it explicitly than let detection guess —
commit an `mcpjam.yaml` at the **root of the repository**.

A committed file is authoritative: MCPJam runs what it says and never falls back
to detection. A file that is present but invalid fails the check rather than
being ignored, because ignoring it would run a server the author never declared.

```yaml mcpjam.yaml theme={"theme":"css-variables"}
version: 1

checks:
  build: npm ci && npm run build
  start: node dist/server.js
  port: 8080
  path: /mcp
  env:
    LOG_LEVEL: debug
```

### Fields

| Field              | Required | What it is                                                                                                           |
| ------------------ | -------- | -------------------------------------------------------------------------------------------------------------------- |
| `version`          | Yes      | Must be `1`.                                                                                                         |
| `checks.build`     | Yes      | The command that installs and compiles. An install-only ecosystem still declares one — `npm ci`, `uv sync --frozen`. |
| `checks.start`     | Yes      | The command that starts the server. It must keep running.                                                            |
| `checks.port`      | Yes      | An integer, 1–65535. The port your server listens on.                                                                |
| `checks.path`      | Yes      | The MCP endpoint path. Must start with `/`.                                                                          |
| `checks.transport` | No       | Only `streamable-http`, which is also the default. Version 1 of this file is HTTP-only.                              |
| `checks.env`       | No       | Environment variables to set before `start`.                                                                         |

Unknown keys **inside `checks:`** are rejected — inside the authoritative
section a stray key is a typo that would otherwise silently change what runs.
Unknown keys at the top level are ignored, so the file can grow new sections
without breaking older checks. The file itself is capped at 32 KB.

### `env`

`checks.env` is a flat mapping of at most 20 entries. Names must be uppercase
environment-variable names (`^[A-Z][A-Z0-9_]*$`, up to 64 characters) and values
must be **strings** of up to 1024 characters — quote anything that looks like a
number, a boolean, or an empty value, or the file is rejected rather than
coerced.

<Warning>
  **Never put a credential in `checks.env`.** This file is committed to the
  repository, so it is readable by anyone who can read the repository, and MCPJam
  stores its values verbatim alongside the check. It is for the flags a server
  needs in order to boot — `LOG_LEVEL`, `FIXTURE_MODE` — and for nothing that
  would matter if it were printed in public.
</Warning>

### Binding and ports

**The port is what has to match; the bind address is not.** MCPJam reaches your
server through the sandbox's own host bridge, from inside the box, so a server
bound to loopback is reachable. What fails is listening on a port other than the
one `checks.port` declares, or a process that exits after startup — both are
reported as an unhealthy server, and the check body says so.

If your framework needs a `HOST` (or equivalent) variable set to bind the way you
want, set it in `checks.env` like any other boot flag. Only add it if your
framework actually needs it — it is not something MCPJam requires.

## The pull-request comment

Alongside the check run, MCPJam posts **one comment per pull request** and
updates that same comment in place on every later push. It is posted for every
conclusion, including a pass — so the comment is always current, and there is
never a thread of stale verdicts to scroll past.

The comment carries:

* **The verdict line** — what the check concluded on this head commit.
* **What the result is not.** A check that could not produce a verdict says so
  in the comment rather than leaving you to infer it from a colour.
* **"What to fix"** — one line per failing case, saying what to go and change.
  Each line names the case and the one thing to look at; it does not claim to
  know why the case failed.
* **A collapsed JSON block**, described below, for coding agents.

Comments are posted for every connected repository by default, and an admin can
turn them off per repository — see [Turning comments
off](#turning-comments-off).

## The machine-readable JSON block

The last section of the comment is a collapsed, fenced JSON block. It exists so a
coding agent working on the pull request can read the result without parsing
prose, and it is stable enough to be worth parsing.

It is a **bounded projection** of MCPJam's eval decision contract — not the whole
contract, and not a dump of the run. It carries:

| Key                         | What it says                                                                                                        |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `schemaVersion`             | The version of this projection. Read it before anything else.                                                       |
| The head SHA                | The commit this result describes. A result whose SHA is not the tip is out of date.                                 |
| The outcome and conclusion  | What MCPJam measured, and what it reported to GitHub.                                                               |
| Trial counts                | How many trials ran, passed, and failed.                                                                            |
| One entry per failing trial | Its **first failed stage**, its failure category, and the tool names the case expected versus the ones it observed. |
| `omittedFields`             | The fields of the contract this projection deliberately leaves out.                                                 |

Two things to know when reading it:

* **It is a projection, and it says so.** Anything the projection drops is named
  in `omittedFields`, so a consumer can tell "MCPJam did not include this" from
  "MCPJam measured nothing here". Do not infer an absent field's value.
* **The first failed stage is a location, not a diagnosis.** It says where the
  chain stopped — connection, discovery, selection, the tool call, the response,
  or the user's request being satisfied — and nothing about why. Treating it as a
  cause is how a change lands in the wrong file.

## Review comments on removed tool names

When a failing case asserts a tool that was never called, MCPJam looks for that
tool name in the pull request's own diff. If the diff **removes** a line
containing the name in quotes, and the name does not come back on any added line,
MCPJam leaves a single review comment on that removed line.

That comment is a **location, not a cause**. It says: the case expected this
tool, this line is where the name stopped existing, start here. It does not claim
the removal is a mistake — renaming or deleting a tool on purpose is an ordinary
thing for a pull request to do, and when it is, the fix is to update the case in
MCPJam rather than to put the line back.

MCPJam never requests changes. Review comments are posted as plain comments, so
they never sit as a blocking review on your pull request.

## What appears in a comment, and what never does

Comments and review comments are **public on a public repository**. What MCPJam
writes there is deliberately bounded.

**What can appear:**

* fixed MCPJam copy — the same sentences on every repository;
* the head SHA, the check-run id, and the pull-request number;
* links back into MCPJam;
* eval case titles, as you wrote them;
* tool names;
* stage reasons and counts;
* the same clamped build and server output the check run already shows.

**What never appears:**

* organization, project, or installation identifiers;
* model or provider names;
* prompts;
* tool-call arguments;
* credentials — evidence strings are credential-redacted before they are
  rendered.

If a case title or a tool name is itself sensitive, treat it as published: those
are the two fields that carry your own words into a public comment.

## Server authentication

Some MCP servers require OAuth login before they will answer the MCP initialize
request. When MCPJam detects a Bearer challenge during the health probe, it
marks the server as requiring authorization and looks for a project-shared OAuth
connection to use.

MCPJam derives the OAuth connection from the repository's selected suite. When
that suite has one authorized project-shared OAuth connection, there is nothing
else connected-repository settings need you to choose.

Settings shows a small action only when attention is needed: **Authorize** for
a suite connection with no saved project authorization, **Reconnect** after a
check reports that authorization no longer works, or a picker when the suite
contains more than one eligible OAuth connection. These actions open the
project's existing Servers page and OAuth flow.

When the required authorization is missing or rejected, the GitHub check
reports **Action required** and skips execution. Personal and unrelated project
connections are never selected automatically.

## Per-repository switches

Each connected repository has three labeled switches:

| Switch          | What it controls                                                                                                                                                                                      |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Checks**      | Whether the eval suite runs on pull requests to this repository.                                                                                                                                      |
| **Conformance** | Whether a conformance check runs alongside the eval suite. Disabled and dimmed while **Checks** is off, because it is a sub-setting of checks.                                                        |
| **Comments**    | Whether MCPJam posts a pull-request comment with the result. Independent of **Checks** — you can pause checks while still allowing MCPJam to comment, or turn comments off while checks keep running. |

## Turning comments off

Comments are **on for every connected repository by default**, including
repositories connected before the feature existed. An admin opts a repository
out; there is nothing to opt in to.

Go to **Settings → Integrations → GitHub**, find the repository in **Connected
repositories**, and turn off the **Comments** switch.

This setting changes nothing about the check itself — whether it runs, what it
measures, and how it reports are all decided elsewhere. Only the comment stops.