Agent Memory Leaderboard 使用统一的端到端流程评估不同记忆系统。我们不限定你使用的数据库、索引、向量模型或内部架构，只要求 Add / Search 接口符合现行规范，且评测过程和检索结果能够复核。

先提交 Evaluation Access Request，完成参测系统、版本、代码或公开接口和鉴权方式的审核。代码提交必须附带完整 README、Docker 启动命令、API 封装说明和原始方法披露；平台会按说明启动并部署后再评测。托管接口提交则需在提交后至少 30 天保持公网可访问且稳定。

1. 01

   提交准入申请

   选择申请新 Key，或输入已有 Key 为其增加版本；填写联系人、系统版本、GitHub 仓库或 Add / Search 接口，并在提交说明中写清运行方式、Docker 命令、API 封装和原始方法信息。

2. 02

   完成 Key 绑定

   审核通过后，新申请会签发 Leaderboard API Key；新增版本申请会直接绑定到原 Key，无需管理员再次手工录入接口信息。

3. 03

   平台部署并运行公开 smoke

   代码提交由平台按 Docker 和 API 说明构建、启动并封装；托管提交直接校验你提供的接口。随后按照现行同步规范执行 Add → Search → Answer → Evaluate smoke 流程，确认接口可用后再进入正式评测。

4. 04

   完成 Full 前置检查

   选择 full 前，必须逐项勾选提交清单：smoke 已通过、API 契约正确、运行说明完整、托管接口可持续 30 天稳定，以及原创性披露和诚信承诺均已完成。

5. 05

   提交评测任务

   在 Evaluation 页面验证 API Key，选择已绑定的版本和评测模式，填写唯一的 Run Label 后提交。正式任务使用统一的数据集套件和 Top K。

6. 06

   查看进度与审计状态

   任务依次执行写入、检索、回答、评分和结果汇总。页面会显示当前阶段、完成进度和错误摘要；我们同时保留复核所需的过程记录。

7. 07

   查看结果与发布状态

   评测结果首先仅对绑定的 API Key 可见。成功完成的 full 任务会进入公榜资格校验和审核队列；审核通过后生成公榜提交记录。

Add 和 Search 的接口地址由你配置，请求和响应格式按照现行规范固定，不随 URL 路径变化。接口必须能够从我们的评测环境访问；生产环境建议使用 HTTPS。URL 中不得包含用户名、密码等凭据，也不得指向私有、回环或链路本地地址。

{
  "request_id": "eval:run_abc123:locomo_refined:conv-0:chunk-0",
  "messages": [{
    "role": "user",
    "timestamp": 1704067200000,
    "content": "memory text"
  }],
  "user_id": "eval:run_abc123:locomo:conv-0",
  "session_id": "eval:run_abc123:sample:0"
}

HTTP 200
{
  "success": true,
  "request_id": "eval:run_abc123:locomo_refined:conv-0:chunk-0",
  "user_id": "eval:run_abc123:locomo:conv-0",
  "session_id": "eval:run_abc123:sample:0"
}

{
  "query": "Which answer best matches the memory?",
  "options": ["A. First answer", "B. Second answer"],
  "user_id": "eval:run_abc123:locomo:conv-0",
  "top_k": 100
}

{
  "data": [{
      "id": "mem_1",
      "content": "remembered fact text",
      "score": 0.87,
      "created_at": "2026-07-01T12:00:00Z"
  }]
}

参测接口应使用标准 HTTP 状态码，并返回便于排查问题且不含密钥的错误信息。平台业务错误通常采用 {"detail":{"reason":"..."}} 格式；字段校验失败时，HTTP 422 会返回结构化错误明细。

Agent Memory Leaderboard compares memory systems under a consistent end-to-end evaluation. The platform does not prescribe a database, index, embedding model, or internal architecture. It requires a conformant external API and verifiable evidence for retrieval and audit.

Industry submissions and academic systems with hosted APIs follow the existing review, Key issuance, and public smoke flow. Code submissions must include a complete README, Docker startup command, API wrapper instructions, and original-method disclosure; maintainers build and start the documented entrypoint before evaluation. Hosted endpoints must remain publicly reachable and stable for at least 30 days after submission.

1. 01

   Submit the access request

   Industry requests provide Add / Search endpoints. Code submissions provide a publicly accessible GitHub repository and explain the Docker command, API wrapper, original authors, technical report, and every method change.

2. 02

   Complete key binding

   Approval either issues a new Leaderboard API Key or binds the submitted version to the verified existing key without administrator re-entry.

3. 03

   Platform deployment and public smoke

   For code submissions, maintainers build and start the documented Docker entrypoint and expose the official API. Hosted submissions are checked at the declared endpoint. The platform then runs the synchronous Add → Search → Answer → Evaluate smoke flow.

4. 04

   Complete the Full gate

   Before selecting full, confirm the interactive checklist: smoke passed, API contract followed, run instructions complete, hosted runtime stable for 30 days, original work disclosed, and no integrity violations.

5. 05

   Start the job

   Verify the API Key on the Evaluation page, select a bound version and mode, assign a unique Run Label, and submit. Formal jobs use the fixed suite and platform Top K.

6. 06

   Monitor progress and audit status

   The job proceeds through ingestion, retrieval, answering, scoring, and aggregation. The interface reports stage, progress, and error summaries while the platform retains evidence required for review.

7. 07

   Review results and publication status

   Results are initially visible only to the bound API Key. A successful full job enters the public-eligibility gate and review queue; approval creates a public submission.

Participants configure the Add and Search URLs; request and response schemas are fixed and do not vary by path. Endpoints must be reachable from the platform network, and production deployments should use HTTPS. URLs must not embed credentials or resolve to private, loopback, or link-local addresses.

{
  "request_id": "eval:run_abc123:locomo_refined:conv-0:chunk-0",
  "messages": [{
    "role": "user",
    "timestamp": 1704067200000,
    "content": "memory text"
  }],
  "user_id": "eval:run_abc123:locomo:conv-0",
  "session_id": "eval:run_abc123:sample:0"
}

HTTP 200
{
  "success": true,
  "request_id": "eval:run_abc123:locomo_refined:conv-0:chunk-0",
  "user_id": "eval:run_abc123:locomo:conv-0",
  "session_id": "eval:run_abc123:sample:0"
}

{
  "query": "Which answer best matches the memory?",
  "options": ["A. First answer", "B. Second answer"],
  "user_id": "eval:run_abc123:locomo:conv-0",
  "top_k": 100
}

{
  "data": [{
      "id": "mem_1",
      "content": "remembered fact text",
      "score": 0.87,
      "created_at": "2026-07-01T12:00:00Z"
  }]
}

Participant endpoints should use standard HTTP status codes and return actionable errors without secrets. Platform business errors normally use {"detail":{"reason":"..."}}; HTTP 422 provides structured field-validation details.