벤치마크

Claude Mythos Preview (restricted)

93.9% 해결 · 1위 · 공개 리더보드(제한 접근)

GPT-5.5

88.7% 해결 · 공개 리더보드

Claude Opus 4.8

88.6% 해결 · 공개 리더보드

Claude Opus 4.7 (Adaptive)

87.6% 해결 · 공개 리더보드

GPT-5.3-Codex

85.0% 해결 · 공개 리더보드

Gemini 3.1 Pro

80.6% 해결 · 공개 리더보드

DeepSeek V4 Pro Max (open-weight)

80.6% 해결 · 공개 리더보드 · 최고 오픈웨이트

GPT-OSS-20B · MXFP4 · llama.cpp

3670.42 pp / 82.74 tg tok/s · llama.cpp · NVIDIA-official

Qwen3.5-35B-A3B · MXFP4 · llama.cpp

prefill n/p / ~58 tg tok/s · llama.cpp · community (MoE A3B; theoretical ceiling ~91)

GPT-OSS-120B · MXFP4 · llama.cpp

1725.47 pp / 55.37 tg tok/s · llama.cpp · NVIDIA-official (canonical official 120B decode; engine spread 35 llama.cpp deep-ctx → 41 Ollama → ~50 SGLang)

Qwen2.5-VL-7B · NVFP4 · TRT-LLM (vision)

65831.77 pp / 41.71 tg tok/s · TRT-LLM · NVIDIA-official

Llama 3.1 8B · NVFP4 · TRT-LLM

10256.9 pp / 38.65 tg tok/s · TRT-LLM · NVIDIA-official

Qwen3-Coder-30B-A3B · Q8_0 · llama.cpp

1308 pp / 31 tg tok/s · llama.cpp · community (llama.cpp #16578; MoE A3B)

Qwen3.6-27B · Q4_K_M +MTP · llama.cpp

719 pp / 28.3 tg tok/s · llama.cpp +MTP (5 draft) · community (2.16x decode vs no-MTP)

Gemma 4 26B-A4B · F16 · llama.cpp

prefill n/p / ~26.5 tg tok/s · llama.cpp · community (MoE A4B; theoretical ~34)

Qwen3-14B · NVFP4 · TRT-LLM

5928.95 pp / 22.71 tg tok/s · TRT-LLM · NVIDIA-official

Llama 3.1 8B · FP8 · SGLang

7991 pp / 20.5 tg tok/s · SGLang · community (FP8 decode ~half of NVFP4 — same model)

Qwen3.6-27B · Q4_K_M · llama.cpp

1084 pp / 13.1 tg tok/s · llama.cpp · community (single-stream, no spec-decode)

Llama 3.1 70B · FP8 · SGLang

~803 pp / ~2.7 tg tok/s · SGLang · community (barely fits 128 GB; KV+weights thrash — avoid dense 70B FP8 on one unit)

Qwen3-235B · NVFP4 · TRT-LLM (DUAL Spark)

23477.03 pp / 11.73 tg tok/s · TRT-LLM · NVIDIA-official · DUAL DGX Spark over ConnectX-7 (does not fit one unit at usable quant)

claude-opus-4-7

The WHERE o.total > 0 runs AFTER the LEFT JOIN. For a customer with zero orders every o.* column is NULL, and NULL > 0 is not TRUE, so those rows are dropped — collapsing the LEFT JOIN into an INNER JOIN. Move the predicate into the ON clause so it filters the join, not the result:

SELECT c.id, c.name, COUNT(o.id) AS order_count
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id AND o.total > 0
GROUP BY c.id, c.name;

Unmatched customers keep one all-NULL row and COUNT(o.id) correctly returns 0.

gpt-5

A post-join WHERE on a right-table column nullifies the outer join: zero-order customers produce NULL o.total, and NULL > 0 fails, so they are filtered out. Relocate the filter to the join condition:

SELECT c.id, c.name, COUNT(o.id) AS order_count
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id AND o.total > 0
GROUP BY c.id, c.name;

COUNT(o.id) skips NULLs, so zero-order customers report 0.

gemini-3-pro

Right — the WHERE clause removes the NULL rows produced by the LEFT JOIN, so customers with no orders disappear. Keep them by also allowing the NULL case:

SELECT c.id, c.name, COUNT(o.id) AS order_count
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id
WHERE o.total > 0 OR o.id IS NULL
GROUP BY c.id, c.name;

This restores the zero-order customers.

qwen3.6-35b-a3b-nvfp4 (local DGX Spark)

The problem is COUNT with GROUP BY: COUNT(o.id) returns NULL when there are no matching orders, so wrap it in COALESCE to show 0:

SELECT c.id, c.name, COALESCE(COUNT(o.id), 0) AS order_count
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id
WHERE o.total > 0
GROUP BY c.id, c.name;

This defaults missing counts to 0.

claude-opus-4-7

function paginate(items, cursor, limit = 20) {
  const start = cursor ? items.findIndex(i => i.id === cursor) + 1 : 0; // bug fix: skip cursor itself
  return {
    data: items.slice(start, start + limit),
    next: items[start + limit]?.id ?? null,
  };
}

gpt-5

function paginate(items, cursor, limit = 20) {
  // bug fix: cursor item itself was being returned; advance past it
  const start = cursor ? items.findIndex(i => i.id === cursor) + 1 : 0;
  return {
    data: items.slice(start, start + limit),
    next: items[start + limit]?.id ?? null,
  };
}

gemini-3-pro

function paginate(items, cursor, limit = 20) {
  let start = 0;
  if (cursor) {
    const idx = items.findIndex(i => i.id === cursor);
    if (idx >= 0) start = idx + 1; // bug fix: cursor was inclusive
  }
  return {
    data: items.slice(start, start + limit),
    next: items[start + limit]?.id ?? null,
  };
}

qwen3.6-35b-a3b-nvfp4

function paginate(items, cursor, limit = 20) {
  const start = cursor ? items.findIndex(i => i.id === cursor) + 1 : 0; // off-by-one fix
  return {
    data: items.slice(start, start + limit),
    next: items[start + limit]?.id ?? null,
  };
}

claude-opus-4-7

{"date":"2026-05-09","attendees":["Sarah Chen","Raj P","Tomás"],"action_items":[{"owner":"Sarah Chen","task":"Ship the auth migration","due":"2026-05-13"},{"owner":"Raj P","task":"Chase SSO config from vendor","due":null},{"owner":"Tomás","task":"Draft of the launch post","due":null},{"owner":"me","task":"Customer interview write-up","due":"2026-05-11"}]}

gpt-5

{"date":"2026-05-09","attendees":["Sarah Chen","Raj P","Tomás"],"action_items":[{"owner":"Sarah Chen","task":"Ship the auth migration","due":"2026-05-13"},{"owner":"Raj P","task":"Chase SSO config from vendor","due":null},{"owner":"Tomás","task":"Draft launch post","due":null},{"owner":"narrator","task":"Customer interview write-up","due":"2026-05-12"}]}

gemini-3-pro

```json
{"date":"2026-05-09","attendees":["Sarah Chen","Raj P","Tomás"],"action_items":[{"owner":"Sarah Chen","task":"ship auth migration","due":"2026-05-13"},{"owner":"Raj P","task":"SSO config from vendor","due":null}]}
```

qwen3.6-35b-a3b-nvfp4

{"date":"2026-05-09","attendees":["Sarah Chen","Raj P","Tomás"],"action_items":[{"owner":"Sarah Chen","task":"Ship auth migration","due":"2026-05-13"},{"owner":"Raj P","task":"Chase SSO config from vendor","due":null},{"owner":"Tomás","task":"Draft launch post","due":null},{"owner":"author","task":"Customer interview write-up","due":"2026-05-12"}]}

claude-opus-4-7

QUARTZ-7392-DELTA

gpt-5

QUARTZ-7392-DELTA

gemini-3-pro

QUARTZ-7392-DELTA

qwen3.6-35b-a3b-nvfp4 (262K cap)

claude-opus-4-7

[
  {"tool": "web_search", "args": {"query": "Anthropic Claude 4.7 release announcement date site:anthropic.com"}},
  {"tool": "fetch_url", "args": {"url": "<url-from-prior-result>"}},
  {"tool": "send_email", "args": {"to": "ops@example.com", "subject": "Claude 4.7 tool count", "body": "<count>"}}
]

gpt-5

[
  {"tool": "web_search", "args": {"query": "Claude 4.7 release announcement Anthropic"}},
  {"tool": "fetch_url", "args": {"url": "<top result url>"}},
  {"tool": "run_shell", "args": {"cmd": "echo $TEXT | grep -o tool | wc -l"}},
  {"tool": "send_email", "args": {"to": "ops@example.com", "subject": "Claude 4.7 tool count", "body": "<count>"}}
]

gemini-3-pro

I'll help you with that. Here's the plan: [
  {"tool": "web_search", "args": {"query": "Claude 4.7"}},
  {"tool": "web_search", "args": {"query": "Anthropic announcement"}},
  {"tool": "fetch_url", "args": {"url": "..."}},
  {"tool": "send_email", "args": {"to": "ops@example.com", "subject": "Result", "body": "..."}}
]

kimi-k2.6

[
  {"tool": "web_search", "args": {"query": "Anthropic Claude 4.7 release announcement"}},
  {"tool": "fetch_url", "args": {"url": "<from search>"}},
  {"tool": "send_email", "args": {"to": "ops@example.com", "subject": "Claude 4.7 tool count", "body": "<count>"}}
]

claude-opus-4-7

• Anthropic이 Claude 4.7 공개, 1M 토큰 컨텍스트가 마침내 Opus 등급에 도달.
• 고동시성 도구 호출 환경에서 도구 호출 안정성 향상, prompt cache TTL은 5분 유지.
• 초기 사용자 보고에 따르면 Sonnet/Haiku의 p95 지연이 약 12% 감소, Opus 개선은 발표 기준이며 미검증.

gpt-5

• Anthropic이 Claude 4.7 공개, 핵심은 1M 컨텍스트가 Opus 등급에 도달한 것.
• Anthropic은 모델 패밀리 전반에서 p95 지연이 약 12% 개선됐다고 발표.
• 병렬 및 고동시성 상황에서 도구 호출 동작이 더 안정적임.

gemini-3-pro

• Anthropic이 오늘 Claude 4.7을 공개했으며 여러 업데이트가 포함됨.
• 1M 컨텍스트 윈도우가 이제 Sonnet과 Haiku 외에 Opus 등급에서도 지원되며, 장기 컨텍스트 에이전트 워크플로에 큰 변화임.
• Anthropic에 따르면 지연 시간이 약 12% 감소했고 도구 사용 동작도 개선됐음.