SWIFTEVAL
A language-specific benchmark for LLM-generated code evaluation
Code LLMs are graded almost exclusively in Python - and the multilingual benchmarks that claim otherwise are mostly machine-translated Python. SwiftEval is the first Swift-oriented benchmark built by hand: 28 problems, 44 models evaluated, and a leaderboard that looks very different once the training-set comfort zone is gone.
35 benchmarks. Three speak Swift.
Across 35 popular natural-language-to-code benchmarks published between 2018 and 2025, robust evaluation exists almost only for Python:
32 support Python 12 Java 8 multiple languages 3 Swift
And four of the five multilingual benchmarks rely on LLM-based translations of Python problems - overlooking the unique features of the adopted languages.
What machine translation breaks
Focusing on Swift, we audited the translated benchmarks. Broadly recognized suites omit key Swift features and introduce critical errors. The share of broken problems:
HumanEval-XL
func sumProduct(numbers: [AnyHashable]) -> [Int] { … AnyHashable breaks Swift static typing
examples left in Python
array instead of tuple
MultiPL-E (HumanEval)
// Round the answer to the nearest integer… // Python: [8, 8, 10] // C++, JavaScript, Swift: [8, 9, 10]
same prompt, different rounding - "it's easy, isn't it?"
MultiPL-E (MBPP)
func list_tuple(listx: [Int]) -> AnyHashable { … completely unsolvable in Swift
HumanEval-XL
func longest(strings: [AnyHashable]) -> String { …
ignores Optional - 27% of all problems share this issue
28 problems, written by hand
Instead of rapid scaling through automatic translation, SwiftEval takes quality over quantity: 28 unique problems (and counting), each crafted by an industry macOS software engineer with deep Swift knowledge - built around the features no other benchmark covers:
Generics · Static Typing · Protocols · Closures · Enums · Optionals
Natural language query
What to build, stated plainly.
Additional context
Protocols or types the solution must fit.
Code entrypoint
The exact signature the model completes.
Unit tests
XCTest assertions that decide pass or fail.
"Write code in Swift programming language. Don't include usage examples or tests. Implement generic stack data structure with push and pop functions."
protocol StackProtocol {
associatedtype Element
func push(_ element: Element)
func pop() -> Element?
}
class Stack<T>: StackProtocol { …
// and the tests that judge it:
XCTAssertEqual(stack.pop(), 2)
XCTAssertNil(emptyStack.pop())So is it good or just overfitted?
44 experiments on open- and closed-source models. Each score is pass@1 with temperature 0.2, top-p 0.95, and 20 completions per problem. The Δ column is what happens when Python leaves the room:
| Model | Size | HumanEval | SwiftEval | Δ score | Δ rank |
|---|---|---|---|---|---|
| GPT-4o | - | 90.3 #2 | 88.9 ±2.6 #1 | -1.4 | +1 |
| GPT-4o Mini | - | 87.2 #7 | 85.6 ±2.9 #3 | -1.6 | +4 |
| GPT-4 Turbo | - | 88.2 #6 | 87.1 ±2.7 #2 | -1.1 | +4 |
| GPT-4 | - | 86.6 #8 | 82.2 ±3.2 #5 | -4.4 | +3 |
| GPT-3.5 Turbo | - | 68.0 #21 | 81.3 ±3.2 #6 | +13.3 | +15 |
| Codestral Mamba | 7B | 75.0 #19 | 58.9 ±4.1 #11 | -16.1 | +8 |
| Codestral | 22B | 81.1 #13 | 77.8 ±3.4 #8 | -3.3 | +5 |
| CodeLlama Instruct | 7B | 34.8 #41 | 28.5 ±3.8 #27 | -6.3 | +14 |
| CodeLlama Instruct | 13B | 42.7 #37 | 43.7 ±3.9 #16 | +1.0 | +21 |
| CodeLlama Instruct | 34B | 41.5 #38 | 42.1 ±3.9 #17 | +0.6 | +21 |
| CodeLlama Instruct | 70B | 67.8 #22 | 52.3 ±4.0 #14 | -15.5 | +8 |
| CodeGemma | 2B | 31.1 #44 | 15.2 ±2.9 #37 | -15.9 | +7 |
| CodeGemma | 7B | 44.5 #35 | 26.9 ±3.8 #30 | -17.6 | +5 |
| CodeGemma Instruct | 7B | 56.1 #31 | 28.1 ±3.7 #28 | -28.0 | +3 |
| CodeGemma 1.1 Instruct | 7B | 60.4 #27 | 36.2 ±4.1 #22 | -24.2 | +5 |
| CodeGeeX2 | 6B | 35.9 #39 | 2.5 ±1.3 #44 | -33.4 | -5 |
| CodeGeeX4 | 9B | 82.3 #12 | 53.7 ±3.9 #12 | -28.6 | 0 |
| CodeQwen1.5 | 7B | 51.8 #32 | 41.5 ±3.8 #18 | -10.3 | +14 |
| CodeQwen1.5 Chat | 7B | 83.5 #11 | 37.9 ±3.9 #21 | -45.6 | -10 |
| Qwen2.5 Coder | 1.5B | 43.9 #36 | 11.3 ±2.7 #39 | -32.6 | -3 |
| Qwen2.5 Coder | 7B | 61.6 #26 | 25.5 ±3.5 #31 | -36.1 | -5 |
| Qwen2.5 Coder Instruct | 7B | 88.4 #5 | 40.0 ±3.8 #19 | -48.4 | -14 |
| Qwen2.5 Coder Instruct | 14B | 89.6 #4 | 62.8 ±4.0 #10 | -26.8 | -6 |
| Qwen2.5 Coder Instruct | 32B | 92.7 #1 | 79.1 ±3.4 #7 | -13.6 | -6 |
| DeepSeek Coder Instruct | 1.3B | 65.2 #24 | 10.8 ±2.6 #40 | -54.4 | -16 |
| DeepSeek Coder Instruct | 6.7B | 78.6 #17 | 17.5 ±3.1 #36 | -61.1 | -19 |
| DeepSeek Coder Instruct | 33B | 79.3 #15 | 32.0 ±3.8 #25 | -47.3 | -10 |
| DeepSeek Coder V2 Instruct | 16B | 81.1 #13 | 69.1 ±3.7 #9 | -12.0 | +4 |
| DeepSeek Coder V2 Instruct | 236B | 90.2 #3 | 82.4 ±3.2 #4 | -7.8 | -1 |
| Granite Code Instruct | 3B | 51.2 #33 | 12.1 ±2.7 #38 | -39.1 | -5 |
| Granite Code Instruct | 8B | 57.9 #30 | 23.1 ±3.6 #32 | -34.8 | -2 |
| Granite Code Instruct | 20B | 60.4 #27 | 17.6 ±3.1 #35 | -42.8 | -8 |
| Granite Code Instruct | 34B | 62.2 #25 | 32.1 ±3.9 #24 | -30.1 | +1 |
| StarCoder 2 | 3B | 31.7 #43 | 21.3 ±3.4 #33 | -10.4 | +10 |
| StarCoder 2 | 7B | 35.4 #40 | 28.9 ±3.6 #26 | -6.5 | +14 |
| StarCoder 2 | 15B | 46.3 #34 | 45.5 ±4.1 #15 | -0.8 | +19 |
| StarCoder 2 Instruct | 15B | 72.6 #20 | 53.6 ±4.0 #13 | -19.0 | +7 |
| Stable Code | 3B | 32.4 #42 | 9.1 ±2.5 #42 | -23.3 | 0 |
| Stable Code Instruct | 3B | 59.0 #29 | 10.2 ±2.4 #41 | -48.8 | -12 |
| OpenCodeInterpreter | 6.7B | 76.2 #18 | 18.9 ±3.1 #34 | -57.3 | -16 |
| OpenCodeInterpreter | 33B | 79.3 #15 | 27.3 ±3.5 #29 | -52.0 | -14 |
| Yi-Coder Chat | 1.5B | 67.7 #23 | 4.9 ±1.7 #43 | -62.8 | -20 |
| Yi-Coder Chat | 9B | 85.4 #10 | 33.9 ±3.9 #23 | -51.5 | -13 |
| Nxcode-CQ | 7B | 86.6 #8 | 39.7 ±4.0 #20 | -46.9 | -12 |
pass@1 on HumanEval (Python) vs SwiftEval (Swift), with per-benchmark ranks. Δ color intensity marks the size of the drop; the rare gains are highlighted. Only 5 of 44 experiments hold within two points of their Python score.
What the numbers say
What the models actually write
In GPT-4 Turbo's HumanEval solutions, just 10 keywords make up 90% of all keyword usage - the fingerprint of purely algorithmic, function-oriented problems:
let · return · if · func · var · in · for · else · true · false
On SwiftEval the vocabulary opens up - the benchmark forces the model into the parts of the language a translated Python problem never touches:
return · func · var · let · if · private · case · self · init · else
Does size buy skill?
On HumanEval, small models match the latest ChatGPT models - a weak correlation that raises the question of benchmark memorization during training. On SwiftEval the trendline is steeper: a better, more natural dependency between model size and score.
We thank the Armed Forces of Ukraine for providing security to complete this work. 🇺🇦
Accepted to FORGE'25 Benchmarking on 15.01.2025, to be published by IEEE under the CC BY-NC-ND 4.0 license. This is the accepted version of the article (5 pages, 2 figures, 1 table). Preprint: arXiv:2505.24324 · DOI: 10.48550/arXiv.2505.24324
28 problems. 44 models.
One language, finally measured.
The paper covers the benchmark survey, the audit of translated suites, the problem set, and the full evaluation.