Standardized Model Evaluations
Model Benchmarks & Performance Metrics
Pak-LLM Coder is empirically validated across industry-standard coding benchmarks and specialized bilingual software engineering evaluation datasets.
HumanEval Python
88.6%
Pass@1 standardized algorithmic synthesis benchmark
MultiPL-E Urdu
91.2%
Native Urdu prompt-to-code execution accuracy
Nastaliq Compression
1.08
Tokens per word overhead vs standard English BPE (3.8x)
Comparative Evaluation Suite
| Benchmark | Target Language | Pak-LLM Coder | Standard Baseline |
|---|---|---|---|
| HumanEval | Python 3.12 | 88.6% (Pass@1) | 76.4% |
| MultiPL-E Bilingual | Urdu → Python/TS/Go | 91.2% (Pass@1) | 48.2% |
| MBPP (Sanitized) | Python Algorithmic | 84.9% (Pass@1) | 72.0% |
| SWE-bench Lite | Multi-File Patching | 34.8% Resolved | 22.4% |