Qwen3.6-27B on RTX 3090: tested 12 GGUF quants across HumanEval+, MBPP+, perplexity, throughput and needle-in-haystack. First-timer results.

Comprehensive Benchmarking of Qwen 3.6-27B GGUF Quantizations on RTX 3090: An Initial Exploration

Introduction

As a newcomer to local Large Language Models (LLMs), I embarked on an exploratory benchmarking journey with the aim of understanding the performance characteristics of various GGUF quantizations of Qwen 3.6-27B. While I do not claim expert status, I hope sharing these initial findings helps others in the community. My methodology reflects practical choices tailored to my hardware environment and may differ from specialized setups.

Key considerations include:
– Using greedy decoding (temperature=0) for reproducibility, which likely yields upper-bound metrics compared to more typical sampling techniques.
– Disabling reasoning features (--reasoning off) to ensure compatibility and consistent results with EvalPlus, understanding that enabling reasoning could improve performance.
– Recognizing the limitations of our testing subset (20-task HumanEval) which is not statistically robust and may not fully discriminate model quality.
– Performing a nuanced analysis of “needle-in-haystack” tests with a single, distinctive prompt, noting that passing all tests may not reflect true equivalence.

Hardware & Software Environment

• GPU: NVIDIA RTX 3090 (24 GiB VRAM)
• CPU: AMD Ryzen 9 9900X
• Software: llama.cpp (commit ad2775726) build b9261
• Sampling Method: Greedy (T=0)
• Evaluation Platform: Windows WSL2 (due to Linux compatibility issues with EvalPlus)

Benchmarking Process Overview

The benchmarking process was conducted in multiple phases:

1. Wide Screening of 10 Candidate Models
2. Introduction of Two Additional MTP Variants for Deeper Testing
3. Full-Scale Evaluation of Top Candidates
4. Validation of Production-Ready Configurations
5. Additional Benchmarks: Perplexity, Throughput, and Needle-in-Haystack Tests

Phase 1: Initial Screening

The initial step involved evaluating 10 GGUF quantizations on a small subset (20 tasks) of HumanEval. The key metric was pass@1 performance, with additional considerations of speed and stability. Notable observations from this phase include:

• Most models achieved 100% pass@1 on the simple subset, indicating limited discrimination at this stage.
• Some models, such as Q6_K without multi-task prompting (MTP), showed higher latency and inconsistency.
• Finetuned variants like abritherated-Gaston underperformed significantly in code tasks, often due to overfitting or reduced generalization.
• Larger models like NEO-CODE-2T-OT-Q5_K_M achieved perfect scores but at the expense of increased computational cost (~3× slower).

Phase 2: Incorporation of MTP Variants

Building on initial insights, two specific MTP-augmented models were added:

• UD-Q4_K_XL-MTP
• IQ4_NL-mtp

These models, incorporating Multi-Task Prompting heads or importance-aware Non-Linear quantization, demonstrated promising early results and were included directly in the comprehensive benchmarks.

Phase 3: Full Benchmarking with HumanEval+ and MBPP+

Using EvalPlus with an 8K context window, we measured pass@1 over extensive datasets:

| Model | HumanEval+ | MBPP+ | Notes |
|———|————–|——–|——-|
| UD-Q4_K_XL (no MTP) | 92.1% | 78.3% | Strong baseline, fastest among top models |
| IQ4_NL-mtp | 91.5% | 76.7% | Slightly smaller size, fastest overall |
| UD-Q4_K_XL-MTP | 90.9% | 78.0% | Close performance, good balance |
| Others (Q5 variants) | ~90.9% | ~76-77% | Consistent but slightly slower |

Notably, all Q5-based models shared similar performance, with minor fluctuations within the margin of error. Failures tended to be concentrated in a common subset of 15 tasks, suggesting inherent difficulty rather than model deficiency.

Model Size & Storage

The models evaluated ranged from approximately 16.3 GB for the smallest candidate (IQ4_NL-mtp) up to around 20 GB for larger variants. These sizes are critical considerations for deployment and resource planning.

Config Validation for Production Use

To simulate operational conditions, the leading candidate, IQ4_NL-mtp, was tested with alternate cache (KV) configurations and extended context windows (up to 128K tokens). Results indicated negligible quality loss when shifting from 8K to 128K context, with performance metrics remaining stable. This validates the potential for longer-context applications without sacrificing accuracy.

Perplexity and Throughput Benchmarks

Perplexity testing on WikiText-2 revealed negligible differences (within measurement error) between the top candidates, confirming their comparable language modeling capabilities at the statistical level.

Throughput measurements, performed with llama-bench, indicated roughly 6% performance gains for IQ4_NL-mtp over UD-Q4_K_XL-MTP across various context lengths, with processing speeds reaching approximately 1500 tokens/sec without MTP. When MTP was engaged, real-world generation slowed but still maintained competitive throughput (~65-100 tokens/sec).

Needle-in-Haystack Recall

A targeted long-context (up to 96K tokens) “needle-in-haystack” test involved searching for a specific password string (the “needle”) embedded within a lengthy prompt. Both top contenders, IQ4_NL-mtp and UD-Q4_K_XL-MTP, achieved perfect recall across multiple context lengths and depths, demonstrating robust long-term retrieval and contextual understanding.

Summary of Leading Models

| Criterion | IQ4_NL-mtp | UD-Q4_K_XL-MTP |
|——————|————–|—————-|
| HumanEval+ | 91.5% | 90.9% |
| MBPP+ | 77.2% | 78.0% |
| Perplexity | ~6.94 | ~6.98 |
| Throughput (tokens/sec) | ~1486 | ~1403 |
| Needle Recall | 100% | 100% |
| Model Size | ~16.3 GB | ~17.9 GB |
| Usable Context | ~110K tokens | ~80K tokens |

Practical Insights and Limitations

While these findings offer valuable guidance, certain aspects remain untested:

• Performance with reasoning or “thinking” enabled, which may enhance code generation.
• Impact of alternative sampling strategies, such as top-p or temperature > 0.
• Evaluation on specialized coding workloads (e.g., Claude Code, Aider).
• Performance with full-length prompts exceeding 128K tokens, beyond current hardware constraints.
• Potential benefits from additional finetuning or model finetuning strategies.

Concluding Remarks

For local deployment on a 24 GiB GPU, the IQ4_NL-mtp quantization appears to provide the most favorable combination of size, speed, and accuracy. Its balanced performance metrics and stability across various tests underscore its suitability for practical applications. The UD-Q4_K_XL-MTP model is also competitive, especially for scenarios favoring higher accuracy over speed.

Future work includes exploring reasoning-enabled performance, optimizing sampling parameters for improved diversity, and extending benchmarks to real-world agentic tasks. I invite the community to review these results, reproduce experiments, or suggest refinements.

---

Note: All commands, configurations, and raw data are documented for reproducibility. Feedback and corrections are welcome—scripted testing and community validation are vital for advancing our collective understanding.