qwen2.5-3b-debug-v3-lora

A LoRA adapter for Qwen2.5-3B-Instruct that fixes buggy Python programs. You give it a broken program (plus its error message, stdin and an execution-trace summary), and it returns the corrected program in a single ```python block.

This is variant v3 (code + error + trace) of a three-way ablation. The ablation tests whether giving a small model a dynamic execution trace (from FocusTracer) helps it repair code.

Variant	Model input	pass@1
v1	buggy code	64.8%
v2	buggy code + error message / stderr + stdin	69.1%
v3	buggy code + error + FocusTracer execution trace summary	68.4%

For the same model as a ready-to-run quantized file (llama.cpp / Ollama), see Pilestin/qwen2.5-3b-debug-v3-GGUF.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen2.5-3B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "Pilestin/qwen2.5-3b-debug-v3-lora")

SYSTEM_PROMPT = (
    "You are an expert Python debugging assistant. "
    "The user gives you a buggy Python program (and optionally its runtime error "
    "and an execution trace). Reply with the corrected program in a single "
    "```python code block. Do not add explanations."
)

buggy = """num = [int(input()) for i in range(10)]
num.sort(reverse=True)
for i in range[0:3]:
    print(num[i])"""

# See "Prompt format" below: v3 also expects the error, stdin and 'Execution trace' sections.
user = f"Fix the following buggy Python program.\n\n```python\n{buggy}\n```"

messages = [{"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True,
                                       return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

To merge the adapter into the base weights, call model = model.merge_and_unload().

Prompt format

The system prompt is:

You are an expert Python debugging assistant. The user gives you a buggy Python program (and optionally its runtime error and an execution trace). Reply with the corrected program in a single ```python code block. Do not add explanations.

The user message is built step by step. Each variant adds a section to the one before it:

Fix the following buggy Python program.

```python
<buggy program>
```
                                                   ── v1 ends here
Runtime error:
<exception message + last 4 lines of stderr>
   (if there is no exception, this whole section is replaced by one of:
    "The program runs without crashing but produces WRONG OUTPUT for the given input."
    "The program is too slow / never terminates (time limit exceeded).")

Program input (stdin):
```
<stdin, if ≤ 400 chars>
```
                                                   ── v2 ends here
Execution trace (FocusTracer, last steps before failure):
```
L3: for i in range[0:3]:
!! EXCEPTION at L3: TypeError: 'type' object is not subscriptable
```
                                                   ── v3 ends here

The trace summary is the last 25 line events before the failure, capped at 1,800 characters. Each line has the form L<line>: <source> [var=value, ...]: the line number (matching the buggy program), its source, and the variables that changed on that line. Any exception is shown as an !! EXCEPTION line.

The assistant reply is always the whole fixed program in a single ```python block.

Evaluation

The benchmark set is 960 held-out CodeNetPy Python bug-fix pairs. The test problems do not appear in training: training covers 466 problems, test covers 306 different ones.

• pass@1: the generated program is executed on the problem's input (5 s timeout), and its stdout is compared with the reference solution's output. This is not text similarity.
• exact: the generated code matches the reference fix after whitespace normalization.
• Decoding is greedy (temperature 0). Models were run as Q4_K_M GGUF through Ollama on an RTX 4060 Laptop GPU. Significance was tested with a paired McNemar exact test.

Input variant	Base Qwen2.5-3B-Instruct (same prompt)	Fine-tuned	Δ
v1 · code only	59.8%	64.8%	+5.0
v2 · code + error	57.2%	69.1%	+11.9
v3 · code + error + trace	59.2%	68.4%	+9.2

Fine-tuned models, by bug category (pass@1):

Model	runtime exception	wrong answer	time limit exceeded	exact	tok/s	J/query
FT-v1	49.4%	73.6%	77.9%	14.4%	80.7	71.7
FT-v2	57.0%	81.0%	74.6%	16.4%	85.0	69.6
FT-v3	60.3%	77.3%	71.1%	18.4%	86.2	65.9

Key findings:

1. Fine-tuning helps clearly. All three variants beat the base model (p < 0.01), with no change in inference speed or energy.
2. Traces help only in some cases. Overall, v3 ≈ v2 (p = 0.62). On runtime exceptions, however, adding the trace raises pass@1 by +3.3 points. The trace helps most when it shows the exact line and variable state where the crash happens. For wrong-output and timeout bugs, the cut-off trace mostly adds noise.
3. Traces produce the fixes closest to the reference. v3 has the highest exact-match rate (18.4 %).

Training data

The training dataset is not published. This section summarizes how it was built.

• Source. The Python subset of CodeNetPy: real buggy → accepted submission pairs from IBM's Project CodeNet (Apache-2.0). The data was deduplicated, empty or identical pairs were dropped, and every fix had to pass ast.parse. The result is 1,959 train / 960 test pairs, with no problem overlap between the splits.
• Labels. Each pair has a bug_category (runtime_exception / wrong_answer / time_limit_exceeded), an error_class (e.g. TypeError, IndexError) and diff line ranges. About 1/3 of the pairs fall in each category.
• Execution traces. Every buggy program was run on its problem's sample input under FocusTracer, a Python execution tracer that records line events, variable changes and exceptions as XML. Programs that ran too long were given a 2 s alarm and a hard-kill watchdog, and yielded partial traces. The trace covers 93 % of train and 96 % of test pairs. The exception seen in the live trace matched the dataset's error_class label in 96–99 % of cases. For training, each trace was shrunk to the compact summary shown in Prompt format.
• Ablation design. The three variants use exactly the same pairs and the same target (the fixed program). Only the amount of information in the user message changes, so any difference comes from that information alone.

Training procedure

QLoRA with TRL SFTTrainer, sized to fit in 8 GB of VRAM.

Setting	Value
Base model	Qwen/Qwen2.5-3B-Instruct (4-bit NF4, double quant, bf16 compute)
LoRA	r = 16, alpha = 32, dropout = 0.05, bias none
Target modules	q, k, v, o, gate, up, down projections
Max sequence length	2048
Epochs	3 (best eval-loss checkpoint kept)
Effective batch size	8 (1 × 8 gradient accumulation)
Optimizer / LR	paged AdamW 8-bit, 2e-4, cosine, 50 warmup steps, weight decay 0.01
Other	gradient checkpointing, no packing

Limitations

• The model was trained only on short, stdin/stdout competitive-programming programs (median ~260 characters). It is not a general-purpose coding assistant, and it has not been tested on multi-file or library code.
• It sometimes returns the input unchanged or makes a plausible but wrong fix. Always run the output before you trust it.
• Prompts are English only. Results are best when you use the exact system prompt and user template shown above.
• The benchmark numbers come from the Q4_K_M GGUF build of this adapter, run through Ollama.
• The license follows the base model (Qwen Research License), which does not allow commercial use.

Links

• Code (training, merging, quantization, benchmark): https://github.com/BitnetTR/llm-fine-tune-for-debugging