Fine-tuning Qwen for C/C++ vulnerability detection

The task: tell a vulnerable function from its patch

Each input contains one function, its filename and its language. The model receives no repository retrieval or tools, and its target answer is either VULNERABLE or SAFE. This keeps the experiment focused on what can be inferred from the function itself.

The central metric is both-correct pair accuracy: a pair counts only when the vulnerable version is flagged and the patched version is accepted. Each version is presented separately. A model that calls everything vulnerable gets zero pairs right, even though it finds every vulnerable function. This makes the metric useful for testing whether a classifier responds to the change that removes a vulnerability.

Training on one consumer GPU

The training set contains 5,458 function-level examples, with another 682 functions forming 341 vulnerable/patched validation pairs. I used supervised fine-tuning with assistant-only loss, so the training objective targets the answer rather than the input code.

LoRA trains a small set of adapter parameters on top of the base model. This run used rank 16, alpha 32 and dropout 0.05, with no bias, targeting language-model attention and MLP projections. The release is checkpoint 540, about 1.58 epochs into training. Checkpoint selection used validation calibrated pair accuracy plus 0.01 times ranking accuracy. The download contains the adapter, so it needs the original Qwen3.5-9B base weights to run.

Results and what is being compared

The PrimeVul population contains 435 vulnerable functions and their 435 patched counterparts. A second evaluation uses 58 curated CVE-linked pairs from 34 projects. All figures below require both functions in a pair to be labelled correctly.

The PrimeVul adapter result uses the frozen native reference threshold. Under the historical calibration procedure used for the base-to-adapter comparison, the same checkpoint scored 73/435 (16.78%). The two adapter results use identical saved benchmark scores with different validation-derived thresholds.

MiMo scored higher on both populations. It uses an investigation harness that generates binary verdicts; Qwen uses candidate-answer likelihoods and validation-derived thresholds. Within each population the input functions are the same, but the prompts and inference procedures differ. These are comparisons of models together with their evaluation procedures, so they do not isolate the effect of model weights.

Building the function-only CVE evaluation

The curated set was selected from CVEfixes v1.0.8 provenance and public before/after source files. Review required the extracted function to contain the linked unsafe operation and its patch to remove that mechanism. Cases needing omitted project-specific context were excluded.

Overlap checks covered recorded local training and evaluation sources, including CVE identities, commits, normalized code and same-name near clones. Both versions had to fit 4,096 tokens without truncation, with a maximum of ten pairs per project. Of 365 collected candidates, review admitted 60; the project cap left 58. The manifest was frozen before either model ran, and thresholds were taken from validation rather than fitted on these cases.

This remains a single-reviewer curated population. Independence from base-model pretraining is unknown, and SAFE means the patched counterpart for the linked CVE. The set does not represent all CVEs.

Why scoring details matter

The frozen reference compares the log probabilities of the complete candidate answers, including ending tokens, using native BF16 output logits. It predicts vulnerable when the log-probability difference exceeds −0.3750172257423401. First-token scoring and free-form generation do not reproduce this procedure. Many margins lie close to the threshold, so small numerical changes can change classifications.

Fitting a threshold directly on PrimeVul produces 80/435 correct pairs, but that is a posthoc diagnostic, excluded from the headline results. PrimeVul and existing CVEfixes results have also informed later diagnostics; they should not be treated as untouched tests for future model selection.

What I take from the experiment

A small adapter trained on consumer hardware improved paired discrimination over the base model in these evaluations. Absolute performance is still weak. On the frozen PrimeVul reference, individual function accuracy is 54.14% and vulnerable-class F1 is 59.66%. Simply flagging every function vulnerable gives 66.67% F1 on this balanced population, while getting no pairs right. Reading these metrics together gives a more useful picture than reporting F1 alone.

The evaluation includes no unrelated benign functions and establishes no performance on the full imbalanced PrimeVul test population or production repositories. The released adapter is a research artifact for function-level classification. It does not establish reliable vulnerability triage or functional patch repair; the separate Q&A and patch-repair experiments are outside this release.