Situation

My Local Muse Glimmer 30B Kept Leaking Its Own Tool-Calling Markers Into Replies, So I Planned To Fix It At The Source With A QLoRA Fine-Tune Instead Of Patching The Parser Again. Before Training Anything I Ran A Baseline Eval And It Said Leak Rate 0%, Parallel Calls 2%. Both Numbers Looked Wrong, So I Opened The Raw Outputs And Found Every Single One Contained <| Tokens. The Metric Was Broken, Not The Model. I Fixed It, Concluded The Parser Was Already Enough, And Trained Anyway As An Experiment. Version One Made Parallel Tool Calls Work Almost Perfectly And Also Made The Model Call A Tool On Two Thirds Of Questions That Needed None.

Result First:

The Training Itself Was The Easy Part. What Mattered Was What The Training Data Never Showed The Model, And How Long It Took Me To Notice My Own Scoreboard Was Lying.

How It Went Wrong

The Scoreboard Lied First

The Model Is A Vision-Language Model With A Chat Template That Routes Output With to=self And to=user Prefixes, And Tool Calls Come Out As <atem:function_calls> Blocks. My Shim Already Had A Parser That Strips All Of That Before The Reply Reaches The User.

The First Baseline Said The Leak Rate Was 0% And Only 2% Of Parallel Questions Were Answered In One Turn. I Did Not Believe Either Number. Raw Outputs Told The Real Story:

• Every Raw Output Contained <| Special Tokens, And Most Contained A to=self Reasoning Channel. That’s The Template Working As Designed, Not A Leak.
• Parallel Questions Were Answered One Call Per Turn. The Model Was Fine. My Metric Required All Calls In A Single Turn.

I Rewrote The Metrics To Score Two Views Separately. raw Is What The Model Emits, And parser Is What The User Would See. Each Has Four Leak Flags: Special Tokens, to= Prefixes, <atem: Fragments And Reasoning Text. Parallel Now Checks Whether The First Call Is In The Expected Set. Re-scored On The Same 500 Outputs:

For The Goal I Started With, The Parser Was Already Enough. The Fine-Tune Was No Longer A Fix. It Was An Experiment, And I Ran It As One.

Version One: Teaching Only The Half That Calls

Setup, Kept Boring On Purpose:

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16,
                         bnb_4bit_use_double_quant=True)
lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
                  target_modules=r".*language_model.*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)")
cfg = SFTConfig(per_device_train_batch_size=1, gradient_accumulation_steps=16,
                learning_rate=1e-4, lr_scheduler_type="cosine",
                completion_only_loss=True, max_length=4096, bf16=True,
                gradient_checkpointing=True)

4-Bit NF4 Base, LoRA On The Language Model Layers Only So The Vision Tower Stays Untouched, And Loss On The Last Assistant Turn Only. The Data Was 8,000 Examples From Three Public Function-Calling Sets (Hermes, xLAM, Glaive). The Eval Set Was Split By Tool Name, Not By Example: About One Eighth Of Tool Names Were Hashed Into Eval And Never Appeared In Training. That Way The Eval Measures Whether The Model Learned The Format, Not Whether It Memorized Tools. It Ran About 340 Steps On One GPU With 48GB Of VRAM, And Loss Went From 0.74 To 0.10.

On The Same-Distribution Eval It Looked Great: Decision 97.0% To 99.4%, Parallel Questions Answered In One Turn From 2% To 100%, And The Reasoning Channel Went From Appearing In 77% Of Outputs To 0%.

Then I Ran The Eval I Almost Skipped. 365 Questions The Training Set Had Nothing Like: 255 From BFCL v3 And 110 Chinese Questions I Wrote Myself, Many Of Them Deliberately Offering A Tool That Doesn’t Fit.

The Upside Was Real Too: Parallel Questions Answered In One Turn Went From 4% To 97%.

The Training Set Was Almost All “Call A Tool” Examples And Had The Reasoning Removed. I Had Taught The Model To Call Tools And Never Showed It Once When Not To.

The Fix

Add The Missing Half

Version Two Added 2,000 Negative Examples To The 8,000, So About 26% Of The Training Set Is Now “Tools Offered, None Needed”:

Each Target Also Got One Or Two Sentences Of Reasoning, Rendered Into The to=self Channel, Saying Whether To Call And Which Tool. I Kept The Reasoning Short On Purpose.

One Deliberate Choice: The Chinese Negatives Use Only Random, Unrelated Tools. My Chinese Eval Questions Are The Kind Where The Tool Looks Related To The Request. If I Had Trained On That Same Pattern, The Eval Would Have Been Measuring Memory Of My Own Trick. I Also Dropped Any Example That Overlapped With Either Eval Set.

Pick A Checkpoint, Don’t Take The Last One

I Saved A Checkpoint Every 50 Steps And Ran Both Evals On Several Of Them. This Mattered More Than I Expected:

On The Same-Distribution 500, ckpt-200 Scored 100% Decision And 0% False Calls.

The Negatives Worked: The Exploding False-Call Rate Came Back To Baseline, And The Chinese Decoy Group Went From 20% To 1.8%. They Did Not Work Everywhere. English live_irrelevance Is Still Three Times Worse Than The Untuned Model, And The Last Checkpoint Is Worse Than ckpt-200 On Almost Every False-Call Row.

Notes

• A Metric That Says 0% Deserves More Suspicion Than One That Says 40%. The First Eval Was Wrong In Both Directions At Once. Reading A Few Raw Outputs Took Minutes And Saved Me From Training Toward A Target That Didn’t Exist.
• The Training Data Is The Behavior. If Every Example Ends In A Tool Call, The Model Learns That Tool Calls Are What Replies Look Like. No Hyperparameter Fixes A Distribution Problem.
• Same-Distribution 100% Tells You Almost Nothing. v1 Scored 99.4% On The Eval Built Like Its Training Set And Called Tools On 67% Of The Irrelevant Questions In The One It Hadn’t Seen.
• Split Evals By The Thing You Want To Generalize Over. Splitting By Tool Name Instead Of By Example Is Cheap, And It’s What Made “Learned The Format” Distinguishable From “Memorized The Tools”.
• The Last Checkpoint Is Not The Best One. With The Same Run, ckpt-200 And Last Differ By 8 Points Of False Calls. Save Often, Score Every One.
• One Chinese Group Of 25 Questions Stayed At About 96% False Calls In Every Version, Including The Untuned One. Fine-Tuning Neither Caused It Nor Fixed It, And I Haven’t Figured Out Why.
• These Are Small Groups From A Single Run. Each Group Is 25 To 70 Questions, There Is One Seed, And A Few Points Of Difference Isn’t A Result. I Haven’t Merged Or Shipped Any Of This.

How To Prevent It

Reference:

• QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)
• Berkeley Function Calling Leaderboard (BFCL)
• PEFT
• TRL