Situation

In The Last Post I Fine-Tuned Muse Glimmer 30B To Call Tools Better, Found It Calling Tools Everywhere, Added Negative Examples, And Picked Checkpoint 200 Out Of The Second Run. It Ended With One Honest Line: I Hadn’t Merged Or Shipped Any Of It.

This Post Is The Shipping Part. The Adapter Was Trained With PEFT On A Single 48GB GPU. The Model Actually Runs On A 32GB Mac Studio, Through MLX, As A 4-Bit Model Behind A Small Shim Service. Those Two Worlds Don’t Share A File Format, And The Mac Doesn’t Have Much Room To Spare.

Result First:

Getting The Adapter To Load Was An Afternoon. Getting It To Run Without Costing Half The Speed Meant Rebuilding The Whole Model — And Then Fighting launchd To Actually Use It.

How It Went

Try The Cheap Way First: Just Bring The Adapter

The Adapter Is 210MB. The Merged Model Would Be 56GB In bf16. So The Obvious First Move Was To Ship Only The Adapter And Load It On Top Of The 4-Bit Model The Mac Already Had.

PEFT And MLX Store The Same LoRA Differently:

So The Converter Is Mostly Renaming And Transposing. I Wrote It With Plain NumPy Reading And Writing Safetensors, Kept The bf16 Bits As They Were, And Listed All 416 Target Modules Explicitly Instead Of Relying On The “Last N Layers” Default. The Test That Mattered: Feed The Same Small Random Input Through The PEFT Math And The MLX Math And Check The LoRA Delta Is Identical.

One Trap: Loading Adapter Weights With strict=False Never Complains. If Half The Names Didn’t Match, It Would Load Silently And Do Nothing. I Compared All 832 Adapter Tensors Against The Model’s LoRA Parameters One By One. Zero Missing, Zero Shape Mismatches.

The Mac Didn’t Have Room To Test It

The Mac Studio Has 32GB. The Running Shim Alone Holds About 19GB, Another Local App And A VM Take About 16GB More, And Swap Was Already Around 25GB. Loading A Second Copy Of The Model To Test Would Have Dragged The Live Service Down With It.

So I Stopped The Shim For About 12 Minutes, Ran The Test In A Separate Process, And Started It Again.

Right, But Slow

10 Questions From The Out-Of-Distribution Set, Two From Each Group:

Running On A 4-Bit Base Instead Of The Base It Was Trained On Didn’t Visibly Hurt Anything Here. Ten Questions Only Tell You It Isn’t Broken, Not That It’s Better.

The Speed Was The Problem:

An Unmerged LoRA Adds Two Small Matrix Multiplications To Every Targeted Layer On Every Token. On A Machine That’s Already Memory-Bound And Swapping, That Cost About 44% Of Generation Speed.

The Fix

Merge, Then Quantize Again

Merging The LoRA Into The Weights Makes The Extra Math Disappear. But You Can’t Cleanly Merge Into Weights That Are Already 4-Bit, So The Path Was:

1. Merge The Adapter Into The bf16 Base With PEFT’s merge_and_unload (CPU Only, Capped At 8 Cores, About 3 Minutes, About 56GB Out).
2. Quantize The Merged Model To MLX 4-Bit With The Same Settings As The Community 4-Bit Build (--q-bits 4 --q-group-size 64).
3. Get The Result Onto The Mac.

The Converter Crashed At The Very End

I Ran The MLX Conversion On Linux With The CPU Build Of MLX, Because The Mac Had No Room For 56GB Of bf16 Weights. It Wrote All Four Weight Shards, About 18.2GiB, And Then Died Saving The Processor Config:

TypeError: Object of type ModelConfig is not JSON serializable

The Weights Were Fine. What Was Broken: processor_config.json Was Empty And config.json Was Missing The quantization Block. Since The Architecture Hadn’t Changed, I Copied Those Two Files From The Community 4-Bit Build And Kept The Original Merged Config Next To Them For Reference.

One Real Difference From The Community Build: Three Vision-Side Tensors Weren’t Quantized And Stayed bf16. Everything On The Text Side Matches. It Costs A Few Tens Of MB And Doesn’t Touch Tool Calling.

Moving 18GB To A Machine With 23GB Free

The Laptop In Between Had Only 12GB Free, So Nothing Could Land There. I Streamed It Straight Through:

# Source → Laptop → Mac, Nothing Written On The Laptop
<remote-exec> tar cf - -C /path/to/mlx-4bit . \
  | ssh mac 'mkdir -p ~/models/new-4bit && tar xf - -C ~/models/new-4bit'

At About 25MB/s That Took Around 13 Minutes. Then The Checksums Didn’t All Match — The Last Two Small Files Had Been Truncated. Resending Those Two Fixed It, And All 10 Files Matched By sha256. A Lazy Load With The Shim’s Own Environment Saw All 2,272 Tensors And A Working Processor.

The Mac’s Data Disk Was Left At 99%, Which Is Its Own Problem For Later.

Switching, And The Restart That Didn’t Restart Anything

The Shim Already Reads The Model Path From An Environment Variable, So Switching Should Have Been One Line In The launchd Plist Plus A Restart. It Wasn’t:

• The Shim’s restart Uses launchctl kickstart -k, Which Restarts The Process With The Job Definition launchd Already Has Loaded. The New Variable Was Never Read.
• The Shim’s start Regenerates The Plist From A Template, Which Would Have Overwritten The Variable I Just Added.

What Actually Works Is Editing The Installed Plist And Reloading The Job Definition:

/usr/libexec/PlistBuddy -c "Add :EnvironmentVariables:MODEL_PATH string /path/to/new-4bit" \
  ~/Library/LaunchAgents/<shim>.plist
launchctl bootout   gui/$(id -u)/<shim>
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/<shim>.plist

About 30 Seconds Of Downtime, 21 Seconds Of That Loading The Model. The Health Check Reported The New Path And loaded: true.

Then The Same 10 Questions, This Time Through The Shim’s Real API Instead Of A Test Script:

The Speed Came All The Way Back. The One Miss Was A Parallel Question Where The Model Made One Call Instead Of Two. Scoring Here Only Compares Tool Names And Call Counts From The Shim’s Parsed Output, Not Arguments. The Median Question Took About 14 Seconds End To End, And The First One About 39 Because Of Warm-Up.

9 Versus 10 On Ten Questions Is One Question. I’m Not Reading Anything Into It Yet.

Rolling Back Is Restoring The Backed-Up Plist And Doing The Same bootout / bootstrap.

Notes

• A 210MB Adapter Isn’t Free At Inference Time. It’s Cheap To Move And Expensive To Run Unmerged — On This Machine, About 44% Of Generation Speed.
• strict=False Will Happily Load Nothing. If You Convert Adapter Names By Hand, Count The Matches Yourself.
• You Can’t Merge Into 4-Bit Weights, So You Re-Quantize The Whole Model. The Adapter Is Tiny, The Path To Shipping It Fast Goes Through The Full bf16 Model Anyway.
• “Wrote All The Weights, Then Crashed” Is Still A Broken Model. Check The Configs, Not Just The Shard Sizes.
• Checksum Every Streamed File. The Big Ones Arrived Fine. The Two Small Ones At The End Didn’t.
• A Service’s restart May Not Reload Its Config. launchctl kickstart -k Restarts The Process, Not The Job Definition. Environment Changes Need bootout And bootstrap.
• 10 Questions Prove It Isn’t Broken, Nothing More. The Real Comparison Is Still The 365-Question Out-Of-Distribution Set From The Last Post, And I Haven’t Re-Run It On The Merged 4-Bit Model.

Which One To Use

Reference:

• Part One: I Fine-Tuned A 30B Model To Call Tools Better, And It Learned To Call Them Everywhere
• PEFT — Merging LoRA Weights
• MLX
• mlx-vlm