Machine learning & NLP
Replacing a 32B GPU model with a fine-tuned small one on CPU
Detecting brands, people and products in TikTok video transcripts ran as a Node service calling Qwen2.5-32B on GPU, in 20 replicas on EKS, the most expensive of our vLLM clusters. The replacement is a GLiNER service on CPU in EKS that consumes from Redpanda.
What I did. I fine-tuned GLiNER with LoRA on about 5,200 examples taken from Qwen's production output plus negatives from an audit, running the open weights locally because the hosted API had no data-processing agreement.
How I measured it. Because the training labels came from Qwen, I did not measure against Qwen's output. I used three references instead. The first is a held-out set of 1,017 posts and the second is 1,024 fresh posts from later days. Two Claude judges labelled both from scratch without seeing either system's output, and an adjudicating model settled their disagreements (the two judges agreed at about 0.9 F1). No person reviewed those two references. I set the bar beforehand for these two: no more than 0.03 F1 below Qwen on people and 0.05 on brands. The third reference is a validity score from 0 to 1 for every entity either system proposed, given by four independent decision models (models that answer a yes/no question with a score: TypeSafe's Jev and models from Perplexity, Liquid and Cloudflare). I averaged their scores and calibrated the average on 153 labels from my team. Before the switch it ran in shadow mode on live traffic, validated with a 568-case differential test.
Result. I shipped it knowing it missed my bar: against the two judge-labelled references it trails Qwen on brands by 0.10 and 0.17 F1 and, on the fresh posts, on people by 0.08 (95% intervals exclude zero). Brands are the bigger gap, and those are the cases most blurred with products; I accepted that trade-off because of the cost. It replaced Qwen in production, an order of magnitude cheaper (from thousands of euros a month to a few hundred) and faster than Qwen was on vLLM. The other measurements are less clear-cut. The validity score puts it within 0.02 F1 of Qwen overall (not statistically significant): better on people, worse on products, slightly behind on brands. A fourth check, using only human labels, tied them at 0.61 F1. The judge references count recall against everything the judges found; the validity score only against what the systems proposed, which flatters both, so the numbers can't be compared directly. I describe it as competitive, not equivalent.
