Evaluating a fine-tuned model rigorously
"It feels better" is not an evaluation, it's a hunch, and it's especially unreliable because people tend to be biased toward liking the output of whatever they just spent time and effort building. A rigorous evaluation starts by holding out a test set, a portion of examples set aside before training and never used to train the model, so performance on them reflects genuine generalization rather than memorization.
From there, the fair comparison isn't fine-tuned model versus no effort at all, it's fine-tuned model versus the base model given a genuinely well-written prompt for the same task. If a strong prompt on the base model gets you 90% of the way there, the fine-tuning may not have been worth its cost. Scoring should be as objective as possible: a fixed rubric, a held-out set of test cases with known-good answers, or another model acting as a consistent judge, rather than a few spot checks by whoever built the system.