Harness or Fine-Tune? An Engineering Guide to What Both Need
Matan Weksler12 min
Every team building AI systems has some version of this argument running. One side wants to take the best frontier model and keep wrapping it in better scaffolding, meaning retrieval, tool calling, memory, sub-agents, and verifiers. The other side wants to stop paying frontier prices for every call and start training smaller models on the task. The argument usually sounds like it is about architecture. It is really about something both sides are skipping.
Both camps are right, on different workflows at different times. The uncomfortable part is that neither approach can work without an input that most teams treat as an afterthought: an executable definition of a good output. The harness needs it to know when a run has succeeded. The fine-tuning loop needs it as the training signal itself. Get the definition right and you can move between the two architectures freely. Without it, both will optimize confidently toward the wrong thing.
For anyone deciding where to invest, the argument runs like this. A harness improves results by changing what surrounds a frozen model. Fine-tuning improves results by changing the model. Each rung of the fine-tuning ladder consumes a different form of "what good looks like", whether labels, preferences, a verifier, a teacher, or a filter. The harness consumes the same thing as a verifier and an eval set. That shared input, the objective made executable, is the contract between the two architectures. Build it first, and the harness-versus-fine-tune question becomes a cost and latency decision instead of a religious one.
Part 1 of our series on the AI-native construction firm, written for the people who build the systems. The later posts take the same argument to the executive side of the business.
What a harness is made of
A harness is the runtime around a frozen model: the components that turn a general reasoner into a system that reliably does one job, and the only place your engineering effort goes until you have the data to train anything.
The components are more standardized than the discourse suggests. Nearly every production system has six:
- Context assembly. Retrieval over your documents, parsers for the formats you actually receive (scanned PDFs, sheet sets, spec books), and the templates that decide what the model sees per call.
- Tool layer. Typed functions the model can invoke, such as searching the spec index, pulling a sheet revision, querying a cost database, or filing a draft RFI. The schema of these tools is a large share of your effective "prompt."
- Memory. State that persists across calls and sessions, including project facts, prior decisions, and the user's corrections.
- Decomposition. A planner or orchestrator that splits a large task into sub-tasks, and often sub-agents that own each one with their own context.
- Verification. Checks that run on the output before it reaches a person, such as schema validation, citation grounding, a cross-check against a second pass, or a small model acting as a judge.
- Feedback capture. Instrumentation that records what the human changed, rejected, or escalated. This is the component most teams skip, and it is the one that makes everything downstream possible.
The strengths of this architecture are practical. The model is a swappable dependency, so a better frontier release upgrades your whole system for free. Iteration is fast, since fixing a workflow means editing a prompt or a tool, not running a training job. And you need no labeled data to start. This is the engineering discipline we have described before as context engineering, and it is where the value is today for almost every construction workflow.
The costs are also real. You pay frontier inference on every call, which matters when a submittal review touches forty documents. Latency is bounded by the number of sequential model calls in the plan. A harness on a general model has a quality ceiling on narrow, high-volume tasks that a specialized model can exceed. And a harness does not compound on its own: if you never capture corrections, month six is no smarter than month one. That last point is the difference between a harness built for your workflow and a generic tool, which we lay out in Pelles vs off-the-shelf AI.
What each rung of the fine-tuning ladder consumes
Fine-tuning changes the model rather than its surroundings, and the ladder of methods is best understood by what each one needs as input, because that input is always a form of the objective.
| Method | What changes | What it consumes | Where the definition of "good" lives |
|---|---|---|---|
| Supervised fine-tuning (SFT) | Model weights | Labeled input/output pairs | The labels |
| Preference optimization (RLHF, DPO) | Model weights | Pairwise judgments: A is better than B | The preference data or reward model |
| RL with verifiable rewards | Model weights | A programmatic pass/fail checker | The verifier |
| Distillation | A smaller model's weights | A stronger teacher's outputs | The teacher, plus your prompt to it |
| Synthetic data generation | The training set | A generator and a filter | The filter |
| Self-improvement loops | All of the above, iteratively | The system's own scored runs | The scoring function |
The pattern in the right-hand column is the point. Supervised fine-tuning needs someone to have decided which outputs are correct. Preference optimization needs someone to have decided which of two outputs is better; the InstructGPT result, where a 1.3-billion-parameter model trained on human preferences beat a 175-billion-parameter base model on human evaluation, was a demonstration that this signal can be worth more than a 100x size difference. Reinforcement learning with verifiable rewards, the approach behind the current generation of reasoning models, needs a checker that can score an output without a human in the loop, which is why it works so well on math and code and so poorly on tasks nobody has written a checker for.
Distillation and synthetic data move the definition one step back: the teacher model and the filter encode the objective instead of a human. Self-improvement loops, where a system generates its own training data, scores it, and trains on the survivors, move it back once more, to whatever scoring function sits at the bottom.
There is no rung where the definition of good disappears. There are only rungs where it becomes easier to forget that you wrote it.
The decision: when to stay on the harness and when to train
Fine-tune when a workflow is high volume, stable in definition, reliably scorable, and blocked by a harness ceiling you can measure; if any one of those is false, keep working on the harness.
| Signal | Stay on the harness | Move toward fine-tuning |
|---|---|---|
| Volume | Hundreds of runs a month | Tens of thousands |
| Task definition | Still changing as you learn | Stable for a quarter or more |
| Scoring | Human review only | Automated eval with agreed thresholds |
| Ceiling | Quality issues fixable in prompt or tools | Cost, latency, or accuracy plateaued |
| Data | Few captured corrections | Months of structured corrections |
| Drift risk | Inputs vary widely | Inputs are homogeneous |
Two failure modes dominate. Fine-tuning too early bakes a moving definition into weights, so every change in what the business wants becomes a retraining job. Fine-tuning without a scorer produces a model you cannot compare to the one it replaced, so "better" becomes a matter of anecdote. Both are far more common than fine-tuning too late.
In practice the most capable systems run both. A general model with a rich harness handles the long tail of tasks that never reach training volume, which in construction is most of them, such as a one-off contract clause question, an unusual coordination conflict, or a new client's submittal format. Specialized models handle the repeated high-volume steps inside those workflows, like classifying a submittal item against the spec, extracting quantities from a schedule, or scoring an RFI draft for completeness. Workflows migrate from the first category to the second as they mature, and the thing that lets them migrate cleanly is the subject of the next section.
The eval is the objective, and the contract between architectures
An objective that cannot be scored cannot be optimized, so the eval suite and verifier are not a testing artifact: they are the organization's goal in executable form, and the only thing that stays fixed when a workflow moves from harness to model.
Consider a submittal-review workflow. "Review the submittal" is not an objective a system can pursue. The engineering team has to decide what done means, and the options are materially different systems:
- Every item matched to a spec section, with a citation.
- Every deviation from the spec flagged for a human.
- Every deviation flagged and ranked by cost and schedule exposure, with the top three surfaced first.
Each of those is a different verifier in the harness and a different label set for a model. Choose the third, and the eval contains cases where a subtle deviation carries large exposure and a glaring one carries none, scored on whether the ranking gets it right. That eval then does three jobs. It gates every harness change. It becomes the acceptance test for any distilled or fine-tuned model that proposes to replace part of the pipeline. And it is the raw material for the training signal, because the scored corrections your reviewers produce are exactly the preference and label data the fine-tuning ladder consumes. This is the mechanism by which software post-trains on your judgment, stated in engineering terms.
Written down, the third option looks something like this:
objective: rank spec deviations by cost and schedule exposure
unit: one submittal package
inputs:
- submittal.pdf
- spec_sections/
- schedule.xer
verifiers:
- type: programmatic
check: every flagged item cites a spec section that exists
- type: programmatic
check: top-3 ranked items match the reviewer's top-3 (Kendall tau >= 0.6)
- type: rubric
check: no material deviation omitted (reviewer-graded, 1-5)
held_out: 20% of packages, never used for prompt or model tuning
acceptance: mean score >= baseline + 0.05 on held_out, at <= 1.0x baseline cost
feedback: reviewer edits logged as (item, verdict, correction) tuples
Every field in that file is a decision the business has to make, and none of them are about model choice. The same file gates the harness today and grades the fine-tuned model tomorrow.
Reward hacking is the failure mode of this whole approach, and it applies to both architectures equally. A verifier that only checks for the presence of a spec citation gets outputs that cite the nearest section rather than the right one. A model rewarded on RFI closure time learns shallow answers. The defenses are standard. Score outcomes rather than activity, hold out an eval set the optimizer never trains against, and route a sample of production outputs to human review indefinitely. An optimizer with a proxy and no oversight will find the gap between the proxy and the goal. The optimizer is doing its job. The gap is in the proxy.
The progression, and where the engineer's job moves
Systems move through four stages of automation, and at each one the execution becomes more automated while the engineer's most consequential work moves up a level, toward defining and defending the scoring function.
- Humans use a model for tasks. An estimator asks for a spec summary. The person chooses the question and judges the answer.
- A harness coordinates tasks toward outcomes. The system reads the bid set, flags gaps, drafts RFIs, assembles the review. The engineer defines the outcome, the tools, and the verifier.
- The system optimizes its own execution. It tracks which prompts, tool orderings, and retrieval strategies produce fewer corrections and adjusts. The engineer defines what counts as a correction.
- The system improves the model. It turns scored runs into training data, evaluates candidates, fine-tunes or distills, and allocates compute to what works. The engineer defines the goal and the held-out eval that keeps the loop honest.
Stage two is where construction agents sit today. The third stage begins the first time a system reads its own correction log, and the fourth is what frontier labs run internally, with the tooling to run it inside a company arriving quickly. At every stage a flaw in the scoring function is amplified rather than diluted. A bad verifier at stage two produces wrong outputs, at stage three it produces a workflow tuned to generate them efficiently, and at stage four it produces a model trained to be very good at the wrong thing.
A reference architecture for a contractor's engineering team
The practical build order is the reverse of the architecture debate. Build the eval first, then the harness, and let captured corrections earn the fine-tune.
- Pick one workflow and write its verifier before its prompt. Decide what done means in the three-option sense above. If you cannot write a checker or a rubric, the workflow is not ready for either architecture.
- Ship the harness on a frozen frontier model. Retrieval over the real documents, typed tools, a verifier, and full instrumentation of inputs, outputs, and human edits. The ready-to-run applications in Pelles Core are harnesses in exactly this sense, wrapping general models in construction-specific context, tools, and review steps, and the Pelles Workshop builds custom ones tuned to a firm's own workflow.
- Turn corrections into an eval set. Every reviewer edit is a labeled example or a preference pair. Curate a held-out set and run it on every harness change. This is the step that makes the system compound.
- Watch the four signals. Volume, definition stability, automated scoring, and a measured ceiling. When all four are green for one step in the pipeline, distill or fine-tune that step and let the same eval decide whether it clears the bar at lower cost.
- Keep the eval as the contract. The harness and the fine-tuned model are interchangeable implementations of the same objective. Never let a training run redefine the goal; only a deliberate change to the eval should do that.
The input neither architecture can produce
Every method for pursuing an objective with AI, from a prompt template to a self-improving training loop, is machinery for getting better at something. None of it can decide what that something should be. The harness asks what its verifier should check. The fine-tuning loop asks what its labels, preferences, or rewards should encode. The self-improving system asks what its scoring function should measure. The answer has to come from the people who own the outcome, written down precisely enough to execute.
That is the layer Pelles is building for construction. It holds the definition of the objective, the measurement that keeps it honest, and the feedback signals that let both harnesses and models improve against it without drifting from it. If your team is working through this decision on a real workflow, talk to our engineering team. Bring the workflow and the eval you have so far, even if it is a spreadsheet of reviewer corrections. That is usually the most valuable artifact in the room.
Frequently asked questions
What is an AI harness, technically?
A harness is the runtime system around a frozen general-purpose model. It is made up of a context-assembly layer (retrieval, document parsing, prompt templates), a tool layer the model can call, memory that persists across calls, a planner or sub-agent structure that decomposes work, verifiers that check outputs before they ship, and instrumentation that captures corrections. The model supplies reasoning; the harness supplies the task definition, the actions, and the accountability. Most production gains over the last two years came from harness engineering rather than from model changes.
When should a team fine-tune instead of improving the harness?
Fine-tune when four conditions hold at once. The task is high volume, its definition is stable, you have a reliable way to score outputs, and the harness has hit a cost, latency, or quality ceiling you can measure. If any one is missing, keep iterating on the harness. Fine-tuning a task whose definition is still moving bakes today's mistakes into weights, and fine-tuning without a scorer gives you no way to know whether the new model is better or just different.
What does each fine-tuning method need as input?
Each rung consumes a different form of 'what good looks like.' Supervised fine-tuning needs labeled examples of correct output. Preference optimization (RLHF, DPO) needs pairwise judgments of which output is better. Reinforcement learning with verifiable rewards needs a programmatic checker that scores an output as pass or fail. Distillation needs a stronger teacher whose outputs define the target. Synthetic data generation needs a generator plus a filter that decides which samples to keep. In every case the input is a definition of the objective in a machine-consumable form.
Why is the eval suite the same thing as the objective?
Because an objective that cannot be scored cannot be optimized. In a harness, the verifier and the eval set are what decide whether a run succeeded. In a fine-tuning loop, the labels, preferences, or reward function are literally the training signal. Both are the organization's goal translated into something executable. That makes the eval the contract between the two architectures: you can move a workflow from harness to fine-tuned model without changing what the system is optimizing for, as long as the eval stays fixed.
How does reward hacking show up in production AI systems?
Whenever the scorer is a proxy for the real goal, an optimizer will eventually satisfy the proxy without achieving the goal. A fine-tuned model rewarded for RFI closure speed learns to produce shallow answers. A harness verifier that only checks for a cited spec section gets outputs that cite the nearest section rather than the right one. The defense is the same in both architectures. Score outcomes rather than activity, hold out an eval the optimizer never sees, and route a sample of production outputs to human review continuously.
How should a construction technology team start?
Start with a harness on a frozen frontier model and instrument it, logging every input, output, and human correction as structured data. Turn the corrections into an eval set and a verifier. Run that eval on every harness change. Once one workflow has stable definitions, high volume, and months of scored corrections, use that same eval to decide whether a distilled or fine-tuned model clears the bar at lower cost. The eval you built for the harness becomes the training and acceptance signal for the model.
