A Nobel laureate is a terrible night watchman
The alarm is boring on purpose. A connection drops. A voltage wanders out of range. A component looks derated. Somewhere on a site controller with no graphics chip and an intermittent link to the outside world, software has to decide whether that event is real, cite the reading that proves it, or say it does not know.
That is a miserable job for a frontier model. It is a plausible job for a small one that has been taught the house rules.
Exowatt sells dispatchable solar into the AI power buildout. The software evidence here is a single AWS write-up by Exowatt engineers Tim Child and Felipe Arce, with AWS architects Purna Sanyal and Jingchao Yang. Every score below is their measurement, not an independent benchmark.
The write-up describes the P3 as optical lenses that capture sunlight, a long-duration thermal battery that stores it as heat, and a heat engine that makes electricity on demand. The pitch is power behind the meter, so a data center is not stuck in a multi-year interconnection queue. The company is scaling from test benches toward manufacturing and a target of thousands of field units. Thousands is a target, not a published deployment count.
Why the big model was the wrong hire
The energy management system throws off a continuous stream of alarms: hardware out of spec, dropped connections, voltages outside range. Calling a frontier model for each one was the wrong tool, for three reasons the write-up is plain about. The task is narrow and proprietary. Many sites have no GPU and only intermittent connectivity. And paying a general-purpose model to remake the same decision, site after site, does not pencil out.
A strategy consultant is wonderful in the boardroom and ruinous if you station one at every loading dock. The model that writes a decent memo is often the wrong employee for a decision you will make thousands of times, in a closet, offline.
Their earlier fine-tuning path made the alternative painful too. Pick an open-weight model, build a container, write a custom training recipe, request a GPU instance, and hope the capacity showed up. A missing instance type could stall the run. Waiting on hardware turned customization into a special occasion.
A line you can rerun
What they built on Amazon SageMaker AI is less a clever model than a repeatable line. Operational data goes in. A checked training set, a task-specific model, and an evaluated deployment artifact come out.
The more durable result of the engagement was the pipeline itself: a repeatable path from raw operational data to a certified training set, a fine-tuned model, and an evaluated deployment artifact.
That line is the AWS write-up’s, and it is the one worth keeping. A model is a spare part. A path you can rerun is the asset. The authors say Exowatt can now run that path in days rather than months. Hold that claim lightly until the clocks are separated below.
Serverless fine-tuning pulls an open model and a ready Low-Rank Adaptation recipe, allocates capacity only while the job runs, and bills for that runtime. No cluster to keep warm just to attempt an experiment. The model they keep is registered. The cloud model is then hosted so it can be called privately and billed only while active. A scheduled ping is meant to keep that hosted model from going cold. That ping is a footnote in the architecture and a live wire in the bill.
The first two products of the line are a fine-tuned Qwen3-1.7B for offline alarm triage and a fine-tuned Qwen3-14B for cloud energy-market reads. The write-up is explicit that these are first outputs. The plan is more small models as new devices and market tasks appear, each with its own data and its own exam, rather than one model stretched across every job.
Read the scoreboard that follows with two brakes on. It is an internal exam, built and graded by the same team, not a public benchmark. And nothing in the results shows these models running a live site: the edge classifier was exercised in shadow mode on a cloud-hosted replica, with no reported alarm volume, uptime, energy saved, or operator hours cut.
A small classifier that can shut up
Alarm triage had a size constraint before it had an accuracy target. The model had to be frozen, quantized, and run on the site’s existing CPU, with no GPU and no internet. It returns one label from a fixed set (offline system, derated component, sensor mismatch, and similar) with a citation, or it abstains.
They did not get there by fine-tuning the obvious mid-size model. A prompted Qwen3-8B with guardrails plateaued at 0.735 macro-F1. Macro-F1 averages F1 across classes, so a model cannot ace the common alarms and quietly fail the rare ones without the score moving. The write-up does not publish the class list or the balance. Thirteen fine-tuned Qwen3-8B candidates, across 12 dataset versions, never beat that prompted 8B. The fine-tuned 1.7B did, at roughly a fifth of the parameters, and it still fit the box.
Size, data recipe, and optimization are tangled in that comparison. It is not evidence that smaller is generally better.
The training set was small and fussy: 2, 522 examples, from 684 verified decisions on generated site scenarios plus 1, 838 paraphrases. Of 2, 218 paraphrase candidates, those 1, 838 passed a deterministic check that the decision, the labels, the evidence citations, and every number were unchanged. One fault family was left out of training entirely. Three hundred twenty cases were reserved for the test. Training took about 32 minutes and roughly $20 to $25.
On that 320-case test, the write-up puts the fine-tuned 1.7B at 0.911 macro-F1. Claude Haiku 4.5 scored 0.519 at about $4.70 per 1, 000 decisions. Claude Opus 4.7 scored 0.680 at about $22. Claude Sonnet 4.6 scored 0.698 at about $15. None of those can run offline. They could answer. On this test they answered worse, online only, and at a poor unit cost.
That 0.911 does not cover the fault family held out of training. No score for that family is published. The base Qwen3-1.7B failed a validity gate and received no macro-F1 at all. A missing score here means the model did not clear the format bar.
The write-up’s $0 inference line means no metered API or GPU charge. The CPU, the pod, and the people who watch it are not free. Do not fold that zero into a 49-times headline. That multiple belongs to a different test.
High marks on a rule-built market exam
The cloud job is a different animal. It advises a timing decision. It does not chat with a customer, and it does not run the plant.
Dispatchable storage is a question of when. Hold heat for the load already promised, or release it when the market pays. For grid-connected sites, the model reads a six-hour packet (about 90 rows of real-time prices, day-ahead forecasts, and grid notices) and returns one of five market events, such as a price spike or a scarcity adder, in about four seconds. Every claim has to point at a specific row.
They chose Qwen3-14B because Bedrock Custom Model Import supports it, a SageMaker recipe for Low-Rank Adaptation already existed, and at about 28 GiB the write-up calls it small enough to serve cost-effectively. The raw material was public ERCOT data: five years, about 163 million rows across 12 tables, rule-labeled into 25, 000 examples (22, 000 for training, 1, 000 for validation, 2, 000 held out). Claude Sonnet 4.6, which later sat in the baseline table, rewrote the prose while labels and numbers stayed fixed, expanding the training set to 86, 281 rows.
One supervised fine-tuning epoch took about 8.5 hours of managed serverless training. The write-up also reports about 155 hours of managed training in the six days from the initial plan to a champion model serving in Cloud EMS. Those are separate clocks. The post does not itemize the 155 hours, so do not multiply the 8.5-hour run until it equals them. The write-up lists active billing at $6.86 per hour only while the custom model is active, with no idle charge. That is the case study’s stated price, not a quote you should treat as this afternoon’s rate card.
The base 14B already spoke the dialect. It produced valid JSON 83.4 percent of the time, with 0.54 event accuracy, 0.415 macro-F1, and a token cost of $0.64 per 1, 000 answers. The authors’ line is that it produced valid JSON but “did not yet know the market.” Fine-tuning is what they credit for the jump. Three rewrites per answer lifted macro-F1 from 0.950 to 0.975.
On the 2, 000-case test, the fine-tuned model scored 0.975 macro-F1, 0.983 event accuracy, and valid JSON on every answer, at $0.60 per 1, 000 answers. That price is Bedrock list prices times the tokens the authors measured. Sonnet 4.6 scored 0.538 macro-F1 and 0.584 event accuracy, with valid JSON on 99.1 percent of answers, at $17.79 per 1, 000. Opus 4.7 scored 0.625 and 0.718, with valid JSON on 99.95 percent of answers, at $29.26.
High agreement with a rule-built exam is not the same as anticipating a market a dispatcher has not already encoded. If rules minted the labels, the model may have learned the labeling script.
The write-up puts the gain over Opus at about 0.35 macro-F1, with a 95 percent confidence interval of 0.324 to 0.371. That interval is only for the macro-F1 difference on this split. It says nothing about cost, and nothing about the edge model. A later energy-market test set held at 0.968 macro-F1, still on the authors’ grading.
$0.60 against $29.26 is about 49 times on that token math. It is not a total cost of ownership. Paraphrase generation, evaluation jobs, registry storage, the keep-warm ping, and the $6.86 active hour sit outside the figure. The post never says how long the model must stay active to hold a four-second response. If it has to sit hot, the pretty per-answer price is the dessert, not the check.
Same steps, a small edge
The alarm-triage comparison is easy to misread, which is why it belongs above a data team’s desk with the arithmetic still attached.
Macro-F1 on the 684 verified decisions alone was 0.584. The paraphrase-expanded set of 2, 522 rows reached 0.911. A control trained on those same 684 rows for 11 epochs (the same number of steps) reached 0.879. Same steps, new wording beat repeated rows by 0.032. Longer training on the original rows moved the score much further.
The authors conclude that curated variety mattered more than extra epochs or raw volume. That is their conclusion, and it is an internal result, not a law of nature. Do not let “mattered more” swallow the arithmetic. The variety edge over the matched-step control is small. The large move was training longer on the rows they already had.
Do not staple that result to the 1.7B-versus-8B runs. Those are different experiments. Size, data recipe, and optimization are confounded, and nothing here shows that smaller is generally better. It is also the opposite of the instinct most budgets still fund, which is to buy a bigger general model and hope the prompt absorbs the domain.
What “autonomous” does not mean
Both models return strict JSON. A claim survives only if its citation literally matches an input row’s source, field, and value. Unsupported claims are dropped before an operator sees them. On the edge, insufficient evidence becomes an abstention, not a guess. A leaderboard score is not the same thing as a system that knows when to shut up.
The write-up says that fence is what lets the team trust small models in autonomous, physical settings, and that the edge classifier can take or recommend the next action. The only autonomy reported in the results is shadow mode on a cloud-hosted replica of a site. There is no described live actuation of hardware, no approval workflow, and no safety case for closing a breaker from a model output. Citing a sensor and refusing to guess are controls. They do not amount to a license to act.
The nine-day claim, from experimentation to production, and the six-day claim, from plan to a serving market model, are both in the post. They are not mapped to the same start and finish. The 32-minute training run, the 8.5-hour run, and the 155 managed training hours are separate clocks. “Days rather than months” is the authors’ summary of a path that no longer waits on a GPU reservation. A reader cannot audit it as one stopwatch.
The evaluation gaps belong in the same drawer. The post does not define the classes, the class balance, how abstention enters macro-F1, who labeled the alarm decisions, or whether the Claude models saw the same schema and examples. One held-out fault family has no published score. No other industry is shown. The closing recommendation even points to SageMaker as an example elastic endpoint, while the cloud path they actually stood up is Bedrock Custom Model Import. Those are different doors.
Questions worth asking first
Did a 1.7B model really beat Claude Opus on alarms?
On the authors’ 320-case test, as reported in the AWS write-up, yes: 0.911 macro-F1 against Opus 4.7 at 0.680, and it ran offline. That is one narrow, author-built task, and the published score does not cover a fault family held out of training.
Is the 49-times-cheaper claim solid?
Only as cloud token math on the energy-market test, $0.60 versus $29.26 per 1, 000 answers at Bedrock list prices the authors measured. It excludes the case study’s $6.86 active hour, keep-warm calls, and the rest of the pipeline, and it does not describe the edge model.
Are these models running the power plant?
Not on anything reported. The edge classifier was exercised in shadow mode on a cloud replica. Citing evidence and abstaining are controls, not a license to actuate live equipment.
What actually moved the score?
On the alarm set, repeating the original 684 rows for the same number of steps moved macro-F1 from 0.584 to 0.879. Certified paraphrases reached 0.911, a 0.032 edge the authors read as proof that curated variety mattered more than extra epochs or volume. That reading overweights a small gap, and it should not be fused with the 1.7B-versus-8B runs, where size, data recipe, and optimization are confounded.
Should every company stand up a model factory?
The AWS post recommends the pattern for tightly scoped work, including outside energy. No other industry deployment is shown. The fit is repeating decisions, labels you can certify, and a place a giant API cannot reliably reach.
Where is the money, really?
Training looked cheap on the figures given: roughly $20 to $25 and about 32 minutes for the edge model, and about 8.5 hours for the highlighted 14B run, without a GPU reservation in the way. Serving is the open bill. A $0.60 answer is not the same thing as $6.86 for every hour the cloud model has to stay active.
A different org chart
Strip the vendor PDF down and the transferable part is ordinary. A checked dataset. A training run that did not wait on a GPU reservation. A deployment matched to the room the decision lives in.
Frontier models still earn their keep on open problems: messy synthesis, unfamiliar documents, the question nobody has labeled yet. Sonnet 4.6 was useful here as a paraphraser. Opus was a serious baseline, and on the market test it was far more accurate than an untuned 14B. The loss came after the small model was given that market’s labeled history and forced to show its work.
For a leadership team pricing AI agents, the practical split is narrower than the brochure. Use the large general model where the work is rare, ambiguous, or worth a human in the loop. Fine-tune a small classifier where the action is narrow, the evidence is structured, and the call will be made whether or not the network feels like cooperating. Then put a fence in front of it that would rather drop a sentence than invent a sensor reading. In that setup, an agent is a narrow advisor with a fence, not an operator.
You end up with a different org chart: many specialists, each examined on its own job, instead of one brilliant generalist billed by the token every time a gauge twitches.