An AI model reads a referral letter. In what form should it hand its answer to the code that acts on it?
As typed fields, such as a choice from a fixed list, a probability for each yes-or-no question, and a score on levels written out in words. Code can check each field and act on it without guessing what a sentence meant.
Name the decisions the code must make.
Becausecode can act only on answers it was built to expect. A clinic sorting referrals might ask three things. How soon should the patient be seen? Did images come with the letter? Is the letter complete enough to book?
Give each decision a type.
Becausea type fixes which answers are possible. Urgency is one of three words, urgent, soon or routine. Images get a probability from 0 to 1, and completeness a score from 1 to 4.
Write each level of the score out in words.
Becausea bare 3 means different things to different readers, and a model is one more reader. Here 1 is “no reason given.” Each level adds one thing, a reason at 2, a history at 3 and test results at 4. Anyone can hold a score up against its words.
Ask for a probability wherever the question is yes or no.
Becausethe code can set its own threshold, and move it later, without asking the model again. For example, a letter at 0.55 goes to a person, and one at 0.98 goes straight through.
Check every field before acting on it.
Becausea value outside its type is caught and sent back, and a valid one leads to exactly one action. Some model services can now hold a reply to a declared shape as the model writes it. That fixes the shape. It cannot make the answer true.
Another way
Let the model write a paragraph, and have code search it for key words.
Becauseit is quick to build. But it breaks on the first summary that says “not urgent,” which contains urgent, and again whenever the model’s wording drifts. A middle way keeps one free-text field, the reason, for people to read.
Check
Test whether the probabilities mean what they say. Take 200 past letters the model scored near 0.8 for images. About 160 of them should have had images. If 195 did, or 120, set the threshold from that record. Models that state their own confidence tend to be overconfident.
KeepFields for code. Sentences for people.