Confidence gates for exposure data.

A wrong OED code does not look wrong. It sits in the file like every other code, and every model downstream reads it the same way. So when a decision model codes exposure rows, the useful question is not “how accurate is it” but “how many wrong codes will it write that nobody looks at”. A confidence gate is the answer to that: the model’s answer is written only when its confidence clears the line, and everything else goes to a person with the suggestion attached.

This is how we set the gates behind Craton’s SOV coder, from a benchmark of 1,430 labeled rows coded by TypeSafe’s Jev. The benchmark itself has its own write-up.

Count the wrong codes you let through

Accuracy on its own hides the cost. A gate trades two things: how many rows get coded without a person, and how many wrong codes slip into the file among them. We count the second per 1,000 rows, because that is what the file ends up carrying.

All three slices pooled. Coded: the share of rows written without a person. Wrong per 1,000: wrong codes among them, per 1,000 rows. Data: gate_tradeoff.csv.

With no real gate (0.5), the model coded 90.7% of occupancies and let through 80.4 wrong ones per 1,000 rows. At 0.95 it coded 66.5% and let through 7.7. Construction starts much cleaner: 12.6 wrong per 1,000 at 0.5, and 1.4 at 0.85.

Why one gate for both fields is wrong

A confidence is only useful if it means something, and the same number meant different things for the two fields. Here is how often the model was right, by the confidence it gave:

Share of answers that were right, by stated confidence

Occupancy

• 0.00 to 0.5080 rows42.5%
• 0.50 to 0.70127 rows58.3%
• 0.70 to 0.8067 rows67.2%
• 0.80 to 0.9076 rows81.6%
• 0.90 to 0.9576 rows80.3%
• 0.95 to 0.99152 rows96.7%
• 0.99 to 1.00799 rows99.3%

Construction

• 0.00 to 0.5045 rows77.8%
• 0.50 to 0.7095 rows89.5%
• 0.70 to 0.8066 rows92.4%
• 0.80 to 0.90114 rows99.1%
• 0.90 to 0.95106 rows98.1%
• 0.95 to 0.99229 rows100.0%
• 0.99 to 1.00668 rows100.0%

Construction confidence runs low: even its hesitant answers are usually right. Occupancy confidence runs high: between 0.8 and 0.95 it was right only about four times in five. Most of those misses were codes we count as acceptable, such as a government office coded as an office; of the rest, half were high-school “academy” buildings read as a college. A single gate at 0.9 would send too many good construction codes to a person and still let too many wrong occupancy codes through.

Where we drew the lines

• Occupancy: 0.95. 66.5% of rows coded, 98.8% of them right, or 99.9% counting codes we marked acceptable.
• Construction: 0.85. 75.0% of rows coded, 99.8% of them right.

We checked both on schedules we never tuned the wording on. On those held-out sources, occupancy at 0.95 coded 59.0% of rows with 98.7% right, and construction at 0.85 coded 75.0% with 99.5% right.

What happens below the line

Nothing below a gate is written as a code. In Craton’s answer the field’s code is empty, the model’s suggestion sits beside it with its confidence and up to two alternatives, and the row is marked for review. Three details matter more than they look:

• Gate each field on its own. A row whose occupancy is certain and whose construction is not should keep its occupancy.
• Never code “unknown”. OED has an Unknown code for both fields. When the model says the row does not say, that is true and useful, and it still goes to a person: a placeholder written as a finding is the worst kind of wrong code, because nobody will think to check it.
• Keep the suggestion. In the benchmark the model’s top answer was still right for most of the rows it sent to review, so the reviewer starts from it and its runner-up, not from a blank cell, and can accept either with one click in the SOV coder.

Rules of thumb

• Pick the gate from your own labeled rows, per field, and count wrong codes per 1,000 rows.
• Hold some sources out while you tune, and check the gate on them.
• Version the gate with the questions. In Craton the gates live in the same question set file as the wording, and every answer carries that file’s digest.
• Measure again when anything moves: the wording, the model behind the alias, or the kind of schedule you receive.

The gate is reported in every answer from POST /exposure-codes, so the rows you get back always say which line they were held to. Code a schedule of values with one API call shows one answer end to end.