What a decision model gets right and wrong on 1,430 exposure rows.

A schedule of values says “2 sty wd frm apts” and a catastrophe model wants 1052 and 5050. Somebody does that translation by hand, row by row, or with a lookup table that breaks on the first typo. We wanted to know whether a decision model could do it, and exactly where it stops being trustworthy, before putting one behind a tool and an API route.

The model is Jev, TypeSafe’s “System One” decision model, reached through Vercel’s AI Gateway. It does not write text. You give it the row and a closed list of choices, each described in plain words, and it picks one and says how confident it is. We gave it forty occupancy classes and fifteen construction classes, each mapped to one OED 5.0.0 code, and asked about every row one at a time.

Three kinds of rows

• Public schedules of values. 654 real rows, verbatim, from ten schedules of values that US public entities have published: school districts in Texas and Louisiana, a Florida county, city and water district, the Pittsburgh housing authority, California’s court facilities, a Kentucky water district and the State of New Mexico’s risk pool. The labels are ours: construction through the ISO classes the schedules use, occupancy read from each building’s name. We scored the 575 rows we could label with confidence and set the rest aside, and where a row fairly supports more than one code we recorded the others as acceptable alternates.
• Rows rendered from the National Structure Inventory. 666 US buildings from the USACE inventory, written out the way schedules are written: abbreviations, tabs, missing cells, typos. The inventory gives a building’s material only as a one-letter class, so the words in these rows come from phrase lists we wrote. The labels come from the inventory’s own occupancy and material, which are partly modeled rather than observed, so this slice’s labels are noisy, and its construction is only as fine as five material classes.
• Hand-written hard cases. 189 rows written to be difficult, with exact labels: mixed use, conversions, “block”, “frame”, tilt-up, PEMB, manufactured homes, typos, codes pasted from other systems, and rows from Japan, Mexico, the Philippines, Germany, Australia and elsewhere, several in their own language.

For comparison we wrote a plain keyword baseline: a table of the words and abbreviations a person would think of first, applied to the same rows.

Exact matches, by slice

Public schedules of values · 575 rows

• OccupancyJev76.2%
• OccupancyKeyword rules28.3%
• ConstructionJev98.4%
• ConstructionKeyword rules37.7%

Hand-written hard cases · 189 rows

• OccupancyJev91.5%
• OccupancyKeyword rules59.8%
• ConstructionJev89.9%
• ConstructionKeyword rules64.0%

NSI rendered rows · 666 rows

• OccupancyJev94.7%
• OccupancyKeyword rules90.1%
• ConstructionJev92.2%
• ConstructionKeyword rules95.9%

• Jev
• Keyword rules

Where it earns its keep

On real schedules the gap is wide. Public schedules write construction as whatever their template calls it (“Masonry Joist”, “MNC”, “Fire Res”, a bare “ISO Class: 6”) and occupancy as a building’s name. The keyword rules got 37.7% of construction codes right; Jev got 98.4%. A lookup written for the construction columns of these particular schedules got 96.5%, which is the fair comparison if you only ever see the same few templates.

The hard cases show why the templates are the easy part:

• Typos (“wood fram singel family”, “reinfored concreet parking strucure”): Jev 100.0% on occupancy, keyword rules 16.7%.
• Rows in other languages and other markets (“木造2階建て 専用住宅”, “Casa de adobe, 1 nivel, Oaxaca”, “Mehrfamilienhaus, Stahlbeton”, “Bungalow type residence, CHB walls, GI sheet roof, Quezon City”): Jev 96.1% on occupancy and 90.2% on construction, across 51 rows. Keyword rules: 37.3% and 33.3%.
• Manufactured and modular homes (“HUD-code manufactured home, tied down”, “Portable classroom (modular trailer)”): Jev 100.0% on construction, keyword rules 70.0%.
• Conversions (“Victorian house converted to dental offices”): Jev 91.7% on occupancy, keyword rules 66.7%. It codes what the building is used for now.

Where it does not

• Codes from other systems. “HAZUS IND2 PC1”, “OED OccupancyCode 1052, ConstructionCode 5050”, “AIR occ 315 const 151”: Jev got 47.1% of occupancies and 41.2% of constructions across 17 rows. Mostly it said “unknown” with low confidence, which is the right thing to say about something it cannot read. If your schedule already carries another scheme’s code, translate it with a table, not a model.
• Metal buildings, by use. It always knew a PEMB was light metal. What it could not always tell is what the building was for: “PEMB shop bldg, road dept equipment maint” came back as a government building rather than a repair shop. Occupancy on PEMB rows: 75.0%.
• The word “block”. Block walls can be plain masonry, reinforced masonry or joisted masonry depending on what sits on them, and the row rarely says. Construction on the 10 “block” rows: 80.0%. Of the two it missed, one came back as plain masonry and one as unknown.
• Occupancy on real schedules. This is the one to watch. Public entities describe buildings by name and role: “Maintenance Storage”, “Master Meter Station”, a building on a university campus known by its department. Jev matched our occupancy label exactly on 76.2% of these rows, or 91.3% counting codes we marked acceptable. The largest groups of disagreement were government offices coded as offices, which we count as acceptable, and high-school “academy” buildings read as a college.

One result runs the other way. On the NSI rows, keyword rules matched construction on 95.9% of rows and Jev on 92.2%. Take that with a grain of salt: we wrote both the phrase lists those rows were built from and the keyword rules, so the rules already knew the vocabulary. Still, the lesson holds: when a schedule uses a small, known set of words, a table does as well as a model, and it is cheaper.

What its confidence is worth

Jev reports a confidence with every answer, and that number is what makes it usable: rows it is sure about can be coded, and the rest can go to a person. But the two fields behave differently.

• Construction confidence runs low. Answers given at 0.5 to 0.7 were right 89.5% of the time.
• Occupancy confidence runs high. Answers given at 0.8 to 0.9 were right 81.6% of the time, and at 0.9 to 0.95 80.3%. Above 0.99 it was right 99.3%.

So one threshold for both fields would be either too strict for construction or too loose for occupancy. Confidence gates for exposure data walks through how we set one per field.

A check on ourselves

We tuned the wording of the choices on some of the public schedules and kept others aside. On the 268 rows from schedules we never tuned on, occupancy matched exactly on 86.2% and construction on 99.6%.

Speed, cost, and a few other fields

The median reply took 0.2 seconds and the 95th percentile 0.35 seconds. Each row cost about 2,325 input tokens, most of them the question set itself, which at the gateway’s list price is $0.10 per 1,000 rows. Replies cost nothing.

We also tried questions beyond the two codes. Number of stories, when the row states it, was right 98.8% of the time. Asked whether a row describes more than one building, it separated the two kinds of row perfectly on our 70 hand-written examples, and raised a false alarm on 0.3% of 300 single-building rows. Those are candidates, not features yet.

What we would tell a colleague

• Use it for the words people type: abbreviations, typos, other languages, conversions, mixed use.
• Do not use it for codes. Map another scheme’s codes with a table.
• Never take the answer without the confidence, and set the bar per field.
• Test it on your own schedules before you trust it on them. Ours were US public entities; yours will be different.