Fictional faxes, real results

Counts on fictional documents from one generator, first run each, reusing none of the system's own saved work. "Serious errors" are a fact given to the wrong patient, a wrong value, date or present-or-absent marking, a value that is not printed where cited, an unreadable value reported as readable, or a required flag not raised. All 4 of Pepper Standard's serious errors on the held-out documents are one pattern: a stopped medication recorded as absent instead of as present and stopped. A manual check of those items against the pages found the drug, dose and action read correctly in each; the marking was one the output rules did not define when these runs were made, and the engine that made them was not consistent about it. The same check found that on the hardest scan Pepper read one drug name with one wrong letter (counted here as a missed fact) and Premium showed it as supported. Model usage is what the provider billed at list rates. The runs were made one after another, so much of each prompt was billed at the provider's lower rate for input it had just seen. With every such token at the ordinary input rate, the held-out figures are $0.00095, $0.00847 and $0.01833 per page. It is not an invoice and is not the customer price. These figures describe these documents; they are not a claim about real faxes.

Which engine. These runs were recorded with engine 0.2.0 and have not been repeated. The engine in service is 0.2.3. It was changed after these runs, partly because of what these documents showed: a medication reported as stopped is now recorded as present with the action stopped, a medication name the second reader spells differently is flagged, a contradiction is carried on the findings it concerns, and the second read no longer counts wording as a disagreement. For the engine in service these documents are regression cases, not untouched test documents, and the figures above do not measure it.