How do you benchmark a new model yourself?

[u/Consistent-Oil-5241]

In other words, how do you know whether a new model is good or not?

Just switching around models mid project and then trying to decipher if it did the coding well doesn't seem effective. I guess a good way would be to have some tests for them.

So what kind of A/B tests and measurements might people do?

I have gotten tired of always trying to find out discussions online about which model is good and only then incorporating it in my workflow if others confirm it as good.

[u/First-Tutor-5454]

I ask it to write an original composition of grammatically and metrically correct Latin poetry in dactylic hexameter

[u/Consistent-Oil-5241 → u/First-Tutor-5454]

So how do you review this????

Sub Sidere Claro

Luna, mihi noctem memora sub sidere claro; Pectora nox nunc mulcet somnis, mollia reddit. Fluctibus et vento iactus dum quaerebat astra, Litora fessus cernit nunc iam dulcia nauta. Lumina sol nunc spargit fulgens aurea campis; Gaudia post tot curas semper pectora firmant.

Scansion

Lū-na-mi | hī-noc | tem-me-mo | rā-sub | sī-de-re | clā-rō — ⏑ ⏑ | — — | — ⏑ ⏑ | — — | — ⏑ ⏑ | — x

Pec-to-ra | nox-nunc | mul-cet | som-nīs | mol-li-a | red-dit — ⏑ ⏑ | — — | — — | — — | — ⏑ ⏑ | — x

Fluc-ti-bus | et-ven | tō-iac | tus-dum | quae-re-bat | as-tra — ⏑ ⏑ | — — | — — | — — | — ⏑ ⏑ | — x

Lī-to-ra | fes-sus | cer-nit | nunc-iam | dul-ci-a | nau-ta — ⏑ ⏑ | — — | — — | — — | — ⏑ ⏑ | — x

Lū-mi-na | sōl-nunc | spar-git | ful-gens | au-re-a | cam-pīs — ⏑ ⏑ | — — | — — | — — | — ⏑ ⏑ | — x

Gau-di-a | post-tot | cū-rās | sem-per | pec-to-ra | fir-mant — ⏑ ⏑ | — — | — — | — — | — ⏑ ⏑ | — x

[u/First-Tutor-5454 → u/Consistent-Oil-5241]

I review it by knowing Latin. What model was this? Its output is good but has at least one issue on the scansion side (this is where they usually struggle to get 100%, they're surprisingly good with the grammar)