Melo Labs

← All articles

Haiku with a UX kit: can open rules and minimal code make a small model as accurate as a larger one?

2026-11-10

haikuuxuicssdata-lakecostpre-registrationagents

Summary. We tested whether the combination of a small model (Haiku 5.5), a UX rules data lake with open sources and a minimal checking code (the “kit”) gets close to a larger model (Sonnet 5.5) on three login-screen tasks: creating, improving and comparing two screens at code level. There were 2,244 items and 2,424 calls, with 0 failures in the end. On creating and improving, Haiku with rules and kit went from 0.744 to 0.998 (create) and from 0.754 to 0.926 (improve), and the combined accuracy of Haiku with the kit was 1.07 times that of plain Sonnet, at 10% of the cost per task. On comparing two screens, the kit did not help and made things slightly worse; pairs with no real difference never produced a false alarm in any condition, so the hypothesis about false alarms could not be tested. Three non-human aesthetics instruments were validated and none passed; for that reason this study claims nothing about beauty. The main threat to validity is circular and is in section 9: the 30 rules the kit checks are the same ones that score create and improve. An independent check with axe-core confirms the gain in create, but not in improve. For the practical summary, there is the simple guide.

Method overview

1. The question and the decision it informs

In studies 1 and 2, Haiku stayed close to larger models when it received the right process and context. What remains unknown is what happens in UX and UI, where two things tend to go wrong with language models: they perceive fine style differences poorly (the literature shows low recall in detecting CSS changes), and taste is subjective. The thesis of this study is that one can use technology to cover what AI does poorly: open-source rules in the prompt and code that checks and compares screens, leaving the model with what it does well, which is writing and fixing code.

Study question: does a Haiku with a UX rules data lake and a code-based checking kit get close to plain Sonnet in accuracy, and at what total cost? The unit under test is the whole (model + rules + kit), not the model alone. This matters: saying that “the code did the detection” is not a defect of the experiment; it is what is being measured.

2. The material

2.1 The login data lake

There are 30 rules for a login screen, each with text, source, how to check it, a good example, a bad example, severity and type (25 objective, 5 semi-objective). All come from open, citable sources: WCAG 2.2 (contrast, labels, touch target, focus, reflow, language), web.dev (login form best practices) and Material Design 3 (consistency, hierarchy, spacing scale). No rule was written from memory as “truth”. The source URLs were written from knowledge of the pages and still need to be checked one by one by opening them (pending item, section 9). Examples:

RuleSourceHow it is checked
L08: button text with a contrast of at least 4.5:1WCAG 2.2, 1.4.3contrast ratio between the text color and the button background
L11: targets of at least 24 px (preferably 44)WCAG 2.2, 2.5.8estimated height of button and fields
L17: link to recover the passwordweb.dev, WCAG 3.3.8an <a> exists with forgot/recover text
L28: fields and button with the same border radiusMaterial 3, text fieldscompares border-radius

2.2 The kit

The kit is minimal code, in Node, with no browser and no layout (degree 0 of a cost ladder): parse5 for the HTML, css-tree for the CSS and culori for colors and contrast. It does two things: (a) it checks the 30 rules and returns a text report with what failed and where; (b) it compares two screens and lists the real differences in structure, content, flow and style, with normalized values (#fff equals rgb(255,255,255), 16px equals 1rem; class names, whitespace, comments and attribute order are ignored). Each run takes about 0.6 s. We left the real browser (Playwright) and images out of the kit: what it does not see, for lack of layout, is declared in section 9.

2.3 The stimuli

All generated by a script with a fixed seed (20261110), which gives an exact answer key by construction and avoids screens the model may have seen in training.

The generator checks the stimuli against the kit itself: the kit saw a difference in every pair with a change and in no noise or identical pair. This validates that the kit is consistent with the generator, not independent proof (section 9).

3. The conditions: what the model receives

ConditionWhat the model receivesThe question it answers
A0only the taskhow much does it do alone? (baseline)
A1task + the 30 data lake rulesdoes knowledge alone, with no tool, help?
A2task + kit reportdoes the tool alone, with no rules in the prompt, help?
A3task + rules + reportthe complete combination
AC (T1 only, Haiku only)a second self-critique pass, without the kitdoes the kit’s gain come from information or only from an extra pass?

In create, conditions A2, A3 and AC have two calls: generate the HTML and then fix it. In improve and compare, the report goes in the same prompt. Corrections come back as patches (ANTES/DEPOIS), applied by code equally to all models: 2,679 of 2,688 Haiku blocks and 2,358 of 2,359 Sonnet blocks were applied.

4. Design decisions

5. The execution

2,244 items ran (Haiku 1,140 and Sonnet 1,104, because AC exists only for Haiku), in 2,424 calls (the two-call ones are create with A2, A3 and AC). Haiku consumed 4.68 million input tokens and 2.65 million output tokens; Sonnet, 4.54 million and 1.10 million. Notably, at the same medium effort, Haiku generated more than twice as many output tokens as Sonnet. The total nominal cost was US$ 19.63 (Haiku US$ 1.69, Sonnet US$ 17.94). There was one error in the middle of the run, from the kit itself, and the item was redone: final result, 0 items with errors.

We recorded three deviations from the pre-registration, described in dados/DESVIOS.md: (1) the kit crashed on an invalid color generated by Haiku and was fixed without changing any rule or threshold; (2) Opus was dropped; (3) the logo was replaced before the run. None of them changes a hypothesis.

6. The hypotheses, one by one

Each hypothesis was written before the run, with a metric and a criterion. The p-values come from the paired Wilcoxon test over the items (means of the 3 samples), with a confidence interval by bootstrap over items (10,000 replicates) and Holm correction across the 12 confirmatory comparisons. Of these, 7 pass Holm.

The hypotheses and the result of each

H4: in create, does Haiku with rules and kit beat plain Haiku? Yes: 0.998 versus 0.744, difference +0.255 (95% CI: +0.239 to +0.274; p = 0.0002, passes Holm).

H3: in improve, the same? Yes: 0.926 versus 0.754, +0.172 (95% CI: +0.083 to +0.267; p = 0.0011, passes). The new defects introduced fall from 0.53 to 0.18 per screen (p = 0.022, does not pass Holm).

H7: do the rules alone already help? Yes, in create (A1 0.988 versus 0.744, p = 0.0002) and in improve (0.894 versus 0.754, p = 0.0034). In compare, the tool (A2) came out below the rules-only condition (0.899 versus 0.939), the opposite of what was predicted.

H6: is the kit’s gain information or just an extra pass? In create, the kit beats self-critique without the kit: 0.954 versus 0.804, +0.150 (p = 0.0002, passes). The independent check with axe-core does not confirm this advantage (section 7.5).

H5: does the kit help Haiku more than Sonnet? In create, the gain of A3 over A0 is +0.255 for Haiku and +0.226 for Sonnet (difference +0.029; p = 0.002, passes); in improve, +0.172 versus −0.025 (difference +0.197; p = 0.0011, passes); in compare, no (p = 0.11). In improve, plain Sonnet is already good (0.935) and gets worse with the rules alone (0.810).

H1: in compare, does the kit raise Haiku’s recall? Recall rises from 0.909 to 0.967 (+0.058; 95% CI: +0.005 to +0.121; p = 0.044), but does not pass Holm. Not supported.

H2: does the kit reduce false alarms on pairs with no real difference? Not testable. Haiku and Sonnet scored 1.00 on the 16 noise and identical pairs in every condition, including without the kit. There was no false alarm to reduce.

H9: the main criterion. Does Haiku with rules and kit reach at least 90% of the combined accuracy of plain Sonnet, at no more than 50% of the cost per task? The combined accuracy is the mean of the create, improve and compare scores. Result: 0.943 versus 0.882, ratio 1.07, and mean cost per task of US$ 0.0022 versus US$ 0.0210, ratio 0.10. Supported, but only against Sonnet: Opus was dropped, so there is no comparison with a larger model than that.

7. Results

Score by task and condition

TaskModelA0A1A2A3AC
T1 createHaiku0.7440.9880.9540.9980.804
T1 createSonnet0.7740.9870.9791.000n.a.
T2 improveHaiku0.7540.8940.9670.926n.a.
T2 improveSonnet0.9350.8100.9560.910n.a.
T3 compareHaiku0.9330.9390.8990.905n.a.
T3 compareSonnet0.9380.9240.8950.897n.a.

7.1 What worked

  1. The rules in the prompt already solve most of create. Haiku goes from 0.744 to 0.988 with just the 30 rules (A1). The kit adds little there (A3 0.998).
  2. In improve, the kit report is what helps Haiku most: from 0.754 (A0) to 0.967 (A2), above the rules alone (0.894), and with fewer new defects introduced.
  3. Haiku with the combination reaches the level of plain Sonnet, for a tenth of the cost. The combined accuracy of Haiku A3 (0.943) is above that of Sonnet A0 (0.882), with US$ 0.0022 per task versus US$ 0.0210.
  4. Patch-based corrections work: 99.7% of the Haiku blocks and 99.96% of the Sonnet blocks were applied without ambiguity.
  5. The response format was not a problem: 100% of the compare JSONs were valid for both models.

7.2 What did not work

T3 by pair category

The likely reason, not tested, is that the report lists several properties derived from a single change (for example, the four sides of a padding) and the model copies them; the answer key has one item, and the rest becomes a false alarm. It is a granularity failure of the kit and of the scoring together, and a rescoring that groups by property would be exploratory, done after seeing the data.

7.3 The cost

Nominal cost per task

ConditionHaiku (US$/task)Sonnet (US$/task)
A00.00170.0210
A10.00220.0241
A20.00290.0359
A30.00220.0238

The mean cost per task is the average of the three task types. Haiku A3 costs 10% of Sonnet A0. This figure is about tokens and does not include the cost of building the kit and the data lake, which was human and AI work and was not monetized; the break-even point against Sonnet depends on how many screens are checked with the same kit.

7.4 Aesthetics: no non-human instrument passed

The author is not a designer, and a human panel was ruled out. The way out was to look for instruments that are neither human nor from the family of the models under test and to validate them before using them, with criteria fixed before computing any score. Result: none validated.

Aesthetics instruments and the thresholds

Conclusion of this part: without validity, aesthetics stays out of the decision and out of the conclusions. The study measures legibility, consistency and conformity with open rules, which are part of UX, but not “beauty”.

7.5 Independent check with axe-core

Since the 30 rules are both in the prompt and in the scoring of create and improve, we ran axe-core (WCAG 2.x A/AA), which the kit and the prompts do not know, on the final screens of T1 and T2. Mean violations per screen:

T1 createA0A1A2A3AC
Haiku0.500.030.080.030.06
Sonnet0.310.000.060.00n.a.

axe-core violations by condition

The gain in create is confirmed independently (Haiku A0 versus A3: p = 0.004). In improve, axe finds almost no violations in any condition, so it does not discriminate. And axe does not confirm the kit’s advantage over self-critique without the kit (0.08 versus 0.06; p = 0.75): that part of H6 is supported only by the 30 rules themselves.

8. Why these results hold, and how far

The design has safeguards: hypotheses, metrics and parameters fixed before the run, with hashes; novel stimuli generated by script with an exact answer key; models without internet access; a self-critique control; multiple-testing correction; an independent check with another tool; reporting of deviations; reporting of negative results (compare, aesthetics) with the same rigor as the positive ones.

9. Threats to validity

10. What this allows us to conclude

With confidence: on the tasks of creating and improving a login screen, giving Haiku the rules (and, in improve, the kit report) brings it to the level of unaided Sonnet, for about a tenth of the cost per task, as measured by the rules themselves and confirmed in create by an independent tool; on comparing two screens at code level, the kit’s gain does not appear, because the models already get almost everything right, and the report can get in the way if it comes out verbose. Without confidence: whether the result holds for other screens, for real screens and for larger models; whether the advantage survives scoring that is independent of the rules; and any claim about aesthetics. Natural next steps: a third set of rules that the kit and the prompt do not know, to score with; a kit with layout (degree 2); Opus; and real screens.

11. Reproducibility

Code in studies/haiku-ui-design/ (kit/, gerador/, executor.py); lake in lake/login.json; stimuli in dados/estimulos/; raw data in dados/bruto/; validation criteria for the aesthetics instruments and the judge in dados/*CRITERIOS.md; deviations in dados/DESVIOS.md; analysis with python gerador/analise.py; hashes in dados/CONGELADO.txt. The WiserUI-Bench images (CC BY-NC-SA 4.0, non-commercial research only) are not redistributed; the downloader gerador/baixar_wiserui.py fetches them from the original hosts. Archive DOI: Archived on Zenodo (version 3.0.0, studies 3 and 3b): 10.5281/zenodo.24132784.

12. References


The experiments, data and conclusions are mine. The writing was AI-assisted and reviewed by me.

The experiments, data and conclusions are the author's. The writing is AI-assisted and reviewed by him.