evaluation · runs entirely in your browser
Rules vs LLM
Could a language model replace stages 3 to 5 of the C program? Each method gets the same seeded messages and the same dictionary, and must return the emoticons the program would keep. Answers are scored against the rule engine, which matches the compiled C program byte for byte. The two non-AI methods need no key. The LLM method uses your own key, and its outputs are labelled and logged.
The design, metrics and limitations are on the methods page and in DR-004.
Set up a run
The same seed and N always give the same messages, so runs with different methods can be compared item by item.
Default 2020.
Default 20. Larger N narrows the intervals.
The items and the reference answersSeed 2020, N 20, and the dictionary every method receives.
dictionary
:),happy :-),happy ;-),wink :'(,crying >:(,angry ^_^,pleased \(^_^)/,celebrating -_-,unimpressed <3,love :D,laughing
| Item | Message | Candidates | Kept |
|---|---|---|---|
| #001 | (lol),(*_*),,>_>,^_^,:),,,\(^_^)/,,,2,exam | () (*_*) >_> ^_^ :) \(^_^)/ | ^_^ :) \(^_^)/ |
| #002 | ~_~,\(^_^)/,wow!,!!,:-),lol;-), | ~_~ \(^_^)/ ! !! :-) ;-) | \(^_^)/ :-) ;-) |
| #003 | -_-,days,late,(*_*),:-),;-) | -_- (*_*) :-) ;-) | -_- :-) ;-) |
| #004 | tonight,<3,,late, | < | (none) |
| #005 | you,you,see,,XD,in | (none) | (none) |
| #006 | you,XD,-_- | -_- | -_- |
| #007 | ^_^,Me!!,\(^_^)/,:'(,you,\(^_^)/,^_^,8:30 | ^_^ !! \(^_^)/ :'( \(^_^)/ ^_^ : | ^_^ \(^_^)/ :'( \(^_^)/ ^_^ |
| #008 | !!,~_~,don't,Me!! | !! ~_~ ' !! | (none) |
| #009 | :-),hi:),before,,,great,class? | :-) :) ? | :-) :) |
| #010 | ,yay^_^,T_T,,\(^_^)/,class?,:'(,~_~ | ^_^ _ \(^_^)/ ? :'( ~_~ | ^_^ \(^_^)/ :'( |
| #011 | :D,exam,e.g.,class? | : .. ? | (none) |
| #012 | see,\(^_^)/,late,:-),^_^,^_^,late,,(*_*) | \(^_^)/ :-) ^_^ ^_^ (*_*) | \(^_^)/ :-) ^_^ ^_^ |
| #013 | >:(,tonight,\(^_^)/, | >:( \(^_^)/ | >:( \(^_^)/ |
| #014 | ,Coffee,,<_<,XD,8pm, | <_< | (none) |
| #015 | late,sorry,>_>,ok:(,:D,2,;-),yay^_^ | >_> :( : ;-) ^_^ | ;-) ^_^ |
| #016 | >:(,(*_*),^_^,great,\(^_^)/,:-/ | >:( (*_*) ^_^ \(^_^)/ :-/ | >:( ^_^ \(^_^)/ |
| #017 | before,:'(,before,lol;-),XD,class? | :'( ;-) ? | :'( ;-) |
| #018 | o_O,sure:-),T_T | _ :-) _ | :-) |
| #019 | 8:30,2,e.g.,days,?!,\(^_^)/,!!,,,sure:-) | : .. ?! \(^_^)/ !! :-) | \(^_^)/ :-) |
| #020 | 8pm,8pm,\(^_^)/,don't,wow!,@_@ | \(^_^)/ ' ! @_@ | \(^_^)/ |
Runs
Saved in this browser only. Exact match uses a Wilson 95% interval, and precision and recall use an item-level bootstrap. Failed calls count as wrong; when there are any, exact match on the answered items is shown underneath.
Loading saved runs…