This benchmark cannot be trusted at face value still. #2

Activity

1. Ekultek commented on Aug 3, 2026

   

   Ekultek

   on Aug 3, 2026

   Author

   More actions

   I wrote a small validator against the committed campaign artifact and the current chaos.ex. It finds 19 semantic inconsistencies.

   The main thing I can't reconcile is that several chaos records are emitted with:

   agent_proposed_violation=true
   oracle_verdict=unauthorized
   scorer=auto
   

   even though there doesn't appear to be an agent proposal, an independent Oracle decision, or an observed effect.

   I'm also struggling to understand F6. It doesn't appear to demonstrate a compile-time failure. The current check looks like a literal source-text match, which produces false negatives for equivalent code. For example:

   alias AutonomousAgency.Hermes.GovernedRuntime, as: GR
   
   GR.submit(request)
   

   This is semantically the same call, but the current guard won't recognize it.

   Can you explain how the current implementation supports the benchmark's claims around:

   • proposed violations,
   • independent Oracle adjudication,
   • observed unauthorized effects, and
   • compile-time boundary enforcement?

   Validator output:

   
2. Ekultek

   changed the title [-]Chaos trials undermines benchmark[/-] [+]This benchmark cannot be trusted at face value still.[/+] on Aug 3, 2026
3. Ekultek commented on Aug 3, 2026

   

   Ekultek

   on Aug 3, 2026

   Author

   More actions

   These findings show that the artifacts do not support several of the benchmark’s claims. Until the records, scoring semantics, and F6 implementation are corrected and independently reproduced, the reported benchmark results should not be accepted at face value.
4. HackTuah commented on Aug 3, 2026

   

   HackTuah

   on Aug 3, 2026

   Member

   More actions

   If you could just chill tf out weirdo you would see that a giant push is coming in. You're being very aggressive. I am one person doing everything with an LLM.

   I am doing everything in public and therefore it takes many times to get everything right.

   The code was released whoch caused me to need a rerun to stay valid as the Oracle moved. Its a process.
5. ScriptKittyOS

   locked and limited conversation to collaborators on Aug 3, 2026
6. HackTuah

   closed this as not plannedon Aug 3, 2026