Order getContext sections stable to volatile so provider prompt caching can take effect - #341
Order getContext sections stable to volatile so provider prompt caching can take effect#341Leul-Negash wants to merge 3 commits into
Conversation
getContext placed LAST_SKILL_USE_RESULTS, which is rewritten every cycle and is one of the largest sections, ahead of HISTORY. Providers that cache prompts only reuse a prefix that is byte-identical to the previous request, so the reusable part ended right after the small static head and almost nothing was served from cache on consecutive calls. Add a stableContextOrder flag (default off) that moves HISTORY ahead of the volatile results section and keeps TIME last. With the flag off the assembled prompt is byte-for-byte identical to before, so behaviour is unchanged unless the flag is turned on. Refs #300
The first version of tests/src_loop.metta drove contextVolatileTail through the global stableContextOrder flag and the &lastresults state cell, toggling the flag with add-atom/remove-atom. That errored out under the PeTTa version CI runs and stopped the suite before any assertion ran. Split the ordering into a pure orderContextTail helper and test that directly with literal section strings, so no loop state, state cell or flag mutation is involved. getContext behaviour is unchanged: with the flag off the tail is still byte-for-byte identical to the original layout.
Building the final string with py-str inside the test (and importing lib_llm to get it) hung under the PeTTa version CI runs and never reached an assertion. Return the ordered sections as a plain tuple from orderedContextSections and let contextVolatileTail wrap that in py-str, then assert on the tuple. The test no longer imports lib_llm or calls py-str. getContext output is unchanged.
|
Thanks for picking this up. One thing from running it, and one pointer.
|
2 participants
Description
getContextplaces LAST_SKILL_USE_RESULTS — which is rewritten every cycle and is one of the largest sections — before HISTORY. Providers that cache prompts only reuse a prefix that is byte-identical to the previous request, so the reusable prefix ends right after the small static head and almost none of the prompt is served from cache on consecutive calls. This is #300.Adds a
stableContextOrderflag (default off). When it is on, HISTORY moves ahead of the volatile results section and TIME stays last, so the stable part of the prompt sits in the reusable prefix. When it is off, the assembled prompt is byte-for-byte identical to before, so nothing changes unless the flag is set. The reorder is isolated in a smallcontextVolatileTailhelper so the rest of getContext is untouched.Note on #284 (Context Frames): that PR also reworks getContext, and #300 calls out the same interaction. This change is default-off and scoped to the current layout, so it can land independently and the same ordering can be carried over when #284 is rebased. This is part 1 (section ordering); the fixed-block HISTORY cut suggested in #300 can follow as a separate change.
How Has This Been Tested?
Added
tests/src_loop.metta, which asserts the flag-off tail is byte-for-byte identical to the previous inline layout and that the flag-on tail places HISTORY ahead of the volatile results section. The full metta test suite passes locally.Checklist