diff --git a/docs/README.md b/docs/README.md index 7869e50..0f8017f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -32,8 +32,11 @@ Start here: - [Scient product identity](product/scient-product-identity.md) - accepted company, app, native-agent, external-agent, and naming vocabulary. - [Product philosophy](product/product-philosophy.md) - draft durable product principles; the accepted PRD governs conflicts. - [Technology stack](architecture/technology-stack.md) - current proposed stack direction. -- [Idea inbox](planning/idea-inbox.md) - categorized intake for unresolved ideas, - including the future Scient memory-architecture discovery. +- [Idea inbox](planning/idea-inbox.md) - lightweight intake for unresolved ideas + before they are evaluated and routed. +- [Memory architecture discovery](planning/memory-architecture-discovery.md) - + draft candidate scopes, questions, scenarios, and discovery sequence; no + memory architecture or storage technology is selected. - [Product roadmap](planning/product-roadmap.md) - current sequence of coherent product outcomes. - [First vertical-slice implementation plan](planning/first-scient-vertical-slice-implementation-plan.md) - bounded plan for the active product slice. - [Scient and external agents implementation plan](planning/scient-and-external-agents-implementation-plan.md) - proposed plan for building the Scient agent as the owned OpenCode-derived first-party agent while preserving external agents independently. diff --git a/docs/architecture/local-first-sync.md b/docs/architecture/local-first-sync.md index ebbf7cf..1b3ad2e 100644 --- a/docs/architecture/local-first-sync.md +++ b/docs/architecture/local-first-sync.md @@ -10,7 +10,8 @@ Doc type: Future home This page will document Scient's local-first and sync architecture when it exists. Unprocessed questions about memory scope, user-selected cloud folders, offline behavior, conversation continuity, and future Scient cloud sync remain -in the [Idea Inbox](../planning/idea-inbox.md#memory-context-and-continuity). +in the draft [Memory Architecture +Discovery](../planning/memory-architecture-discovery.md). No canonical memory store or sync engine is selected. Document here: diff --git a/docs/architecture/project-format.md b/docs/architecture/project-format.md index 55382d0..5f475d6 100644 --- a/docs/architecture/project-format.md +++ b/docs/architecture/project-format.md @@ -9,9 +9,8 @@ Doc type: Future home This page will document the Scient project format when it exists. Unprocessed questions about project memory, conversations, files, portability, Git, cloud -folders, and storage boundaries remain in the -[Idea Inbox](../planning/idea-inbox.md#memory-context-and-continuity) until a -dedicated memory-architecture discovery begins. +folders, and storage boundaries remain in the draft +[Memory Architecture Discovery](../planning/memory-architecture-discovery.md). Document here: diff --git a/docs/architecture/technology-stack.md b/docs/architecture/technology-stack.md index 61afb23..d73973d 100644 --- a/docs/architecture/technology-stack.md +++ b/docs/architecture/technology-stack.md @@ -237,8 +237,8 @@ Scient scientific project database. Scient-owned project persistence has not been selected, designed, or implemented. The future memory-architecture project will decide the roles of conversations, user memory, project memory, raw history, files, local application storage, and cloud storage before evaluating -their persistence technologies. Unprocessed questions remain in the -[Idea Inbox](../planning/idea-inbox.md#memory-context-and-continuity). +their persistence technologies. Open questions remain in the draft +[Memory Architecture Discovery](../planning/memory-architecture-discovery.md). Scient should distinguish: @@ -477,7 +477,7 @@ Completed historical experiments remain evidence, not the roadmap. | Synara-derived application | Standalone owned source, build, isolated Scient identity and state, reviewed upstream process | Scientific-product fit, sustainable domain UI divergence, and long-term maintenance cost | Gate 1 and Gate 1.5 lab reports; ADR-0001 owns adoption; ADR-0002 owns repository authority | | Scient source foundation | Owned OpenCode build, Synara compatibility, project-root fidelity, transcript fidelity, and approval flow for a constrained action | Scient identity and packaging, owned capabilities, isolated Scient state, durable task behavior, and justified inherited-core changes | Gate 1.5 report proves the source baseline; ADR-0001 owns Scient adoption | | External agents | Nine inherited adapters and external OpenCode settings/adapter paths are present in source | Per-agent live compatibility, subscription/auth behavior, project-task certification, and migration protection | [Scient and external agents implementation plan](../planning/scient-and-external-agents-implementation-plan.md) | -| Scient project state and memory | Product responsibilities, high-level memory principles, approved non-Git recovery requirement, and trust boundary are documented | Memory scopes, canonical representation, conversation relationship, package seam, portability, recovery, cloud sync, and first real scientific object relationship | PRD, [Idea Inbox](../planning/idea-inbox.md#memory-context-and-continuity), and future focused architecture work | +| Scient project state and memory | Product responsibilities, high-level memory principles, approved non-Git recovery requirement, and trust boundary are documented | Memory scopes, canonical representation, conversation relationship, package seam, portability, recovery, cloud sync, and first real scientific object relationship | PRD, [Memory Architecture Discovery](../planning/memory-architecture-discovery.md), and future focused architecture work | | Scient-agent and Scient-app boundary | Scient-agent identity plus context, proposal, review, provenance, and permission responsibilities are documented | Actual contract, code placement, event mapping, isolated Scient-agent state, and accepted write-back path | ADR-0001 and linked implementation plans; `agent-runtime.md` remains a future home | | Goose | Source seams, ACP path, and safety risks inspected | Incremental capabilities or architecture lessons for Scient; any future external Goose path is a separate decision | Goose source-depth inspection | | Cloud sync | Postgres, object storage, and local-first sync are proposed directions | Authority, offline behavior, conflicts, revocation, and recovery | Later roadmap and focused architecture work | diff --git a/docs/planning/README.md b/docs/planning/README.md index 1354b52..4700e25 100644 --- a/docs/planning/README.md +++ b/docs/planning/README.md @@ -3,7 +3,7 @@ Status: Active Owner: Yaacov Created: 2026-06-27 -Last updated: 2026-07-17 +Last updated: 2026-07-18 Purpose: Defines where Scient planning documents live and how they relate to product truth, architecture, design, quality, and research documents. Doc type: Repo orientation @@ -16,6 +16,9 @@ Do not use planning docs as product truth, accepted architecture, or current imp Current planning docs: - `idea-inbox.md` - temporary intake for raw, unprocessed ideas before evaluation and routing. +- `memory-architecture-discovery.md` - draft discussion of candidate memory + scopes, authority, lifecycle, agent access, local/cloud boundaries, and the + questions to resolve before architecture or storage selection. - `product-roadmap.md` - active sequence of coherent product outcomes, beginning with the first Scient scientific project slice. - `first-scient-vertical-slice-implementation-plan.md` - draft source-tracing, implementation, and verification plan for the active product slice. - `scient-and-external-agents-implementation-plan.md` - proposed end-to-end plan for building the Scient agent as the owned OpenCode-derived first-party agent while preserving external agents independently. diff --git a/docs/planning/first-scient-vertical-slice-implementation-plan.md b/docs/planning/first-scient-vertical-slice-implementation-plan.md index e904f7b..4b3c879 100644 --- a/docs/planning/first-scient-vertical-slice-implementation-plan.md +++ b/docs/planning/first-scient-vertical-slice-implementation-plan.md @@ -59,8 +59,8 @@ Phase 2 source tracing is complete in state-ownership gaps and candidate seams. Yaacov clarified that conversations, project memory, user memory, recovery, cloud synchronization, and their storage boundaries belong to a dedicated future memory-architecture project, not an -immediate persistence decision. Those unprocessed ideas now live in the -[`Idea Inbox`](idea-inbox.md#memory-context-and-continuity). The permanent +immediate persistence decision. Those questions now live in the draft +[`Memory Architecture Discovery`](memory-architecture-discovery.md). The permanent scientific-operation package and fake-executor product proof also remain deferred. Non-Git recovery, trusted project filesystem scope, and complete native-Scient/external-OpenCode runtime independence remain product @@ -444,8 +444,8 @@ scopes and relationships among conversations, user memory, project memory, raw history, files, local storage, and future cloud storage must be discovered together in a dedicated future project. -The candidate questions are preserved in the -[`Idea Inbox`](idea-inbox.md#memory-context-and-continuity). They do not select +The candidate questions are preserved in the draft +[`Memory Architecture Discovery`](memory-architecture-discovery.md). They do not select SQLite, define a project ledger, authorize memory implementation, or block the independent T3 reliability work. The permanent scientific-operation package and deterministic fake-executor product proof remain separate deferred decisions. diff --git a/docs/planning/idea-inbox.md b/docs/planning/idea-inbox.md index 579df53..a73cbe0 100644 --- a/docs/planning/idea-inbox.md +++ b/docs/planning/idea-inbox.md @@ -57,91 +57,3 @@ not analysis or architecture, and should move with the idea when it is promoted. | Idea | Raised by | Date added | Context or source | Possible area | |---|---|---|---|---| | **Visual literature map.** Add an interactive, Obsidian-style graph view of the literature sources in a Scient project. Sources would appear as nodes, with inspectable relationships such as citations, shared topics, project links, or researcher-created connections. The view should help researchers explore clusters, identify central or isolated sources, review gaps, filter the collection, and open each source in its normal detail view. | Yishai | 2026-07-18 | Spoken idea. The initial scope should focus on sources already imported into the project; broader scholarly-network discovery and the exact relationship types remain open questions. | Source-library product planning, literature-review UX, design, and future source-relationship architecture | - -### Memory, Context, And Continuity - -| Idea | Raised by | Date added | Context or source | Possible area | -|---|---|---|---|---| -| **Future Scient memory architecture.** Discuss Scient's complete memory architecture as a dedicated future product and architecture project before selecting schemas, databases, or synchronization machinery. | Yaacov | 2026-07-18 | Questions about project records, conversations, SQLite, Git, user-selected cloud folders, recovery, and future Scient cloud sync arose during the first-slice source review. They belong to the broader memory-system discussion, not the completed T3 reliability work or an immediate persistence decision. Reusable questions from an oversized standalone persistence brief were condensed here before that out-of-scope architecture file was removed. | Product planning, future memory architecture, agent runtime, project format, security, provenance, synchronization, and product design | - -#### Candidate scopes and questions to preserve - -Candidate scopes to discuss, not accepted layers: - -- **Conversation memory:** short-lived or derived context from one conversation, - distinct from the complete transcript. -- **Task or run memory:** working state for one delegated task or agent run, - including what may expire when the run completes. -- **Project memory:** durable project direction, decisions, source judgments, - analysis choices, unresolved questions, collaborator decisions, and prior - work needed for continuity. -- **User memory:** personal preferences, recurring choices, and working style - that may apply across projects without silently overriding project rules. -- **Team or organization memory:** shared methods, conventions, and - institutional knowledge with explicit membership, permission, and ownership - boundaries. -- **Scient-maintained knowledge:** built-in product guidance, scientific skills, - and maintained procedures; this may require different authority and update - rules from user-generated memory. -- **Raw history and provenance:** complete conversations, events, actions, and - evidence that may support memory but are not automatically trusted memory. - -Questions for the future discovery project: - -- Which candidate scopes are actually needed, and what are their precise names - and responsibilities? -- Who owns, reads, edits, shares, exports, deletes, or promotes information in - each scope? -- What is ephemeral, retained for continuity, durable, canonical, derived, or - rebuildable? -- What is the difference between a complete conversation transcript, - conversation context, a summary, and trusted memory? -- Can a conversation or task propose project memory, and which promotions - require explicit researcher review? -- How are source, authority, confidence, freshness, conflict, staleness, - distrust, archival, forgetting, and supersession represented? -- What happens when user memory conflicts with project memory, or project - memory conflicts with current files and evidence? -- How can researchers inspect, correct, pin, challenge, archive, forget, or - disable remembered information? -- Which memory can Scient use, and which memory may be disclosed to an external - agent for one bounded task? -- How does task context include only the appropriate user, project, - conversation, and run information? -- How do project memory and provenance relate to ordinary researcher-owned - files without turning generated summaries into project authority? -- How do projects remain friendly to Git while never requiring Git for ordinary - use, history, or recovery? -- How should projects behave in iCloud Drive, Dropbox, OneDrive, external - drives, network locations, and other user-selected folders? -- Which information belongs in the project folder, local application storage, - a user account, or future Scient cloud storage? -- What remains fully useful offline, and what is restored or synchronized when - cloud access returns? -- How are concurrent edits, offline divergence, deletion, restoration, device - loss, and collaborator removal handled? -- How are conversations, memory, decisions, and provenance retained, backed up, - exported, transferred, encrypted, redacted, or permanently deleted? -- Which privacy, sensitive-data, institutional-control, and regional-storage - requirements constrain the design? -- What scale, latency, reliability, recovery, migration, portability, and exit - requirements must be accepted before evaluating storage technologies? -- Only after the memory model is understood, which persistence approaches - should be evaluated, including SQLite, structured files, derived indexes, - local databases, cloud databases, and combinations of them? -- If SQLite remains a candidate at that later stage, what evidence is required - for crash behavior, journal/WAL handling, active copying, cloud-folder sync, - backup and restore, integrity checks, locking, migrations, packaging, - performance, readable export, and exit from the format? - -Explicit non-decisions: - -- These scopes are prompts for discussion, not accepted memory architecture. -- Conversation, user, project, task, team, and system memory are not yet formal - product objects or storage boundaries. -- No database, schema, project ledger, cloud-sync protocol, or retention policy - is selected or authorized. -- The existing application SQLite database remains current app/session - implementation; it does not settle the future memory architecture. -- This future discovery does not block T3 reliability work or unrelated product - work. diff --git a/docs/planning/memory-architecture-discovery.md b/docs/planning/memory-architecture-discovery.md new file mode 100644 index 0000000..cea5460 --- /dev/null +++ b/docs/planning/memory-architecture-discovery.md @@ -0,0 +1,467 @@ +# Memory Architecture Discovery + +Status: Draft +Owner: Yaacov +Created: 2026-07-18 +Last updated: 2026-07-18 +Purpose: Preserves the candidate scopes, vocabulary, product questions, trust boundaries, and investigation sequence required before Scient proposes or implements a memory architecture. +Doc type: Planning note + +## Document Rules + +- This is a discovery document, not accepted product truth, architecture, an + ADR, a database design, or implementation authorization. +- The accepted [PRD](../product/PRD.md), especially Project Memory And + Continuity, owns current product principles. This document must not weaken or + silently extend them. +- Candidate memory scopes are discussion tools. Their names, number, + responsibilities, and relationships are not selected. +- Storage technology comes after the memory model. SQLite, structured files, + indexes, cloud databases, and other mechanisms remain later candidates. +- Do not make this discovery a dependency of the completed T3 reliability work + or unrelated product development. +- When the discovery becomes active, record accepted product outcomes in the + appropriate product/planning document, proposed technical direction in a + focused architecture document, and consequential accepted choices in ADRs. + +## Why This Discovery Exists + +Scient intends to provide continuity across conversations, tasks, agent runs, +devices, collaborators, and the long life of a scientific project. The PRD +already requires memory that is inspectable, correctable, challengeable, and +supported by trust metadata. It also distinguishes helpful summaries from +canonical memory when summaries obscure the underlying project record. + +Those principles do not yet define one memory system. Open questions include: + +- which kinds or scopes of memory Scient needs; +- how complete conversation history differs from remembered context; +- what belongs to a user, project, task, team, organization, or device; +- how information is proposed, reviewed, promoted, inherited, corrected, + forgotten, or superseded across scopes; +- what Scient and external agents may access; +- how memory relates to ordinary files, provenance, decisions, and accepted + project records; +- how local-first behavior, user-selected cloud folders, Git, and future Scient + cloud synchronization interact; and +- only after those answers, which persistence technologies are appropriate. + +These questions surfaced during the first-slice source trace and were briefly +framed too narrowly as a project-persistence or SQLite decision. They belong +together as a future memory-architecture discovery. + +## What Is Already Product Direction + +The following are existing product-level directions, not discoveries made by +this file: + +- the research project is the durable center of work; +- project memory should support continuity across sessions, collaborators, and + agent runs; +- memory should be inspectable, editable, correctable, challengeable, and + recoverable when it affects project work; +- researchers should be able to pin, distrust, update, archive, forget, or + supersede remembered information; +- memory needs enough source, authority, confidence, freshness, conflict, and + staleness information to be evaluated; +- summaries may help continuation but must not silently become authority; +- ordinary researchers must not depend on Git for basic use or recovery; +- local work should remain useful without Scient cloud access; and +- external tools and agents must not become the only owners of Scient project + meaning or accepted scientific work. + +The discovery must translate these directions into a coherent model without +mistaking them for a schema. + +## Vocabulary That Must Be Separated + +The future discussion should avoid using `memory` as a synonym for every form +of stored data. + +| Term | Candidate meaning | Why it must remain distinct | +|---|---|---| +| Complete transcript | The full messages and events from a conversation | Retaining history does not make every message trusted memory | +| Conversation context | The bounded material available while continuing one conversation | It may be temporary, summarized, truncated, or reconstructed | +| Task/run state | Inputs, progress, tool evidence, outputs, and status for one delegated action | Operational continuity and audit evidence are not automatically reusable memory | +| Memory | Information intentionally retained to improve future continuity or decisions | It needs scope, authority, lifecycle, and correction semantics | +| Summary | A compressed interpretation of larger material | It may omit nuance and must expose its sources and limitations | +| Provenance | Evidence of where information or an artifact came from and how it changed | Provenance supports evaluation but is not itself always remembered guidance | +| Decision/history record | A durable account of a choice, revision, or prior state | Historical truth and current guidance can diverge | +| Project file | Researcher-owned source, data, code, analysis, figure, note, or manuscript material | Files remain ordinary project material and should not all be duplicated as memory | +| Application state | Settings, registered paths, UI projections, caches, queues, and sessions | Useful implementation state must not silently define project memory | +| Scient-maintained knowledge | Product guidance, built-in scientific skills, and maintained procedures | Its authority and update path differ from user-generated memory | + +The discovery may revise these terms, but it should not proceed while their +meanings remain collapsed. + +## Candidate Memory Scopes + +These are candidate scopes to compare, combine, rename, or reject. + +### Conversation memory + +Possible purpose: + +- continue one conversation coherently; +- remember temporary conversational goals, references, and unresolved threads; +- retain or derive context without repeatedly replaying the full transcript. + +Open distinction: the transcript, a generated summary, and remembered context +may be three different objects with different retention and trust rules. + +### Task or run memory + +Possible purpose: + +- continue or retry one delegated task; +- preserve bounded inputs, decisions, tool evidence, outputs, and failure state; +- help an agent resume without receiving unrelated project or user context. + +Open distinction: run evidence may be important provenance without becoming +future guidance. + +### Project memory + +Possible purpose: + +- preserve project direction, protocol decisions, source judgments, extraction + choices, analysis decisions, writing choices, unresolved questions, + collaborator decisions, and relevant prior work; +- support continuity across conversations, agents, users, and devices; +- remain inspectable and correctable by project members. + +Open distinction: accepted project records, current files, historical events, +and memory may overlap without being identical. + +### User memory + +Possible purpose: + +- preserve personal working preferences, recurring choices, communication + style, accessibility needs, and preferred review behavior across projects; +- reduce repetitive setup without placing private preferences into every + project. + +Open distinction: user memory must not silently override project rules, +evidence, collaborator decisions, institutional policy, or explicit current +instructions. + +### Team or organization memory + +Possible purpose: + +- preserve shared methods, terminology, templates, conventions, and + institutional knowledge; +- support continuity across projects and changing membership. + +Open distinction: this scope requires explicit governance, permission, +attribution, revocation, and ownership. It should not be assumed necessary just +because user and project memory exist. + +### Scient-maintained knowledge + +Possible purpose: + +- provide maintained product guidance, scientific procedures, and built-in + skills; +- give users a stable baseline that is versioned and updated by Scient rather + than learned implicitly from one user's behavior. + +Open distinction: maintained product knowledge may belong in documentation or +skills rather than the same memory subsystem as personal or project memory. + +### Raw history and provenance + +Possible purpose: + +- preserve complete conversations, events, tool activity, proposal history, + decisions, and source relationships; +- allow memory claims and project changes to be audited or reconstructed. + +Open distinction: raw history can support memory without being injected into +future context or treated as current guidance. + +## Candidate Scope Matrix + +Every candidate scope should eventually answer the following. Empty or vague +cells indicate unresolved architecture, not permission to inherit defaults. + +| Scope | Candidate owner | Typical lifetime | Candidate sharing | Authority question | +|---|---|---|---|---| +| Conversation | Participant or project, undecided | One conversation to retained history | Usually bounded to participants/project | What survives after the conversation, and why? | +| Task/run | Project or execution owner, undecided | One run through audit/recovery retention | Bounded participants and reviewers | Which evidence may influence later work? | +| Project | Project and authorized members | Project lifetime and export/retention period | Project members and bounded agents | What can become trusted project memory? | +| User | Individual user | Account/device lifetime subject to user control | Private by default | When may it affect a shared project? | +| Team/organization | Governed shared entity | Policy-defined | Members according to roles | Who may publish, revoke, or supersede it? | +| Scient-maintained knowledge | Scient product governance | Versioned product lifetime | Product users | How are versions, authority, and updates exposed? | +| Raw history/provenance | Depends on originating object | Retention-policy dependent | Auditors, participants, or project members | What must be retained versus deleted or redacted? | + +## Relationships Between Scopes + +### Proposal and promotion + +The discovery should determine whether lower-scope information can propose a +higher-scope memory. Example questions: + +- Can a conversation propose project memory? +- Can repeated user corrections propose a user preference? +- Can one task produce project memory, or only evidence for researcher review? +- Which promotions require explicit approval? +- Is automatic promotion ever acceptable, and how is it reversed? + +### Context assembly and inheritance + +An agent should not receive every available memory item. The discovery should +define how a task assembles only appropriate context from: + +- explicit current instructions; +- project rules and accepted project memory; +- user preferences that are permitted in that project; +- conversation context; +- task/run state; +- relevant files and evidence; and +- Scient-maintained guidance or skills. + +The resulting context should remain inspectable enough to understand why the +agent behaved as it did. + +### Conflict and precedence + +Candidate conflicts include: + +- user preference versus project rule; +- remembered project direction versus a newer explicit decision; +- summary versus source evidence; +- project memory versus changed project files; +- two collaborators recording incompatible judgments; +- team convention versus project-specific method; and +- stale memory versus current instructions. + +The discovery must not assume that one universal priority order resolves every +case. Some conflicts may require visible coexistence and researcher judgment. + +### Correction, forgetting, and history + +The discussion should distinguish: + +- correcting current memory; +- superseding it while preserving history; +- distrusting it without deleting evidence; +- archiving it from active retrieval; +- forgetting or permanently deleting it; +- redacting sensitive content while retaining an audit event; and +- removing a collaborator's access without falsifying prior attribution. + +## Core Product Questions + +### Scope and ownership + +- Which candidate scopes are actually necessary? +- Is conversation memory a separate scope or a view of project/user memory? +- Is task/run memory a memory scope, operational state, provenance, or some + combination? +- Can one memory item belong to more than one scope? +- Who owns and administers each scope? +- What happens to personal contributions when a user leaves a project or team? + +### Authority and trust + +- Which memories can affect future agent behavior automatically? +- Which require researcher confirmation before use? +- What source, authority, confidence, freshness, and conflict metadata are + mandatory? +- How are inferred memories distinguished from explicit user statements and + accepted project decisions? +- How is stale or contradicted memory detected and surfaced? +- Can an agent challenge memory, and can it change memory without approval? + +### Conversation lifecycle + +- Are complete conversations always retained, optionally retained, or governed + by project/account policy? +- Can users delete a conversation while preserving promoted project memory and + necessary provenance? +- What must be available to continue a conversation on another device? +- How are summaries regenerated, versioned, corrected, or distrusted? +- Which provider-native transcript data must be normalized or exported so + continuity does not depend on one provider? + +### Project and user interaction + +- When may private user memory influence shared project work? +- Should collaborators see that a private preference affected an output without + seeing the preference itself? +- Can a project prohibit particular categories of user memory? +- How are project-specific preferences prevented from leaking into other + projects? +- What happens when one user corrects a project memory that other collaborators + still rely on? + +### Agent access and disclosure + +- Which memory scopes may the native Scient agent access by default? +- Which scopes may an external agent receive for one bounded task? +- How are sensitive or unrelated memories excluded from prompts and tools? +- How does the researcher inspect the memory/context receipt supplied to an + agent? +- How are retrieval, disclosure, use, and resulting changes audited? +- Can an agent propose a memory without gaining permission to write it? + +### Files, Git, and user-selected folders + +- How does project memory refer to ordinary files without duplicating all file + content? +- What happens when referenced files move, change, disappear, or conflict? +- Which memory or provenance material should be Git-friendly? +- How does non-Git recovery work for ordinary users? +- How should projects behave in iCloud Drive, Dropbox, OneDrive, external + drives, network locations, and other folders selected by the researcher? +- Can Scient detect divergent copies without blocking user control over folder + location? +- What is included when a researcher copies, exports, archives, or transfers a + project? + +### Local-first and future Scient cloud + +- What remains fully usable offline? +- Which scopes synchronize across a user's devices? +- Which scopes synchronize with project collaborators? +- How are offline divergence, concurrent edits, deletion, and restoration + represented? +- Is the cloud a mirror, collaboration authority, backup, or different things + for different scopes? +- How do Scient cloud and user-selected folder synchronization coexist without + silently overwriting work? +- What happens when cloud access, account access, or project membership is + revoked? +- How can a researcher leave Scient while retaining understandable project + material and memory exports? + +### Privacy, security, and retention + +- Which memory categories may contain unpublished research, health data, + credentials, personal preferences, or institutional information? +- What must remain local-only or institution-controlled? +- Which scopes require encryption at rest or end to end? +- How are retention, legal hold, export, redaction, and permanent deletion + expressed? +- How are embeddings, derived summaries, caches, and backups deleted when their + source memory is forgotten? +- What diagnostics can be shared without disclosing memory content? +- Which regional-storage or institutional requirements may constrain cloud + behavior? + +## Scenarios The Discovery Should Test + +1. A conversation produces a useful project decision. The researcher approves + it as project memory, later corrects it, and can still inspect its source. +2. A user's private writing preference affects their own draft assistance but + does not become a collaborator's preference or override a project style rule. +3. An agent run fails halfway through. Operational state supports retry and + audit without turning every tool event into active memory. +4. A researcher deletes a conversation while preserving an independently + accepted project decision and the minimum provenance required to understand + it. +5. Two collaborators record conflicting interpretations. Scient shows the + conflict rather than silently choosing one memory. +6. A project moves into Dropbox or iCloud Drive, remains usable offline, and + later reconnects to Scient cloud without silent data loss. +7. An external agent receives only the project and task context authorized for + one action, not unrelated user or organization memory. +8. A collaborator loses access. Their prior attributed contributions remain + intelligible while private or revoked memory is no longer disclosed. +9. A researcher exports or leaves Scient and can retain understandable project + files, relevant memory, decisions, and provenance without requiring the + original provider transcript. + +## Persistence Questions Come Later + +Only after the scope, authority, lifecycle, privacy, portability, and sync +requirements are understood should the project compare persistence models. +Candidates may include: + +- existing application SQLite for operational app/session concerns; +- project-owned structured files; +- a local database; +- canonical files with a derived local index; +- app-local storage with explicit portable export; +- a cloud database or event service; or +- different mechanisms for different memory scopes. + +If SQLite remains a candidate, later technical evaluation should cover crash +behavior, journal/WAL handling, active copying, cloud-folder behavior, backup +and restore, integrity checks, locking, migrations, packaging, performance, +readable export, and exit from the format. Those are evaluation questions, not +reasons to select SQLite now. + +## Discovery Sequence + +When Yaacov starts the memory project, proceed in this order: + +1. **Vocabulary and scenarios:** refine the terms and select representative + user scenarios without discussing tables or libraries. +2. **Scope model:** compare, combine, rename, or reject the candidate memory + scopes and define ownership boundaries. +3. **Authority and lifecycle:** define promotion, precedence, trust, + correction, forgetting, retention, and provenance behavior. +4. **Access and privacy:** define native/external-agent access, collaborator + permissions, sensitive-data boundaries, and inspectable context assembly. +5. **Local/cloud behavior:** define offline continuity, portability, Git and + cloud-folder behavior, device sync, collaboration, conflict, export, and + exit requirements. +6. **Technology evaluation:** compare persistence and sync candidates using the + accepted behavioral requirements and realistic failure cases. +7. **Architecture proposal:** write a focused architecture document with + rejected alternatives, limitations, and migration/exit paths. +8. **Acceptance and implementation planning:** create ADRs where required and a + bounded implementation plan only after explicit human review. + +## Expected Outputs Of The Future Discovery + +- accepted vocabulary or a clearly documented set of remaining terminology + disputes; +- a memory-scope and ownership model; +- promotion, conflict, correction, forgetting, and retention rules; +- conversation/history/provenance boundaries; +- an agent access and context-assembly model; +- local, project-folder, device, collaborator, and cloud authority boundaries; +- portability, Git, cloud-folder, export, privacy, and recovery requirements; +- a technology-neutral test and evaluation plan; +- a proposed architecture document; and +- explicit decisions about what is deferred. + +## Explicit Non-Decisions + +This discovery does not currently decide: + +- that every candidate scope should exist; +- that conversation memory differs from task or project memory; +- whether complete transcripts are project-owned, user-owned, or provider-owned; +- how memory is represented in code or storage; +- whether SQLite, structured files, Postgres, embeddings, a graph, or an event + log should be used; +- whether all memory synchronizes through Scient cloud; +- a schema, package boundary, API, sync protocol, retention schedule, or + encryption design; +- the complete relationship between memory and accepted scientific records; +- implementation sequencing; or +- any dependency on T3 Code. + +## Related Documents + +- [Product Requirements](../product/PRD.md), especially Project Memory And + Continuity, Local-First Ownership, and Provenance/Recovery. +- [Product Planning](product-planning.md), especially the Project Memory feature + row and cross-project memory question. +- [Agent Runtime](../architecture/agent-runtime.md) for future runtime context + and agent boundaries. +- [Security And Permissions](../architecture/security-and-permissions.md) for + trust, disclosure, and sensitive-data questions. +- [Project Format](../architecture/project-format.md) for future project-folder + and portable-record boundaries. +- [Local-First Sync](../architecture/local-first-sync.md) for future offline, + cloud-mirror, and conflict semantics. +- [First-Slice Source Trace](../../lab/notes/first-slice-source-trace-2026-07-18.md) + for the source evidence that exposed—but did not resolve—the memory boundary. +- [T3 Code Targeted Review](../../lab/notes/t3-code-targeted-review-2026-07-18.md) + for the independent T3 reliability intake and its explicit stop boundary. diff --git a/docs/planning/product-planning.md b/docs/planning/product-planning.md index a163f0e..c942f0a 100644 --- a/docs/planning/product-planning.md +++ b/docs/planning/product-planning.md @@ -3,7 +3,7 @@ Status: Draft Owner: Yaacov Created: 2026-06-28 -Last updated: 2026-07-17 +Last updated: 2026-07-18 Purpose: Tracks current product planning after the accepted PRD, including candidate features, open product questions, and cross-document handoffs. Doc type: Planning note @@ -101,7 +101,7 @@ This is the active feature inventory. It should stay compact. Add detail only wh | Agent delegation and safe automation | Object-scoped tasks, context receipts, project-aware tools, proposed artifacts, task queue, durable runs, approvals, retries, cancellation, recovery. | Core | Foundation to early validation | Approval later | Architecture handoffs: `docs/architecture/agent-runtime.md` and `docs/architecture/security-and-permissions.md`. | | Model access and routing | Provider-connected subscriptions, bring-your-own API keys, Scient-managed access, manual model choice, and later task-aware routing. | Core | Foundation to early expansion | None first | Sequencing and commercial options: `model-access-and-routing-evolution.md`. Candidate portfolio: `../research/source-evaluations/model-portfolio-and-provider-routing.md`. | | Scientific skills | Built-in bounded skills for evidence extraction, drafting, citation checking, data analysis, figure creation, method guidance, journal adaptation, project mentoring. | Important | Early expansion | Approval later | Start with a tiny skill set; defer marketplace/registry mechanics. | -| Project memory | Inspectable memory, source/authority/confidence/freshness metadata, pin/archive/forget, conflict/staleness handling, project continuity summaries. | Core | Foundation to early expansion | Capture and review later | Architecture handoff: memory may need its own doc after agent runtime pressure clarifies boundaries. | +| Project memory | Inspectable memory, source/authority/confidence/freshness metadata, pin/archive/forget, conflict/staleness handling, project continuity summaries. | Core | Foundation to early expansion | Capture and review later | Discovery handoff: [`memory-architecture-discovery.md`](memory-architecture-discovery.md); no memory scopes or storage technology are selected. | | Identity, sharing, collaboration, and mobile | Account/device identity, roles, permissions, invitations, shared review, comments, assignments, attribution, cloud mirror, sync/conflict states, mobile continuation. | Core | Design early, implement in phases | Read/review/capture/approval | Architecture handoffs: collaboration model, local-first sync, security. | | Provenance, versioning, and recovery | Event history, source/evidence/citation/action provenance, diffs, checkpoints, snapshots, rollback, failed-run recovery, optional Git-like workflows. | Core | Foundation | Approval later | Normal users should not need Git. Architecture and quality handoffs required. | | External interoperability and open science | Reference managers, citation formats, scholarly databases, document formats, repositories, drives, code/data tools, archives, deposit records. | Important | Early where it unblocks core workflows | Read/review later | Name specific targets in roadmap/architecture only when compatibility is the requirement. | @@ -122,7 +122,7 @@ The PRD intentionally leaves these open. Resolve them in the right document when | What cloud mirroring and collaboration semantics come first? | Local-first ownership, backup, sharing, conflicts, revocation, and restore must be coherent. | Collaboration, sync, and security architecture. | | What mobile actions are allowed first? | Mobile should continue project work without becoming a second source of truth. | Product planning and design. | | Which sensitive data classes are supported, unsupported, or institution-gated? | Security posture must be explicit before real sensitive projects are encouraged. | Security architecture and product planning. | -| How should cross-project memory or organization-level methods work, if at all? | Useful later, but dangerous before project-level memory is trustworthy. | Product planning and future architecture. | +| How should cross-project memory or organization-level methods work, if at all? | Useful later, but dangerous before project-level memory is trustworthy. | [Memory Architecture Discovery](memory-architecture-discovery.md) and future architecture. | ## Handoffs @@ -131,6 +131,7 @@ Use this section to route work out of product planning. Do not let this file bec | Handoff area | Destination | Product planning input | |---|---|---| | Project format | `docs/architecture/project-format.md` | Durable project records, local files, artifacts, source/evidence/citation/analysis relationships, export/deposit records. | +| Memory architecture discovery | `docs/planning/memory-architecture-discovery.md` | Candidate memory scopes, authority, lifecycle, conversation/history boundaries, agent access, local/cloud behavior, and technology-neutral requirements. | | Agent runtime | `docs/architecture/agent-runtime.md` | Object-scoped tasks, context receipts, project-aware tools, proposed artifacts, durable runs, retries, checkpoints, recovery. | | Security and permissions | `docs/architecture/security-and-permissions.md` | High-impact action review, permission scope, unknown-data handling, local execution, imported-file trust, sensitive data classes. | | Collaboration model | `docs/architecture/collaboration-model.md` | Roles, membership, invitations, attribution, shared review, comments, assignments, conflict states. | diff --git a/lab/notes/first-slice-source-trace-2026-07-18.md b/lab/notes/first-slice-source-trace-2026-07-18.md index 276d659..04d38f9 100644 --- a/lab/notes/first-slice-source-trace-2026-07-18.md +++ b/lab/notes/first-slice-source-trace-2026-07-18.md @@ -32,7 +32,7 @@ or representation. Conversations, task/run context, project memory, user memory, raw history, files, recovery, and cloud sync must be discussed together in a dedicated future memory-architecture project. The raw candidate layers and questions are preserved in the -[`Idea Inbox`](../../docs/planning/idea-inbox.md#memory-context-and-continuity). +[`Memory Architecture Discovery`](../../docs/planning/memory-architecture-discovery.md). The existing Git-backed checkpoint mechanism is not suitable for the approved non-Git guarantee. A future memory and recovery design must determine how @@ -232,7 +232,7 @@ kind of project record before comparing storage technologies. Its questions include portability, Git independence and compatibility, user-selected cloud folders, recovery, conversation retention, future Scient cloud sync, privacy, and export. They are preserved in the -[`Idea Inbox`](../../docs/planning/idea-inbox.md#memory-context-and-continuity). +[`Memory Architecture Discovery`](../../docs/planning/memory-architecture-discovery.md). Secrets and provider credentials must never become project memory merely for convenience. @@ -353,7 +353,7 @@ It hands the unresolved relationships among conversation history, task/run context, user memory, project memory, raw history, provenance, files, recovery, and cloud sync to a dedicated future discovery project. The candidate layers, questions, and explicit non-decisions are preserved in the -[`Idea Inbox`](../../docs/planning/idea-inbox.md#memory-context-and-continuity). +[`Memory Architecture Discovery`](../../docs/planning/memory-architecture-discovery.md). The permanent scientific-operation package and deterministic fake-executor product proof remain separate deferred questions. Native Scient isolation is an @@ -374,5 +374,5 @@ The source map is complete, but its former SQLite and hybrid-package selection is withdrawn. The approved requirements are non-Git recovery, trusted filesystem scope, and complete native-Scient/external-OpenCode runtime independence. Fake-executor work is deferred. Future memory questions live in -the [`Idea Inbox`](../../docs/planning/idea-inbox.md#memory-context-and-continuity); +the [`Memory Architecture Discovery`](../../docs/planning/memory-architecture-discovery.md); do not infer memory architecture or implementation authorization from this note. diff --git a/lab/notes/t3-code-targeted-review-2026-07-18.md b/lab/notes/t3-code-targeted-review-2026-07-18.md index 9c13e0a..bf284d8 100644 --- a/lab/notes/t3-code-targeted-review-2026-07-18.md +++ b/lab/notes/t3-code-targeted-review-2026-07-18.md @@ -103,7 +103,7 @@ The independent T3 intake is complete and has no dependency on the future memory-architecture project. Raw questions about conversation, user, project, and task/run memory; portability; recovery; Git and cloud folders; and future Scient cloud sync are preserved separately in the -[`Idea Inbox`](../../docs/planning/idea-inbox.md#memory-context-and-continuity). +[`Memory Architecture Discovery`](../../docs/planning/memory-architecture-discovery.md). ## Triggered Shelf