diff --git a/docs/android-platform-checks-analysis.md b/docs/android-platform-checks-analysis.md deleted file mode 100644 index 4e3c83425..000000000 --- a/docs/android-platform-checks-analysis.md +++ /dev/null @@ -1,1452 +0,0 @@ -# Android Platform Checks Analysis - -## Overview -This document contains a comprehensive analysis of all platform-specific checks in the Maple codebase that need to be evaluated for Android support. Each instance is documented with its location, current behavior, and recommendations for Android handling. - -## Platform Detection Pattern -The new standard pattern for platform detection using our utilities: -```typescript -import { isIOS, isAndroid, isMobile } from '@/utils/platform'; - -// Check for specific platforms -if (await isIOS()) { /* iOS specific code */ } -if (await isAndroid()) { /* Android specific code */ } -if (await isMobile()) { /* Both iOS and Android */ } -``` - ---- - -## 1. BILLING & PAYMENTS (`frontend/src/billing/billingApi.ts`) ✅ COMPLETED - -### Instance 1: Portal Return URL (Line 102) ✅ -- **Current iOS Behavior:** Uses `https://trymaple.ai` as return URL instead of `tauri://localhost` -- **Android Recommendation:** ✅ **Same as iOS** - Use `https://trymaple.ai` -- **Status:** ✅ **IMPLEMENTED** - Now uses `isMobile()` platform utility -- **Implementation:** - ```typescript - import { isMobile } from '@/utils/platform'; - - if (await isMobile()) { - returnUrl = "https://trymaple.ai"; - } - ``` - -### Instance 2: Stripe Checkout Opening (Lines 166-186) ✅ -- **Current iOS Behavior:** Forces external Safari browser via `plugin:opener|open_url` with no fallback -- **Android Recommendation:** ✅ **Same as iOS** - Use external browser for payments -- **Reasoning:** Google Play Store has similar payment restrictions as Apple App Store -- **Status:** ✅ **IMPLEMENTED** - Now uses `isMobile()` platform utility -- **Implementation:** - ```typescript - import { isMobile } from '@/utils/platform'; - - if (await isMobile()) { - await invoke("plugin:opener|open_url", { url: checkout_url }); - return; - } else { - window.location.href = checkout_url; - } - ``` - -### Instance 3: Zaprite Checkout Opening (Lines 228-247) ✅ -- **Current iOS Behavior:** Forces external browser for crypto payments -- **Android Recommendation:** ✅ **Same as iOS** - Use external browser -- **Status:** ✅ **IMPLEMENTED** - Now uses `isMobile()` platform utility -- **Implementation:** Same as Stripe checkout - use `isMobile()` check - ---- - -## 2. PROXY SERVICE (`frontend/src/services/proxyService.ts`) ✅ COMPLETED - -### Instance 1: Desktop Platform Check (Lines 99-106) ✅ -- **Current Behavior:** Returns true only for `macos`, `windows`, `linux` -- **Android Recommendation:** ❌ **Different from iOS** - Keep as desktop-only -- **Reasoning:** Proxy functionality not needed on mobile platforms -- **Status:** ✅ **VERIFIED** - Current logic correctly excludes mobile platforms (returns `false` for both `ios` and `android`) -- **No changes needed** - The `isTauriDesktop()` method properly handles all mobile exclusions - ---- - -## 3. AUTHENTICATION (`frontend/src/routes/login.tsx` & `signup.tsx`) ✅ COMPLETED - -### Instance 1: Platform Detection ✅ -- **Current iOS Behavior:** Sets `isIOS` state when `platform === "ios"` -- **Android Recommendation:** ➕ **Add Android detection** -- **Status:** ✅ **IMPLEMENTED** - Now uses platform utility hooks -- **Implementation:** - ```typescript - import { useIsIOS, useIsAndroid, useIsTauri } from '@/hooks/usePlatform'; - - function LoginPage() { - const { isIOS } = useIsIOS(); - const { isAndroid } = useIsAndroid(); - const { isTauri: isTauriEnv } = useIsTauri(); - // No need for manual useEffect or state management - } - ``` - -### Instance 2: Email Authentication ✅ -- **Current Behavior:** Standard email/password flow -- **Android Recommendation:** ✅ **Same across all platforms** -- **Status:** ✅ **No changes needed** - Works identically on all platforms - -### Instance 3: GitHub OAuth ✅ -- **Current Behavior:** External browser for all Tauri platforms -- **Android Recommendation:** ✅ **Same as iOS** - Use external browser -- **Status:** ✅ **Already works** - Current `isTauri` check handles Android correctly -- **Implementation:** Opens `https://trymaple.ai/desktop-auth?provider=github` in external browser - -### Instance 4: Google OAuth ✅ -- **Current Behavior:** External browser for all Tauri platforms -- **Android Recommendation:** ✅ **Same as iOS** - Use external browser (no native Google Sign-In yet) -- **Status:** ✅ **Already works** - Current `isTauri` check handles Android correctly -- **Future Enhancement:** Could implement native Google Play Services integration later -- **Implementation:** Opens `https://trymaple.ai/desktop-auth?provider=google` in external browser - -### Instance 5: Apple Sign In ✅ -- **Current Behavior:** Platform-specific implementations -- **Android Recommendation:** ✅ **Same as Desktop** - Use external browser -- **Status:** ✅ **IMPLEMENTED** - Android now uses external browser like desktop -- **Implementation:** - ```typescript - // login.tsx and signup.tsx - if (isTauriEnv && isIOS) { - // Native iOS flow using plugin:sign-in-with-apple - await invoke("plugin:sign-in-with-apple|get_apple_id_credential"); - } else if (isTauriEnv) { - // Desktop and Android Tauri flow - external browser - await invoke("plugin:opener|open_url", { - url: "https://trymaple.ai/desktop-auth?provider=apple" - }); - } else { - // Web flow only - use AppleAuthProvider component - // This renders the Apple JS SDK button - } - - // Button rendering logic - {isTauriEnv ? ( - - ) : ( - // Web SDK for Web only - )} - ``` - -### Summary of Auth Behavior -- **Email**: ✅ Same on all platforms -- **GitHub**: ✅ External browser on mobile (iOS & Android) and desktop, web flow on web -- **Google**: ✅ External browser on mobile (iOS & Android) and desktop, web flow on web -- **Apple**: - - iOS: ✅ Native Apple Sign In plugin - - Android: ✅ External browser (via `/desktop-auth`) - - Desktop: ✅ External browser (via `/desktop-auth`) - - Web: ✅ AppleAuthProvider web SDK - ---- - -## 4. PRICING PAGE (`frontend/src/routes/pricing.tsx`) ✅ COMPLETED - -### Instance 1: Platform Detection ✅ -- **Current iOS Behavior:** Sets `isIOS` state -- **Android Recommendation:** ➕ **Add Android detection** -- **Status:** ✅ **IMPLEMENTED** - Now uses platform utility hooks -- **Implementation:** - ```typescript - import { useIsIOS, useIsAndroid, useIsMobile } from '@/hooks/usePlatform'; - import { isMobile } from '@/utils/platform'; - - const { isIOS } = useIsIOS(); - const { isAndroid } = useIsAndroid(); - const { isMobile: isMobilePlatform } = useIsMobile(); - ``` - -### Instance 2: Bitcoin Toggle Auto-Enable ✅ -- **Current iOS Behavior:** Prevents auto-enabling Bitcoin payments -- **Android Recommendation:** ✅ **Same as iOS** - Prevent auto-enable on mobile -- **Status:** ✅ **IMPLEMENTED** - Now checks for all mobile platforms -- **Implementation:** - ```typescript - // Auto-enable Bitcoin toggle for Zaprite users (except on mobile platforms) - useEffect(() => { - if (freshBillingStatus?.payment_provider === "zaprite" && !isMobilePlatform) { - setUseBitcoin(true); - } - }, [freshBillingStatus?.payment_provider, isMobilePlatform]); - ``` - -### Instance 3: Product Availability ✅ -- **Current iOS Behavior:** Shows "Not available in app" for paid plans when `is_available === false` -- **Android Recommendation:** ❌ **Different from iOS** - Android can support paid plans -- **Status:** ✅ **IMPLEMENTED** - iOS restrictions maintained, Android allows paid plans -- **Implementation:** - ```typescript - // Show "Not available in app" for iOS paid plans if server says not available - // Android can support paid plans (no App Store restrictions) - if (isIOS && !isFreeplan && product.is_available === false) { - return "Not available in app"; - } - // Android will allow paid plans when server returns is_available: true - ``` - -### Instance 4: Payment Success URLs ✅ -- **Current iOS Behavior:** Uses Universal Links `https://trymaple.ai/payment-success` -- **Android Recommendation:** ✅ **Same as iOS** - Use deep links for callbacks -- **Status:** ✅ **IMPLEMENTED** - Both mobile platforms use deep links -- **Implementation:** - ```typescript - // For mobile platforms (iOS and Android), use Universal Links / App Links - const isMobilePlatform = await isMobile(); - - if (isMobilePlatform) { - // Use trymaple.ai URLs for deep linking back to app - successUrl = "https://trymaple.ai/payment-success?source=stripe"; - cancelUrl = "https://trymaple.ai/payment-canceled?source=stripe"; - } else { - // Use origin URLs for web/desktop - successUrl = `${window.location.origin}/pricing?success=true`; - cancelUrl = `${window.location.origin}/pricing?canceled=true`; - } - ``` - -### Instance 5: Portal Opening ✅ -- **Current iOS Behavior:** Uses `plugin:opener|open_url` to launch Safari -- **Android Recommendation:** ✅ **Same as iOS** - Use external browser -- **Status:** ✅ **IMPLEMENTED** - Both mobile platforms use external browser -- **Implementation:** - ```typescript - if (portalUrl) { - // Open in external browser for mobile platforms (iOS and Android) - if (isMobilePlatform) { - console.log("[Billing] Mobile platform detected, using opener plugin"); - await invoke("plugin:opener|open_url", { url: portalUrl }); - } else { - // Desktop and web platforms - window.open(portalUrl, "_blank"); - } - } - ``` - -### Instance 6: Product Fetching with Version ✅ -- **Current iOS Behavior:** Sends app version to server for iOS builds -- **Android Recommendation:** ✅ **Same as iOS** - Send version for mobile builds -- **Status:** ✅ **IMPLEMENTED** - Both mobile platforms send version -- **Implementation:** - ```typescript - queryKey: ["products", isIOS, isAndroid], - queryFn: async () => { - const billingService = getBillingService(); - // Send version for mobile builds (iOS needs it for App Store restrictions) - if (isIOS || isAndroid) { - const version = `v${packageJson.version}`; - return await billingService.getProducts(version); - } - return await billingService.getProducts(); - } - ``` - ---- - -## 5. ACCOUNT MENU (`frontend/src/components/AccountMenu.tsx`) ✅ COMPLETED - -### Instance 1: API Management Visibility (Lines 276-281) ✅ -- **Current iOS Behavior:** Hides API Management menu item -- **Android Recommendation:** ✅ **Same as iOS** - Hide API Management on mobile -- **Reasoning:** Mobile platforms don't need API management features -- **Status:** ✅ **IMPLEMENTED** - Now uses `useIsMobile()` hook -- **Implementation:** - ```typescript - import { useIsMobile } from '@/hooks/usePlatform'; - - const { isMobile } = useIsMobile(); - - // Hide for all mobile platforms (iOS and Android) - {!isMobile && ( - API Management - )} - ``` - -### Instance 2: Manage Subscription (Lines 155-178) ✅ -- **Current iOS Behavior:** Uses external browser via opener plugin -- **Android Recommendation:** ✅ **Same as iOS** - Use external browser -- **Status:** ✅ **IMPLEMENTED** - Now uses `isMobile()` platform utility -- **Implementation:** - ```typescript - import { isMobile } from '@/utils/platform'; - - if (await isMobile()) { - await invoke("plugin:opener|open_url", { url: portalUrl }); - } else { - window.open(url, "_blank"); - } - ``` - ---- - -## 6. TEAM INVITES (`frontend/src/components/team/TeamInviteDialog.tsx`) ✅ COMPLETED - -### Instance 1: Billing Portal (Lines 48-55) ✅ -- **Current iOS Behavior:** Uses `plugin:opener|open_url` for external browser -- **Android Recommendation:** ✅ **Same as iOS** - Use external browser -- **Status:** ✅ **IMPLEMENTED** - Now uses `isMobile()` platform utility -- **Implementation:** - ```typescript - import { isMobile } from '@/utils/platform'; - import { invoke } from '@tauri-apps/api/core'; - - // Use external browser for mobile platforms (iOS and Android) - if (await isMobile()) { - await invoke("plugin:opener|open_url", { url }); - return; - } - - // Web or desktop flow - window.open(url, "_blank", "noopener,noreferrer"); - ``` - ---- - -## 7. MARKETING (`frontend/src/components/Marketing.tsx`) ✅ COMPLETED - -### Instance 1: Platform Detection ✅ -- **Current iOS Behavior:** Sets `isIOS` state when `platform === "ios"` -- **Android Recommendation:** ✅ **Uses platform hook** - No Android-specific detection needed -- **Status:** ✅ **IMPLEMENTED** - Now uses `useIsIOS()` hook from platform utilities -- **Implementation:** - ```typescript - import { useIsIOS } from '@/hooks/usePlatform'; - - export function Marketing() { - // Use the platform detection hook for iOS - // Android doesn't have App Store restrictions, so we only need to check for iOS - const { isIOS } = useIsIOS(); - // No manual useEffect or state management needed - } - ``` - -### Instance 2: Pricing Tier Buttons (Lines 141-152) ✅ -- **Current iOS Behavior:** Shows "Coming Soon" for paid plans -- **Android Recommendation:** ❌ **Different from iOS** - Enable paid plans -- **Reasoning:** Android doesn't have App Store payment restrictions -- **Status:** ✅ **IMPLEMENTED** - iOS shows "Coming Soon", Android shows normal "Get Started" -- **Implementation:** - ```typescript - // In PricingTier component: - // Only iOS shows "Coming Soon" for paid plans - {isIOS && !isFreeplan ? ( - - ) : ( - - )} - ``` - ---- - -## 8. API CREDITS (`frontend/src/components/apikeys/ApiCreditsSection.tsx`) ✅ COMPLETED - -### Instance 1: Payment URLs (Lines 98-112) ✅ -- **Current iOS Behavior:** Uses Universal Links for payment callbacks -- **Android Recommendation:** ✅ **Same as iOS** - Use deep links -- **Status:** ✅ **IMPLEMENTED** - Now uses `useIsMobile()` hook -- **Implementation:** - ```typescript - import { useIsMobile } from '@/hooks/usePlatform'; - - const { isMobile } = useIsMobile(); - - // For mobile platforms (iOS and Android), use Universal Links - if (isMobile) { - successUrl = `https://trymaple.ai/payment-success-credits?source=${method}`; - cancelUrl = method === "stripe" ? `https://trymaple.ai/payment-canceled?source=stripe` : undefined; - } else { - // For web or desktop, use regular URLs with query params - const baseUrl = window.location.origin; - successUrl = `${baseUrl}/?credits_success=true`; - cancelUrl = method === "stripe" ? `${baseUrl}/` : undefined; - } - ``` - -### Instance 2: Feature Availability ✅ -- **Current iOS Behavior:** Feature hidden on mobile platforms -- **Android Recommendation:** ✅ **Same as iOS** - Hide feature on mobile -- **Reasoning:** API credits feature not exposed on mobile platforms -- **Status:** ✅ **VERIFIED** - Feature not accessible on mobile platforms -- **Note:** The page/feature is not accessible on mobile, so the component won't be rendered - ---- - -## 9. CHAT BOX (`frontend/src/components/ChatBox.tsx`) ✅ COMPLETED - -### Instance 1: Document Processing (Line 410) ✅ -- **Current Behavior:** Tauri environments support PDF processing -- **Android Recommendation:** ✅ **Same as iOS** - Support local document processing -- **Status:** ✅ **IMPLEMENTED** - Now uses `useIsTauri()` hook from `@/hooks/usePlatform` -- **Implementation:** - ```typescript - import { useIsTauri } from '@/hooks/usePlatform'; - - const { isTauri: isTauriEnv } = useIsTauri(); - - // All Tauri platforms (desktop and mobile) support document processing - if (isTauriEnv && (file.type === "application/pdf" || ...)) { - // Process documents locally using Rust in Tauri - const { invoke } = await import("@tauri-apps/api/core"); - // ... document processing - } - ``` -- **Verified:** Android will work correctly since it's a Tauri environment -- **Note:** This was primarily a refactoring change - the functionality already worked for Android - ---- - -## 10. PROXY CONFIGURATION (`frontend/src/components/apikeys/ProxyConfigSection.tsx`) ✅ COMPLETED - -### Instance 1: Component Visibility (Lines 28-40) ✅ -- **Current Behavior:** Only shows on desktop platforms -- **Android Recommendation:** ✅ **Same as iOS** - Hide proxy config -- **Reasoning:** Proxy not needed on mobile -- **Status:** ✅ **IMPLEMENTED** - Now uses `useIsTauriDesktop()` hook -- **Implementation:** - ```typescript - import { useIsTauriDesktop } from '@/hooks/usePlatform'; - - export function ProxyConfigSection({ apiKeys, onRequestNewApiKey }: ProxyConfigSectionProps) { - const { isTauriDesktop } = useIsTauriDesktop(); - - if (!isTauriDesktop) { - return null; // Don't show proxy config on non-desktop platforms (includes mobile) - } - } - ``` - ---- - -## 11. API KEY DASHBOARD (`frontend/src/components/apikeys/ApiKeyDashboard.tsx`) ✅ COMPLETED - -### Instance 1: Proxy Tab Visibility (Lines 215-230) ✅ -- **Current Behavior:** Shows proxy tab only on desktop -- **Android Recommendation:** ✅ **Same as iOS** - Hide proxy tab -- **Status:** ✅ **IMPLEMENTED** - Now uses `useIsTauriDesktop()` hook -- **Implementation:** - ```typescript - import { useIsTauriDesktop } from '@/hooks/usePlatform'; - - export function ApiKeyDashboard() { - const { isTauriDesktop } = useIsTauriDesktop(); - - // Only show proxy tab on desktop - - {/* Credits and API Keys tabs always shown */} - {isTauriDesktop && ( - Local Proxy - )} - - - // Conditionally render proxy content - {isTauriDesktop && ( - - - - )} - } - ``` - ---- - -## 12. AUTH CALLBACK (`frontend/src/routes/auth.$provider.callback.tsx`) ✅ COMPLETED - -### Instance 1: Native App Redirect (Lines 38-59) ✅ -- **Current Behavior:** Checks `redirect-to-native` flag for deep linking -- **Android Recommendation:** ✅ **Same as iOS** - Use native app flow -- **Status:** ✅ **VERIFIED** - Works correctly for Android -- **Implementation:** - ```typescript - // Platform-agnostic approach works for both iOS and Android - const isTauriAuth = localStorage.getItem("redirect-to-native") === "true"; - - if (isTauriAuth) { - // Deep link back to app with auth tokens - window.location.href = `cloud.opensecret.maple://auth?access_token=...`; - } - ``` -- **Notes:** - - Uses localStorage flag approach instead of platform detection - - This design is intentionally platform-agnostic - - No platform utility imports needed in this component - ---- - -## 13. APPLE AUTH PROVIDER (`frontend/src/components/AppleAuthProvider.tsx`) ✅ COMPLETED - -### Instance 1: Component Rendering (Lines 95, 393) ✅ -- **Current Behavior:** Returns null for ALL Tauri environments (`window.location.protocol === "tauri:"`) -- **Android Recommendation:** ✅ **Working correctly** - Component not used by mobile/desktop apps -- **Status:** ✅ **NO CHANGES NEEDED** - Working as designed -- **Explanation:** - - This component is **ONLY for the Web JS SDK** (Apple's browser-based auth) - - It correctly returns `null` for ALL Tauri platforms (iOS, Android, Desktop) - - Mobile and desktop apps handle Apple auth differently: - - **iOS**: Uses native Apple Sign In plugin directly - - **Android/Desktop**: Opens external browser to `/desktop-auth?provider=apple` - - The external browser then loads the web page which DOES use this component - - The check `window.location.protocol === "tauri:"` is intentionally platform-agnostic -- **How it works:** - ```typescript - // AppleAuthProvider.tsx - Web SDK component - if (window.location.protocol === "tauri:") { - return null; // Don't render in ANY Tauri app - } - - // Login/Signup pages handle the platform routing: - if (isTauriEnv) { - // Show custom button that: - // - iOS: calls native plugin - // - Android/Desktop: opens external browser - } else { - // Web only: render AppleAuthProvider component - - } - ``` - ---- - -## 14. DEEP LINK HANDLER (`frontend/src/components/DeepLinkHandler.tsx`) ✅ COMPLETED - -### Instance 1: Deep Link Events (Lines 31-84) ✅ -- **Current Behavior:** Sets up listeners in Tauri environments -- **Android Recommendation:** ✅ **Same as iOS** - Handle deep links -- **Status:** ✅ **IMPLEMENTED** - Now uses platform utilities -- **Implementation:** - ```typescript - import { isTauri } from '@/utils/platform'; - import { listen } from '@tauri-apps/api/event'; - - useEffect(() => { - const setupDeepLinks = async () => { - if (await isTauri()) { - // Set up deep link listeners for both iOS and Android - const unlisten = await listen('deep-link-received', (event) => { - // Handle deep links - }); - } - }; - setupDeepLinks(); - }, []); - ``` - ---- - -## DEEP LINKING CONFIGURATION - -### Overview -Deep linking is critical for handling OAuth callbacks, payment redirects, and team invites. iOS uses Universal Links (HTTPS) and custom URL schemes. Android needs equivalent App Links and custom schemes. - -### Current iOS Configuration - -#### 1. Universal Links (HTTPS Links) -**File**: `/frontend/src-tauri/gen/apple/maple_iOS/maple_iOS.entitlements` -```xml -com.apple.developer.associated-domains - - applinks:trymaple.ai - -``` - -**Server Requirements**: -- Host `.well-known/apple-app-site-association` at https://trymaple.ai -- This file tells iOS which paths should open in the app - -#### 2. Custom URL Scheme -**File**: `/frontend/src-tauri/gen/apple/maple_iOS/Info.plist` -```xml -CFBundleURLTypes - - - CFBundleURLName - cloud.opensecret.maple - CFBundleURLSchemes - - cloud.opensecret.maple - - - -``` - -#### 3. Tauri Configuration -**File**: `/frontend/src-tauri/tauri.conf.json` -```json -"deep-link": { - "desktop": { - "schemes": ["cloud.opensecret.maple"] - }, - "mobile": [{ - "host": "trymaple.ai" - }] -} -``` - -### Android Requirements (To Be Implemented) - -#### 1. App Links (HTTPS Links - Android Equivalent of Universal Links) - -**AndroidManifest.xml** additions needed: -```xml - - - - - - - - -``` - -**Server Requirements**: -- Host `.well-known/assetlinks.json` at https://trymaple.ai -- Example content: -```json -[{ - "relation": ["delegate_permission/common.handle_all_urls"], - "target": { - "namespace": "android_app", - "package_name": "cloud.opensecret.maple", - "sha256_cert_fingerprints": ["YOUR_APP_SIGNING_CERT_SHA256"] - } -}] -``` - -#### 2. Custom URL Scheme -**AndroidManifest.xml** additions: -```xml - - - - - - - - -``` - -### Deep Link Flow in the App - -#### Current Implementation (`DeepLinkHandler.tsx`) -The app listens for deep links and handles: -1. **Auth callbacks**: `cloud.opensecret.maple://auth?access_token=...&refresh_token=...` -2. **Payment success**: `https://trymaple.ai/payment-success?source=stripe` -3. **Payment canceled**: `https://trymaple.ai/payment-canceled?source=stripe` - -#### Code Flow: -```typescript -// DeepLinkHandler.tsx -listen("deep-link-received", (event) => { - const url = event.payload; - // Parse and handle auth tokens, payment callbacks, etc. -}); -``` - -### URLs That Need Deep Link Support - -| Flow | iOS URL | Android URL (Should Be Same) | Purpose | -|------|---------|------------------------------|---------| -| **Stripe Payment Success** | `https://trymaple.ai/payment-success?source=stripe` | Same | Return from Stripe checkout | -| **Stripe Payment Cancel** | `https://trymaple.ai/payment-canceled?source=stripe` | Same | Canceled Stripe payment | -| **Zaprite Success** | `https://trymaple.ai/payment-success?source=zaprite` | Same | Bitcoin payment success | -| **API Credits Success** | `https://trymaple.ai/payment-success-credits?source=stripe` | Same | API credit purchase | -| **OAuth Callback** | `cloud.opensecret.maple://auth?access_token=...` | Same | OAuth provider returns | -| **Desktop Auth** | `https://trymaple.ai/desktop-auth?provider=...` | Same | Desktop OAuth flow | - -### Current Status of Android Deep Linking ✅ FULLY WORKING - -#### ✅ What's Completed: -1. **HTTPS App Links intent filter** - AndroidManifest has `https://trymaple.ai` configured -2. **Custom URL Scheme** - `cloud.opensecret.maple://` intent filter in AndroidManifest -3. **Rust deep link handler** - Set up in `lib.rs` to emit events to frontend -4. **Frontend listener** - `DeepLinkHandler.tsx` listens for `deep-link-received` events -5. **Payment/auth URLs** - Using `isMobile()` for correct URL generation -6. **Digital Asset Links** - `assetlinks.json` deployed with upload key SHA256 -7. **App verification** - Android verified ownership of trymaple.ai domain -8. **Testing** - Both custom scheme and HTTPS deep links tested and working - -#### ✅ Deep Linking Test Results: -- **Custom scheme** (`cloud.opensecret.maple://`) - Working perfectly -- **HTTPS links** (`https://trymaple.ai/*`) - Auto-verified, no chooser dialog -- **OAuth callbacks** - Ready for GitHub, Google, Apple authentication -- **Payment redirects** - Stripe/Zaprite success/cancel URLs working - -### Implementation Steps for Android - -#### 1. Configure Android Manifest ✅ COMPLETED -- [x] ✅ App Links intent filter for `https://trymaple.ai` (already existed) -- [x] ✅ Custom scheme intent filter for `cloud.opensecret.maple` (just added) -- [x] ✅ `android:launchMode="singleTask"` already set - -#### 2. Digital Asset Links Configuration ✅ COMPLETED -- [x] ✅ Created `.well-known/assetlinks.json` file with upload key SHA256 -- [x] ✅ Deployed to https://trymaple.ai/.well-known/assetlinks.json -- [x] ✅ Android verified domain ownership automatically -- [x] ✅ HTTPS deep links tested and working without chooser dialog - -#### 3. Update Payment/Auth Flows ✅ COMPLETED -- [x] ✅ All instances updated to use `isMobile()` platform utilities -- [x] ✅ Callback URLs use `https://trymaple.ai` for mobile platforms -- [ ] Test OAuth flows (GitHub, Google, Apple) - needs device testing -- [ ] Test payment flows (Stripe, Zaprite) - needs device testing - -#### 4. Native Android Code ✅ ALREADY WORKING -- [x] ✅ Intent handling via Tauri's deep link plugin -- [x] ✅ Deep links passed to WebView/Tauri runtime -- [x] ✅ "deep-link-received" event emitted to JavaScript - -### Testing Deep Links - -#### Android Testing Commands: -```bash -# Test custom scheme -adb shell am start -W -a android.intent.action.VIEW \ - -d "cloud.opensecret.maple://auth?access_token=test" \ - cloud.opensecret.maple - -# Test App Links (HTTPS) -adb shell am start -W -a android.intent.action.VIEW \ - -d "https://trymaple.ai/payment-success?source=stripe" \ - cloud.opensecret.maple -``` - -#### Verification Tools: -1. Android App Links Assistant (in Android Studio) -2. Digital Asset Links API Validator -3. Chrome DevTools for testing web-to-app navigation - -### Security Considerations - -1. **App Links Verification**: Android automatically verifies App Links ownership via assetlinks.json -2. **Certificate Pinning**: Consider pinning the SHA256 cert in assetlinks.json -3. **Token Handling**: Ensure auth tokens in deep links are: - - One-time use - - Short-lived - - Properly validated - ---- - -## Android App Signing & Distribution (Tauri v2) - -### Overview -This section covers how to properly sign and distribute your Android app both through Google Play Store and as direct APK downloads, including setting up deep linking to work with both distribution methods. - -### Understanding Android Signing Keys - -#### What You Need to Know: -- **Upload Key**: The keystore you create and manage. Signs APKs you upload to Google Play or distribute directly. -- **App Signing Key**: Google's key (if using Play App Signing). Google re-signs your app with this for Play Store distribution. -- **SHA256 Fingerprint**: A public identifier derived from a key. Safe to share, goes in `assetlinks.json`. - -#### Key Security: -- 🔒 **KEEP SECRET**: Keystore files (.jks), passwords -- ✅ **PUBLIC/SAFE**: SHA256 fingerprints, certificates in APKs -- 📝 **NEVER COMMIT**: keystore.properties, *.jks files - -### Step 1: Create Your Upload Keystore ✅ COMPLETED - -```bash -# Create a new keystore for your app (run this once) -keytool -genkey -v \ - -keystore ~/maple-android-upload.jks \ - -keyalg RSA \ - -keysize 2048 \ - -validity 10000 \ - -alias upload - -# You'll be prompted for: -# - Keystore password (remember this!) -# - Your name, organization, etc. (can use generic values like "Unknown") -# - Key password (can be same as keystore password) -``` - -**Important**: -- Store this file somewhere safe (NOT in your git repo) -- Back it up! If you lose it and aren't using Play App Signing, you can't update your app -- Remember the password and alias (upload) - -**Status**: ✅ Created `~/maple-android-upload.jks` and backed up - -### Step 2: Configure Tauri for Signing ✅ COMPLETED - -Create `frontend/src-tauri/gen/android/keystore.properties`: -```properties -password=your-keystore-password -keyAlias=upload -storeFile=/absolute/path/to/maple-android-upload.jks -``` - -**Add to `.gitignore`**: -```gitignore -# Android signing -*.jks -*.keystore -keystore.properties -``` - -**Status**: -- ✅ Created `keystore.properties` with actual password -- ✅ Updated `.gitignore` to exclude sensitive files -- ✅ Added signing configuration to `build.gradle.kts` - -### Step 3: Build Signed Apps ✅ COMPLETED - -```bash -# For development/testing (uses debug keystore automatically) -bun tauri android dev - -# For release APK (direct distribution) -bun tauri android build - -# For release AAB (Google Play Store) -bun tauri android build -- --aab -``` - -Output locations: -- **APK**: `frontend/src-tauri/gen/android/app/build/outputs/apk/universal/release/app-universal-release.apk` -- **AAB**: `frontend/src-tauri/gen/android/app/build/outputs/bundle/universalRelease/app-universal-release.aab` - -**Status**: ✅ Successfully built signed APK and AAB, installed on physical device - -### Step 4: Get SHA256 Fingerprints ✅ COMPLETED - -```bash -# For your upload keystore (direct APK distribution) -keytool -list -v \ - -keystore ~/maple-android-upload.jks \ - -alias upload \ - | grep SHA256 - -# Output: -# SHA256: C6:12:09:59:0A:27:73:F9:EA:EC:80:0A:C1:09:07:54:4A:56:6C:62:A5:68:7D:DF:9D:B3:DE:91:19:E4:3B:2A - -# For debug builds (testing only) -keytool -list -v \ - -keystore ~/.android/debug.keystore \ - -alias androiddebugkey \ - -storepass android \ - -keypass android \ - | grep SHA256 -``` - -If using Google Play App Signing: -1. Upload your first AAB to Google Play Console -2. Go to: Setup → App Integrity → App signing -3. Copy the "SHA-256 certificate fingerprint" shown there - -### Step 5: Create assetlinks.json for Deep Linking ✅ COMPLETED - -Created `frontend/public/.well-known/assetlinks.json`: - -```json -[{ - "relation": ["delegate_permission/common.handle_all_urls"], - "target": { - "namespace": "android_app", - "package_name": "cloud.opensecret.maple", - "sha256_cert_fingerprints": [ - "C6:12:09:59:0A:27:73:F9:EA:EC:80:0A:C1:09:07:54:4A:56:6C:62:A5:68:7D:DF:9D:B3:DE:91:19:E4:3B:2A" - ] - } -}] -``` - -**Status**: ✅ Deployed to https://trymaple.ai/.well-known/assetlinks.json - -**Current Configuration**: -- ✅ Contains upload key SHA256 only (for direct APK distribution) -- ⏳ **TODO: Google Play SHA256** - Needs to be added when Play Store is configured - -**Next Steps for Google Play Store**: -1. Create app in Google Play Console -2. Upload first AAB: `bun tauri android build -- --aab` -3. Enable Play App Signing (recommended) -4. Get Google's SHA256 from Console (Setup → App Integrity → App signing) -5. Update `assetlinks.json` to include both fingerprints: - ```json - "sha256_cert_fingerprints": [ - "C6:12:09:59:0A:27:73:F9:EA:EC:80:0A:C1:09:07:54:4A:56:6C:62:A5:68:7D:DF:9D:B3:DE:91:19:E4:3B:2A", // Upload key - "GOOGLE_PLAY_SHA256_HERE" // Google's signing key - ] - ``` -6. Redeploy assetlinks.json to server - -### Step 6: Distribution Strategy - -#### Option A: Google Play Store Only -1. Build AAB: `bun tauri android build -- --aab` -2. Upload to Play Console -3. Enable Play App Signing (recommended) -4. Get SHA256 from Play Console -5. Add only Google's SHA256 to assetlinks.json - -#### Option B: Direct APK Only -1. Build APK: `bun tauri android build` -2. Distribute app-universal-release.apk -3. Get SHA256 from your keystore -4. Add only your SHA256 to assetlinks.json - -#### Option C: Both (Recommended) -1. Upload AAB to Play Store with Play App Signing -2. Also distribute APK directly -3. Add BOTH SHA256s to assetlinks.json -4. Both distribution methods work with deep links! - -### Testing Deep Links ✅ VERIFIED WORKING - -Once the APK is installed (for multiple devices, use `-s `): - -```bash -# Test custom scheme (works immediately, no assetlinks needed) -adb -s shell am start -W -a android.intent.action.VIEW \ - -d "cloud.opensecret.maple://auth?access_token=test" \ - cloud.opensecret.maple - -# Test HTTPS links (verified working with deployed assetlinks.json) -adb -s shell am start -W -a android.intent.action.VIEW \ - -d "https://trymaple.ai/payment-success?source=stripe" \ - cloud.opensecret.maple -``` - -**Status**: ✅ Both custom scheme and HTTPS deep links tested and working on physical device - -**Note**: Android may cache App Links verification. If updating assetlinks.json, force re-verification: -```bash -adb shell pm clear-app-links cloud.opensecret.maple -adb shell pm verify-app-links --re-verify cloud.opensecret.maple -``` - -### Common Issues & Solutions - -#### "App isn't verified" dialog for HTTPS links -- **Cause**: assetlinks.json missing or wrong SHA256 -- **Fix**: Ensure SHA256 in assetlinks.json matches your signing certificate -- **Workaround**: Users can tap "Open in app anyway" and select "Always" - -#### Can't update app after reinstall -- **Cause**: Different signing keys (e.g., debug vs release) -- **Fix**: Uninstall first, then install the new version - -#### Lost keystore file -- **If using Play App Signing**: Contact Google Play support to reset upload key -- **If NOT using Play App Signing**: You cannot update the app anymore -- **Prevention**: Always use Play App Signing for Play Store releases - -### CI/CD Setup (GitHub Actions Example) - -Store in GitHub Secrets: -- `ANDROID_KEYSTORE_BASE64`: `base64 -i maple-upload-keystore.jks` -- `ANDROID_KEY_PASSWORD`: Your keystore password -- `ANDROID_KEY_ALIAS`: `upload` - -`.github/workflows/android.yml`: -```yaml -- name: Setup Android signing - run: | - cd frontend/src-tauri/gen/android - echo "${{ secrets.ANDROID_KEYSTORE_BASE64 }}" | base64 -d > keystore.jks - echo "password=${{ secrets.ANDROID_KEY_PASSWORD }}" > keystore.properties - echo "keyAlias=${{ secrets.ANDROID_KEY_ALIAS }}" >> keystore.properties - echo "storeFile=$(pwd)/keystore.jks" >> keystore.properties - -- name: Build signed APK - run: bun tauri android build -``` - -### Version Code Management - -Android `versionCode` is not user-visible; `versionName`/Tauri `version` is what users see. -Google Play only requires that each uploaded build uses a higher `versionCode` than previous uploads -and stays at or below `2100000000`. - -Set the override in `tauri.conf.json`: -```json -{ - "bundle": { - "android": { - "versionCode": 42 - } - } -} -``` - ---- - -## Google Play Store Setup ✅ IN PROGRESS - -### Prerequisites Completed: -- ✅ Upload keystore created and configured -- ✅ Signed AAB can be built with `bun tauri android build --aab` -- ✅ Deep linking configured with both SHA256 fingerprints -- ✅ App tested on physical device -- ✅ Google Play Developer Account created -- ✅ App created in Play Console ("Maple AI") -- ✅ First AAB uploaded to internal testing -- ✅ Play App Signing enabled -- ✅ Google's SHA256 certificate obtained -- ✅ assetlinks.json updated with both certificates -- ✅ Privacy policy URL configured (https://opensecret.cloud/privacy) -- ✅ API level 35 requirement met - -### Current Status: -- **Internal Testing Release**: Version 1.3.2 (Build 1003002002) is live -- **Package Name**: cloud.opensecret.maple -- **Google's App Signing SHA256**: `36:B3:1C:A3:CC:DD:CA:9A:DD:47:8A:8F:86:70:DB:11:E3:56:E7:90:09:6E:CC:7D:8C:43:38:F4:55:13:B1:0A` -- **Upload Key SHA256**: `C6:12:09:59:0A:27:73:F9:EA:EC:80:0A:C1:09:07:54:4A:56:6C:62:A5:68:7D:DF:9D:B3:DE:91:19:E4:3B:2A` - -### Version Code Management: -To avoid conflicts with Tauri's automatic version code generation and Google Play's -`2100000000` cap, Android builds use a simple sequential `versionCode`: -- Increment by one for each Play Store upload -- Keep the user-facing app version in Tauri `version` -- Configured in `tauri.conf.json` under `bundle.android.versionCode` - -### Still Required for Full Release: -1. **Add Testers to Internal Testing Track** -2. **Complete Content Rating Questionnaire** -3. **Complete Store Listing** - - App description (short & full) - - Screenshots for different device sizes - - Feature graphic (1024x500) - - App icon (512x512) -4. **Test Deep Linking and OAuth Flows** -5. **Promote to Production** - - Submit for review - - First review may take several days - ---- - -## Android CI/CD Build Configuration - -### Build Warnings and Version Compatibility - -#### Current Warnings Observed: -1. **Gradle Plugin Version**: - ``` - WARNING: We recommend using a newer Android Gradle plugin to use compileSdk = 36 - This Android Gradle plugin (8.5.1) was tested up to compileSdk = 34. - ``` - - **Impact**: Build succeeds but may have compatibility issues - - **TODO**: Wait for Tauri to update their Android template or manually update Gradle plugin - -2. **Deprecated targetSdk in library DSL**: - ``` - 'targetSdk: Int?' is deprecated. Will be removed from library DSL in v9.0 - ``` - - **Location**: tauri-plugin-fs Android build.gradle.kts - - **Impact**: Warning only, will need update when Gradle 9.0 releases - - **TODO**: Wait for Tauri plugin updates - -3. **Unused Rust Functions**: - ``` - warning: function `get_config_path` is never used - warning: function `save_proxy_config` is never used - ``` - - **Impact**: No runtime impact, just compilation warnings - - **TODO**: Clean up unused proxy functions or add #[cfg] attributes for desktop-only - -#### Version Information: -- **Current Gradle**: 8.9 -- **Android Gradle Plugin**: 8.5.1 -- **compileSdk**: 35 (Updated for Google Play requirements) -- **targetSdk**: 35 (Required by Google Play as of Aug 2025) -- **minSdk**: 24 (Android 7.0+, covers 99%+ of devices) -- **NDK**: r25c - -#### Build Performance: -- **Initial build time**: ~9-10 minutes (includes downloading dependencies) -- **Cached build time**: ~4-5 minutes -- **Architectures built**: arm64-v8a, armeabi-v7a, x86_64, x86 - -### GitHub Actions Workflow Status - -#### ✅ Completed: -1. **Android SDK and Java setup** -2. **NDK installation and configuration** -3. **Rust targets for all Android architectures** -4. **Cross-compilation environment variables** -5. **Keystore configuration for signing** -6. **Comprehensive caching**: - - Bun/Node dependencies - - Gradle cache - - Rust compilation cache - - Cargo registry and binaries - - APT packages - -#### 🔧 Known Issues Fixed: -1. **OpenSSL cross-compilation** - Fixed by setting up proper NDK paths and ranlib symlinks -2. **Missing i686 target** - Added i686-linux-android to Rust targets -3. **NDK toolchain not found** - Fixed by adding NDK bin to PATH and setting env vars - -#### 📋 TODO for Production: -1. **Update Gradle Plugin** when Tauri updates templates -2. **Google Play Store Setup**: - - Create developer account - - Configure Play App Signing - - Add Google's SHA256 to assetlinks.json -3. **Version Management**: - - Implement automatic version bumping - - Consider using semantic-release -4. **Testing**: - - Add Android emulator tests - - Implement UI testing with Espresso -5. **Optimization**: - - Consider using `--target` flags to build specific architectures only - - Implement APK size optimization - - Add ProGuard/R8 rules if needed - -### Secrets Required for CI/CD - -Add these to GitHub repository secrets: -- `ANDROID_KEYSTORE_BASE64` - Base64 encoded keystore file -- `ANDROID_KEY_ALIAS` - Key alias (usually "upload") -- `ANDROID_KEY_PASSWORD` - Keystore password - ---- - -## Summary of Android Behavior Patterns - -### ✅ Same as iOS (Mobile Behavior) -1. External browser for payments/billing -2. Deep linking for auth callbacks -3. Document processing capabilities -4. No proxy functionality -5. Return URLs using trymaple.ai domain - -### ❌ Different from iOS (Android-Specific) -1. **Enable paid plans** - No App Store payment restrictions -2. **Show API Management** - No restrictions on API key features -3. **Apple Sign In** - Use web flow, not native -4. **Product availability** - Server should return `is_available: true` - -### 🔄 Future Enhancements -1. Native Google Sign-In using Google Play Services -2. Android-specific payment integration options - ---- - -## Implementation Checklist - -### High Priority (Required for Basic Android Support) -- [x] ✅ Add Android platform detection alongside iOS -- [x] ✅ Update billing API to treat Android like iOS for external browser -- [x] ✅ Ensure deep linking works for auth callbacks -- [ ] Configure opener plugin for Android in Tauri (platform-side config needed) - -### Medium Priority (Feature Parity) -- [ ] Enable paid plans on Android (server-side changes) -- [x] ✅ Show API Management features on Android (hidden on mobile) -- [x] ✅ Handle Apple auth with external browser on Android -- [x] ✅ Document processing support verified for Android - -### Low Priority (Enhancements) -- [ ] Consider native Google Sign-In integration -- [ ] Optimize Android-specific UI elements -- [ ] Add Android-specific telemetry - ---- - -## Platform Detection - NEW UNIFIED DESIGN - -### CRITICAL ISSUES WITH CURRENT IMPLEMENTATION -The current platform detection has fundamental flaws that must be fixed: - -1. **INCORRECT VALUES DURING LOADING**: Hooks return `false` while detecting platform, causing: - - Tauri-only code running on web (CRASHES) - - Web-only code running in Tauri (CRASHES) - - Wrong UI showing temporarily (flickering) - -2. **MULTIPLE CONFLICTING APIS**: - - Async functions: `await isIOS()` - - React hooks: `useIsIOS()` returning potentially wrong values - - Both can be incorrect during initialization - -3. **NO SINGLE SOURCE OF TRUTH**: Platform can be detected differently in different parts of the app - -### NEW UNIFIED PLATFORM DETECTION DESIGN - -Platform detection happens ONCE before the app renders, guaranteeing correctness: - -#### Implementation (platform.ts): -```typescript -// Platform info - NOT nullable, ALWAYS set before app renders -let platformInfo: PlatformInfo; - -// Initialize immediately when module loads -const platformReady = (async () => { - try { - const tauriEnv = await import("@tauri-apps/api/core") - .then(m => m.isTauri()) - .catch(() => false); - - if (tauriEnv) { - const { type } = await import("@tauri-apps/plugin-os"); - const platform = await type(); - platformInfo = { - platform, - isTauri: true, - isIOS: platform === "ios", - isAndroid: platform === "android", - isMobile: platform === "ios" || platform === "android", - isDesktop: platform === "macos" || platform === "windows" || platform === "linux", - isMacOS: platform === "macos", - isWindows: platform === "windows", - isLinux: platform === "linux", - isWeb: false, - isTauriDesktop: true && (platform === "macos" || platform === "windows" || platform === "linux"), - isTauriMobile: true && (platform === "ios" || platform === "android") - }; - } else { - platformInfo = { - platform: "web", - isTauri: false, - isIOS: false, - isAndroid: false, - isMobile: false, - isDesktop: false, - isMacOS: false, - isWindows: false, - isLinux: false, - isWeb: true, - isTauriDesktop: false, - isTauriMobile: false - }; - } - } catch { - // Default to web on any error - platformInfo = { - platform: "web", - isTauri: false, - isIOS: false, - isAndroid: false, - isMobile: false, - isDesktop: false, - isMacOS: false, - isWindows: false, - isLinux: false, - isWeb: true, - isTauriDesktop: false, - isTauriMobile: false - }; - } -})(); - -// Export for main.tsx to await -export const waitForPlatform = () => platformReady; - -// Simple, synchronous, ALWAYS correct -export function isIOS(): boolean { - return platformInfo.isIOS; -} - -export function isAndroid(): boolean { - return platformInfo.isAndroid; -} - -export function isMobile(): boolean { - return platformInfo.isMobile; -} - -export function isTauri(): boolean { - return platformInfo.isTauri; -} - -export function isDesktop(): boolean { - return platformInfo.isDesktop; -} - -export function isWeb(): boolean { - return platformInfo.isWeb; -} - -// Get full platform info if needed -export function getPlatformInfo(): PlatformInfo { - return platformInfo; -} -``` - -#### App Initialization (main.tsx): -```typescript -import { waitForPlatform } from '@/utils/platform'; - -// Platform MUST be ready before rendering -await waitForPlatform(); - -// NOW platform is guaranteed correct - no loading states, no wrong values -createRoot(document.getElementById("root")!).render( - - - -); -``` - -#### Usage in Components - SIMPLE AND ALWAYS CORRECT: -```typescript -import { isIOS, isAndroid, isMobile, isTauri } from '@/utils/platform'; - -// Direct usage - ALWAYS correct, never wrong, no await needed -function MyComponent() { - if (isMobile()) { - // This is GUARANTEED correct - no loading states - return ; - } - - if (isTauri()) { - // SAFE to use Tauri APIs - will NEVER run on web - const { invoke } = await import("@tauri-apps/api/core"); - await invoke("some_command"); - } - - return ; -} - -// No more hooks with loading states! -function PricingComponent() { - // Just use the functions directly - they're always correct - if (isIOS()) { - return ; - } - - if (isAndroid()) { - return ; - } - - return ; -} -``` - -### BENEFITS OF NEW DESIGN: -1. **SINGLE API**: Just `isIOS()`, `isMobile()`, etc. No hooks, no await, no variants -2. **ALWAYS CORRECT**: Platform is detected before app exists, can NEVER be wrong -3. **INSTANT**: All checks are synchronous after initialization -4. **CRASH-PROOF**: Tauri APIs only called when definitely in Tauri, web APIs only on web -5. **SIMPLE**: No loading states, no undefined handling, no complexity -6. **NO FLICKERING**: UI is correct from first render - -### MIGRATION PLAN: -1. Update `platform.ts` with new implementation -2. Add `await waitForPlatform()` to `main.tsx` -3. Remove ALL `useIsIOS()`, `useIsMobile()` etc. hooks -4. Remove ALL `await isIOS()` async calls -5. Replace with direct `isIOS()`, `isMobile()` function calls -6. Delete the hooks file entirely - no longer needed - -## Platform Utility Migration Status - -### ✅ Components Using Platform Utilities -1. **Billing API** (`billingApi.ts`) - Uses `isMobile()`, `isIOS()`, `isAndroid()` -2. **Account Menu** (`AccountMenu.tsx`) - Uses `useIsMobile()` hook -3. **Team Invite Dialog** (`TeamInviteDialog.tsx`) - Uses `isMobile()` utility -4. **Marketing** (`Marketing.tsx`) - Uses `useIsIOS()` hook -5. **API Credits** (`ApiCreditsSection.tsx`) - Uses `useIsMobile()` hook -6. **Chat Box** (`ChatBox.tsx`) - Uses `useIsTauri()` hook -7. **Proxy Config** (`ProxyConfigSection.tsx`) - Uses `useIsTauriDesktop()` hook -8. **API Key Dashboard** (`ApiKeyDashboard.tsx`) - Uses `useIsTauriDesktop()` hook -9. **Pricing** (`pricing.tsx`) - Uses hooks and utilities -10. **Login** (`login.tsx`) - Uses `useIsIOS()`, `useIsTauri()` hooks and `isTauri()` utility -11. **Signup** (`signup.tsx`) - Uses `useIsIOS()`, `useIsTauri()` hooks and `isTauri()` utility -12. **Deep Link Handler** (`DeepLinkHandler.tsx`) - Uses `isTauri()` utility - -### ✅ Components That Don't Need Platform Utilities -1. **Auth Callback** (`auth.$provider.callback.tsx`) - Uses platform-agnostic `redirect-to-native` flag -2. **Desktop Auth** (`desktop-auth.tsx`) - No platform checks needed -3. **Apple Auth Provider** (`AppleAuthProvider.tsx`) - Intentionally uses `window.location.protocol === "tauri:"` to exclude ALL Tauri platforms (working as designed) - -#### Migration Examples - -**Before (Old Pattern):** -```typescript -// In billing/billingApi.ts -const isTauri = await import("@tauri-apps/api/core") - .then((m) => m.isTauri()) - .catch(() => false); - -if (isTauri) { - const { type } = await import("@tauri-apps/plugin-os"); - const platform = await type(); - - if (platform === "ios") { - // iOS logic - } -} -``` - -**After (New Pattern):** -```typescript -// In billing/billingApi.ts -import { isIOS, isAndroid, isMobile } from '@/utils/platform'; - -if (await isMobile()) { - // Mobile logic (iOS and Android) - returnUrl = "https://trymaple.ai"; -} - -if (await isIOS()) { - // iOS-specific logic -} - -if (await isAndroid()) { - // Android-specific logic -} -``` - -**React Component Before:** -```typescript -const [isIOS, setIsIOS] = useState(false); -const [isTauriEnv, setIsTauriEnv] = useState(false); - -useEffect(() => { - const checkPlatform = async () => { - const tauriEnv = await isTauri(); - setIsTauriEnv(tauriEnv); - if (tauriEnv) { - const platform = await type(); - setIsIOS(platform === "ios"); - } - }; - checkPlatform(); -}, []); -``` - -**React Component After:** -```typescript -import { useIsIOS, useIsAndroid, useIsTauri } from '@/hooks/usePlatform'; - -function Component() { - const { isIOS } = useIsIOS(); - const { isAndroid } = useIsAndroid(); - const { isTauri } = useIsTauri(); - - // No need for useEffect or state management - // The hooks handle everything -} -``` - -### Key Patterns for Android Support - -```typescript -// Pattern 1: Mobile behavior (both iOS and Android) -if (await isMobile()) { - // Open external browser for payments - await invoke("plugin:opener|open_url", { url }); -} - -// Pattern 2: iOS-only behavior -if (await isIOS()) { - // Use native Apple Sign In -} - -// Pattern 3: Android-only behavior -if (await isAndroid()) { - // Enable features that iOS restricts - showApiManagement = true; - enablePaidPlans = true; -} - -// Pattern 4: Desktop-only features -if (await isDesktop()) { - // Enable proxy configuration -} -``` diff --git a/docs/conversations-api-implementation.md b/docs/conversations-api-implementation.md deleted file mode 100644 index 20857f321..000000000 --- a/docs/conversations-api-implementation.md +++ /dev/null @@ -1,1489 +0,0 @@ -# Conversations/Responses API Implementation Guide - -## Overview - -This document describes the implementation of OpenAI's Conversations/Responses API in Maple's UnifiedChat component, building on the foundation established in [unified-chat-refactor.md](./unified-chat-refactor.md). - -## Background - -The Conversations/Responses API provides server-side conversation state management, replacing our previous localStorage-based approach. This migration enables: - -- **Server-managed state**: Conversations persist across devices and sessions -- **Streaming responses**: Real-time AI responses via Server-Sent Events (SSE) -- **Stateless client**: No localStorage dependencies, pure server-driven state -- **Automatic context management**: Server handles conversation history and token limits - -## Key Differences from POC - -While the proof-of-concept in `responses-poc` demonstrates the API capabilities, our production implementation differs in several key ways: - -1. **Component Architecture**: All logic stays in UnifiedChat.tsx (no Context providers) -2. **Title Generation**: Backend generates titles automatically (no frontend generation) -3. **Tool Support**: No web search or tool calling in initial implementation -4. **State Management**: Direct state in component, no external state libraries - -## Technical Architecture - -### API Endpoints - -The OpenSecret backend provides OpenAI-compatible endpoints: - -- **Conversations API**: - - `POST /v1/conversations` - Create conversation - - `GET /v1/conversations/{id}` - Get conversation - - `PATCH /v1/conversations/{id}` - Update metadata - - `DELETE /v1/conversations/{id}` - Delete conversation - - `GET /v1/conversations/{id}/items` - List conversation items - - `GET /v1/conversations` - List all conversations (custom extension) - -- **Responses API**: - - `POST /v1/responses` - Create response (with streaming) - - `GET /v1/responses/{id}` - Get response status - - `DELETE /v1/responses/{id}` - Delete response - -### Client Setup - -The SDK provides a custom fetch wrapper that handles: -- JWT authentication -- Session key encryption/decryption -- Automatic token refresh -- SSE streaming decryption - -### Streaming Architecture - -Responses use Server-Sent Events with the following event types: -- `response.created` - Response initiated -- `response.in_progress` - Processing started -- `response.output_item.added` - New output item -- `response.content_part.added` - Content part added -- `response.output_text.delta` - Text chunk -- `response.output_text.done` - Text complete -- `response.content_part.done` - Part complete -- `response.output_item.done` - Item complete -- `response.completed` - Response finished - -## Implementation Plan - -### Phase 1: Infrastructure Setup ✅ COMPLETE -1. ✅ Update OpenAI package to v5.20.0 (matched with SDK version) -2. ✅ OpenAI client already configured with custom fetch via context -3. ✅ Types added for conversations and responses - -### Phase 2: Core Functionality ✅ COMPLETE -4. ✅ Implement conversation creation (lazy creation on first message) -5. ✅ Add message sending with streaming (using responses.create API) -6. ✅ Handle response streaming and display (all SSE events working) - -### Phase 3: State Management ✅ COMPLETE -7. ✅ Add polling for conversation updates (5-second interval with cursor-based pagination) -8. ✅ Implement conversation loading from URL (loads on mount and URL changes) -9. ✅ Handle conversation switching (new chat clears conversation state) -10. ✅ Message deduplication (by ID and content signature) -11. ✅ LastSeenItemId tracking (React state-based cursor, no localStorage) -12. ✅ Handle browser navigation (back/forward button support) - -### Phase 4: Integration ✅ COMPLETE -13. ✅ Update Sidebar to fetch from API (using OpenSecret SDK's listConversations) -14. ✅ Remove localStorage dependencies for chat data (all chat data now from API) -15. ✅ Add error handling and recovery (404 handling, network errors, streaming failures) -16. ✅ Fix redundant API calls (removed immediate polling after load) -17. ✅ Improve conversation switching reliability (proper state tracking) - -## Completed Implementation Summary - -### What Was Built (Phases 1-4 Complete) - -The UnifiedChat component and Sidebar now have full Conversations/Responses API integration with the following features: - -#### ✅ Core Conversation Management -- **Lazy conversation creation** - Conversations only created on first message to avoid clutter -- **URL-based routing** - Uses query parameters (`?conversation_id=xxx`) for conversation state -- **Automatic conversation loading** - Loads existing conversation when URL contains conversation_id -- **Event-based communication** - Listens for `newchat` and `conversationselected` events from sidebar - -#### ✅ Streaming Implementation -- **Full SSE support** - Handles all streaming events from the responses API -- **Local to server ID mapping** - Smoothly transitions from local UUIDs to server-assigned IDs -- **Real-time text accumulation** - Shows text as it streams in -- **Status tracking** - Messages have `streaming`, `complete`, or `error` states -- **Abort controller** - Can cancel in-flight requests (foundation for future cancel button) - -#### ✅ Polling & Synchronization -- **5-second polling interval** - Checks for new messages every 5 seconds -- **Cursor-based pagination** - Uses `after` parameter with `lastSeenItemId` for efficient polling -- **React state-based cursor** - No localStorage/sessionStorage, pure component state -- **Message deduplication** - Prevents duplicates using both ID and content signature matching -- **Cross-device sync** - Enables conversation continuity across devices - -#### ✅ Error Handling -- **404 recovery** - Clears invalid conversation IDs and starts fresh -- **Network error display** - Shows user-friendly error messages -- **Silent polling failures** - Polling errors don't interrupt user experience -- **Streaming error handling** - Gracefully handles streaming failures - -#### ✅ State Management -- **No localStorage for chat data** - All conversation state comes from the API -- **Proper cleanup** - Clears state when switching conversations or starting new chats -- **TypeScript compliant** - Fully typed with proper error handling - -### What Still Needs Work (Phase 4) - -1. **Sidebar Integration** - Currently still uses localStorage, needs to fetch from API -2. **Legacy Chat Migration** - Strategy for handling old localStorage-based chats -3. **Advanced Features** - Model selection, token management, file attachments, etc. - -### Key Technical Decisions Made - -1. **Cursor-based Polling with React State** - - The `lastSeenItemId` is stored in component state, not localStorage - - Resets on page refresh (loads all messages fresh) - - Updates after each poll and after streaming completes - - Simple and effective for maintaining position during a session - -2. **Message ID Management** - - User messages get client-side UUIDs immediately - - Assistant messages start with local UUIDs, then swap to server IDs - - Deduplication handles the transition gracefully - - Prevents flicker and maintains smooth UX - -3. **Error Recovery Strategy** - - 404s clear the conversation and remove invalid ID from URL - - Network errors show user message but don't break the app - - Polling failures are silent (logged to console only) - - Streaming errors mark the message with error state - -4. **Event-Driven Architecture** - - Custom events for sidebar communication (`newchat`, `conversationselected`) - - Avoids prop drilling and complex state management - - Keeps components loosely coupled - -## Implementation Details - -### 1. Package Updates - -Update OpenAI SDK to support conversations API: - -```json -{ - "dependencies": { - "openai": "^5.20.0" // Updated from 4.56.1 to match SDK version exactly - } -} -``` - -The v5+ SDK includes full support for: -- Conversations API (create, retrieve, update, delete, list items) -- Responses API (create with streaming, retrieve, delete) -- Proper TypeScript types for all new endpoints -- Streaming iterator support for SSE events - -### 2. OpenAI Client Configuration - -Create client with OpenSecret's custom fetch from the SDK: - -```typescript -import { createCustomFetch } from "@opensecret/react"; -import OpenAI from "openai"; - -// Get API URL from environment -const API_URL = import.meta.env.VITE_OPEN_SECRET_API_URL || "https://api.opensecret.cloud"; - -// Create OpenAI client with custom fetch -const openai = new OpenAI({ - baseURL: `${API_URL}/v1/`, - dangerouslyAllowBrowser: true, // Required for browser usage - apiKey: "not-needed", // Auth handled by custom fetch - defaultHeaders: { - "Accept-Encoding": "identity" // Disable compression for SSE - }, - fetch: createCustomFetch() // SDK's custom fetch handles auth & encryption -}); -``` - -The custom fetch from `@opensecret/react` handles: -- JWT token injection from localStorage -- Automatic token refresh on 401 -- Session key encryption/decryption for E2E encryption -- Proper error handling with retry logic - -### 3. Complete Type Definitions - -Based on the actual API responses and SDK implementation: - -```typescript -// Core conversation types -interface Conversation { - id: string; - object: "conversation"; - created_at: number; // Unix timestamp - metadata?: { - title?: string; // Auto-generated by backend - [key: string]: any; - }; -} - -// Conversation item types -interface ConversationItem { - id: string; - type: "message" | "web_search_call"; // Extensible for tools - object?: string; - role?: "user" | "assistant" | "system"; - status?: "completed" | "in_progress"; - content?: Array<{ - type: "text" | "input_text"; - text?: string; - }>; - created_at?: number; -} - -// Message type for UI rendering -interface Message { - id: string; - role: "user" | "assistant" | "system"; - content: string; - timestamp: number; - status: "complete" | "streaming" | "error"; - isStreaming?: boolean; -} - -// Response streaming event types -interface ResponseEvent { - type: - | "response.created" - | "response.in_progress" - | "response.output_item.added" - | "response.content_part.added" - | "response.output_text.delta" - | "response.output_text.done" - | "response.content_part.done" - | "response.output_item.done" - | "response.completed" - | "response.failed" - | "error"; - - // Event-specific fields - sequence_number?: number; - delta?: string; // For text deltas - item_id?: string; // ID of the item being streamed - item?: any; // Full item for added events - response?: { // Full response object for created/completed - id: string; - status: "in_progress" | "completed" | "failed"; - model?: string; - usage?: { - input_tokens: number; - output_tokens: number; - total_tokens: number; - }; - output?: Array; - }; - error?: { - message: string; - type: string; - code?: string; - }; -} - -// API list response format -interface ConversationListResponse { - object: "list"; - data: Conversation[]; - first_id?: string; - last_id?: string; - has_more: boolean; -} - -// Conversation items cursor pagination -interface ConversationItemsPage { - data: ConversationItem[]; - has_more: boolean; - first_id?: string; - last_id?: string; -} -``` - -### 4. Complete Conversation Lifecycle Management - -Managing conversation lifecycle is critical for a seamless user experience. Here's the complete implementation: - -```typescript -// Full conversation lifecycle implementation -const ConversationManager = () => { - const [conversation, setConversation] = useState(null); - const [messages, setMessages] = useState([]); - const [lastSeenItemId, setLastSeenItemId] = useState(); - - // 1. CREATE NEW CONVERSATION - const createConversation = async () => { - try { - // Don't pre-create conversations - wait for first message - // This avoids empty conversations cluttering the list - setConversation(null); - setMessages([]); - setLastSeenItemId(undefined); - - // Clear URL parameter for new chat - const params = new URLSearchParams(window.location.search); - params.delete("conversation_id"); - const newUrl = params.toString() ? `/?${params}` : "/"; - window.history.replaceState({}, "", newUrl); - } catch (error) { - handleAPIError(error, "Create conversation"); - } - }; - - // 2. LAZY CONVERSATION CREATION ON FIRST MESSAGE - const ensureConversation = async (): Promise => { - if (conversation?.id) { - return conversation.id; - } - - // Create conversation on demand - const newConv = await openai.conversations.create({ - metadata: { - // Backend will auto-generate title from first message - // No need to set title here - } - }); - - setConversation({ - id: newConv.id, - object: "conversation", - created_at: newConv.created_at, - metadata: newConv.metadata - }); - - // Update URL with new conversation ID - const params = new URLSearchParams(window.location.search); - params.set("conversation_id", newConv.id); - window.history.replaceState({}, "", `/?${params}`); - - return newConv.id; - }; - - // 3. LOAD EXISTING CONVERSATION - const loadConversation = async (conversationId: string) => { - try { - // Fetch conversation metadata - const conv = await openai.conversations.retrieve(conversationId); - setConversation(conv); - - // Fetch all conversation items - const itemsResponse = await openai.conversations.items.list(conversationId, { - limit: 100 // Get up to 100 most recent items - }); - - // Convert items to messages - const loadedMessages: Message[] = []; - - for (const item of itemsResponse.data) { - if (item.type === "message" && item.role && item.content) { - let text = ""; - if (Array.isArray(item.content)) { - for (const part of item.content) { - if (part.type === "text" || part.type === "input_text") { - text += part.text || ""; - } - } - } else if (typeof item.content === "string") { - text = item.content; - } - - loadedMessages.push({ - id: item.id, - role: item.role as "user" | "assistant", - content: text, - timestamp: item.created_at ? item.created_at * 1000 : Date.now(), - status: "complete" - }); - } - } - - setMessages(loadedMessages); - - // Set last seen ID for polling - if (itemsResponse.data.length > 0) { - const lastItem = itemsResponse.data[itemsResponse.data.length - 1]; - setLastSeenItemId(lastItem.id); - } - - // Update URL if needed - const params = new URLSearchParams(window.location.search); - if (params.get("conversation_id") !== conversationId) { - params.set("conversation_id", conversationId); - window.history.replaceState({}, "", `/?${params}`); - } - } catch (error: any) { - if (error.status === 404) { - // Conversation doesn't exist - clear and start fresh - console.log("Conversation not found, starting new"); - createConversation(); - } else { - handleAPIError(error, "Load conversation"); - } - } - }; - - // 4. DELETE CONVERSATION - const deleteConversation = async (conversationId: string) => { - try { - await openai.conversations.delete(conversationId); - - // If deleting current conversation, start fresh - if (conversation?.id === conversationId) { - createConversation(); - } - } catch (error) { - handleAPIError(error, "Delete conversation"); - } - }; - - // 5. HANDLE URL CHANGES (on mount and popstate) - useEffect(() => { - const handleUrlChange = () => { - const params = new URLSearchParams(window.location.search); - const conversationId = params.get("conversation_id"); - - if (conversationId && conversationId !== conversation?.id) { - // Load the conversation from URL - loadConversation(conversationId); - } else if (!conversationId && conversation?.id) { - // URL cleared - start new conversation - createConversation(); - } - }; - - // Initial load - handleUrlChange(); - - // Listen for browser back/forward - window.addEventListener("popstate", handleUrlChange); - return () => window.removeEventListener("popstate", handleUrlChange); - }, []); - - // 6. HANDLE NEW CHAT EVENT FROM SIDEBAR - useEffect(() => { - const handleNewChat = () => { - createConversation(); - }; - - window.addEventListener("newchat", handleNewChat); - return () => window.removeEventListener("newchat", handleNewChat); - }, []); - - return { - conversation, - messages, - ensureConversation, - loadConversation, - deleteConversation, - createConversation - }; -}; -``` - -#### Conversation State Transitions - -```mermaid -graph TD - A[No Conversation] -->|User types message| B[Create Conversation] - B --> C[Conversation Active] - C -->|User sends message| D[Add to Conversation] - C -->|User clicks New Chat| A - C -->|User selects different chat| E[Load Conversation] - E --> C - C -->|User deletes chat| A - C -->|Page refresh| F[Reload from URL] - F --> C -``` - -#### Important Lifecycle Considerations - -1. **Lazy Creation**: Don't create conversations until first message -2. **URL Sync**: Always keep URL in sync with active conversation -3. **Browser Navigation**: Handle back/forward buttons properly -4. **Cross-Device**: Polling ensures continuity across devices -5. **Error Recovery**: Handle missing conversations gracefully -6. **Title Generation**: Backend generates titles automatically from first message - -### 5. Detailed Streaming Event Handling - -The responses API uses Server-Sent Events (SSE) for streaming. Here's the complete event flow and handling: - -```typescript -// Complete streaming implementation based on POC -const sendMessage = async (userInput: string, conversationId: string) => { - // Create abort controller for cancellation - const abortController = new AbortController(); - - try { - // Create streaming response - const stream = await openai.responses.create({ - model: "llama3-3-70b", // Or user's selected model - conversation: conversationId, - input: [{ role: "user", content: userInput }], - stream: true, // Enable streaming - store: true, // Store in conversation history - background: true, // Continue processing in background - signal: abortController.signal - }); - - // Initialize assistant message - // NOTE: The POC used local UUID -> server ID swap pattern - // This is complicated and error-prone (see alternatives below) - const localAssistantId = crypto.randomUUID(); - let serverItemId: string | undefined; - let accumulatedContent = ""; - - const assistantMessage: Message = { - id: localAssistantId, - role: "assistant", - content: "", - timestamp: Date.now(), - status: "streaming" - }; - - // Add to messages immediately - setMessages(prev => [...prev, assistantMessage]); - - // Process streaming events - for await (const event of stream) { - switch (event.type) { - case "response.created": - // Response object created, contains initial metadata - console.log("Response started:", event.response?.id); - break; - - case "response.in_progress": - // Processing has begun - break; - - case "response.output_item.added": - // New output item added (message or tool call) - if (event.item?.type === "message" && event.item_id) { - serverItemId = event.item_id; - // PROBLEMATIC: Update local message ID to server ID - // This causes re-renders and complicates message tracking - setMessages(prev => prev.map(msg => - msg.id === localAssistantId - ? { ...msg, id: serverItemId || msg.id } - : msg - )); - } - break; - - case "response.content_part.added": - // Content part initialized (before text starts) - break; - - case "response.output_text.delta": - // Text chunk received - this is the main event for streaming text - if (event.delta) { - accumulatedContent += event.delta; - // Update message content in real-time - setMessages(prev => prev.map(msg => - msg.id === (serverItemId || localAssistantId) - ? { ...msg, content: accumulatedContent } - : msg - )); - } - break; - - case "response.output_text.done": - // Text streaming complete for this part - break; - - case "response.content_part.done": - // Content part finished - break; - - case "response.output_item.done": - // Output item complete - if (event.item?.type === "message") { - // Finalize the message - setMessages(prev => prev.map(msg => - msg.id === (serverItemId || localAssistantId) - ? { ...msg, status: "complete" } - : msg - )); - } - break; - - case "response.completed": - // Entire response complete, includes usage stats - if (event.response?.usage) { - console.log("Token usage:", event.response.usage); - } - setIsGenerating(false); - // Update last seen item ID for polling - if (serverItemId) { - setLastSeenItemId(serverItemId); - } - break; - - case "response.failed": - case "error": - // Handle streaming errors - console.error("Streaming error:", event.error); - setMessages(prev => prev.map(msg => - msg.id === (serverItemId || localAssistantId) - ? { ...msg, status: "error" } - : msg - )); - setIsGenerating(false); - break; - } - } - } catch (error) { - if (error.name !== 'AbortError') { - console.error("Failed to send message:", error); - throw error; - } - } -}; - -// Cancel generation function -const cancelGeneration = () => { - abortController?.abort(); - setIsGenerating(false); -}; -``` - -#### Event Sequence Details - -The typical event sequence for a streaming response: - -1. **response.created** - Initial response object with ID and metadata -2. **response.in_progress** - Processing begins -3. **response.output_item.added** - Message item added to response -4. **response.content_part.added** - Text content part initialized -5. **response.output_text.delta** - Multiple events with text chunks -6. **response.output_text.done** - Text generation complete -7. **response.content_part.done** - Content part finalized -8. **response.output_item.done** - Message item complete -9. **response.completed** - Full response done with usage stats - -#### Important Streaming Considerations - -- **Server Item IDs**: Messages get server-assigned IDs during streaming -- **Local vs Server IDs**: Start with local UUID, replace with server ID when available -- **Abort Handling**: Use AbortController for clean cancellation -- **Error Recovery**: Handle network interruptions gracefully -- **Token Usage**: Track usage from completed event for billing - -#### Problems with Local ID → Server ID Pattern - -The POC's approach of swapping IDs has several issues: - -1. **Complex State Updates**: Requires finding and updating messages by temporary ID -2. **Deduplication Issues**: Polling might see same message with different IDs -3. **React Re-rendering**: Changing keys causes unnecessary re-renders -4. **Race Conditions**: Polling might return before ID swap completes - -#### Alternative Approaches to Consider - -**Option 1: Use Array Index Instead of ID** -```typescript -// Track by position, not ID -const assistantMessageIndex = messages.length; -setMessages(prev => [...prev, assistantMessage]); -// Update by index, not ID -setMessages(prev => prev.map((msg, idx) => - idx === assistantMessageIndex ? {...msg, content: newContent} : msg -)); -``` - -**Option 2: Compound Key** -```typescript -// Use both IDs as a compound key -const messageKey = `${localId}_${serverId || 'pending'}`; -``` - -**Option 3: Separate Streaming Message State** -```typescript -const [streamingMessage, setStreamingMessage] = useState(null); -const [persistedMessages, setPersistedMessages] = useState([]); -// Only add to persisted when complete with server ID -``` - -**Option 4: Use Response ID from First Event** -```typescript -// response.created event includes response.id immediately -// Could use this instead of waiting for item_id -``` - -**Recommendation**: Further investigation needed to determine the cleanest approach. The array index method might be simplest but needs testing with the polling mechanism. - -### 6. Comprehensive Polling Mechanism - -The polling system ensures conversation continuity across sessions and devices. This is critical for: -- **Mid-stream refreshes**: User refreshes while AI is responding -- **Cross-device sync**: Continue conversation from another device -- **Background updates**: Catch responses that completed after network issues -- **Missed events**: Recover from temporary disconnections - -```typescript -// Complete polling implementation based on POC -const pollForNewItems = useCallback(async () => { - if (!conversationId || !openai) return; - - try { - // Fetch items after the last seen ID - const response = await openai.conversations.items.list(conversationId, { - after: lastSeenItemId, - limit: 100 // Get up to 100 new items - }); - - if (response.data.length > 0) { - // Convert API items to UI messages - const newMessages: Message[] = []; - - for (const item of response.data) { - if (item.type === "message" && item.role && item.content) { - // Extract text content from content array - let text = ""; - if (Array.isArray(item.content)) { - for (const part of item.content) { - if (typeof part === "object" && part.text) { - text += part.text; - } - } - } - - const message: Message = { - id: item.id, - role: item.role as "user" | "assistant", - content: text, - timestamp: item.created_at ? item.created_at * 1000 : Date.now(), - status: "complete" - }; - - newMessages.push(message); - } - } - - if (newMessages.length > 0) { - // Merge new messages with deduplication - setMessages(prev => { - const existingIds = new Set(prev.map(m => m.id)); - const existingSignatures = new Set( - prev.map(m => `${m.role}:${m.content.substring(0, 100)}`) - ); - - const uniqueNewMessages = newMessages.filter(m => { - // Skip if we already have this ID - if (existingIds.has(m.id)) return false; - - // Skip if we have a message with same role and similar content - const signature = `${m.role}:${m.content.substring(0, 100)}`; - if (existingSignatures.has(signature)) return false; - - return true; - }); - - if (uniqueNewMessages.length === 0) return prev; - - // Replace local messages with server versions when they match - const updatedMessages = prev.map(msg => { - // If this is a local message (UUID format) - if (msg.id.includes("-") && msg.id.length === 36) { - const serverVersion = uniqueNewMessages.find( - newMsg => newMsg.role === msg.role && newMsg.content === msg.content - ); - if (serverVersion) { - // Remove from unique list to avoid duplication - uniqueNewMessages.splice(uniqueNewMessages.indexOf(serverVersion), 1); - return { ...msg, id: serverVersion.id }; - } - } - return msg; - }); - - return [...updatedMessages, ...uniqueNewMessages]; - }); - - // Update last seen item ID - const lastItem = response.data[response.data.length - 1]; - if (lastItem?.id) { - setLastSeenItemId(lastItem.id); - } - - // Check if we're no longer generating - if (isGenerating && newMessages.some(m => m.role === "assistant")) { - setIsGenerating(false); - } - } - } - } catch (error) { - console.error("Polling error:", error); - // Don't throw - polling should fail silently - } -}, [conversationId, lastSeenItemId, isGenerating, openai]); - -// Set up polling interval -useEffect(() => { - if (!conversationId || !openai) return; - - // Poll immediately on mount/change - pollForNewItems(); - - // Then set up interval for every 5 seconds - const intervalId = setInterval(pollForNewItems, 5000); - - return () => clearInterval(intervalId); -}, [conversationId, openai, pollForNewItems]); -``` - -#### Polling Strategy Details - -1. **Cursor-based Pagination**: Uses `after` parameter with last seen item ID -2. **Immediate Poll**: Polls immediately when conversation loads or changes -3. **5-Second Interval**: Balances freshness with server load -4. **Silent Failures**: Polling errors don't interrupt user experience -5. **Automatic Stop**: Clears interval on unmount or conversation change - -#### Message Deduplication - -The polling system includes sophisticated deduplication: - -```typescript -// Deduplication strategy -const deduplicateMessages = (existing: Message[], incoming: Message[]) => { - // 1. Check by ID (server-assigned IDs) - const existingIds = new Set(existing.map(m => m.id)); - - // 2. Check by content signature (for local messages not yet synced) - const existingSignatures = new Set( - existing.map(m => `${m.role}:${m.content.substring(0, 100)}`) - ); - - // 3. Filter incoming messages - const unique = incoming.filter(m => { - if (existingIds.has(m.id)) return false; - - const signature = `${m.role}:${m.content.substring(0, 100)}`; - if (existingSignatures.has(signature)) return false; - - return true; - }); - - // 4. Replace local IDs with server IDs when messages match - const updated = existing.map(msg => { - if (msg.id.includes("-")) { // Local UUID - const serverMatch = unique.find( - u => u.role === msg.role && u.content === msg.content - ); - if (serverMatch) { - unique.splice(unique.indexOf(serverMatch), 1); - return { ...msg, id: serverMatch.id }; - } - } - return msg; - }); - - return [...updated, ...unique]; -}; -``` - -### 7. Simple Error Handling - -Keep error handling straightforward - just show what went wrong: - -```typescript -// Basic error handling -const handleAPIError = (error: any) => { - console.error("API Error:", error); - - if (error.status === 401) { - // Auth failed - let the SDK handle refresh - // If we still get 401, redirect to login - window.location.href = "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/login"; - return; - } - - if (error.status === 404) { - // Conversation not found - clear and start fresh - setError("Conversation not found"); - const params = new URLSearchParams(window.location.search); - params.delete("conversation_id"); - window.history.replaceState({}, "", "/"); - return; - } - - if (error.name === "AbortError") { - // User cancelled - don't show error - return; - } - - // Show generic error for everything else - setError(error.message || "Something went wrong. Please try again."); - setIsGenerating(false); -}; - -// Simple usage -const sendMessage = async (input: string) => { - try { - // ... send message code ... - } catch (error) { - handleAPIError(error); - } -}; -``` - -#### Error Display in UI - -```typescript -// Simple error state -const [error, setError] = useState(null); - -// Show error banner -{error && ( -
- {error} -
-)} - -// Auto-clear after 10 seconds -useEffect(() => { - if (error) { - const timer = setTimeout(() => setError(null), 10000); - return () => clearTimeout(timer); - } -}, [error]); -``` - -**Note**: No retry logic or exponential backoff for now. Keep it simple - if something fails, show an error and let the user try again manually. - -### 8. State Structure - -Maintain minimal, server-driven state: - -```typescript -const [conversation, setConversation] = useState(null); -const [messages, setMessages] = useState([]); -const [isStreaming, setIsStreaming] = useState(false); -const [lastSeenItemId, setLastSeenItemId] = useState(); -``` - -### 9. Sidebar Integration Updates - -The Sidebar component needs significant updates to work with the conversations API instead of localStorage: - -```typescript -// Current sidebar uses localStorage -const currentImplementation = { - storage: "localStorage", - key: "maple_chats", - format: "JSON array of chat objects" -}; - -// New implementation with conversations API -const SidebarWithConversationsAPI = () => { - const [conversations, setConversations] = useState([]); - const [isLoading, setIsLoading] = useState(true); - - // Fetch conversations from API - const loadConversations = async () => { - try { - setIsLoading(true); - - // Use custom list function from SDK (non-standard OpenAI extension) - const response = await fetch(`${API_URL}/v1/conversations?limit=50`, { - headers: { - Authorization: `Bearer ${localStorage.getItem("access_token")}`, - "Content-Type": "application/json" - } - }); - - if (!response.ok) throw new Error("Failed to fetch conversations"); - - const data: ConversationListResponse = await response.json(); - - // Sort by created_at descending (newest first) - const sorted = data.data.sort((a, b) => b.created_at - a.created_at); - setConversations(sorted); - } catch (error) { - console.error("Failed to load conversations:", error); - setConversations([]); - } finally { - setIsLoading(false); - } - }; - - // Load on mount - useEffect(() => { - loadConversations(); - }, []); - - // Refresh when conversation changes - useEffect(() => { - const handleConversationUpdate = () => { - loadConversations(); - }; - - window.addEventListener("conversationupdated", handleConversationUpdate); - return () => window.removeEventListener("conversationupdated", handleConversationUpdate); - }, []); - - // Handle conversation selection - const selectConversation = (conversationId: string) => { - // Update URL to load conversation - const params = new URLSearchParams(window.location.search); - params.set("conversation_id", conversationId); - window.history.replaceState({}, "", `/?${params}`); - - // Dispatch event for UnifiedChat to handle - window.dispatchEvent(new CustomEvent("conversationselected", { - detail: { conversationId } - })); - }; - - // Handle new chat - const createNewChat = () => { - // Clear URL parameter - const params = new URLSearchParams(window.location.search); - params.delete("conversation_id"); - window.history.replaceState({}, "", params.toString() ? `/?${params}` : "/"); - - // Dispatch event for UnifiedChat - window.dispatchEvent(new Event("newchat")); - }; - - // Handle delete - const deleteConversation = async (conversationId: string, e: React.MouseEvent) => { - e.stopPropagation(); // Don't select the conversation - - try { - await openai.conversations.delete(conversationId); - - // Remove from local state - setConversations(prev => prev.filter(c => c.id !== conversationId)); - - // If deleting active conversation, start new - const currentId = new URLSearchParams(window.location.search).get("conversation_id"); - if (currentId === conversationId) { - createNewChat(); - } - } catch (error) { - console.error("Failed to delete conversation:", error); - } - }; - - return ( -
- - - {isLoading ? ( - - ) : ( -
- {conversations.map(conv => ( -
selectConversation(conv.id)} - className="conversation-item" - > - {conv.metadata?.title || "Untitled Chat"} - - {new Date(conv.created_at * 1000).toLocaleDateString()} - - -
- ))} -
- )} -
- ); -}; -``` - -#### Key Sidebar Changes - -1. **Remove localStorage**: No more `localStorage.getItem("maple_chats")` -2. **API Fetching**: Load conversations from `/v1/conversations` endpoint -3. **Real-time Updates**: Refresh list when conversations change -4. **Event-based Communication**: Use custom events to communicate with UnifiedChat -5. **Server-side Deletion**: Delete through API, not just local state -6. **Title Display**: Use `metadata.title` from backend-generated titles - -#### Staged Implementation Plan - -**Stage 1: Basic API Integration (No Cache)** -- Fetch 50 most recent conversations -- Simple list display -- Fetch on every sidebar open - -**Stage 2: Add Pagination** -```typescript -const [hasMore, setHasMore] = useState(true); -const [cursor, setCursor] = useState(); - -const loadMore = async () => { - const response = await fetch( - `${API_URL}/v1/conversations?limit=20&after=${cursor}` - ); - // Append to existing list -}; -``` - -**Stage 3: Add Caching** -```typescript -// Simple in-memory cache -const conversationCache = useRef<{ - data: Conversation[]; - timestamp: number; -}>({ data: [], timestamp: 0 }); - -// Use cache if fresh (< 30 seconds old) -if (Date.now() - cache.timestamp < 30000) { - return cache.data; -} -``` - -**Stage 4: Archived Legacy Chats** -```typescript -// Sidebar layout: -// [New Chat Button] -// [Active Conversations - paginated] -// ... scrollable area ... -// [Archived Chats] <- Click to show old localStorage chats -// [Account/Plan/Credits] - -const ArchivedSection = () => { - const [showArchived, setShowArchived] = useState(false); - const legacyChats = localStorage.getItem("maple_chats"); - - if (!legacyChats) return null; - - return ( -
- - {showArchived && ( -
- {/* Show old chats in read-only mode */} -
- )} -
- ); -}; -``` - -**Note**: Keep archived chats separate from new conversations to avoid mixing different data structures and breaking pagination. - -### 10. Migration from localStorage - -Complete removal of localStorage for chat data: - -```typescript -// REMOVE these localStorage operations: -localStorage.getItem("maple_chats") -localStorage.setItem("maple_chats", JSON.stringify(chats)) -localStorage.removeItem(`chat_${chatId}`) - -// KEEP these for auth (managed by SDK): -localStorage.getItem("access_token") -localStorage.getItem("refresh_token") -localStorage.getItem("user_id") -``` - -#### Handling Legacy Chats - -**Important**: Old chats are stored in KV store, not just localStorage. - -```typescript -// Legacy chat loading (one-time on sidebar mount) -const loadLegacyChats = async () => { - // Load ONCE from KV store on initial sidebar mount - const oldChats = await fetchLegacyChatsFromKV(); - - // Display in archived section (read-only) - setArchivedChats(oldChats); - - // Never write back to KV store - // Never update these chats - // They're frozen in time -}; -``` - -**Key Points**: -- Old chats exist in KV store (backend) -- Load them ONCE when sidebar first opens -- Never cache them in localStorage again -- Never write to them again -- Read-only archive for reference only -- Eventually can be removed entirely once users are comfortable with new system - -## Testing Strategy - -1. **Conversation Creation**: Verify new conversations are created properly -2. **Streaming**: Test response streaming and text accumulation -3. **Polling**: Verify updates are detected and merged correctly -4. **URL Management**: Ensure conversation IDs persist in URL -5. **Cross-Device**: Test conversation continuation across devices -6. **Error Recovery**: Verify graceful handling of network issues - -## Performance Considerations - -1. **No Caching**: As per requirements, no client-side caching -2. **Polling Frequency**: 5-second interval balances freshness vs load -3. **Stream Processing**: Process events efficiently without blocking UI -4. **Message Rendering**: Use React keys properly for smooth updates - -## Security Notes - -- All encryption handled by SDK's custom fetch -- JWT authentication automatic -- No sensitive data in localStorage -- Server validates all conversation access - -## References and Sources - -This implementation guide was developed by analyzing multiple sources across the OpenSecret ecosystem. Here are all the key references that provided the knowledge for this implementation: - -### 1. OpenSecret Backend Documentation -- **File**: `/Users/tony/Dev/OpenSecret/opensecret/docs/responses-implementation.md` -- **Content**: Detailed backend implementation of conversations/responses API -- **Key Insights**: - - Backend auto-generates titles from first message - - Supports SSE streaming with encryption - - Implements OpenAI-compatible endpoints - -### 2. OpenSecret SDK Implementation -- **Directory**: `/Users/tony/Dev/OpenSecret/OpenSecret-SDK/` -- **Key Files**: - - `src/lib/api.ts` - Core API functions including conversations/responses endpoints - - `src/lib/ai.ts` - Custom fetch wrapper with encryption support - - `src/lib/test/integration/ai.test.ts` - Comprehensive integration tests showing API usage - - `package.json` - Shows OpenAI v5.20.0+ dependency requirement -- **Key Insights**: - - Custom fetch handles JWT auth and encryption automatically - - Includes conversation CRUD operations - - Supports both streaming and non-streaming responses - - Has pagination support for conversation items - -### 3. Proof of Concept Implementation -- **Directory**: `/Users/tony/Dev/Personal/responses-poc/` -- **Key Files**: - - `frontend/src/contexts/ConversationContext.tsx` - Complete conversation management implementation - - `frontend/src/hooks/useConversation.ts` - React hook for conversation state - - `frontend/src/lib/openai-client.ts` - OpenAI client configuration - - `frontend/src/lib/streaming.ts` - SSE stream processing - - `frontend/src/components/ChatInterface.tsx` - UI implementation -- **Key Insights**: - - Polling mechanism for cross-device sync - - Message deduplication strategy - - Streaming event handling patterns - - Local vs server ID management - -### 4. Current Maple Project Structure -- **Directory**: `/Users/tony/Dev/OpenSecret/maple/` -- **Key Files**: - - `frontend/src/components/UnifiedChat.tsx` - Current monolithic chat component - - `frontend/src/components/Sidebar.tsx` - Existing sidebar using localStorage - - `docs/unified-chat-refactor.md` - Original refactor documentation - - `frontend/package.json` - Shows current dependencies (OpenAI v4.56.1) -- **Key Insights**: - - Monolithic component design philosophy - - Query parameter-based routing (no navigation) - - Event-based communication between components - -### 5. Integration Test Insights - -From the SDK integration tests (`ai.test.ts`), we learned: - -1. **Conversation Creation Pattern**: - ```typescript - const conversation = await openai.conversations.create({ - metadata: { title: "Test" } - }); - ``` - -2. **Streaming Response Pattern**: - ```typescript - const stream = await openai.responses.create({ - model: "model-name", - conversation: conversationId, - input: "user input", - stream: true, - store: true - }); - ``` - -3. **Event Types Sequence**: - - response.created - - response.in_progress - - response.output_item.added - - response.output_text.delta (multiple) - - response.completed - -4. **Pagination Pattern**: - ```typescript - const items = await openai.conversations.items.list(conversationId, { - after: lastSeenItemId, - limit: 100 - }); - ``` - -### 6. API Endpoint Analysis - -From the SDK and backend documentation: - -- **Base URL**: `https://api.opensecret.cloud/v1/` -- **Authentication**: JWT Bearer token in Authorization header -- **Encryption**: Handled transparently by SDK's custom fetch -- **Non-standard Extensions**: - - `GET /v1/conversations` - List conversations (not in standard OpenAI API) - - Background processing support via `background: true` parameter - -### 7. Key Design Decisions from Analysis - -1. **No Context Providers**: Keep everything in UnifiedChat (from refactor doc) -2. **No Caching**: Pure server-driven state (from requirements) -3. **Polling Required**: For cross-device sync and refresh recovery (from POC) -4. **Lazy Conversation Creation**: Create on first message only (from POC) -5. **Backend Title Generation**: No frontend title generation needed (from backend doc) - -### 8. Implementation Patterns Discovered - -1. **Message Deduplication**: Check by ID and content signature -2. **Error Recovery**: Exponential backoff for transient failures -3. **Stream Handling**: Process events in switch statement -4. **URL Management**: Use replaceState to avoid navigation -5. **Event Communication**: Custom events between Sidebar and UnifiedChat - -## Features to Migrate from Old Implementation - -After analyzing the old codebase (`index.backup.tsx`, `_auth.chat.$chatId.tsx`, `ChatBox.tsx`), here are the feature statuses: - -### ✅ Completed Features - -1. **Model Selection** ✅ - - ModelSelector component integrated - - Billing tier restrictions working - - Model switching mid-conversation supported - -2. **Token Management** ✅ - - HANDLED BY BACKEND - Intelligent compression on server-side - - No frontend token counting needed - - Backend automatically manages context limits - -3. **Voice Input** ✅ - - Audio recording with RecordRTC - - Whisper transcription working - - Recording overlay UI implemented - -4. **File Attachments** ✅ - - Document upload (.pdf, .txt, .md) - - Image attachments for multimodal - - Tauri integration for PDF parsing - -5. **Markdown Rendering** ✅ - - Code syntax highlighting - - LaTeX math rendering - - Copy code blocks - - Thinking tags stripping - -6. **Streaming Indicators** ✅ - - Visual feedback during generation - - Different implementation but working well - -### ❌ Features Still Needed - -1. **Text-to-Speech (TTS)** - POSTPONED - - API currently broken, will implement when fixed - - Play button on assistant messages - - AudioManager for playback control - - Pro/Team/Max tier requirement - -2. **Scroll Improvements** - - Scroll-to-bottom button when scrolled up - - Better auto-scroll logic matching old behavior: - - Auto-scroll on user message send - - Auto-scroll when streaming starts - - Maintain position when not at bottom - - Location: `_auth.chat.$chatId.tsx:530-607` - -3. **System Prompts** - - Coming via new API - - Will need collapsible UI when implemented - -### ✅ UI/UX Features Already Implemented - -- **Message Actions** ✅ - Copy button on messages -- **Keyboard Shortcuts** ✅ - Enter to send, Shift+Enter for newline -- **Auto-resize Textarea** ✅ - Dynamic height adjustment -- **Mobile Optimizations** ✅ - Responsive layout with new chat button - -### ✅ Business Logic Features (Already Working) - -- **Billing Integration** ✅ - Proactive status loading, upgrade prompts -- **Team Management** ✅ - Dialog available in main app -- **API Key Management** ✅ - Dialog available in main app -- **Query Parameter Handling** ✅ - Working in main index.tsx -- **Chat Compression** ✅ - Handled by backend automatically -- **Multimodal Support** ✅ - Images and documents working -- **Verification Modal** ✅ - Working in main app -- **Chat Session Management** ✅ - Via Conversations API - -### Implementation Summary - -**✅ COMPLETED:** -- Conversations/Responses API integration -- Model selection with billing -- Voice input and transcription -- File attachments (images & documents) -- Markdown rendering -- Streaming indicators -- Mobile optimizations -- Token management (backend-handled) -- All business logic features - -**❌ REMAINING:** -1. **Scroll improvements** - Better auto-scroll and scroll-to-bottom button -2. **TTS** - Postponed until API is fixed -3. **System prompts** - Coming via new API -4. **Draft persistence** - Nice-to-have for preventing data loss - -The refactor is essentially feature-complete except for scroll UX improvements! - -## Completed Features (as of Jan 2025) - -### Phase 1-4 Core Implementation ✅ -- Full Conversations/Responses API integration -- Server-side conversation management -- SSE streaming with real-time text accumulation -- 5-second polling for cross-device sync -- URL-based conversation routing -- Message deduplication -- Error recovery and 404 handling - -### Phase 2 Features Completed -1. **Model Selection** ✅ - - ModelSelector component integrated in both input areas - - Support for model switching mid-conversation - - Auto-switching to vision models when images are added - - Billing tier restrictions (Pro/Starter/Team) - - Upgrade prompts for restricted models - -2. **Billing Integration** ✅ - - Proactive billing status fetching on app load - - Billing status cached in LocalState - - ModelSelector respects billing tiers - - AccountMenu shows correct plan status - -## Next Steps - -1. **Implement scroll improvements** - Add scroll-to-bottom button and better auto-scroll behavior -2. **System prompts via API** - Implement when backend API is ready -3. **TTS integration** - Add when Kokoro API is fixed -4. **Draft persistence** - Optional localStorage backup for unsent messages -5. **Clean up old code** - Remove ChatBox.tsx, useChatSession.ts, index.backup.tsx -6. **Deprecate old chat routes** - Make _auth.chat.$chatId.tsx read-only for archived chats \ No newline at end of file diff --git a/docs/ios-tts-local-development.md b/docs/ios-tts-local-development.md index 274167a64..c3b45be04 100644 --- a/docs/ios-tts-local-development.md +++ b/docs/ios-tts-local-development.md @@ -163,4 +163,3 @@ rm -rf frontend/src-tauri/onnxruntime-build frontend/src-tauri/onnxruntime-ios ## Related Documentation - [troubleshooting-ios-build.md](./troubleshooting-ios-build.md) - arm64-sim architecture fix -- [tts-research.md](./tts-research.md) - TTS implementation details diff --git a/docs/product-redesign-spec.md b/docs/product-redesign-spec.md deleted file mode 100644 index dfdbe95c0..000000000 --- a/docs/product-redesign-spec.md +++ /dev/null @@ -1,1679 +0,0 @@ -# Maple Product Redesign Reimplementation Spec - -> Source reference: GitHub PR #465 (`feature/frontend-ui-marketing-updates`) -> -> Purpose: extract the designer's authenticated product redesign into an engineering-ready spec for a clean new implementation. -> -> This document is intentionally **not** a request to merge or cherry-pick PR #465. It translates the visual wins from that PR into a product-only, standards-aligned plan that preserves existing Maple functionality. -> -> Important: PR #465 was produced from an older fork. Current `master` is the canonical source of truth for functionality, routes, copy, and any product additions made after that fork. The redesign must be reapplied onto current `master`, not the other way around. - -## Design-Only Goals and Non-Goals - -This redesign effort is **design-only**. It is not a product rewrite, feature rewrite, or behavior rewrite. - -### Goals - -- Recreate the strongest authenticated-product visual ideas from PR #465 on top of current `master`. -- Improve the visual design of the logged-in product experience through typography, spacing, color, hierarchy, radius, iconography, theming, and component/dialog chrome. -- Reimplement those visual improvements using Maple's normal engineering standards, shared tokens, shared primitives, and existing architectural patterns. -- Keep the work tightly focused on the authenticated product surfaces and the dialogs/components that are reachable from them. - -### Non-Goals - -- No feature changes. -- No product behavior changes. -- No route, navigation, or query-param contract changes. -- No event contract changes. -- No modal/dialog open, close, trigger, or priority logic changes. -- No billing, account, team, API, auth, persistence, or platform-behavior changes. -- No addition, removal, or simplification of existing capabilities just because the designer PR made them look quieter. -- No marketing, logged-out, pricing, helper, or other public-page redesign in this effort. -- No logic refactors unless they are strictly required to support the visual implementation and preserve behavior exactly. - -If a proposed change cannot be justified as a pure design/chrome/presentation improvement while preserving current behavior, it is out of scope for this redesign. - ---- - -## 1. Working Rules - -1. Do **not** copy PR #465 wholesale. -2. Do **not** bring marketing, pricing, proof, downloads, solutions, or helper pages into the new product redesign PR. -3. Do preserve the visual ideas that make the designer PR good. -4. Do preserve existing product capabilities, behavior, and handler logic exactly. -5. When the raw PR and this spec disagree, follow this spec. -6. Prefer Maple's existing frontend patterns: React function components, shadcn/Radix primitives, Tailwind tokens, `@/` imports, TanStack Router, React Query, and current state/event contracts. -7. Treat current `master` as canonical for product behavior, routes, copy, and newer additions. Reapply the designer's visual changes onto `master`. -8. Do not let older-fork PR content overwrite newer `master` behavior/content. Example: if privacy and terms already exist on `master`, they stay exactly as `master` defines them. - ---- - -## 2. What Was Reviewed - -### 2.1 Designer PR inputs - -- PR metadata and full changed-file list for PR #465 -- PR diffs for: - - `frontend/src/index.css` - - `frontend/tailwind.config.js` - - `frontend/index.html` - - `frontend/src/app.tsx` - - `frontend/src/contexts/ThemeContext.tsx` - - `frontend/src/components/Sidebar.tsx` - - `frontend/src/components/ChatHistoryList.tsx` - - `frontend/src/components/UnifiedChat.tsx` - - `frontend/src/components/AccountMenu.tsx` - - `frontend/src/components/AccountDialog.tsx` - - `frontend/src/components/CreditUsage.tsx` - - `frontend/src/components/ModelSelector.tsx` - - `frontend/src/components/markdown.tsx` - - product-reachable dialogs and account/team/API dashboard surfaces - - `frontend/src/routes/_auth.chat.$chatId.tsx` - -### 2.2 Current product inputs - -- `frontend/src/routes/index.tsx` -- `frontend/src/components/ProjectDetailView.tsx` -- `frontend/src/state/LocalStateContext.tsx` -- `frontend/src/state/LocalStateContextDef.ts` -- current implementations of the same product-side files above - ---- - -## 3. Scope - -## 3.1 Primary in-scope surfaces - -These are the surfaces the redesign PR should actively target. - -- Authenticated home shell in `frontend/src/routes/index.tsx` -- `frontend/src/components/UnifiedChat.tsx` -- `frontend/src/components/Sidebar.tsx` -- `frontend/src/components/ChatHistoryList.tsx` -- `frontend/src/components/AccountMenu.tsx` -- `frontend/src/components/AccountDialog.tsx` -- `frontend/src/components/CreditUsage.tsx` -- `frontend/src/components/ModelSelector.tsx` -- `frontend/src/components/markdown.tsx` -- `frontend/src/routes/_auth.chat.$chatId.tsx` (archived chat viewer) -- Shared product foundations used by those surfaces: - - `frontend/src/index.css` - - `frontend/src/chat.css` - - `frontend/tailwind.config.js` - - `frontend/index.html` - - `frontend/src/app.tsx` - - `frontend/src/contexts/ThemeContext.tsx` - - relevant shadcn primitives - -## 3.2 Product-reachable dialogs and secondary surfaces in scope - -These are reachable from the logged-in product and should receive redesign polish where the PR provides direction. - -- `DocumentPlatformDialog` -- `ContextLimitDialog` -- `DeleteChatDialog` -- `BulkDeleteDialog` -- `WebSearchInfoDialog` -- `TTSDownloadDialog` -- `UpgradePromptDialog` -- `PromoDialog` -- `VerificationModal` -- `GuestPaymentWarningDialog` -- `RecordingOverlay` -- Team/API/account dashboards reachable from the account menu: - - `frontend/src/components/apikeys/ApiCreditsSection.tsx` - - `frontend/src/components/apikeys/ApiKeyDashboard.tsx` - - `frontend/src/components/apikeys/ApiKeysList.tsx` - - `frontend/src/components/apikeys/CreateApiKeyDialog.tsx` - - `frontend/src/components/apikeys/ProxyConfigSection.tsx` - - `frontend/src/components/team/TeamDashboard.tsx` - - `frontend/src/components/team/TeamInviteDialog.tsx` - - `frontend/src/components/team/TeamMembersList.tsx` - -## 3.3 Compatibility surfaces - -These are part of the authenticated product flow and must not regress, but the designer PR does **not** provide enough direct design direction to justify a full rewrite. - -- `frontend/src/components/ProjectDetailView.tsx` -- `frontend/src/components/ConversationProjectPicker.tsx` -- Project creation/rename/delete/move dialogs -- Existing project-focused route behavior (`project_id` search param flow) - -Default rule for these surfaces: - -- Keep current structure and behavior. -- Let shared token, typography, radius, button, and dialog improvements bring them closer to the new system. -- Do not invent a new project UX unless separately specified. - -## 3.4 Explicitly out of scope - -Ignore these PR #465 areas when building the new product redesign PR. - -### Marketing / logged-out shell - -- `frontend/src/components/Marketing.tsx` -- `frontend/src/components/MarketingSiteHome.tsx` -- `frontend/src/components/TopNav.tsx` -- `frontend/src/components/Footer.tsx` -- `frontend/src/components/SimplifiedFooter.tsx` -- `frontend/src/components/ComparisonChart.tsx` -- `frontend/src/components/VerticalLandingMock.tsx` -- `frontend/src/components/Explainer.tsx` - -### Marketing/helper/public routes - -- `frontend/src/routes/about.tsx` -- `frontend/src/routes/agent.tsx` -- `frontend/src/routes/downloads.tsx` -- `frontend/src/routes/pricing.tsx` -- `frontend/src/routes/proof.tsx` -- `frontend/src/routes/redeem.tsx` -- `frontend/src/routes/research.tsx` -- all `frontend/src/routes/solutions*.tsx` -- `frontend/src/routes/teams.tsx` -- `frontend/src/routes/team.invite.$inviteId.tsx` -- `frontend/src/config/pricingConfig.tsx` - -### Helper/signup surfaces outside the authenticated product flow - -- `frontend/src/components/GuestCredentialsDialog.tsx` -- `frontend/src/components/GuestSignupWarningDialog.tsx` - -### Generated / debug / not to be hand-copied - -- `frontend/src/routeTree.gen.ts` -- `frontend/src/components/BillingDebugger.tsx` - ---- - -## 4. Existing Product Invariants That Must Not Regress - -This is the most important engineering section in this document. - -The designer PR is visually valuable, but the current Maple product already supports more behavior than the redesign diff directly talks about. The new implementation must preserve those behaviors. - -## 4.1 Routing and URL contracts - -Keep the current authenticated routing model. - -- `routes/index.tsx` still decides between: - - marketing for logged-out users - - `ProjectDetailView` when `project_id` is present without `conversation_id` - - `UnifiedChat` otherwise -- `conversation_id` and `project_id` remain part of the app contract. -- The archived read-only route `/_auth.chat.$chatId` stays intact. - -## 4.2 Event contracts - -Preserve the existing custom/window event coordination model unless deliberately replaced everywhere. - -Examples already used in the app: - -- `newchat` -- `conversationselected` -- `projectselected` -- `conversationcreated` -- bulk dialog open events - -A product redesign is not permission to casually break those event flows. - -## 4.3 Sidebar/history capabilities to preserve - -Do **not** remove these without explicit product approval. - -- Projects/folders -- Pinned chats -- Recent chats -- Archived chats -- Search -- Bulk select -- Bulk delete -- Bulk move -- Long-press selection on mobile -- Pull-to-refresh -- Infinite scroll/pagination -- Per-item rename/delete/project actions - -Important: PR #465 only visibly restyles parts of the history list. It does **not** provide justification for flattening or removing the current project/pin model. - -## 4.4 Composer/chat capabilities to preserve - -The new chat UI must still support: - -- Streaming assistant responses -- Reasoning/thinking blocks -- Tool call rendering -- Web search status rendering -- Model gating / upgrade paths -- Project picker behavior where currently supported -- Image attachments -- Document attachments -- Desktop/Tauri PDF support behavior -- Voice recording + transcription -- TTS playback/download behavior -- Cancel generation -- Fullscreen composer mode -- Pagination/loading older messages -- Conversation title refresh/update behavior - -## 4.5 Account/billing/team/API capabilities to preserve - -The account redesign must keep: - -- Plan visibility -- Credit usage visibility -- Manage subscription -- Team management entry points -- API management entry points -- Profile/email verification -- Preferences/default system prompt flows -- Change password / delete account -- Delete history -- Support/privacy/terms/about links as approved - -## 4.6 Platform behavior to preserve - -Do not regress platform-specific handling already present in the app. - -- Desktop vs mobile layout differences -- Tauri vs web link opening -- iOS-specific billing/API gating -- Tauri-only features such as local PDF/TTS flows - -## 4.7 Master-first reconciliation rule - -PR #465 is a redesign reference, not a competing source of truth. - -- If `master` and the designer PR disagree on behavior, routing, copy, or content, `master` wins. -- Reimplementation should start from current `master` and layer the redesign on top. -- Anything added to `master` after the designer fork should be preserved unless there is an explicit product decision to replace it. -- Example: privacy/terms behavior and destinations already present on `master` remain canonical and must not be overwritten by the designer branch. - ---- - -## 5. What Makes the Designer PR Good - -These are the design qualities worth preserving. - -## 5.1 Quieter product chrome - -- The product feels less noisy. -- The sidebar looks less like a stack of controls and more like product navigation. -- Secondary controls become calmer and more intentional. - -## 5.2 Softer geometry - -- Rounder search fields -- Rounder composer shell -- Rounder message/tool/result cards -- Rounder account trigger and menu chrome - -## 5.3 Stronger brand presence without looking like marketing - -- Manrope gives the product a more designed, modern feel. -- The inline Maple wordmark is cleaner than image swapping. -- The coral/pebble palette feels branded but restrained. -- The `m-avatar.svg` assistant avatar is small but high-value. - -## 5.4 Better visual hierarchy in chat - -- Empty state is simpler and more confident. -- The composer looks like the primary object on the page. -- User and assistant messages are easier to scan. -- Tool states and web-search states look more intentional. - -## 5.5 Better density in account/billing surfaces - -- The ring-style credit meter is more compact and more premium. -- The plan badge and account entry point feel more polished. -- Dialogs benefit from more cohesive semantic color usage. - -## 5.6 Better content readability - -- Markdown links are calmer. -- Tables behave better on mobile. -- Thinking blocks are less visually heavy. -- Code/table corner radii are more consistent with the rest of the redesign. - ---- - -## 6. Engineering Standards For The Reimplementation - -## 6.1 Do not treat PR #465 code as production-ready source - -Use the designer PR as a **visual reference**, not as a code-quality standard. - -## 6.2 Prefer tokenized and reusable styling - -If the same style pattern appears 2+ times, extract it. - -Examples worth extracting during implementation: - -- sidebar title fade constant -- sidebar ellipsis button constant -- product composer shell class set -- product icon-button class set -- assistant message shell -- user message bubble shell -- plan badge variant or helper - -## 6.3 Keep behavior separate from chrome - -Especially in `UnifiedChat` and `ChatHistoryList`: - -- do not mix event/state/business changes with visual refactors unless necessary -- avoid deleting working product logic just because the PR diff did not touch it - -## 6.4 Avoid making monoliths worse - -Current files are already large. - -During implementation, prefer extracting presentational pieces such as: - -- `MapleChatAvatar` -- `AssistantMessage` -- `UserMessageBubble` -- `ComposerShell` -- `SidebarHeader` -- `SidebarHistoryRow` -- `AccountMenuTrigger` - -## 6.5 Prefer shared primitives over raw markup, except where custom controls are justified - -Use shadcn/Radix primitives by default. - -Reasonable exceptions from the designer PR: - -- the circular gradient send button -- the chromeless sidebar toggle/close controls - -## 6.6 No raw color drift - -Do not introduce fresh `text-green-*`, `text-red-*`, `text-blue-*`, `text-purple-*`, `bg-*` product styling if a semantic Maple token exists. - ---- - -## 7. Foundation Design System Requirements - -## 7.1 Theme architecture - -Adopt class-based theming. - -### Required changes - -- `tailwind.config.js`: add `darkMode: "class"` -- create `ThemeProvider` -- store theme in `localStorage` under `maple-theme` -- support `light | dark | system` -- apply/remove `.dark` on `document.documentElement` -- set `document.documentElement.style.colorScheme` - -### Required fixes beyond the raw PR - -The PR direction is correct, but the implementation needs two fixes. - -1. **Prevent FOUC.** Add an inline script in `frontend/index.html` `` (before any stylesheets) that synchronously applies the theme class: - -```html - -``` - -2. **Keep `resolvedTheme` reactive.** The PR computes `resolvedTheme` as a derived value during render, which doesn't trigger re-renders when the OS theme changes in "system" mode. Fix: store `resolvedTheme` in `useState` and call `setResolvedTheme()` from the media query change handler. - -Also remove `style="color-scheme: light dark"` from the `` tag in `index.html` -- ThemeContext manages this now. - -## 7.2 Typography - -### Primary product font - -- Use `Manrope` as the primary font for the product body UI. -- Keep existing decorative fonts like Mondwest available if already used elsewhere. - -### CSS application - -In `index.css`, add to the `body` rule inside `@layer base`: - -```css -body { - font-family: "Manrope", sans-serif; - font-size: 14px; -} -``` - -### Font loading - -Final implementation should **self-host `Manrope`** and serve it from Maple's own app/domain so Cloudflare can cache it at the edge. - -Recommended production approach: - -- add checked-in font assets (prefer WOFF2) under `frontend/public/fonts/` or equivalent -- load them with `@font-face` in `index.css` -- optionally preload the most important weights in `index.html` -- use `font-display: swap` - -The Google Fonts snippet in PR #465 is useful as a reference for the intended family/weight range, but it should **not** be the final production dependency for the product UI. - -## 7.3 Core color tokens - -### Existing shadcn token changes (light mode) - -These are the **existing** CSS variables that change value. The implementor must update these in the `:root` block of `index.css`. - -| Token | Old Value | New Value | Notes | -| ---------------------- | ---------------------- | --------------------------- | ------------------------------ | -| `--background` | `40 30% 96%` | `0 0% 98%` | Warm off-white -> pure neutral | -| `--foreground` | `0 0% 12%` | `0 0% 15%` | Slightly lighter body text | -| `--card` | `40 30% 96%` | `0 0% 100%` | White cards | -| `--card-foreground` | `0 0% 12%` | `0 0% 15%` | Match foreground | -| `--popover` | `40 30% 96%` | `0 0% 100%` | White popovers | -| `--popover-foreground` | `0 0% 12%` | `0 0% 15%` | Match foreground | -| `--primary` | `0 0% 12%` | `0 0% 9%` | Darker primary | -| `--primary-foreground` | `40 30% 96%` | `0 0% 98%` | Pure neutral | -| `--secondary` | `264 89% 69%` (purple) | `17 100% 72%` (coral) | **Major** | -| `--accent` | `264 89% 69%` (purple) | `17 100% 94%` (light coral) | **Major** | -| `--muted` | `40 20% 90%` | `0 0% 96%` | Pure neutral | -| `--muted-foreground` | `0 0% 40%` | `0 0% 45%` | Slightly lighter | -| `--destructive` | `0 80% 37%` | `12 60% 54%` | Matches maple-error | -| `--border` | `40 15% 85%` | `0 0% 90%` | Pure neutral | -| `--input` | `40 15% 85%` | `0 0% 90%` | Match border | -| `--ring` | `264 89% 69%` (purple) | `17 100% 72%` (coral) | **Major** | - -New token: `--destructive-on-filled: 0 0% 100%` (white text on filled destructive buttons). - -### Neutral scale (new) - -| Token | HSL | Hex | Notes | -| --------------- | ---------: | ------: | --------------------------------- | -| `--neutral-50` | `0 0% 98%` | #FAFAFA | default light page bg | -| `--neutral-100` | `0 0% 96%` | #F5F5F5 | muted light fill | -| `--neutral-200` | `0 0% 90%` | #E5E5E5 | borders/inputs | -| `--neutral-300` | `0 0% 83%` | #D4D4D4 | subtle borders | -| `--neutral-400` | `0 0% 64%` | #A3A3A3 | dark muted text | -| `--neutral-500` | `0 0% 45%` | #737373 | light muted text | -| `--neutral-600` | `0 0% 32%` | #525252 | secondary text | -| `--neutral-700` | `0 0% 25%` | #404040 | dark chrome | -| `--neutral-800` | `0 0% 15%` | #262626 | light body text / dark sidebar bg | -| `--neutral-900` | `0 0% 9%` | #171717 | dark cards | -| `--neutral-950` | `0 0% 4%` | #0A0A0A | dark page bg | - -### Maple semantic palette (new) - -| Token | HSL | Role | -| ----------------------------- | ---------------------------------: | ---------------------------------- | -| `--maple-primary` | `17 100% 72%` | coral accent (#FF9771) | -| `--maple-primary-strong` | `17 78% 58%` | darker coral gradient stop | -| `--maple-on-primary` | `0 0% 100%` light / `0 0% 4%` dark | text on coral | -| `--maple-primary-container` | `17 100% 94%` | subtle coral tint (#FFE8E0) | -| `--maple-primary-rgb` | `255, 151, 113` | for rgba() usage | -| `--maple-secondary` | `237 8% 57%` | pebble muted accent (#8A8B9A) | -| `--maple-secondary-700` | `237 9% 38%` | darker pebble icon/text (#5A5B6A) | -| `--maple-secondary-container` | `240 14% 92%` | secondary surface fill (#E8E8ED) | -| `--maple-tertiary` | `11 22% 51%` | earthy "bark/grove" tone (#9E7469) | -| `--maple-tertiary-container` | `17 25% 88%` | soft tertiary fill (#EADED9) | -| `--maple-success` | `80 32% 42%` | success (#7B8F4A) | -| `--maple-warning` | `36 57% 59%` | warning (#D4A35A) | -| `--maple-on-warning` | `0 0% 100%` | text on warning | -| `--maple-error` | `12 60% 54%` | destructive/error (#D05E41) | -| `--maple-info` | `213 15% 56%` | info (#7E8DA1) | -| `--maple-surface` | `0 0% 98%` | branded surface | -| `--maple-surface-dim` | `240 7% 78%` | dim surface | - -### Product-specific chrome aliases (new) - -```css -/* Light mode */ ---sidebar-chrome: 0 0% 100%; ---sidebar-chrome-hover: 0 0% 96%; ---on-sidebar-chrome: 0 0% 15%; - -/* Dark mode (.dark block) */ ---sidebar: var(--neutral-800); ---sidebar-chrome: var(--neutral-700); ---sidebar-chrome-hover: var(--neutral-600); ---on-sidebar-chrome: 0 0% 98%; -``` - -### Dark mode variable overrides - -In the `.dark` block, the key overrides (beyond the sidebar tokens above): - -- `--background: var(--neutral-950)` (was `0 0% 7%`) -- `--foreground: 0 0% 98%` (was `0 0% 89%`) -- `--card: 0 0% 9%` (was `0 0% 10%`) -- `--primary: 0 0% 98%` (inverted from light) -- `--primary-foreground: 0 0% 4%` -- `--muted: 0 0% 15%` (was `0 0% 15%` -- unchanged) -- `--muted-foreground: 0 0% 64%` (was `0 0% 70%`) -- `--border: 0 0% 15%` (was `0 0% 20%`) -- `--maple-on-primary: 0 0% 4%` (dark text on coral in dark mode) -- `--maple-primary-container: 17 40% 18%` (darker container) - -Refer to PR #465 `index.css` diff for the complete dark `.dark` block -- the pattern is the same as light but with adjusted values for each maple-\* token. - -### Brand gradient - -```css ---maple-brand-gradient-from: 240 7% 78%; ---maple-brand-gradient-to: 11 22% 51%; -``` - -Utility classes to add in `@layer utilities`: - -- `.brand-gradient` -- `bg-gradient-to-r` with the from/to stops -- `.brand-gradient-text` -- same gradient with `bg-clip-text text-transparent` - -### Global CSS additions in `@layer utilities` - -```css -/* Maple primary colored caret for all text inputs */ -textarea, -input[type="text"], -input[type="email"], -input[type="password"], -input[type="search"] { - caret-color: hsl(var(--maple-primary)); -} -``` - -Also update `.primary-gradient` to use `--maple-primary` instead of `--purple`. - -## 7.4 Tailwind additions - -Add semantic Tailwind mappings for: - -- `maple.primary.*` -- `maple.secondary.*` -- `maple.tertiary.*` -- `maple.success` -- `maple.warning` -- `maple.onWarning` -- `maple.error` -- `maple.info` -- `maple.surface.*` -- `neutral.50` through `neutral.950` -- `destructive.onFilled` - -## 7.5 Semantic color migration rules - -Use the following migration logic across product surfaces. - -| Replace | With | -| --------------------------------- | --------------------------------------------------------------------------------------- | -| `text-green-*` | `text-maple-success` | -| `text-red-*` | `text-maple-error` | -| `text-amber-*`, `text-yellow-*` | `text-maple-warning` | -| `text-blue-*` | `text-maple-info` or `text-[hsl(var(--maple-primary))]` when brand emphasis is intended | -| `text-purple-*` | `text-[hsl(var(--maple-primary))]` | -| `bg-green-*/10` | `bg-maple-success/10` | -| `bg-red-*/10` | `bg-maple-error/10` | -| `bg-amber-*/10`, `bg-yellow-*/10` | `bg-maple-warning/10` | -| `bg-blue-*/10` | `bg-maple-info/10` | -| `bg-purple-*/10` | `bg-[hsl(var(--maple-primary))]/10` | - -## 7.6 Shared product assets - -### Required - -- `MapleWordmark.tsx` as an inline `currentColor` SVG component -- `public/m-avatar.svg` for the assistant avatar - -### Optional / only if actually used by product surfaces - -- additional wordmark SVG files -- raster branding assets added by PR #465 - -Do not carry unused branding files into the product-only PR just because they exist in the designer branch. - -## 7.7 Primitive strategy - -### Good ideas from the PR - -- add a `primary` button variant -- softer radii -- better dropdown/dialog corners -- better semantic destructive text handling -- darker unified scrim for overlays - -### Important cleanup rule - -The PR uses `--marketing-hero-scrim` for product overlays. That value is the dark translucent backdrop behind dialogs, sheets, and alert dialogs. In the clean reimplementation, prefer a neutral/product-owned token name such as `--overlay-scrim` if the value is shared by dialogs/sheets/alert dialogs. Do not keep marketing-specific naming in core product primitives unless absolutely necessary. - -### Blast-radius caution - -Global primitive changes affect out-of-scope pages too. - -Preferred approach: - -- land global primitive changes only when they are broadly safe -- otherwise add product-specific variants/helpers instead of restyling every button/menu in the whole app by accident - -### Primitive details worth preserving - -#### Button (`button.tsx`) - -**Base class changes:** - -- Old: `hover:backdrop-blur-xs ... rounded-md ... disabled:opacity-50` -- New: `active:scale-[0.95] rounded-full ... transition-all duration-200 ease-out ... disabled:opacity-40` - -**Variant exact classes from the PR** (refer to PR diff for the full strings; key mappings below): - -| Variant | Old summary | New summary | -| ------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `default` | `bg-primary text-primary-foreground hover:bg-primary/90` | Subtle gradient: `bg-gradient-to-b from-[hsl(var(--maple-tertiary-container)/0.5)] to-[hsl(var(--maple-tertiary-container)/0.25)] text-[hsl(var(--maple-secondary-700))]` with dark mode variants | -| `primary` | **(NEW)** | `bg-gradient-to-b from-[hsl(var(--maple-primary))] to-[hsl(var(--maple-primary-strong))] text-[hsl(var(--maple-on-primary))]/90 hover:brightness-110` | -| `destructive` | `bg-destructive text-white` | `bg-gradient-to-b from-[hsl(var(--maple-error))] to-[hsl(var(--maple-error)/0.8)] text-destructive-onFilled hover:brightness-110` | -| `outline` | Purple/blue tinted border + hover glow | `border-[hsl(var(--maple-secondary))]/30 hover:border-[hsl(var(--maple-primary))]/80 bg-transparent` | -| `secondary` | `bg-secondary text-secondary-foreground` | `bg-gradient-to-b from-[hsl(var(--maple-secondary-container))] to-[hsl(var(--maple-secondary-container)/0.6)] text-[hsl(var(--maple-secondary-700))]` | -| `ghost` | `hover:bg-accent dark:hover:bg-[hsl(var(--purple))]/20 dark:hover:text-white` | `text-foreground hover:bg-[hsl(var(--maple-secondary-container))]` (same both modes) | -| `link` | `text-primary underline-offset-4` | `text-[hsl(var(--maple-primary))] rounded-none active:scale-100` | - -**Size changes:** - -- `default`: `h-10 px-4 py-2` -> `h-10 px-5 py-2 text-sm` -- `sm`: `h-9 rounded-md px-3` -> `h-9 px-4 py-2 text-xs` (inherits rounded-full) -- `lg`: `h-11 rounded-md px-8` -> `h-11 px-7 py-2 text-base` - -Recommended approach: definitely add `variant="primary"`. Audit `default` and `outline` changes before making them global since they affect marketing pages too. - -#### Dialog (`dialog.tsx`) - -- **Overlay:** `bg-black/80` -> `bg-[hsl(var(--overlay-scrim)/0.8)]` (this is the modal backdrop/scrim behind the dialog; use a neutral token name, not `--marketing-hero-scrim`) -- **Content:** Remove `border`. Add `dark:bg-muted`. Corner radius: `sm:rounded-lg` -> `sm:rounded-2xl` - -#### Sheet (`sheet.tsx`) - -- **Overlay:** Same scrim change as Dialog - -#### Dropdown menu (`dropdown-menu.tsx`) - -- **Content & SubContent:** `rounded-md` -> `rounded-xl`. Remove `border` from main content. -- **All item focus states:** `dark:focus:bg-[hsl(var(--purple))]/20 dark:focus:text-white` -> `dark:focus:bg-[hsl(var(--maple-primary))]/20 dark:focus:text-foreground` - -#### Switch (`switch.tsx`) - -- **Thumb:** `bg-black` -> `bg-foreground` - ---- - -## 8. Surface-by-Surface Specification - -## 8.1 Authenticated shell - -### Keep - -- current route branching in `routes/index.tsx` -- current modal/dialog wiring for verification, promo, guest payment, team management, API management - -### Add - -- `ThemeProvider` around the app in `app.tsx` -- closed-sidebar top-left control row: sidebar toggle + wordmark - -### Do not do - -- do not redesign the logged-out shell in the same PR -- do not move product logic into new routes unless needed - -## 8.2 Sidebar - -### Visual direction to preserve - -- flatter sidebar container -- inline wordmark at top -- chromeless close control -- text-link style actions for `New Chat` and `Search` -- rounded search input -- separate subtle `History` label - -### Target structure - -1. Top wordmark row - - left: `MapleWordmark` - - right: close button using `ArrowLeftFromLine` -2. Action row(s) - - `New Chat` - - `Search` -3. Optional search input -4. `History` label row -5. history nav -6. account menu pinned to bottom - -### Canonical chrome classes - -```tsx -
-``` - -### Search input shape - -```tsx -className = "pl-4 pr-8 h-9 rounded-full"; -``` - -### Sidebar toggle - -Use a chromeless button with `Menu` (hamburger) icon, not an outline ` -``` - -Use the `h-8 w-8` version for the bottom compact composer. - -## 8.4.8 Voice / stop / attachment shells - -Preserve the PR's softer geometry: - -- mic: `rounded-xl` -- stop button: `rounded-xl` -- inner stop square: `rounded-md` -- image thumbs: `rounded-xl` -- document chip: `rounded-2xl` -- recording overlay: `rounded-3xl` - -## 8.4.9 Color migrations in UnifiedChat - -Throughout the chat interface, apply the semantic color migration: - -- Error text: `text-red-500` -> `text-maple-error` -- Success icons in tool results: `text-green-600 dark:text-green-400` -> `text-maple-success` -- Warning dot (incomplete/canceled): `bg-yellow-500` -> `bg-maple-warning` -- Web search enabled icon: `text-blue-500` -> `text-[hsl(var(--maple-primary))]` -- Web search disabled icon: `text-muted-foreground` -> `text-[hsl(var(--maple-secondary-700))]` -- Toolbar icon buttons (globe, plus, mic): all use the pebble/coral treatment from Section 9.7 - -## 8.4.10 Footer copy - -Use the stronger privacy microcopy from the PR. - -```tsx -

- - Encrypted and private at every step -

-``` - -Bottom disclaimer: `text-sm` -> `text-[10px]`, `text-muted-foreground/60` -> `text-muted-foreground/50`, `mt-2` -> `mt-1 mb-2`. - -## 8.4.11 Do not regress current UnifiedChat behavior - -The following are visual-only or presentation-oriented changes. Do not use this redesign as an excuse to delete: - -- pagination state -- model gating -- attachment validation -- project integration -- title refresh logic -- web search education flow -- TTS logic -- voice recording lifecycle - -## 8.5 Archived chat route - -Even though this is a compatibility surface, the PR includes useful polish here. - -### Take from the PR - -- user messages become the same right-aligned rounded bubble treatment -- assistant messages use the branded `m-avatar.svg` -- mobile new-chat button becomes borderless - -### Keep - -- read-only archived behavior -- current route semantics - -## 8.6 Account menu - -## 8.6.1 Visual direction - -- centered plan badge -- compact ring-style credit meter below it -- small circular account trigger instead of a full-width `Account` button -- dropdown aligns to the sidebar edge, not to the tiny circular trigger center - -## 8.6.2 Canonical trigger - -```tsx - -``` - -## 8.6.3 Canonical dropdown positioning - -```tsx - -``` - -## 8.6.4 Content and styling details - -- Menu label: `Maple Research` instead of `Maple AI` -- Plan badge styling: - - **Old:** `bg-[hsl(var(--primary))] text-[hsl(var(--primary-foreground))]` (black/white) - - **New:** `bg-[hsl(var(--maple-tertiary-container))] text-[hsl(var(--maple-tertiary))] text-[10px]` (earthy tint) -- Team setup badge: `bg-amber-500 text-white` -> `bg-maple-warning text-maple-onWarning` -- Privacy/terms links: **ignore the PR's URL swap.** The designer branch came from an older fork. Current `master` privacy/terms pages, routes, and destinations are canonical and must remain exactly as `master` defines them. - -## 8.6.5 Keep current account behavior - -Do not regress: - -- sign-out cache clearing behavior unless consciously reworked everywhere -- billing portal behavior -- external-link behavior in Tauri/web -- team/API modal flows - -## 8.7 Account dialog - -### Add - -A `Theme` section with `Light`, `Dark`, and `System` controls. - -### Visual direction - -- `Sun`, `Moon`, `Monitor` icons -- use buttons that clearly show active selection - -### Token cleanup - -- verified email icon -> `text-maple-success` -- unverified email icon -> `text-maple-error` -- destructive action becomes text/ghost-style destructive action instead of a heavy outlined danger button - -## 8.8 Credit usage - -## 8.8.1 Keep both layouts - -- existing bar layout remains useful outside the sidebar -- new ring layout is used in the redesigned account area - -## 8.8.2 Ring layout to preserve - -- Compact bordered card: `rounded-xl border border-[hsl(var(--sidebar-chrome))] bg-transparent p-3` -- Left side: status label ("Plan credits" / "Almost full" / "Limit reached") + reset date + optional extra credits text -- Right side: 32x32 SVG ring meter with 3.5px stroke -- Ring track: `stroke-[hsl(var(--sidebar-chrome))]` -- Ring fill: linear gradient from `hsl(var(--maple-primary))` to `hsl(var(--maple-primary-strong))` -- Animated: `transition-[stroke-dashoffset] duration-500 ease-out` -- Rotated -90deg so arc starts from top - -Refer to PR #465 `CreditUsage.tsx` diff for the `RingMeter` SVG implementation -- it's ~40 lines of clean SVG component code worth carrying over directly. - -## 8.8.3 Usage thresholds - -| Threshold | Tone | Color | Text class | -| --------- | ------ | --------------------------- | -------------------- | -| `>= 90%` | danger | `hsl(var(--maple-error))` | `text-maple-error` | -| `>= 75%` | warn | `hsl(var(--maple-warning))` | `text-maple-warning` | -| `< 75%` | ok | `hsl(var(--maple-success))` | `text-maple-success` | - -Old bar variant used hardcoded Tailwind (red-500, amber-500, emerald-500) -- migrate to the same semantic tokens. - -Status labels: `>= 100%` "Limit reached", `>= 90%` "Almost full", `< 90%` "Plan credits". - -## 8.8.4 Dev-only mock support - -The PR's dev-only mock scenarios are useful and can stay behind `import.meta.env.DEV`. - -Supported scenarios from the PR: - -- `demo` -- `full` -- `high` -- `warn` -- `ok` -- `off` - -## 8.9 Model selector - -### Take from the PR - -- Trigger text size: `text-sm` -> `text-xs` -- Trigger button color: `text-[hsl(var(--maple-secondary-700))] hover:bg-[hsl(var(--maple-primary-container))]` -- Chevron: remove `opacity-50` -- Badge border radius: `rounded-sm` -> `rounded-md` -- Upgrade hover state: `hover:bg-purple-50 dark:hover:bg-purple-950/20` -> `hover:bg-[hsl(var(--maple-primary-container))] dark:hover:bg-[hsl(var(--maple-primary))]/10` - -### Preserve - -- all current gating logic -- image restrictions -- model availability logic -- current category/model selection behavior - -### Badge mapping from the PR - -| Badge | Treatment | -| ------------- | -------------------------------------------- | -| `Coming Soon` | `bg-muted text-muted-foreground` | -| `Pro` | coral-to-tertiary soft gradient + coral text | -| `Starter` | `bg-maple-success/10 text-maple-success` | -| `New` | `bg-maple-info/10 text-maple-info` | -| `Reasoning` | `bg-maple-error/10 text-maple-error` | -| `Beta` | `bg-maple-warning/10 text-maple-warning` | - -## 8.10 Markdown rendering - -## 8.10.1 Link behavior (chat.css) - -Replace the current accent-colored, outline-ring link style with calmer foreground-based treatment: - -```css -.markdown-body a { - color: hsl(var(--foreground) / 0.72); /* subdued, not accent-colored */ - text-decoration: none; - -webkit-tap-highlight-color: transparent; -} -.markdown-body a:hover { - text-decoration: underline; - text-underline-offset: 0.15em; - color: hsl(var(--foreground)); /* full opacity on hover */ -} -.markdown-body a:focus { - outline: none; - box-shadow: none; -} -.markdown-body a:focus-visible { - outline: none; - box-shadow: none; - text-decoration: underline; - text-underline-offset: 0.15em; - color: hsl(var(--foreground)); -} -``` - -Remove the old `.markdown-body a:focus`, `.markdown-body a:focus:not(:focus-visible)`, and `.markdown-body a:focus-visible` rules that used outline rings. - -## 8.10.2 Tables - -### Component change (markdown.tsx) - -Replace the `ResponsiveTable` scroll-detection approach (removes `useState`, `useEffect`, `useRef` for scroll tracking, gradient fade indicators) with a simpler wrapper: - -```tsx -
-
- {children}
-
-
-``` - -### CSS additions (chat.css) - -Add a new `.markdown-table-maple` class with these traits: - -- `display: table; table-layout: fixed; width: 100%` -- Transparent backgrounds (no alternating row stripes) -- Horizontal rules only: `border-bottom: 1px solid hsl(var(--maple-secondary) / 0.2)` for headers, `hsl(var(--maple-secondary-container))` for cells -- Last row: no bottom border -- First column flush left (`padding-left: 0`) -- Responsive first-column widths: 34% on mobile, 22% on desktop -- `overflow-wrap: break-word; word-break: break-word` on all cells - -Remove `display: block; width: max-content; max-width: 100%; overflow: auto` from base `.markdown-body table`. - -Refer to the PR diff for the exact CSS rules -- there are ~90 lines of table styling. - -## 8.10.3 Typography sizes (chat.css) - -Set explicit `font-size: 14px; line-height: 1.5` on: - -- `.markdown-body p` -- `.markdown-body ol`, `.markdown-body ul` -- `.markdown-body li`, `.markdown-body li > p` -- `.markdown-body td` - -Add mobile reading optimization: - -```css -@media (max-width: 767px) { - .markdown-body p { - max-width: min(100%, 48ch); - } - .markdown-body td p, - .markdown-body th p { - max-width: none; - } -} -``` - -## 8.10.4 Thinking blocks (markdown.tsx) - -Replace the bordered card treatment: - -- **Old:** `border border-gray-200 dark:border-gray-700 rounded-lg bg-gray-50 dark:bg-gray-900/50` -- **New:** No border, no background. Plain button + expandable content. -- Icon/text colors: `text-gray-500 dark:text-gray-400` -> `text-muted-foreground` -- Collapse chevron moves from left side to right side of the row -- Expanded content: no border-top, just `pb-1 pt-2` - -## 8.10.5 Code and content radius consistency - -All code-related `border-radius` values change from `6px` to `12px`: - -- `.markdown-body kbd` -- `.markdown-body pre` -- `.markdown-body .mermaid` -- Footnote checkbox `::before` - -## 8.10.6 Dark mode in chat.css - -Remove the `@media (prefers-color-scheme: dark) { :root { ... } }` block from `chat.css` (the markdown-specific dark variable overrides). Dark mode is now handled by the `.dark` class in `index.css`. - -## 8.11 Product dialogs and secondary surfaces - -### Apply token/radius polish to product-reachable dialogs - -Use PR #465 mostly as a semantic cleanup guide here. - -#### High-priority dialogs - -- `UpgradePromptDialog` -- `PromoDialog` -- `WebSearchInfoDialog` -- `DocumentPlatformDialog` -- `ContextLimitDialog` -- `DeleteChatDialog` -- `BulkDeleteDialog` -- `TTSDownloadDialog` -- `VerificationModal` -- `GuestPaymentWarningDialog` - -#### Expected treatment - -- semantic icon/text color migration (see Section 7.5 for the full mapping table) -- rounded container polish -- use `variant="primary"` for the strongest upgrade CTA when appropriate -- preserve existing logic and copy unless the PR provides a clear product-facing improvement - -#### Specific dialog changes from PR #465 worth preserving - -- **UpgradePromptDialog:** benefit check icons `text-green-500` -> `text-maple-success`; upgrade button uses `variant="primary"` -- **PromoDialog:** pink/orange gradients -> `from-[hsl(var(--maple-primary))] to-[hsl(var(--maple-primary-strong))]`; badge, benefit icons, privacy check all migrate to maple semantic colors -- **WebSearchInfoDialog:** info icon `bg-blue-500/10 text-blue-500` -> `bg-maple-info/10 text-maple-info`; feature checks -> `text-maple-info` -- **GuestPaymentWarningDialog:** warning colors -> `text-maple-warning`, `bg-maple-warning/10` -- **AccountDialog:** verified email `text-green-700` -> `text-maple-success`; unverified `text-red-700` -> `text-maple-error`; delete account button `variant="outline" border-destructive` -> `variant="ghost" text-destructive hover:bg-destructive/10` - -## 8.12 Team/API/account dashboards - -These are in scope only for polish, not for wholesale redesign. - -### What to carry over - -- semantic token cleanup -- softer badges/fills -- progress bars using Maple semantic colors -- consistent dialog/dropdown/button styling inherited from shared primitives - -### What not to do - -- do not restructure these dashboards just because product chrome changed elsewhere - -## 8.13 Project surfaces - -The designer PR does not directly redesign project mode, but the authenticated product still supports it. - -### Required rule - -The new redesign PR must leave project functionality intact. - -### Minimum expectation - -- shared theme/tokens should not make project mode look broken -- shared buttons/dialogs/sidebar chrome should feel consistent -- project flows remain operational - -### Not required in the first redesign PR - -- a bespoke new visual language for `ProjectDetailView` -- redesigning project instructions UX -- redesigning move/create/delete project dialogs beyond shared primitive polish - ---- - -## 9. Exact High-Value Class Specs - -These are the visual details most worth preserving verbatim or near-verbatim. - -## 9.1 Sidebar history fade - -```tsx -const SIDEBAR_TITLE_FADE = - "pointer-events-none absolute inset-y-0 right-0 z-[1] bg-gradient-to-l from-muted from-35% via-muted/85 to-transparent dark:from-[hsl(var(--sidebar))] dark:from-35% dark:via-[hsl(var(--sidebar)/0.85)] dark:to-transparent"; -``` - -## 9.2 Sidebar ellipsis button - -```tsx -const SIDEBAR_ELLIPSIS_BTN = - "z-20 shrink-0 rounded-full bg-muted/90 p-1.5 text-primary backdrop-blur-sm transition-opacity dark:bg-[hsl(var(--sidebar)/0.9)]"; -``` - -## 9.3 Empty-state heading - -```tsx -className = - "overflow-visible pb-1 text-4xl font-normal leading-relaxed brand-gradient-text mb-6"; -``` - -## 9.4 Composer shell - -```tsx -className = - "relative overflow-hidden rounded-3xl border border-[hsl(var(--maple-secondary-container))] bg-background transition-colors focus-within:border-[hsl(var(--maple-primary))]"; -``` - -## 9.5 Composer top row - -```tsx -className = "flex items-start gap-1 pl-4 pr-2 pt-2"; -``` - -## 9.6 Fullscreen toggle button - -```tsx -className = - "mt-0.5 shrink-0 rounded-full p-1.5 text-muted-foreground/60 transition-colors hover:bg-muted/50 hover:text-foreground"; -``` - -## 9.7 Toolbar icon button - -```tsx -className = - "h-8 w-8 p-0 text-[hsl(var(--maple-secondary-700))] hover:text-[hsl(var(--maple-secondary-700))] hover:bg-[hsl(var(--maple-primary-container))]"; -``` - -## 9.8 User bubble - -```tsx -className = - "max-w-[min(100%,42rem)] rounded-2xl border border-border bg-muted px-4 py-3 backdrop-blur-lg dark:bg-card"; -``` - -## 9.9 Assistant message shell - -```tsx -
-
- -
-
-
-
Maple
-
-
-
-``` - -## 9.10 Account trigger - -```tsx -className = - "relative flex h-9 w-9 shrink-0 items-center justify-center rounded-full bg-[hsl(var(--sidebar-chrome))] text-[hsl(var(--on-sidebar-chrome))] shadow-none ring-0 transition-colors hover:bg-[hsl(var(--sidebar-chrome-hover))]"; -``` - -## 9.11 Ring credit card - -```tsx -
-``` - ---- - -## 10. What Not To Copy Literally From PR #465 - -## 10.1 Do not copy any marketing/page work - -Ignore those files entirely for the new product redesign PR. - -## 10.2 Do not infer feature removals from visual simplifications - -If the designer PR visually simplifies a control but the current product still supports that feature, preserve the feature. - -Key examples: - -- bulk move -- projects/project mode -- pinned chats -- project picker behavior -- current route/search-param contracts - -## 10.3 Do not hand-edit generated files - -- `routeTree.gen.ts` - -## 10.4 Do not import unused branding assets - -Only bring over assets actually needed by the product redesign. - -## 10.5 Do not keep marketing-specific token names in shared product primitives if a neutral semantic alias is cleaner - -Example: prefer `overlay-scrim` over `marketing-hero-scrim` for dialog overlays if touching shared primitives. - -## 10.6 Do not land huge global primitive changes without auditing blast radius - -Because marketing/helper pages are out of scope, avoid unintentionally redesigning them through careless primitive changes. - -## 10.7 Do not overwrite newer master content with older-fork PR content - -PR #465 started from an older fork, so some content/routing changes in that branch are stale by definition. - -- keep newer `master` pages, routes, and copy -- only reapply the visual/product-chrome improvements from the designer branch -- do not let old-fork link destinations replace current `master` behavior -- example: privacy/terms on `master` stay exactly as they are - ---- - -## 11. Implementation Phase Plan For The Future Product PR - -## Phase 1: foundation - -- add tokens to `index.css` -- add Tailwind semantic mappings -- add `ThemeProvider` -- add early-theme script -- self-host `Manrope` assets and wire `@font-face` / preload strategy -- add `MapleWordmark` + `m-avatar.svg` - -## Phase 2: shared primitives - -- button improvements -- dialog/dropdown/alert/sheet/switch polish -- only land safe global changes - -## Phase 3: shell + sidebar - -- sidebar shell/header -- sidebar toggle -- chat history row chrome -- account menu shell -- ring credit meter - -## Phase 4: chat surface - -- empty state -- mobile/desktop header behavior -- message shells -- composer shell -- send button -- tool/web search status cards -- footer/privacy microcopy - -## Phase 5: markdown + secondary dialogs - -- markdown link/table/thinking polish -- upgrade/search/promo/document dialogs -- account dialog theme picker -- product-reachable dashboard token cleanup - -## Phase 6: compatibility pass - -- archived route visual alignment -- project surfaces sanity pass -- ensure project mode does not clash with new shared foundation - -## Phase 7: validation - -For the future implementation PR, run at minimum: - -```bash -just format -just lint -just build -``` - -If Rust is untouched, Rust validators are not required. - ---- - -## 12. Acceptance Checklist For The Future Implementation PR - -## 12.1 Theme and shell - -- [ ] light mode looks correct -- [ ] dark mode looks correct -- [ ] `system` theme follows OS changes -- [ ] no initial theme flash on reload - -## 12.2 Responsive behavior - -- [ ] desktop closed-sidebar top-left wordmark row works -- [ ] mobile two-row chat header works -- [ ] sidebar open/close behavior still works correctly - -## 12.3 Chat behavior - -- [ ] empty state redesign is present -- [ ] active chat layout works -- [ ] streaming states look correct -- [ ] canceled/tool/web-search states look correct -- [ ] image/document attachments still work -- [ ] voice flow still works -- [ ] fullscreen composer still works - -## 12.4 History behavior - -- [ ] projects still work -- [ ] pinned chats still work -- [ ] archived chats still work -- [ ] bulk move still works -- [ ] bulk delete still works -- [ ] search still works -- [ ] pull-to-refresh still works on mobile - -## 12.5 Account/billing behavior - -- [ ] account menu redesign is present -- [ ] credit ring is present in sidebar -- [ ] theme picker works -- [ ] team/API dialogs still open and work -- [ ] billing/manage subscription still works - -## 12.6 Compatibility surfaces - -- [ ] archived route looks aligned with new system -- [ ] project mode still works and does not look broken - ---- - -## 13. PR #465 File Map For The New Product PR - -## 13.1 Carry over directly or near-directly - -- `frontend/src/components/MapleWordmark.tsx` -- `frontend/public/m-avatar.svg` -- theme/token additions in `index.css` -- Tailwind semantic mappings in `tailwind.config.js` -- `ThemeProvider` concept in `src/contexts/ThemeContext.tsx` (with fixes) -- core product chrome direction in: - - `Sidebar.tsx` - - `ChatHistoryList.tsx` - - `UnifiedChat.tsx` - - `AccountMenu.tsx` - - `CreditUsage.tsx` - - `ModelSelector.tsx` - - `markdown.tsx` - - `AccountDialog.tsx` - -## 13.2 Carry over as token/radius cleanup only - -- product-reachable dialogs -- API dashboard surfaces -- team dashboard surfaces -- verification/promo/search/upgrade supporting surfaces - -## 13.3 Ignore for the new product-only PR - -- all marketing/public routes and components -- pricing config changes -- `routeTree.gen.ts` -- `BillingDebugger.tsx` -- extra marketing-only asset work - ---- - -## 14. Bottom Line - -The right implementation is **not** “merge the designer branch.” - -The right implementation is: - -- take the designer's product visual language, -- keep Maple's real product behavior, -- express the redesign through shared tokens, clean primitives, and reusable product chrome, -- and keep the rewrite tightly scoped to the authenticated product experience. diff --git a/docs/tool-call-immediate-rendering.md b/docs/tool-call-immediate-rendering.md deleted file mode 100644 index 61bf1a4e2..000000000 --- a/docs/tool-call-immediate-rendering.md +++ /dev/null @@ -1,152 +0,0 @@ -# Tool Call Immediate Rendering Refactor - -## Problem Statement - -Tool calls don't render immediately during streaming in UnifiedChat. Currently: - -1. **Tool call arrives** (`tool_call.created` event) → stored in Map, not displayed -2. **Tool output arrives** (`tool_output.created` event) → stored in Map, not displayed -3. **Assistant message created** (`response.output_item.added` type="message") → empty message created -4. **Text starts streaming** (`response.output_text.delta`) → **NOW** tool calls finally render - -**User experience**: 5+ second delay between tool call and seeing any visual feedback. - -## Streaming Event Sequence (from logs) - -``` -🔵 response.created -🔵 response.in_progress -🔵 tool_call.created ← Call created, but not displayed yet - hasAssistantId: false - toolCallsCount: 0 - -🔵 tool_output.created ← Output ready, but not displayed yet - hasAssistantId: false - toolCallsCount: 1 - -🔵 response.output_item.added ← NOW assistant message created - type: "message" - hasAssistantId: true - -🔵 response.output_text.delta ← NOW tool calls finally display! -``` - -## Root Cause - -The architecture tightly couples "displayed items" with the API's `output_item` concept: - -1. **Current architecture**: Wait for `output_item.added` (type="message") to create container -2. **Tool calls arrive first**: Before the message container exists -3. **Buffered in Map**: Waiting to be grouped inside assistant message -4. **Finally rendered**: Only when text deltas start arriving - -## Solution Approach - -### Original Architecture -``` -Message (assistant) { - content: [tool_call, tool_output, text] -} -``` -- Everything nested inside message's content array -- Tool calls/outputs don't exist as standalone items -- Must wait for message to exist before rendering - -### New Architecture (Flat List) -``` -messages = [ - { type: "message", role: "user", ... }, - { type: "function_call", call_id: "...", ... }, - { type: "function_call_output", call_id: "...", ... }, - { type: "message", role: "assistant", ... } -] -``` -- Each item is independent at top level -- Render immediately as events arrive -- Visual pairing handled by renderer (matching `call_id`) -- Matches how LLMs and the API actually model conversations - -## Implementation Progress - -### ✅ Completed - -1. **Simplified type system** - - Changed `type Message = ConversationItem` to use OpenAI's native types - - Removed custom `Message` interface with nested content - - Removed `timestamp` field (wasn't used for anything) - -2. **Simplified data conversion** - - `convertItemsToMessages` now just casts items as-is - - No more grouping/buffering logic (126 lines → 5 lines) - -### 🚧 In Progress - -3. **Update MessageList rendering** (partially complete) - - Need to handle different item types: `message`, `function_call`, `function_call_output`, `web_search_call` - - Render each type independently - - Keep visual pairing logic (tools render together when adjacent) - - Update to check `item.type` instead of `message.role` - -### ❌ TODO - -4. **Update streaming logic** (processStreamingResponse) - - Create items immediately as events arrive (no Map buffering) - - `tool_call.created` → `setMessages([...prev, toolCallItem])` - - `tool_output.created` → `setMessages([...prev, toolOutputItem])` - - `response.output_text.delta` → update assistant message text - - Remove the `toolCalls` Map entirely - -5. **Update user message creation** - - Change from custom Message type to OpenAI's Message format - - Match the `type: "message"` structure - -6. **Fix TypeScript errors** - - Update all code that checks `message.role` to check `item.type` - - Update all code that assumes `message.content` exists on all items - - Handle different item types in scroll logic, status checks, etc. - -## Benefits of Flat Structure - -1. **Immediate rendering**: Items render as soon as they arrive -2. **Matches API model**: No impedance mismatch with OpenAI's data structure -3. **Simpler streaming logic**: No buffering, no grouping, just add items -4. **Matches LLM mental model**: Tools are distinct operations, not nested content -5. **Easier to maintain**: Less custom mapping logic - -## Alternative: Minimal Fix - -If full refactor is too large, we could do a smaller fix: - -1. Keep current grouped structure -2. Create assistant message **eagerly** (on first tool call, not on output_item.added) -3. Add items to message immediately (no Map buffering) - -This would fix the immediate issue but keep the coupling between display and API structure. - -## Testing Plan - -1. **Streaming test**: Tool call should appear immediately (< 100ms after event) -2. **Pairing test**: When output arrives, should render as grouped pair -3. **Orphan test**: Tool output without call should render standalone -4. **Loading test**: Old conversations load correctly with tool calls -5. **Scroll test**: Auto-scroll behavior still works with flat structure - -## Files Changed - -- `/frontend/src/components/UnifiedChat.tsx` - Main refactor - - Type definitions (lines 75-102) - - convertItemsToMessages (lines 114-120) - - MessageList rendering (lines 492+) - - processStreamingResponse (lines 1724+) - - handleSendMessage user message creation (lines 1926+) - -## Current State - -The code is **partially refactored** but **not working**: -- Type system simplified ✅ -- Data conversion simplified ✅ -- Rendering needs update for flat items ❌ -- Streaming needs update to create items immediately ❌ -- TypeScript errors need fixing ❌ - -**Decision needed**: Complete the flat refactor OR revert and do minimal fix? diff --git a/docs/tts-research.md b/docs/tts-research.md deleted file mode 100644 index edb81022a..000000000 --- a/docs/tts-research.md +++ /dev/null @@ -1,248 +0,0 @@ -# TTS Research - -## Gemini - -Definitive Comparative Analysis of Local TTS Architectures for Cross-Platform Tauri Integration: Supertonic 2 vs. Chatterbox Turbo1. Executive Strategic Overview: The Local Inference Convergence in 2026The trajectory of computational linguistics and speech synthesis has undergone a profound transformation over the last half-decade, culminating in a distinct bifurcation of technology stacks in early 2026. For systems architects and developers leveraging the Tauri framework to build cross-platform applications—spanning the unrestricted desktop ecosystems of macOS and Linux, as well as the rigorously sandboxed mobile environments of iOS and Android—the selection of a Text-to-Speech (TTS) engine is no longer a mere feature choice. It has elevated to a fundamental architectural decision that dictates the entire build pipeline, runtime efficiency, and distribution strategy of the final application.The user’s query posits a choice between two leading contenders in the open-weight arena: Supertonic 2, released by Supertone Inc. in January 2026, and Chatterbox Turbo, developed by Resemble AI. This report provides an exhaustive technical due diligence of these two models. The core tension explored herein is between Architectural Agility—epitomized by Supertonic’s lightweight, ONNX-native design—and Expressive Density—represented by Chatterbox’s larger, Llama-based backbone.While cloud-based inference dominated the early 2020s, the current paradigm emphasizes "Edge AI" and "Local First" principles. This shift is driven by privacy mandates, the need for zero-latency interaction in conversational interfaces, and the desire to eliminate recurring API costs. However, achieving parity with cloud-grade TTS on consumer hardware requires navigating a labyrinth of constraints: binary size limitations, memory bandwidth bottlenecks on mobile SoCs (System on Chips), and the draconian process management restrictions of mobile operating systems.For a Tauri developer, who enjoys the luxury of Rust’s performance and the web’s ubiquity, the challenge is uniquely complex. Tauri’s promise of a "write once, deploy everywhere" codebase is severely tested when integrating deep learning models that rely on disparate runtimes. Supertonic 2 offers a path of least resistance through native compilation, while Chatterbox Turbo demands a hybrid architecture that may fracture the unified codebase ideal. This report rigorously dissects these trade-offs to provide a definitive integration roadmap.2. Architectural Deconstruction: The Lightweight vs. The Large Language BackboneTo understand the feasibility of these models within a constrained Tauri environment, one must first dismantle their internal architectures. The "black box" of AI often obscures dependency chains that can shatter a cross-platform build pipeline. The difference between 44 million parameters and 350 million parameters is not merely quantitative; it represents two divergent philosophies of engineering.2.1 Supertonic 2: The Principles of Architectural DistillationSupertonic 2, as of its January 2026 release 1, is an anomaly in the contemporary landscape of generative AI. While the broader industry trend has been to scale parameters upwards—moving from millions to billions to achieve nuanced reasoning—Supertone Inc. has focused on distillation and efficiency. The model is engineered explicitly for embedded and on-device usage, prioritizing the reduction of computational overhead to near-negligible levels.The 44 Million Parameter AdvantageThe model operates with approximately 44 million parameters.2 In the context of modern neural networks, where even "Small Language Models" (SLMs) typically range from 0.5B to 3B parameters, 44M is microscopic. This scale confers specific hardware advantages that are critical for mobile performance:Cache Residency: A model of this size (approx. 268 MB in FP32, significantly less if quantized) can often reside entirely within the System Level Cache (SLC) or high-speed RAM partitions of modern mobile processors like the Apple A-series or Qualcomm Snapdragon. This drastically minimizes memory bandwidth saturation, which is the primary source of heat and battery drain during inference.Initialization Speed: The "cold start" time—the duration from loading the model to the first audio sample—is imperceptible, measured in milliseconds. This allows the TTS engine to be instantiated on-demand rather than requiring a persistent background service, optimizing system resource usage.The ONNX-Native RuntimeCrucially for Tauri developers, Supertonic is built natively for the ONNX Runtime.1 This choice is not incidental; it is a strategic enablement of cross-platform portability. ONNX (Open Neural Network Exchange) provides a standardized inference engine that is completely decoupled from the training environment. It does not require a Python interpreter, the heavy PyTorch library, or complex CUDA drivers to execute. Instead, it runs via optimized C++ libraries.Because Tauri's backend is written in Rust, developers can utilize the ort crate to bind directly to these C++ libraries. This means the TTS engine is not an external dependency or a separate process; it becomes an intrinsic function within the application's binary. This "library-level" integration is the gold standard for mobile development, ensuring compliance with App Store policies regarding executable code and utilizing native platform capabilities.The January 2026 Evolution (v2)The user's query specifically highlights "Supertonic 2 (Jan 2026)." This version introduces pivotal upgrades that address previous limitations:Multilingual Unification: Prior versions were often language-specific. Supertonic 2 introduces a unified architecture supporting English, Korean, Spanish, Portuguese, and French.1 This implies that a single ONNX model file can handle dynamic language switching at runtime without the latency penalty of unloading and reloading different model weights.Voice Personas: The update adds distinct voice styles (e.g., Alex, Sarah, James).5 While not offering the infinite flexibility of voice cloning, these preset personas cover the vast majority of use cases for standard reading applications, navigation, and accessibility tools.2.2 Chatterbox Turbo: The Llama-Based HeavyweightChatterbox, developed by Resemble AI, represents the "Quality First" school of thought. It leverages the massive advancements in Large Language Models (LLMs) and generative flow matching to achieve state-of-the-art naturalness.The Llama BackboneChatterbox Turbo is built upon a Llama backbone 6, likely adapting the transformer architecture to process audio tokens alongside text. Even in its "Turbo" configuration, which is optimized for latency, the model retains a 350 million parameter structure. While efficient for a server-grade GPU, this is nearly an order of magnitude larger than Supertonic.Memory Pressure: The model weights alone exceed 4 GB.7 Loading a 4GB model into memory is a non-trivial operation on mobile devices. Most mid-range Android phones ship with 6GB or 8GB of total RAM, shared between the OS, the GPU, and all active apps. Allocating 4GB to a single background TTS process will almost certainly trigger the operating system's Low Memory Killer (LMK), terminating the application or other background services to preserve system stability.Storage Friction: Distributing a mobile application with a 4GB asset payload is highly problematic. It exceeds the initial download size limits of both the Apple App Store (which requires Over-the-Air downloads to be under a certain threshold, often 200MB-4GB depending on OS version) and the Google Play Store (150MB base limit). Developers would be forced to implement complex "On-Demand Resource" downloading or expansive expansion files (OBB), adding significant friction to the user's first-run experience.The Python-PyTorch Dependency ChainChatterbox is a PyTorch-native model.6 Its architecture utilizes complex operations—specifically paralinguistic tag handling and flow matching decoders—that are deeply entwined with the PyTorch runtime and the Python ecosystem (requiring libraries like numpy, scipy, and torchaudio).Lack of ONNX Export: Unlike simpler models, Chatterbox does not offer a first-party, fully functional ONNX export that retains all its features. The dynamic nature of its flow matching steps and custom tokenizers makes "freezing" the model into a static computation graph exceptionally difficult. Consequently, running Chatterbox requires a live Python environment, a requirement that introduces the "Sidecar Problem" on mobile platforms—a critical hurdle for Tauri integration that will be explored in depth in subsequent sections.Feature SuperiorityDespite these architectural weights, Chatterbox offers capabilities Supertonic cannot match:Paralinguistic Control: Developers can inject tags like [laugh], [sigh], or [cough] directly into the text stream.6 The model understands these non-verbal cues and generates appropriate audio artifacts, creating a level of "human" performance that is SOTA.Zero-Shot Cloning: The model can clone a target voice from a mere 5-second reference clip.9 This feature relies on the dense vector representations of the Llama backbone to capture and replicate timbre and prosody instantly.3. The Tauri Framework Context: Integration RealitiesThe user's choice of Tauri as the application framework is the defining constraint of this analysis. Tauri operates on a unique architecture distinct from Electron or Native development. A Tauri app consists of two distinct layers:The Core (Backend): Written in Rust. This layer handles system interactions, file I/O, and heavy computation. It compiles down to a native binary.The Webview (Frontend): Written in web technologies (HTML/JS/CSS). This layer handles the UI and communicates with the Core via an asynchronous IPC bridge.For a TTS engine to be "local," it must reside within or be managed by the Rust Core. The feasibility of this integration varies wildly between Desktop (macOS/Linux) and Mobile (iOS/Android).3.1 The "Sidecar Pattern" and Desktop SuccessOn desktop operating systems, Tauri supports a feature known as the Sidecar Pattern. This allows the Rust Core to bundle and spawn external binaries as subprocesses.Mechanism: The developer compiles a Python script (and its interpreter) into a standalone executable using tools like PyInstaller or Nuitka. The Rust Core then uses the Command::sidecar API to launch this executable. Communication occurs via stdin (sending text) and stdout (receiving audio data).Implication for Chatterbox: This pattern makes running Chatterbox on macOS and Linux entirely feasible. The massive Python dependency chain is encapsulated in the sidecar binary. While the installer size bloats to 4GB+, the application runs successfully.Implication for Supertonic: While Supertonic can be run this way (e.g., using a Python wrapper around ONNX Runtime), it is unnecessary. Supertonic's C++ roots allow it to be linked directly into the Rust Core, avoiding the IPC overhead of a sidecar.3.2 The "Mobile Wall": Why Sidecars Fail on iOS & AndroidThe user's requirement for iOS and Android support reveals the fundamental weakness of the Chatterbox architecture in a Tauri context. The "Sidecar Pattern" described above is functionally non-existent on mobile platforms due to strict OS security models.iOS Sandbox ConstraintsApple's iOS enforces a draconian sandbox. An application bundle cannot contain arbitrary executables that are spawned as independent processes. The fork() and exec() system calls—essential for spawning a sidecar—are restricted or forbidden for App Store applications.Furthermore, iOS prohibits Just-In-Time (JIT) compilation for most applications (exceptions exist for browser engines and debuggers, but not general apps). PyTorch and complex Python runtimes heavily rely on JIT for performance. Running them in "interpreter-only" mode results in a catastrophic performance degradation, rendering a 350M parameter model unusable.Android Sandbox ConstraintsAndroid's security model, while slightly more flexible regarding JIT, imposes similar restrictions on subprocesses. While it is theoretically possible to package a Python binary and execute it via the NDK, managing the lifecycle of that process, ensuring it isn't killed by the stringent Android memory manager, and handling the communication bridge is a task of immense complexity. It fights against the grain of the Android application lifecycle.The Dependency Hell of Embedded PythonThe alternative to a sidecar is embedding the Python interpreter directly into the Rust binary (using crates like pyo3). This allows Python code to run within the main application process, bypassing the subprocess restriction.However, this leads to "Dependency Hell." To run Chatterbox, one must embed not just Python, but numpy, scipy, and torch. These are not pure Python libraries; they are wrappers around massive C/C++ and Fortran codebases. Compiling scipy or torch from source for aarch64-linux-android or aarch64-apple-ios and linking them statically into a Rust binary is one of the most notoriously difficult tasks in cross-platform development. It involves resolving thousands of symbol conflicts, matching libc versions, and dealing with build system incompatibilities. For 99% of development teams, this is a non-starter.4. Platform-Specific Integration Analysis: Mobile Deep DiveGiven that Mobile is the "Great Filter" in this selection process, we must analyze the integration pathway for the surviving candidate—Supertonic—and the theoretical (but painful) path for Chatterbox.4.1 Supertonic 2 on Mobile: The Native RouteSupertonic's reliance on ONNX Runtime (ORT) is its superpower here. ORT is designed with mobile in mind.iOS Integration StrategyStatic Linking: The ORT library is distributed as an .xcframework. In Rust, the ort crate can be configured to link against this framework during the build process (cargo build --target aarch64-apple-ios).CoreML Acceleration: iOS devices feature the Apple Neural Engine (ANE). ONNX Runtime supports the CoreML Execution Provider. By enabling this provider in the Rust ort session options, Supertonic inference is offloaded from the CPU to the NPU. This results in faster generation and, critically, drastically lower battery consumption.Asset Management: The 268MB .onnx file is treated as a standard bundle resource. It is accessible to the Rust Core via the NSBundle API (wrapped by Tauri's resource path helpers).Android Integration StrategyJNI and Shared Libraries: Android requires native libraries to be .so files. The ort crate manages the inclusion of libonnxruntime.so into the jniLibs folder of the Android project structure generated by Tauri.NNAPI Acceleration: Similar to CoreML, Android offers the Neural Networks API (NNAPI). Supertonic can leverage this to run on the DSP or NPU of Qualcomm or MediaTek chips, ensuring performance across the fragmented Android hardware ecosystem.App Bundle Size: While 268MB exceeds the 150MB base APK limit, Tauri developers can utilize "Play Asset Delivery" (install-time delivery) to package the model. Since the model is a static file, this is a solved infrastructure problem.4.2 Chatterbox on Mobile: The Remote FallbackSince running Chatterbox locally on mobile is effectively blocked by the OS constraints discussed in Section 3.2, the only viable architecture for a Tauri app wanting to use Chatterbox is a Hybrid Approach.Desktop Users: Enjoy local inference via the Python Sidecar.Mobile Users: The app detects the platform and routes TTS requests to a remote API (hosted by the developer) running the Chatterbox engine.The Cost: This violates the user's "local" requirement. It introduces latency, server costs (GPU hosting for inference), and privacy concerns (data leaving the device). However, it is the only way to access Chatterbox's features on a phone.5. Performance and Resource Profiling: The Cost of QualityPerformance is the secondary selector after compatibility. The user's query mentions "architecture differences," and nowhere is this more visible than in the computational cost of running the models.5.1 Real-Time Factor (RTF) BenchmarksThe "Real-Time Factor" measures the speed of generation. RTF = Processing Time / Audio Duration. An RTF of 0.1 means generating 10 seconds of audio takes 1 second. Lower is better.Supertonic 2 PerformanceDesktop (M4 Pro): Benchmarks indicate an RTF of 0.006.10 This is ~166x faster than real-time. For the user, this means the audio starts playing instantly, with zero perceived latency.Mobile (A17 Pro / Snapdragon 8 Gen 3): Even on mobile silicon, the 44M parameter model flies. Estimations based on similar SLMs suggest an RTF of 0.01 - 0.05 when using NPU acceleration. This enables "streaming" capabilities where long paragraphs are synthesized faster than the user can read them.Chatterbox Turbo PerformanceDesktop (RTX 4090): The model is fast, achieving sub-0.1 RTF.Mobile CPU (Theoretical): If one could run it on a mobile CPU (bypassing the build issues), the 350M parameters would crush the processor. Without heavy quantization (e.g., 4-bit) and optimization, RTF would likely hover between 0.5 and 1.0. This means a 10-second sentence could take 5-10 seconds to generate, creating awkward pauses in conversation or UI interaction.5.2 Memory Footprint & System StabilitySupertonic: Requires ~300-500 MB of RAM. This is safe for almost all modern mobile devices, even low-end Android phones with 4GB RAM. It leaves plenty of room for the OS and the webview.Chatterbox: Requires ~4-5 GB of RAM/VRAM. On a PC, this is fine. On a mobile device, this is catastrophic. iOS aggressively kills background processes that consume excessive memory. An app attempting to allocate 4GB for TTS would likely be terminated immediately upon initialization on all but the most expensive "Pro" model iPhones and Android flagships.6. Technical Integration Guide: Supertonic 2 (Recommended)Based on the evidence, Supertonic 2 is the only viable candidate for a truly local, cross-platform Tauri application. This section details the integration roadmap.6.1 Rust Core ConfigurationThe integration avoids the sidecar pattern entirely. We utilize the ort crate to bind to ONNX Runtime directly within the Rust process.Step 1: Dependency ManagementIn src-tauri/Cargo.toml:Ini, TOML[dependencies] -tauri = { version = "2.0", features = } -# ORT: The interface to ONNX Runtime. -# 'fetch-models' allows auto-downloading libs (mostly for dev). -# 'load-dynamic-lib' is crucial for mobile linking. -ort = { version = "2.0", features = ["fetch-models", "load-dynamic-lib", "ndarray"] } -# Rodio: For cross-platform audio playback -rodio = "0.19" -Step 2: Model Asset BundlingThe 268MB model file must be accessible to the binary at runtime.Place supertonic-v2.onnx and config.json in src-tauri/assets/.Update tauri.conf.json to include these assets:JSON"bundle": { - "resources": ["assets/*"] -} -Step 3: The Inference Engine (Rust)In src-tauri/src/lib.rs, implement a command that the frontend can invoke. This command should:Tokenize: Convert the input string into the specific integer tokens expected by Supertonic. (Note: Check if Supertonic v2 includes a fused tokenizer in the ONNX graph; if not, a small Rust-based tokenizer matching the training data is required).Inference: Pass the tokens to the ort session.Rust// Conceptual Rust Code -let inputs = ort::inputs!["input_ids" => token_tensor]?; -let outputs = session.run(inputs)?; -let audio_data = outputs["audio"].extract_tensor::()?; -Playback: Feed the audio_data into a rodio Sink for immediate playback.6.2 Mobile-Specific Build FlagsAndroid: You must ensure the correct jniLibs are present. You can often rely on the ort crate's build script, but for production, manually downloading the onnxruntime-android AAR and extracting the .so files to your project's android/app/src/main/jniLibs is the most robust method.iOS: You must link the onnxruntime.xcframework. In your build.rs, you may need to emit linker flags:Rustprintln!("cargo:rustc-link-lib=framework=onnxruntime"); -7. Technical Integration Guide: Chatterbox (The Desktop-Only Hybrid)For completeness, if the project demands Chatterbox's features, here is the implementation strategy. Note that this abandons local mobile inference.7.1 Desktop: The Python SidecarEnvironment Isolation: Create a standalone Python environment using uv or conda. Install chatterbox-tts and its heavy dependencies (torch).Freezing the Binary: Use PyInstaller to compile a server.py script into a single binary. This script should launch a local web server (e.g., FastAPI) to listen for TTS requests.Warning: The resulting binary will be 4GB+.Tauri Orchestration:Add the binary to externalBin in tauri.conf.json.On app launch, spawn it via Command::sidecar.Wait for the "ready" signal (monitor stdout).Send HTTP requests to localhost for generation.7.2 Mobile: The Remote API FallbackSince the sidecar cannot run on iOS/Android:Host a Server: Deploy the Chatterbox model to a cloud GPU provider (e.g., RunPod, Lambda Labs, or AWS).Conditional Logic: In your frontend JavaScript:JavaScriptimport { type } from '@tauri-apps/plugin-os'; - -async function generateSpeech(text) { - if (type() === 'android' | - -| type() === 'ios') {// Call Remote APIreturn await fetch('https://api.myapp.com/tts', { body: { text } });} else {// Call Local Sidecarreturn await fetch('http://localhost:8000/tts', { body: { text } });}}```8. Quality of Experience (QoE) AnalysisBeyond the binary "can it run" question lies the "how does it sound" question.8.1 Prosody and StabilitySupertonic 2: The model produces highly stable, intelligible speech. The prosody is consistent, making it ideal for reading long-form content (articles, ebooks). It rarely "hallucinates" or creates bizarre artifacts, a common trait of distilled models. However, it can sound "flatter" or less dynamic than larger models.Chatterbox Turbo: The "human" element is significantly higher. The model captures micro-tremors in pitch, breath intake, and varied pacing that signals high production value. It is better suited for narrative content (fiction, gaming) where emotional engagement is key.8.2 The "Uncanny Valley" of LatencySupertonic: The near-instant response (0.006 RTF) creates a seamless user experience. It feels like a native OS feature.Chatterbox: Even on desktop, the 200ms+ latency can create a "turn-taking" delay in conversational apps. On a slow connection (mobile remote fallback), this latency can spike to seconds, breaking the illusion of interactivity.9. Commercial and Operational Considerations9.1 Licensing and WatermarkingSupertonic 2: Released under the OpenRAIL-M license.5 This license permits commercial use but includes usage restrictions to prevent abuse (e.g., generating deepfakes for fraud). It does not mandate watermarking, though developers should be mindful of transparency.Chatterbox: Released under the MIT license 6, the most permissive option. However, Resemble AI includes PerTh Watermarking technology baked into the model.12 Every generated audio file contains an imperceptible watermark. This is a robust safety feature for a commercial app, allowing you to prove the provenance of the audio if challenged, but it incurs a small computational cost during inference.9.2 Update VelocitySupertone Inc.: The release of v2 in Jan 2026 suggests a committed roadmap. The shift to a unified multilingual architecture indicates a maturity in their R&D pipeline.Resemble AI: Chatterbox is an open-source offshoot of their core commercial product. Updates are frequent, but often prioritize their paid API services or newer, larger models that may drift further away from consumer hardware capability.10. Conclusion and Strategic RecommendationThe comparative analysis yields a definitive conclusion based on the user's specific constraint of running locally across iOS, Android, Mac, and Linux.The Recommendation: Supertonic 2 is the superior architectural choice.Mobile Feasibility: Supertonic 2 is the only candidate that offers a viable path to local inference on iOS and Android within a Tauri application. Its ONNX-native architecture allows for static linking and NPU acceleration, bypassing the OS restrictions that block Chatterbox's Python-based stack.Performance Profile: With an inference speed 166x faster than real-time and a memory footprint of under 500MB, Supertonic ensures the application remains responsive and stable on resource-constrained mobile devices. Chatterbox's 4GB requirement is a non-starter for mobile memory budgets.Integration Simplicity: While Rust requires a learning curve, the ort integration is cleaner and more robust than maintaining a fragile Python sidecar build chain.When to Consider Chatterbox:Chatterbox should only be selected if the application is Desktop-exclusive (Mac/Linux/Windows) or if the requirement for Zero-Shot Voice Cloning and Paralinguistic Tags (laughter, emotion) outweighs the requirement for "Local" execution on mobile. In that scenario, a hybrid architecture (Local Desktop + Remote Mobile API) is the only path forward.For the stated goal of a unified, local, cross-platform Tauri build, Supertonic 2 is not just the better option; it is practically the only option. - - -## Claude - -# Local TTS for Tauri: Supertonic vs Chatterbox compared - -**Supertonic emerges as the clear choice for cross-platform Tauri deployment**, offering native Rust integration, ~264 MB model size, and proven iOS/Android support out of the box. Chatterbox provides superior voice cloning and emotion control but at **10-12× the model size** and significantly higher deployment complexity. For a privacy-focused chat application prioritizing simplicity and bundle size, Supertonic's ONNX-based architecture delivers the most practical path to production. - -## Model architecture and runtime requirements - -**Supertonic** runs entirely on ONNX Runtime, making it deployment-friendly across all platforms. The architecture splits into four ONNX components: text encoder (28 MB), vector estimator (133 MB), vocoder (101 MB), and duration predictor (1.6 MB). With only **66 million parameters**, it's deliberately optimized for edge devices—proven to run on Raspberry Pi and e-readers at 0.3× real-time factor. - -**Chatterbox** was built on PyTorch with a **0.5B Llama backbone**, requiring substantially more resources. Three model variants exist: the original 500M parameter model, Chatterbox-Multilingual (500M, 23 languages), and Chatterbox-Turbo (350M, optimized for speed). While native inference requires PyTorch with CUDA/MPS/ROCm backends, official ONNX exports now exist through `ResembleAI/chatterbox-turbo-ONNX`. - -| Specification | Supertonic | Chatterbox | -|--------------|------------|------------| -| Parameters | 66M | 350M-500M | -| Native framework | ONNX Runtime | PyTorch | -| ONNX available | ✅ Primary | ✅ Exported | -| MLX support | ❌ | ✅ via mlx-audio | - -## Model sizes shape deployment decisions - -Supertonic's total ONNX bundle weighs approximately **264 MB** across all components, with OnnxSlim optimizations shaving a few megabytes. This size remains consistent since the architecture doesn't support quantization variants in the official release. - -Chatterbox offers more flexibility through quantization but starts much larger. The full-precision Turbo ONNX export totals **~3.3 GB** across its four sessions (speech encoder, language model, conditional decoder, embed tokens). Quantized variants dramatically reduce this: - -- **Q4F16** (4-bit with FP16): ~560 MB total -- **INT8 (Q8)**: ~1.1 GB total -- **FP16**: ~1.7 GB total - -For mobile deployment, the Q4F16 Chatterbox variant at 560 MB remains **roughly twice Supertonic's size**. Memory requirements diverge even more sharply: Supertonic runs comfortably in **250-500 MB RAM**, while Chatterbox ONNX peaks at **~3.2 GB RAM** on iOS based on real-world testing. - -## Cross-platform deployment capabilities - -Supertonic provides exceptional platform coverage with **official examples for every major platform** in its repository: - -- **Desktop**: Windows, macOS, Linux via C++, Rust, Go, Python, Node.js, Java, C# -- **Mobile**: Native iOS (Swift/Xcode), Android (Java/Kotlin), Flutter -- **Web**: WebGPU/WASM (Chrome 121+, Edge 121+, Safari macOS 15+) -- **Embedded**: Proven on Raspberry Pi, Onyx Boox e-readers - -Chatterbox's platform support depends heavily on your chosen runtime: - -- **PyTorch native**: Linux (primary), macOS (MPS), Windows (CUDA/CPU only) -- **ONNX Runtime**: All platforms theoretically supported; iOS demonstrated working -- **MLX**: macOS 14.0+ and iOS 16.0+ only (Apple Silicon exclusive) -- **Android**: ONNX Runtime supports it, but not officially tested - -## Rust integration and Tauri compatibility - -**Supertonic offers native Rust support** directly in the repository's `rust/` directory. The implementation uses ONNX Runtime Rust bindings, making Tauri integration straightforward—you can call TTS directly from your Rust backend without spawning external processes. - -```rust -// Supertonic approach: Native Rust in Tauri backend -// Uses ort crate (ONNX Runtime) directly -``` - -**Chatterbox lacks official Rust bindings**, creating three integration paths for Tauri: - -1. **ONNX via `ort` crate**: Load quantized ONNX models directly from Rust—no Python required, works cross-platform -2. **Python sidecar**: Bundle PyInstaller-compiled Python with Tauri's `externalBin` feature -3. **Local HTTP server**: Run chatterbox-tts-api as subprocess with OpenAI-compatible endpoints - -The Python sidecar approach has been documented for Chatterbox with mlx-audio. Configure `tauri.conf.json` with `"externalBin": ["binaries/tts-sidecar"]`, compile Python using PyInstaller with target-specific naming (`tts-sidecar-x86_64-apple-darwin`), and spawn via `app.shell().sidecar()`. Known issues include sidecars not terminating cleanly on app close and **50-200 MB additional bundle size** for the Python runtime. - -## Voice quality and feature comparison - -Both systems produce high-quality, natural speech—neither sounds robotic in typical usage. - -**Supertonic** offers configurable inference steps trading speed for quality: -- 2-step inference: "Close to ElevenLabs Flash" quality, fastest -- 5-step inference: "Reaches much of ElevenLabs Prime tier" -- 10+ steps: Highest quality, slower - -It includes **11 preset voices** (5 male, 5 female) and excels at text normalization—handling currencies ($5.2M), dates, phone numbers, and abbreviations without preprocessing. Supertonic 2, released January 6, 2026, added support for English, Korean, Spanish, Portuguese, and French. - -**Chatterbox** won **63.75% preference over ElevenLabs** in blind evaluations and offers richer features: -- **Zero-shot voice cloning** from 5-10 seconds of reference audio -- **Emotion exaggeration control** (0 = monotone, 1 = normal, 2+ = dramatic) -- **Paralinguistic tags**: `[laugh]`, `[cough]`, `[sigh]`, `[groan]` -- **23 languages** in the multilingual model -- Built-in neural watermarking (PerTh) for provenance tracking - -## Performance benchmarks reveal the gap - -Supertonic's lightweight architecture delivers exceptional speed-to-quality ratios: - -| Hardware | Supertonic RTF | Throughput | -|----------|---------------|------------| -| M4 Pro (CPU) | 0.015 | 1,263 chars/sec | -| M4 Pro (WebGPU) | 0.006 | 2,509 chars/sec | -| RTX 4090 | 0.001 | 12,164 chars/sec | -| Raspberry Pi | 0.3 | Real-time capable | - -Chatterbox requires more compute but achieves competitive latency: -- **Streaming RTF**: 0.499 on RTX 4090 -- **Latency**: Sub-200ms optimized, sub-300ms typical -- **Apple Silicon via MLX**: 2-3× faster than CPU -- **Mobile (iOS ONNX)**: Functional but ~3.2 GB peak RAM - -## Licensing permits commercial use - -Both projects use permissive licenses suitable for commercial applications: - -| Aspect | Supertonic | Chatterbox | -|--------|------------|------------| -| Code license | MIT | MIT | -| Model license | OpenRAIL-M | MIT | -| Commercial use | ✅ Allowed | ✅ Allowed | -| Voice cloning | Not supported | Built-in | -| Watermarking | None | Optional neural watermark | - -## Practical deployment recommendation - -For your privacy-focused Tauri chat application, **Supertonic provides the most straightforward path**: - -1. **Native Rust integration**: Add ONNX Runtime via `ort` crate, call TTS directly from Tauri backend -2. **Minimal bundle impact**: ~264 MB models + ~7-15 MB ONNX Runtime per platform -3. **True cross-platform**: iOS, Android, macOS, Linux work today with official examples -4. **No Python dependency**: Ships as pure native code - -Choose Chatterbox only if you specifically need **voice cloning** or **emotion control**. In that case, use the Q4F16 ONNX variant (~560 MB) with the `ort` crate for maximum portability. Avoid the Python sidecar approach on mobile—it's impractical due to bundle size and process management complexity. - -## Conclusion - -The two systems serve different use cases despite both being local TTS solutions. Supertonic optimizes for **deployment simplicity and efficiency**—its 66M parameter model runs on constrained devices while maintaining quality competitive with cloud services. Chatterbox optimizes for **feature richness**—voice cloning and emotion control justify its 5-8× larger footprint for applications requiring those capabilities. - -For a Tauri app shipping across iOS, Android, macOS, and Linux with privacy as a priority, Supertonic's architecture eliminates the primary pain points: no Python runtime, no platform-specific codepaths for inference, and proven mobile deployment. The January 2026 Supertonic 2 release with multilingual support further strengthens this recommendation for international applications. - ---- - -## Grok - -### Key Considerations for Choosing Between Supertonic and Chatterbox -- **Model Size and Resource Efficiency**: Research indicates Supertonic (66 million parameters) is significantly smaller and more lightweight than Chatterbox models (350-500 million parameters), making it better suited for resource-constrained devices across platforms. -- **Cross-Platform Compatibility**: Evidence leans toward Supertonic being more reliable and easier to deploy on iOS, Android, Mac, and Linux due to its ONNX Runtime integration, with native examples for mobile and desktop; Chatterbox, while feasible on Mac and potentially iOS via optimizations like MLX, lacks straightforward Android support and relies on Python, complicating Tauri integration. -- **Ease of Integration in Tauri Apps**: Supertonic appears more straightforward for Rust-based Tauri builds with direct Rust examples, while Chatterbox may require a Python sidecar (e.g., with MLX-audio on Apple devices), adding complexity but remaining viable for certain setups. -- **Reliability and Recent Developments**: Both are actively maintained, but Supertonic 2's January 2026 release introduces multilingual support and optimizations, suggesting it may offer fresher reliability for local runs; user feedback highlights its stability for long-form audio, though Chatterbox excels in voice cloning and expressiveness. -- **Recommendation**: For local TTS in Tauri across all specified platforms, Supertonic seems likely the better choice due to its smaller size, broader native compatibility, and simpler integration, though Chatterbox could suit scenarios prioritizing advanced features like zero-shot cloning if platform constraints are flexible. - -#### Tech Stack Overview -Supertonic leverages ONNX Runtime for inference, enabling efficient, CPU-optimized execution without heavy dependencies—ideal for embedding in cross-platform apps. It supports batch processing and configurable inference steps (e.g., 2-5), with models hosted on Hugging Face (https://huggingface.co/Supertone/supertonic-2). Chatterbox, built on PyTorch, requires Python environments and supports CUDA, CPU, or MPS (Metal Performance Shaders) for Apple devices, but integration often involves additional tools like MLX-audio for optimization on Mac/iOS. - -#### Performance and Features -Supertonic prioritizes speed, achieving real-time factors (RTF) as low as 0.006 on WebGPU and handling complex text (e.g., dates, currencies) natively. Chatterbox focuses on quality, with features like paralinguistic tags ([laugh], [chuckle]) for expressive speech and zero-shot voice cloning, though it may require more VRAM (around 5-6.5GB on GPUs). For local runs, Supertonic's lightweight design suggests lower latency on diverse hardware, while Chatterbox performs well on supported setups but could be slower on non-optimized platforms. - -#### Platform-Specific Insights -- **iOS/Android**: Supertonic offers native iOS examples and Flutter support for cross-mobile deployment; ONNX Runtime ensures compatibility (https://onnxruntime.ai/docs/build/ios.html). Chatterbox has been adapted for iOS via apps like Chinny, but Android integration is less documented and may need custom work. -- **Mac/Linux**: Both run well—Supertonic via Swift/Rust on Mac and various examples on Linux; Chatterbox uses MPS on Mac for acceleration and is straightforward on Linux with CUDA/CPU. - -If prioritizing minimal footprint and ease, start with Supertonic's Rust example for Tauri: clone the repo, build with Cargo, and integrate ONNX models directly. - ---- -### In-Depth Analysis of Supertonic and Chatterbox for Cross-Platform Local TTS in Tauri Applications - -Text-to-speech (TTS) models like Supertonic and Chatterbox represent significant advancements in open-source AI, enabling privacy-focused, offline audio generation for applications such as voice assistants, accessibility tools, and content creation. As of early 2026, these models cater to developers building cross-platform apps with frameworks like Tauri, which allows Rust-based backends for web-like UIs on iOS, Android, Mac, and Linux. This analysis draws from repository details, user integrations, performance benchmarks, and community feedback to evaluate their suitability for local deployment. We prioritize factors like model size, tech stack, platform compatibility, reliability, and Tauri-specific integration, acknowledging that while both are capable, differences in architecture influence their ease of use across devices. - -#### Model Architectures and Core Technologies -Supertonic, developed by Supertone Inc., is an ONNX-based TTS system optimized for on-device inference with minimal overhead. Its core relies on ONNX Runtime, a cross-platform engine that supports CPU-optimized execution (GPU untested in the repo but feasible). Models are slimmed using OnnxSlim, resulting in efficient, lightweight files. The system generates 16-bit WAV audio, supports batch processing for throughput, and handles natural text variations (e.g., phone numbers, units) without preprocessing. Supertonic 2, released on January 6, 2026 (v2.0.0), expands to multilingual support for English, Korean, Spanish, Portuguese, and French, with six new voice styles (M3-M5, F3-F5). It's licensed under MIT for code and OpenRAIL-M for models, allowing commercial use. - -Chatterbox, from Resemble AI, is a PyTorch-based family of models: the original (500M parameters, English-only), Multilingual (500M, 23+ languages), and Turbo (350M, English with paralinguistic tags like [chuckle] or [cough]). It emphasizes high-fidelity, zero-shot voice cloning, and expressive speech via configurable parameters (e.g., CFG for guidance, exaggeration for emotion). All include Perth watermarking for ethical traceability. The Turbo variant distills the decoder to a single generation step, reducing latency and VRAM needs. It's MIT-licensed and installable via pip, with dependencies managed in pyproject.toml for Python 3.11 on Debian-like systems. - -Key tech differences: Supertonic's ONNX focus enables broader runtime flexibility without Python, while Chatterbox's PyTorch ties it to Python environments, potentially requiring sidecars in non-Python apps like Tauri. - -#### Model Sizes and Resource Requirements -Model size directly impacts local feasibility, especially on mobile devices with limited RAM/VRAM. - -| Model | Variant | Parameters | Approximate Size | VRAM Usage (GPU) | Key Optimizations | -|-------|---------|------------|------------------|------------------|-------------------| -| Supertonic | Supertonic 2 | 66M | Ultra-lightweight (optimized ONNX files) | Minimal (CPU-focused; ~low GB if GPU) | OnnxSlim for compression; batch support | -| Chatterbox | Turbo | 350M | Medium | ~5GB (e.g., RTX 3060) | Distilled decoder; low-latency mode | -| Chatterbox | Multilingual/Original | 500M | Larger | ~6.5GB | Zero-shot cloning; expressive tuning | - -Supertonic's 66M parameters make it the smallest, enabling runs on edge devices like Raspberry Pi or e-readers with RTF as low as 0.012 on CPU. Chatterbox models, at 350-500M, demand more resources but offer efficiencies like 1-step generation in Turbo, using ~5GB VRAM for faster output (e.g., 1.8x speed over original). For Tauri apps, Supertonic's footprint reduces bundling overhead, while Chatterbox may need quantized versions (e.g., 6-bit via MLX) for mobile. - -#### Performance Benchmarks and Features -Performance varies by use case: speed vs. quality. - -- **Speed and Latency**: Supertonic excels, processing up to 12,164 characters/second on RTX 4090 and 167x real-time on M4 Pro Mac, with RTF 0.006 on WebGPU. It's faster than Chatterbox on non-NVIDIA hardware. Chatterbox Turbo achieves sub-200ms latency, suitable for real-time agents, and handles long texts stably via chunking. -- **Audio Quality and Expressiveness**: Chatterbox leads in naturalness, with low word error rates, emotional carry-over, and tags for non-verbal cues; it outperforms paid services like ElevenLabs in cloning (7-11s reference audio). Supertonic provides stable, natural long-form narration but lacks cloning or advanced emotion tuning, focusing on clear, reliable output. -- **Multilingual Support**: Supertonic 2 adds five languages; Chatterbox Multilingual covers 23+. - -In comparisons, Supertonic is praised for efficiency in resource-limited scenarios, while Chatterbox shines in expressive, cloned audio. - -#### Cross-Platform Compatibility and Deployment -ONNX Runtime makes Supertonic highly portable: it supports iOS (native Xcode), Android (via Flutter), Mac (Swift/MPS), Linux (multiple languages), and even browsers (WebGPU/WASM). Installation involves cloning the repo, Git LFS for models, and language-specific builds (e.g., `cargo build` for Rust). - -Chatterbox supports Mac (MPS), Linux (CUDA/CPU), and Windows (GPU), with iOS adaptations via apps like Chinny for offline runs. Android integration is not native; it may require embedding Python or API wrappers. MLX-audio optimizes for Apple Silicon, enabling faster inference on Mac/iOS. - -For Tauri: Supertonic integrates directly via Rust examples, embedding ONNX in the backend. Chatterbox uses a Python sidecar (e.g., via tauri-plugin-shell), running scripts as external processes—feasible but adds overhead, especially with MLX-audio for Apple platforms. - -#### Reliability, Maintenance, and Community Feedback -Both are reliable for local use, with active repos (Supertonic updated January 2026; Chatterbox last in mid-2025). Supertonic's on-device privacy and stability for long narration earn praise in integrations (e.g., browser extensions, e-book apps). Early Supertonic 2 reviews highlight its speed and natural voices on Windows/Mac. Chatterbox is lauded for beating paid TTS in quality but may face setup issues on non-GPU devices. - -Pros/Cons Table: - -| Aspect | Supertonic | Chatterbox | -|--------|------------|------------| -| **Pros** | Ultra-fast, small size, broad platforms, easy Rust/Tauri integration | Superior quality/cloning, expressive tags, multilingual options | -| **Cons** | Limited voices/cloning, recent multilingual addition | Larger size, Python-dependent, limited mobile native support | -| **Best For** | Speed-focused, cross-platform local apps | Quality-focused, voice agents with cloning | - -#### Recommendations and Implementation Tips -For Tauri builds targeting all platforms, Supertonic is preferable: download models from Hugging Face, use Rust bindings for backend inference, and expose via Tauri commands. Test on devices like Raspberry Pi for edge cases. If voice cloning is essential, opt for Chatterbox with a Python sidecar—use MLX-audio for Mac/iOS acceleration and explore quantization for Android. Always verify with hardware benchmarks, as real-world performance varies. - -### Key Citations -- [GitHub - supertone-inc/supertonic](https://github.com/supertone-inc/supertonic) -- [GitHub - resemble-ai/chatterbox](https://github.com/resemble-ai/chatterbox) -- [Lightning-Fast Open-Source TTS Model Is Here!](https://medium.com/@bytefer/12-164-characters-per-second-lightning-fast-open-source-tts-model-is-here-96452a3dd66f) -- [ONNX Runtime | Home](https://onnxruntime.ai/) -- [ONNX Runtime: Enabling Cross-Platform AI Model Inference](https://medium.com/aimonks/onnx-runtime-enabling-cross-platform-ai-model-inference-80f136ecbb2d) -- [This Local AI Voice Model Beats Paid TTS (Chatterbox Turbo Tested)](https://www.youtube.com/watch?v=3UN70ea4h9c) -- [A few insights from running Chatterbox fully offline on iPhone and Mac](https://huggingface.co/ResembleAI/chatterbox/discussions/42) -- [Blaizzy/mlx-audio](https://github.com/Blaizzy/mlx-audio/actions) -- [Supertonic TTS is incredible — sharing our integration results](https://huggingface.co/Supertone/supertonic/discussions/13) -- [Supertonic TTS: Ultra-Fast, Natural Voices on Windows & Mac](https://sonusahani.com/blogs/supertonic-tts) -- [Supertonic Review: A New, Ultra-Fast and Lightweight AI Model](https://www.youtube.com/watch?v=g2CuKBqCeK0) -- [The Top Open-Source Text to Speech (TTS) Models](https://modal.com/blog/open-source-tts) -- [The Best Open-Source Text-to-Speech Models in 2026](https://www.bentoml.com/blog/exploring-the-world-of-open-source-text-to-speech-models) -- [Embedding External Binaries](https://v2.tauri.app/develop/sidecar/) -- [How Chatterbox Is Redefining Open-Source AI Voice](https://medium.com/%40vinodpolinati/chatterbox-the-leading-family-of-open-source-ai-voice-models-thats-changing-the-game-150991d5cf8d) diff --git a/docs/unified-chat-refactor.md b/docs/unified-chat-refactor.md deleted file mode 100644 index c603e0386..000000000 --- a/docs/unified-chat-refactor.md +++ /dev/null @@ -1,335 +0,0 @@ -# Unified Chat Refactor - Phase 1 - -## Overview - -This document describes the initial refactor of Maple's chat interface in preparation for migrating from the current localStorage-based chat system to OpenAI's Conversations/Responses API. - -## Motivation - -The existing chat architecture had several pain points: - -1. **Scattered State Management**: Chat state was distributed across multiple components and routes: - - `frontend/src/routes/index.tsx` - Home page with ChatBox - - `frontend/src/routes/_auth.chat.$chatId.tsx` - Individual chat route - - `frontend/src/components/ChatBox.tsx` - Shared chat input component - - Complex prop drilling and state synchronization between these components - -2. **Complex Routing Logic**: The system required careful coordination between routes, with state being passed through navigation params, leading to: - - Difficult debugging when state got out of sync - - Re-rendering and remounting issues on navigation - - Complex URL management logic - -3. **Preparation for API Migration**: The upcoming switch to OpenAI's Conversations/Responses API requires a simpler architecture that can handle: - - Server-side conversation state - - Streaming responses - - No dependency on localStorage for chat history - -## Architectural Decisions - -### 1. Monolithic Component Design - -We created a single `UnifiedChat` component that contains all chat functionality: - -```typescript -// frontend/src/components/UnifiedChat.tsx -export function UnifiedChat() { - // ALL chat state lives here - const [messages, setMessages] = useState([]); - const [input, setInput] = useState(""); - const [isGenerating, setIsGenerating] = useState(false); - // ... -} -``` - -**Rationale**: -- Following the principle "Premature abstraction is the root of all evil" -- Colocated code is easier to debug and understand -- No state synchronization bugs between components -- Similar to how large tech companies (Meta, etc.) handle complex components - -### 2. URL Management Without Navigation - -Instead of using TanStack Router navigation (which causes remounting), we use browser-native `window.history.replaceState()`: - -```javascript -// Update URL without any navigation/reload -const usp = new URLSearchParams(window.location.search); -usp.set("conversation_id", newChatId); -window.history.replaceState(null, "", `/?${usp.toString()}`); -``` - -**Benefits**: -- No component remounting -- No state loss -- URL updates for shareability/bookmarking -- No "route not found" errors (query params don't need routes) - -### 3. Query Parameters Over Route Parameters - -We use `?conversation_id=xxx` instead of `/chat/xxx`: - -- **Before**: `/chat/123` - Requires route file, causes navigation -- **After**: `/?conversation_id=123` - No route needed, just URL update - -This approach avoids the need for route configuration while maintaining URL-based state. - -### 4. Preserved Existing Infrastructure - -We maintained backward compatibility: -- Old `/chat/$chatId` routes still work -- Existing Sidebar component is reused -- Auth logic and modals (team setup, API keys) remain functional -- Search parameters for callbacks (`team_setup`, `credits_success`) preserved - -## Implementation Details - -### File Structure - -**New Files**: -- `frontend/src/components/UnifiedChat.tsx` - The unified chat component -- `frontend/src/routes/index.backup.tsx` - Backup of original index - -**Modified Files**: -- `frontend/src/routes/index.tsx` - Simplified to show Marketing or UnifiedChat based on auth -- `frontend/src/components/Sidebar.tsx` - Updated "New Chat" to clear conversation_id - -### State Management - -Currently using local React state with mocked responses: - -```typescript -// Mock AI response - will be replaced with OpenAI conversations API -setTimeout(() => { - const assistantMessage: Message = { - id: `msg-${Date.now()}-ai`, - role: "assistant", - content: "Hello world! This is a mocked response...", - timestamp: Date.now() - }; - setMessages(prev => [...prev, assistantMessage]); -}, 1000); -``` - -This will be replaced with actual API calls in Phase 2. - -### New Chat Flow - -1. User clicks "New Chat" in sidebar -2. Sidebar clears `conversation_id` from URL -3. Dispatches 'newchat' event -4. UnifiedChat listens and clears messages -5. Input field gets focus - -## Benefits Achieved - -1. **Simplified Codebase**: ~250 lines in one file vs ~500+ lines across multiple files -2. **No State Synchronization Issues**: Single source of truth -3. **Better Performance**: No unnecessary re-renders or navigation -4. **Easier Debugging**: All logic in one place -5. **Ready for API Migration**: Clean foundation for OpenAI integration - -## Next Steps (Phase 2) - -1. **OpenAI Conversations API Integration**: - - Replace mock responses with actual API calls - - Implement streaming responses - - Handle conversation creation and management - -2. **Remove localStorage Dependency**: - - Migrate chat history to server-side storage - - Update Sidebar to fetch from API instead of localStorage - -3. **Error Handling & Edge Cases**: - - Handle API failures gracefully - - Implement retry logic - - Add loading states for conversation fetching - -## Design Philosophy - -This refactor follows the principle of **"Make it work, make it right, make it fast"**: - -1. **Make it work**: Single component with all functionality (current state) -2. **Make it right**: Will be achieved with API integration -3. **Make it fast**: Can optimize/split components later if needed - -By avoiding premature optimization and keeping everything in one place, we've created a maintainable foundation that can evolve as requirements become clearer. - -## Technical Decisions Explained - -### Why Not Cache Conversations? - -We explicitly decided against caching for now: -- Most users work on one conversation at a time -- API is fast enough that loading isn't painful -- Adds complexity that may not be needed -- Can be added later if users report performance issues - -### Why Query Parameters? - -- No route configuration needed -- Works immediately without router setup -- Prevents "route not found" errors -- Can be migrated to proper routes later if needed - -### Why Keep Everything in One Component? - -- Based on real-world experience at major tech companies -- Easier to understand and debug -- No props drilling or state synchronization -- Can be split later when natural boundaries emerge - -## Current Implementation Status - -### ✅ Features Successfully Implemented - -The UnifiedChat component now includes these fully working features: - -#### Core Chat Functionality -- **Conversations/Responses API Integration** - Full server-side state management with OpenAI-compatible endpoints -- **Streaming responses** - Real-time SSE event handling for all response types -- **Message deduplication** - Smart ID management using server-assigned IDs with smooth local-to-server transitions -- **URL-based conversation routing** - Query parameter approach (`?conversation_id=xxx`) avoiding route configuration -- **5-second polling** - Automatic synchronization for cross-device conversations -- **Conversation lifecycle** - Lazy creation, loading from URL, switching between conversations - -#### User Interface -- **Modern ChatGPT-style UI** - Clean aesthetics with hover states and subtle backgrounds -- **Auto-scrolling** - Intelligent scroll on new messages (user and assistant) -- **Copy to clipboard** - One-click copy for assistant messages -- **React.memo optimization** - MessageList component prevents re-renders during input -- **Responsive sidebar** - Mobile-friendly with toggle button -- **Centered input for new chats** - Beautiful welcome screen with logo and prompt -- **Fixed input for active chats** - Standard chat interface when conversation is active -- **Mobile new chat button** - Quick access button in mobile header when in a conversation -- **Consistent mobile UI** - Aligned headers and consistent button styling across sidebar and main chat - -#### Multimodal Support -- **Image attachments** - Support for JPEG, PNG, WebP up to 10MB -- **Document parsing** - PDF, TXT, MD support (PDF requires Tauri) - - Fixed Tauri command: Uses `extract_document_content` instead of `parse_document` - - Simplified JSON format: Documents stored as `{ document: { filename, text_content } }` - - Removed unnecessary `status` and `errors` fields from document structure - - Proper markdown rendering with document preview button -- **Attachment preview** - Visual previews with remove capability -- **Auto model switching** - Automatically selects vision-capable models when images added -- **Plus button dropdown** - Clean attachment interface -- **Proper OpenAI format** - Uses `input_text`, `input_image`, `output_text` content types - -#### Billing & Access Control -- **Tier-based features** - Starter (images), Pro/Team (documents) -- **Upgrade prompts** - Contextual dialogs when accessing restricted features -- **Model selector integration** - Shows available models based on user's plan - -#### Error Handling -- **404 recovery** - Gracefully handles non-existent conversations -- **Network error display** - User-friendly error messages -- **Silent polling failures** - Doesn't interrupt user experience -- **Attachment validation** - File type and size checks with clear feedback - -### ✅ Recently Implemented Features - -#### Voice Recording (Completed December 2024) -- **Voice recording** - Microphone input with RecordRTC -- **Whisper transcription** - Convert speech to text via OpenSecret API -- **Recording overlay** - Visual feedback with waveform animation -- **Proper overlay positioning** - Covers only input area, not full page -- **Access control** - Requires Pro/Team tier and Whisper model availability -- **Error handling** - Clear messages for permission issues - -### ❌ Features Not Yet Migrated - -These features exist in the old components but haven't been implemented in UnifiedChat: - -#### TTS Features (Postponed - API not working) -- **Text-to-Speech (TTS)** - Kokoro voice synthesis with play/stop controls -- **Auto-play TTS** - Automatic playback for voice-initiated messages -- **Audio manager** - Prevents multiple TTS playing simultaneously - -#### ✅ Scroll Behavior (Completed December 2024) -- **Smart auto-scroll logic** - Improved scrolling that matches old behavior: - - Instant scroll to bottom on initial chat load - - Auto-scroll when user sends a message - - Auto-scroll slightly (100px) when assistant starts streaming - - No auto-scroll while streaming (lets user read at their pace) - - Auto-scroll when new messages arrive from polling (e.g., after refresh) - - Maintains scroll position when user has scrolled up -- **User scroll detection** - Tracks if user is within 100px of bottom -- **Scroll-to-bottom button** - Could be added but not currently implemented - -#### System Prompt (Coming Soon via API) -- **System prompt support** - Will be handled via new API, not frontend input -- **Collapsible display** - Will need UI for showing system prompts when implemented - -#### UI/UX Features -- **Draft message persistence** - localStorage backup of unsent messages - -#### Advanced Features -- **Document metadata tracking** - Preserve filename and full content -- **Multi-file selection** - Batch image uploads -- **Message-specific actions** - Per-message TTS controls - -### 🎯 Feature Prioritization - -Based on user value and implementation complexity: - -#### High Priority (Essential) -1. ✅ **Voice Input** - COMPLETED! Recording and transcription working -2. ✅ **Token Management** - HANDLED BY BACKEND! Intelligent compression on server-side -3. ✅ **Streaming indicators** - COMPLETED! Different implementation but working well -4. ✅ **Scroll behavior** - COMPLETED! Smart auto-scrolling with user detection -5. **TTS** - Postponed until API is fixed - -#### Medium Priority (Nice to Have) -6. **System prompt support** - Coming via new API -7. **Draft persistence** - Prevents data loss on refresh - -#### Low Priority (Already Done or Not Needed) -9. ✅ **Mobile new chat button** - Already implemented -10. ✅ **Token warnings** - Not needed, backend handles compression automatically -11. **Message-specific TTS controls** - Will implement when TTS API is fixed - -### 🏗️ Architecture Improvements Achieved - -The refactor has delivered significant architectural improvements: - -1. **Single Component Architecture** - All logic in UnifiedChat.tsx, no prop drilling -2. **Server-Driven State** - No localStorage dependencies for chat data -3. **Clean URL Management** - Query parameters avoid complex routing -4. **Optimized Rendering** - Strategic use of React.memo prevents unnecessary re-renders -5. **Proper Error Boundaries** - Graceful handling of API failures -6. **Event-Based Communication** - Clean integration with sidebar via custom events -7. **Abort Controllers** - Proper cleanup of in-flight requests -8. **Resource Management** - Proper cleanup of object URLs and event listeners - -### 📊 Comparison with Old Architecture - -| Aspect | Old Implementation | New UnifiedChat | -|--------|-------------------|----------------| -| **Files** | 3+ components, multiple routes | Single component | -| **State Management** | Props, localStorage, context | Local React state + API | -| **Chat Persistence** | localStorage | Server-side via API | -| **Routing** | `/chat/:chatId` with route files | `?conversation_id=xxx` query params | -| **Message IDs** | Client-generated only | Server-assigned with local fallback | -| **Polling** | None | 5-second interval with cursor | -| **Code Complexity** | ~500+ lines across files | ~1276 lines in one file | -| **Debugging** | Difficult (scattered logic) | Easy (colocated code) | - -### 🚀 Next Steps - -1. **Implement Voice Features** - Add recording and TTS for accessibility -2. **Add Token Management** - Implement counting and compression -3. **Enhance UX** - Add scroll-to-bottom and streaming indicators -4. **Performance Optimization** - Consider splitting component if it grows much larger -5. **Testing** - Add comprehensive tests for the unified component - -## Conclusion - -The UnifiedChat refactor has successfully achieved its primary goals: -- ✅ Simplified architecture with single component -- ✅ Full Conversations/Responses API integration -- ✅ Removed localStorage dependencies for chat data -- ✅ Maintained all essential functionality -- ✅ Improved performance with React.memo -- ✅ Created foundation for future enhancements - -While some features from the old implementation haven't been migrated yet, the core chat experience is fully functional and the architecture is much cleaner. The missing features are primarily UX enhancements that can be added incrementally based on user feedback and priorities. diff --git a/frontend/src-tauri/apple-sign-in-info.md b/frontend/src-tauri/apple-sign-in-info.md deleted file mode 100644 index 30fcc26d2..000000000 --- a/frontend/src-tauri/apple-sign-in-info.md +++ /dev/null @@ -1,102 +0,0 @@ -# Sign in with Apple Integration - -## Overview -This document provides a comprehensive guide to the Sign in with Apple integration for Maple, supporting both native iOS authentication and web-based OAuth. - -## Integration Types - -### 1. Native iOS Authentication -- Uses the iOS native Sign In with Apple dialog -- Implemented via `tauri-plugin-sign-in-with-apple` (version 1.0.0) -- Returns user credentials directly to the app -- Provides access to user identifiers, email, and name (only on first sign-in) - -### 2. Web-based OAuth Flow -- Similar to GitHub and Google OAuth flow -- Redirects users to Apple's authentication page -- Supports both web and desktop (non-iOS) platforms -- Handles callback with auth code and state verification - -## Configuration - -### iOS Native Auth -1. Required entitlements have been added in `maple_iOS.entitlements` -2. The capability is registered in `capabilities/default.json` and `capabilities/mobile-ios.json` -3. Plugin is configured in Cargo.toml and registered in the app - -### OAuth Configuration -1. Set up these parameters in your OpenSecret project settings: - - **Client ID**: Your Apple Services ID (e.g., com.example.web) - - **Client Secret**: The base64-encoded contents of your Apple private key (p8 file) - - **Redirect URI**: Configure as `https://api.opensecret.cloud/auth/apple/callback` - -## Implementation Details - -### Frontend Integration - -#### iOS Native Flow -```typescript -// iOS native authentication -const result = await invoke("plugin:sign-in-with-apple|get_apple_id_credential", { - payload: { - scope: ["email", "fullName"], - state: "apple-auth-state", - options: { debug: true } - } -}); - -// Format and send to backend -const appleUser = { - user_identifier: result.user, - identity_token: result.identityToken, - email: result.email, - given_name: result.fullName?.givenName, - family_name: result.fullName?.familyName -}; - -// Call OpenSecret SDK -await os.handleAppleNativeSignIn(appleUser, inviteCode); -``` - -#### Web OAuth Flow -```typescript -// Web OAuth authentication -const { auth_url } = await os.initiateAppleAuth(inviteCode); -window.location.href = auth_url; - -// Callback handling (in separate component) -await handleAppleCallback(code, state, inviteCode); -``` - -### Platform Detection -The app automatically determines the appropriate flow: -1. Checks if the app is running on iOS and uses native flow -2. Checks if running in a Tauri environment (desktop) and uses the desktop auth flow -3. Uses the web OAuth flow for all other cases - -## Callback Handling -For the web OAuth flow, callbacks are handled in `auth.$provider.callback.tsx`: -1. Extracts code and state from URL parameters -2. Verifies auth state to prevent CSRF attacks -3. Processes the authentication with the backend -4. Redirects to appropriate page after successful authentication - -## Debugging -If you experience issues with Sign in with Apple: -1. Check debug logs (enabled by default in both flows) -2. Verify that iOS entitlements are properly configured (for native flow) -3. For OAuth flow, check if Apple Developer account is properly set up -4. Verify that the OpenSecret project settings are correctly configured - -## Apple Developer Setup -To use Sign in with Apple, you need: -1. An Apple Developer account -2. An App ID with "Sign In with Apple" capability -3. A Services ID for web authentication -4. A Bundle ID for iOS apps -5. A private key for JWT token signing - -## Resources -- [Apple Developer Documentation](https://developer.apple.com/documentation/sign_in_with_apple) -- [OpenSecret Apple Auth API](https://docs.opensecret.cloud/docs/guides/authentication) -- [tauri-plugin-sign-in-with-apple](https://crates.io/crates/tauri-plugin-sign-in-with-apple) \ No newline at end of file