M3DPIS: Multimodal 3D Printing Intelligence System

Transform speech into printable 3D models using cutting-edge AI and contrastive learning.

---

🎯 Project Vision

Create an AI-powered system that enables natural language interaction with 3D printing workflows through multimodal contrastive learning. Users can generate and modify 3D printable models through conversational speech, with a professional, elegant web interface that handles everything from narration to printer-ready G-code.

---

🚀 What We've Built (Planning Phase)

1. Comprehensive Product Requirements Document (PRD)

Location: .taskmaster/docs/prd.txt

A detailed technical specification covering:

• Phase 1: G-code dataset acquisition & preprocessing (ORNL Peregrine, open datasets)
• Phase 2: MeshLib integration for G-code → 3D model reconstruction
• Phase 3: VLM-based 3D model description (DeepSeek-VL, Qwen 2.5 VL)
• Phase 4: Contrastive learning model training (CLIP-style for text/3D/gcode)
• Phase 5: CLI tool with local speech-to-text (Voxtral, Whisper alternatives)

Key Innovation: Apply CLIP-style contrastive learning to create aligned embeddings across:

• Natural language descriptions
• 3D mesh geometry
• G-code toolpaths

2. Complete User Journey & UX Design

Location: .taskmaster/docs/ux-user-journey.md

A professional, elegant web application flow:

Speech Input → Model Generation → Visualization → Validation → Optimization → Slicing → Printing

10 Detailed Stages:

1. Welcome & Onboarding: First-time user experience with tutorials
2. Voice Input Interface: Pulsing mic button, real-time transcription, waveform viz
3. Model Generation: Animated progress with 4 stages
4. Model Visualization: PBR rendering, orbit controls, multiple view modes
5. Iterative Refinement: Voice commands for modifications, version history
6. Validation & Analysis: Automated printability checks, issue detection
7. Optimization: Smart optimization wizard (quality, material, supports, orientation)
8. Slicing Preparation: Layer-by-layer preview, toolpath visualization
9. Final Review & Export: Pre-flight checklist, direct printer integration
10. Print Monitoring: Live webcam, failure detection (stretch feature)

Tech Stack Recommendations:

• Frontend: React + TypeScript + Vite
• 3D: Three.js + React Three Fiber
• UI: Radix UI + Tailwind CSS
• State: Zustand
• Voice: Web Speech API / AssemblyAI

3. Development Workflow with Trae + Figma

Location: .taskmaster/docs/development-workflow-trae-figma.md

A cutting-edge design-to-development pipeline:

Figma → Trae Integration:

• Design system in Figma (colors, typography, components)
• Export design tokens as JSON
• AI-powered code generation in Trae IDE
• Automatic sync: Figma changes → React components

Key Workflow Features:

• Voice commands in Trae (meta-experience!)
• Screenshot → React component generation
• Figma API integration for automation
• Live Figma preview in split-screen
• Design-code feedback loop

Example Workflow:

Designer creates component in Figma (4 hrs)
  ↓
Export & handoff to Trae
  ↓
Trae AI generates React component (5 min)
  ↓
Test & iterate
  ↓
Designer reviews → feedback
  ↓
Trae updates code instantly
  ↓
Ship to production

Total Time: 6 hours (vs. 2-3 days traditional)

4. Task Master Implementation Plan

Location: .taskmaster/tasks/tasks.json

30 AI-Generated Tasks organized by dependency:

• ✅ Task #1 (Next): Set up project repository and dev environment
• 20 High Priority Tasks: Core ML and frontend implementation
• 10 Medium Priority Tasks: Testing, docs, optimization

Dependency Metrics:

• Tasks with no dependencies: 1
• Tasks ready to work on: 1
• Tasks blocked by dependencies: 29
• Most depended-on task: #1 (12 dependents)

---

📁 Project Structure

mac3d/
├── .taskmaster/
│   ├── config.json                 # AI model configuration
│   ├── state.json                  # Project state
│   ├── tasks/
│   │   └── tasks.json              # 30 implementation tasks
│   ├── docs/
│   │   ├── prd.txt                 # Product Requirements Document
│   │   ├── ux-user-journey.md      # Complete UX design
│   │   └── development-workflow-trae-figma.md  # Dev workflow
│   ├── reports/                    # Task Master reports
│   └── templates/                  # PRD templates
├── src/                            # Source code (to be created)
├── public/                         # Static assets (to be created)
├── .gitignore
├── .env.example                    # API keys template
└── README.md                       # This file

---

🛠 Setup & Installation

Prerequisites

• Node.js v18+ (for Task Master)
• Python 3.10+ (for ML backend)
• GPU (recommended): NVIDIA RTX 3060+ for inference, RTX 4090 for training

Task Master Setup (Completed ✅)

Task Master v0.32.2 is installed and initialized with:

• Global npm package
• Project configuration
• MCP integrations for: Amp, Claude, Cursor, Gemini, Kiro, OpenCode, Kilo, Roo, VSCode, Windsurf, Zed
• Shell aliases (if .bashrc exists)

Environment Variables

Create .env file with your API keys:

# Required
ANTHROPIC_API_KEY="sk-ant-api03-..."  # ✅ Configured

# Optional (for enhanced features)
OPENAI_API_KEY="sk-proj-..."          # ❌ Not set
PERPLEXITY_API_KEY="pplx-..."        # ❌ Not set
GOOGLE_API_KEY="..."                  # For Gemini VLM

---

📋 Getting Started

View Current Tasks

task-master list                     # View all 30 tasks
task-master next                     # See next task to work on
task-master show 1                   # View details of task #1

Start Working

# Mark task as in-progress
task-master set-status --id=1 --status=in-progress

# Expand task into subtasks for detailed breakdown
task-master expand --id=1 --num=5

# When complete
task-master set-status --id=1 --status=done

Analyze Complexity

# Analyze all tasks for complexity
task-master analyze-complexity

# View the complexity report
task-master complexity-report

# Expand complex tasks with research
task-master expand --id=2 --research --num=8

---

🎨 Design & Development Workflow

Phase 1: Design in Figma

1. Create Figma workspace: "Speech-to-3D-Print App"
2. Build design system (colors, typography, components)
3. Design key screens (landing, voice input, 3D viewer, optimization)
4. Create interactive prototype
5. Export design tokens

Phase 2: Develop in Trae IDE

1. Initialize React + TypeScript + Vite project
2. Import design tokens → Tailwind config
3. Use Trae AI to generate components from Figma:

   "Build VoiceInputPanel component from Figma design [link]"
   

4. Implement 3D viewer with Three.js
5. Integrate backend ML API for model generation
6. Test and iterate

Phase 3: Backend ML Development

1. Acquire G-code datasets (ORNL Peregrine, Thingiverse)
2. Implement MeshLib G-code reconstruction
3. Generate 3D model descriptions with VLM
4. Train contrastive learning model (3D-CLIP-G)
5. Deploy inference API

Phase 4: Integration & Testing

1. Connect frontend to ML backend
2. Implement validation & optimization
3. Integrate slicer (CuraEngine WASM)
4. User testing & feedback
5. Polish and optimize

---

🏗 Technical Architecture

System Overview

┌─────────────────────────────────────────────────────────────┐
│                     Frontend (React + Three.js)              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │ Voice Input  │  │  3D Viewer   │  │ Optimization │      │
│  │   Module     │  │    Module    │  │    Wizard    │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└─────────────┬───────────────┬───────────────┬───────────────┘
              │               │               │
          Web Speech API   Three.js      CuraEngine WASM
              │               │               │
┌─────────────▼───────────────▼───────────────▼───────────────┐
│                    Backend API (FastAPI)                     │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │  3D-CLIP-G   │  │   MeshLib    │  │ Validation   │      │
│  │    Model     │  │ Integration  │  │   Engine     │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└──────────────────────────────────────────────────────────────┘

Key Components

Frontend (Trae + Figma)

• React + TypeScript for type safety
• Three.js for 3D rendering
• Tailwind CSS for styling
• Zustand for state management
• Web Speech API for voice input

ML Backend (Python)

• PyTorch for deep learning
• 3D-CLIP-G contrastive model
• DeepSeek-VL or Qwen 2.5 VL for descriptions
• MeshLib (C++ with Python bindings) for G-code processing

Deployment

• Frontend: Vercel / Netlify
• Backend: AWS Lambda / Google Cloud Run (serverless)
• Model: Hugging Face Inference Endpoints / Modal
• Storage: AWS S3 for datasets and models

---

📊 Success Metrics

User Experience

• Time to First Model: < 30 seconds from speech to preview
• Semantic Accuracy: > 80% match between description and generated model
• Print Success Rate: > 85% of generated models print successfully
• User Satisfaction: > 4.5/5 stars

Technical Performance

• Inference Speed: < 5 seconds per generation
• Model Size: < 5GB (deployment-friendly)
• Frontend Load Time: < 2 seconds (3D viewer ready)
• Accessibility: WCAG 2.1 Level AA compliance

Development Efficiency (Trae + Figma)

• Design-to-Code Time: < 1 hour per component
• Code Generation: 80% AI-generated
• Design-Implementation Match: 99%+ pixel-perfect

---

🗺 Roadmap

✅ Phase 0: Planning (COMPLETED)

• Product Requirements Document
• UX/UI Journey Design
• Development Workflow (Trae + Figma)
• Task Master implementation plan (30 tasks)

🚧 Phase 1: Foundation (Current)

• Task #1: Set up project repository (NEXT)
• Task #2: G-code dataset collection
• Task #3: MeshLib integration
• Task #5: VLM setup (DeepSeek-VL)

📅 Phase 2: ML Pipeline (Weeks 4-8)

• Implement 3D-CLIP-G model architecture
• Train contrastive learning model
• Model evaluation and benchmarking

📅 Phase 3: Frontend (Weeks 7-10)

• Design system in Figma
• React components with Trae AI
• 3D viewer with Three.js
• Voice input interface
• Optimization wizard

📅 Phase 4: Integration (Weeks 9-12)

• Backend API integration
• Validation & slicing
• User testing
• Polish & deployment

🔮 Future Enhancements

• Multi-material support
• AR preview (view model in real-world scale)
• Collaborative editing
• Community marketplace
• Generative design optimization

---

🤝 Contributing

This is currently a solo project in the planning phase. Contributions will be welcome once the MVP is complete.

---

📄 License

MIT License - See LICENSE file for details

---

🙏 Acknowledgments

• Task Master AI by @eyaltoledano - Project management
• ORNL Peregrine Dataset - High-quality 3D printing data (2025)
• MeshLib by MeshInspector - G-code processing
• CLIP by OpenAI - Contrastive learning inspiration
• Trae IDE - AI-powered development environment
• Figma - Professional design tool

---

📞 Contact

• Project Lead: [Your Name]
• Email: [Your Email]
• GitHub: [Your GitHub]
• Twitter/X: [Your Handle]

---

Built with ❤️ and AI

Making 3D printing accessible to everyone, one voice command at a time.