OCTv2/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

OCTv2 (Oreo Cookie Thrower v2) - A complete system for automated Oreo delivery targeting open mouths.

System Components:

1. iOS SwiftUI App - Remote control interface with video streaming
2. Raspberry Pi Server - Computer vision and motor control coordinator
3. ESP32 Controller - Precise stepper motor control via serial communication
4. Camera-Follows-Aim System - Moves camera with targeting mechanism for accuracy

Project Status

✅ COMPLETE - All components implemented and documented

System Architecture

iOS App (OreoLauncher/)

• SwiftUI interface with real-time video streaming
• Manual controls: Left/Right aim, Fire, Angle (0-60°)
• Auto mode: Enables computer vision targeting
• TCP networking to Raspberry Pi server
• Photo capture functionality

Raspberry Pi Server (raspberry_pi_server/)

• Advanced mouth detection using OpenCV + dlib facial landmarks
• State classification: CLOSED, SPEAKING, SMILING, WIDE_OPEN
• Only targets WIDE_OPEN mouths for firing
• Distance estimation using face size and camera focal length
• ESP32 serial communication at 115200 baud
• Camera-follows-aim targeting with centering algorithm

ESP32 Firmware (esp32_firmware/)

• Stepper motor control with AccelStepper library
• Serial command protocol: HOME, MOVE, REL, FIRE, POS
• Limit switch homing for precise positioning
• A4988 drivers with microstepping support

File Structure

freds_first_iphone_app/
├── OreoLauncher/
│   ├── ContentView.swift          # Main SwiftUI interface
│   ├── NetworkService.swift       # TCP networking
│   └── OreoLauncher.xcodeproj    # Xcode project
├── raspberry_pi_server/
│   ├── octv2_server.py           # Main Pi server
│   ├── requirements.txt          # Python dependencies
│   ├── camera_aim_calibration.md # Targeting calibration guide
│   ├── wide_mouth_detection_guide.md # Detection tuning guide
│   └── setup_mouth_detection.md  # Setup instructions
├── esp32_firmware/
│   ├── octv2_motor_controller.ino # ESP32 stepper control firmware
│   └── ESP32_COMMANDS.md          # Complete command reference
└── CLAUDE.md                     # This file

Development Commands

iOS App

# Build and run in simulator
xcodebuild -project OreoLauncher/OreoLauncher.xcodeproj -scheme OreoLauncher -destination 'platform=iOS Simulator,name=iPhone 15' build

# Open in Xcode for development
open OreoLauncher/OreoLauncher.xcodeproj

Raspberry Pi Server

# Install dependencies
cd raspberry_pi_server
pip3 install -r requirements.txt

# Run server
python3 octv2_server.py

# Download facial landmark model (if using advanced detection)
wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
bunzip2 shape_predictor_68_face_landmarks.dat.bz2

ESP32 Firmware

# Upload via Arduino IDE or PlatformIO
# Configure board: ESP32 Dev Module
# Baud rate: 115200

Key Features

Mouth Detection Algorithm

• dlib facial landmarks for precise mouth analysis
• Mouth Aspect Ratio (MAR) calculation
• Lip separation measurement for open mouth detection
• Real-time classification with visual feedback

Camera-Follows-Aim Targeting

• Centering algorithm instead of angle calculation
• Distance estimation using face size
• Trajectory compensation for different distances
• Mechanical offset correction for hardware alignment

Communication Protocol

• iOS ↔ Pi: TCP JSON commands
• Pi ↔ ESP32: Serial text commands at 115200 baud
• Real-time video streaming with detection overlays

Calibration & Tuning

Targeting Sensitivity

# In octv2_server.py
self.target_deadzone_pixels = 30      # Targeting tolerance
pixels_per_degree_rotation = 15       # Movement sensitivity

Mouth Detection Thresholds

# Wide-open mouth detection
inner_aspect_ratio > 0.6              # Mouth opening ratio
avg_lip_thickness > 8                 # Lip separation pixels

Hardware Offsets

# Mechanical compensation
self.rotation_offset_degrees = 0.0    # Camera/launcher alignment
self.pitch_offset_degrees = 0.0   # Gravity compensation

Troubleshooting

Common Issues

• Xcode crashes: Use sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
• dlib import errors: Install cmake, libopenblas-dev, liblapack-dev
• Serial communication: Check ESP32 connection and baud rate
• Detection sensitivity: Adjust thresholds in mouth detection algorithm

Testing Commands

# Test mouth detection
python3 -c "import cv2, dlib; print('CV2:', cv2.__version__, 'dlib:', dlib.DLIB_VERSION)"

# Test ESP32 serial
python3 -c "import serial; s=serial.Serial('/dev/ttyUSB0', 115200); print('Serial OK')"

Development Notes

• Use Xcode for iOS development
• Test on actual Pi for camera/motor functionality
• Calibrate targeting for specific hardware setup
• Monitor CPU usage on Pi during operation
• Follow safety protocols when testing firing mechanism