# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview
**OCTv2 (Oreo Cookie Thrower v2)** - A complete system for automated Oreo delivery targeting open mouths.

### System Components:
1. **iOS SwiftUI App** - Remote control interface with video streaming
2. **Raspberry Pi Server** - Computer vision and motor control coordinator
3. **ESP32 Controller** - Precise stepper motor control via serial communication
4. **Camera-Follows-Aim System** - Moves camera with targeting mechanism for accuracy

## Project Status
✅ **COMPLETE** - All components implemented and documented

## System Architecture

### iOS App (OreoLauncher/)
- **SwiftUI interface** with real-time video streaming
- **Manual controls**: Left/Right aim, Fire, Angle (0-60°)
- **Auto mode**: Enables computer vision targeting
- **TCP networking** to Raspberry Pi server
- **Photo capture** functionality

### Raspberry Pi Server (raspberry_pi_server/)
- **Advanced mouth detection** using OpenCV + dlib facial landmarks
- **State classification**: CLOSED, SPEAKING, SMILING, WIDE_OPEN
- **Only targets WIDE_OPEN mouths** for firing
- **Distance estimation** using face size and camera focal length
- **ESP32 serial communication** at 115200 baud
- **Camera-follows-aim targeting** with centering algorithm

### ESP32 Firmware (esp32_firmware/)
- **Stepper motor control** with AccelStepper library
- **Serial command protocol**: HOME, MOVE, REL, FIRE, POS
- **Limit switch homing** for precise positioning
- **A4988 drivers** with microstepping support

## File Structure
```
freds_first_iphone_app/
├── OreoLauncher/
│   ├── ContentView.swift          # Main SwiftUI interface
│   ├── NetworkService.swift       # TCP networking
│   └── OreoLauncher.xcodeproj    # Xcode project
├── raspberry_pi_server/
│   ├── octv2_server.py           # Main Pi server
│   ├── requirements.txt          # Python dependencies
│   ├── camera_aim_calibration.md # Targeting calibration guide
│   ├── wide_mouth_detection_guide.md # Detection tuning guide
│   └── setup_mouth_detection.md  # Setup instructions
├── esp32_firmware/
│   ├── octv2_motor_controller.ino # ESP32 stepper control firmware
│   └── ESP32_COMMANDS.md          # Complete command reference
└── CLAUDE.md                     # This file
```

## Development Commands

### iOS App
```bash
# Build and run in simulator
xcodebuild -project OreoLauncher/OreoLauncher.xcodeproj -scheme OreoLauncher -destination 'platform=iOS Simulator,name=iPhone 15' build

# Open in Xcode for development
open OreoLauncher/OreoLauncher.xcodeproj
```

### Raspberry Pi Server
```bash
# Install dependencies
cd raspberry_pi_server
pip3 install -r requirements.txt

# Run server
python3 octv2_server.py

# Download facial landmark model (if using advanced detection)
wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
bunzip2 shape_predictor_68_face_landmarks.dat.bz2
```

### ESP32 Firmware
```bash
# Upload via Arduino IDE or PlatformIO
# Configure board: ESP32 Dev Module
# Baud rate: 115200
```

## Key Features

### Mouth Detection Algorithm
- **dlib facial landmarks** for precise mouth analysis
- **Mouth Aspect Ratio (MAR)** calculation
- **Lip separation measurement** for open mouth detection
- **Real-time classification** with visual feedback

### Camera-Follows-Aim Targeting
- **Centering algorithm** instead of angle calculation
- **Distance estimation** using face size
- **Trajectory compensation** for different distances
- **Mechanical offset correction** for hardware alignment

### Communication Protocol
- **iOS ↔ Pi**: TCP JSON commands
- **Pi ↔ ESP32**: Serial text commands at 115200 baud
- **Real-time video streaming** with detection overlays

## Calibration & Tuning

### Targeting Sensitivity
```python
# In octv2_server.py
self.target_deadzone_pixels = 30      # Targeting tolerance
pixels_per_degree_rotation = 15       # Movement sensitivity
```

### Mouth Detection Thresholds
```python
# Wide-open mouth detection
inner_aspect_ratio > 0.6              # Mouth opening ratio
avg_lip_thickness > 8                 # Lip separation pixels
```

### Hardware Offsets
```python
# Mechanical compensation
self.rotation_offset_degrees = 0.0    # Camera/launcher alignment
self.pitch_offset_degrees = 0.0   # Gravity compensation
```

## Troubleshooting

### Common Issues
- **Xcode crashes**: Use `sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer`
- **dlib import errors**: Install cmake, libopenblas-dev, liblapack-dev
- **Serial communication**: Check ESP32 connection and baud rate
- **Detection sensitivity**: Adjust thresholds in mouth detection algorithm

### Testing Commands
```bash
# Test mouth detection
python3 -c "import cv2, dlib; print('CV2:', cv2.__version__, 'dlib:', dlib.DLIB_VERSION)"

# Test ESP32 serial
python3 -c "import serial; s=serial.Serial('/dev/ttyUSB0', 115200); print('Serial OK')"
```

## Development Notes
- **Use Xcode** for iOS development
- **Test on actual Pi** for camera/motor functionality
- **Calibrate targeting** for specific hardware setup
- **Monitor CPU usage** on Pi during operation
- **Follow safety protocols** when testing firing mechanism