audiobook/backend/README.md

# Text to Audiobook Backend

FastAPI backend service for converting text to multi-voice audiobooks using Kokoro TTS (hexgrad/Kokoro-82M).

## Overview

The backend provides REST API endpoints for:
- Role/speaker detection in text
- Multi-voice audio synthesis
- Audio streaming and download

## Prerequisites

- Python 3.10 or higher
- espeak-ng (system dependency for grapheme-to-phoneme conversion)
- Piper TTS (optional, for alternative TTS engine with pitch control)
- Docker (optional, for containerized deployment)

## Local Development Setup

### 1. Install espeak-ng

**macOS:**
```bash
brew install espeak-ng
```

**Linux (Ubuntu/Debian):**
```bash
sudo apt-get update
sudo apt-get install espeak-ng
```

**Linux (Fedora/RHEL):**
```bash
sudo dnf install espeak-ng
```

**Windows:**
Download from [espeak-ng releases](https://github.com/espeak-ng/espeak-ng/releases)

Verify installation:
```bash
espeak-ng --version
```

### 2. Python Environment Setup

```bash
# Create virtual environment
python -m venv venv

# Activate virtual environment
# macOS/Linux:
source venv/bin/activate
# Windows:
venv\Scripts\activate

# Upgrade pip
pip install --upgrade pip

# Install dependencies
pip install -r requirements.txt
```

### 3. Piper TTS Setup (Optional)

For an alternative TTS engine with pitch control support:

**Installation:**
```bash
# Run the installation script
./install_piper.sh
```

Or install manually:
- Visit [Piper releases](https://github.com/rhasspy/piper/releases)
- Download the appropriate binary for your platform
- Extract and place in your PATH

**Verify installation:**
```bash
piper --version
```

**Benefits of Piper TTS:**
- ✨ Pitch shifting support (-12 to +12 semitones per role)
- ✨ Fast, lightweight neural TTS
- ✨ Multiple voice models available
- ✨ Lower resource usage than Kokoro

### 4. Ollama Setup (Optional but Recommended)

For AI-powered speaker detection and text segmentation, install Ollama:

**Installation:**
- Visit [ollama.com](https://ollama.com) and download for your platform
- Or use package managers:
  ```bash
  # macOS
  brew install ollama
  
  # Linux
  curl -fsSL https://ollama.com/install.sh | sh
  ```

**Pull the required model:**
```bash
ollama pull llama3
```

**Start Ollama service:**
```bash
ollama serve
```

Verify Ollama is running:
```bash
curl http://localhost:11434/api/tags
```

**Benefits of using Ollama:**
- ✨ Intelligently detects speakers in unmarked text
- ✨ Automatically segments dialogue and assigns to correct characters
- ✨ Handles narrative text without explicit role markers
- ✨ Falls back to pattern matching if unavailable

### 5. Environment Configuration

```bash
cp .env.example .env
```

Edit `.env` if needed:
```bash
PORT=8000
LOG_LEVEL=info
OLLAMA_BASE_URL=http://localhost:11434  # Optional: Ollama API URL
TTS_ENGINE=kokoro  # Options: kokoro, piper (default: kokoro)
```

### 6. Run the Server

```bash
uvicorn main:app --reload --host 0.0.0.0 --port 8000
```

The server will start on http://localhost:8000

**Available URLs:**
- API Base: http://localhost:8000
- Interactive API Docs: http://localhost:8000/docs
- Alternative API Docs: http://localhost:8000/redoc
- Health Check: http://localhost:8000/api/health

## Docker Deployment

### Using Docker Compose (Recommended)

```bash
# Build and start
docker-compose up -d

# View logs
docker-compose logs -f backend

# Stop
docker-compose down

# Rebuild after code changes
docker-compose up -d --build
```

### Using Docker Directly

```bash
# Build image
docker build -t text-to-audiobook-backend .

# Run container
docker run -d \
  --name text-to-audiobook-backend \
  -p 8000:8000 \
  -e PORT=8000 \
  -e LOG_LEVEL=info \
  text-to-audiobook-backend

# View logs
docker logs -f text-to-audiobook-backend

# Stop and remove
docker stop text-to-audiobook-backend
docker rm text-to-audiobook-backend
```

### Docker Configuration

The Dockerfile includes:
- Python 3.10 slim base image
- Automatic espeak-ng installation
- Python dependencies from requirements.txt
- Health check monitoring
- Uvicorn ASGI server

**Resource Requirements:**
- Memory: 2GB minimum (4GB recommended)
- Storage: ~500MB for Kokoro models
- CPU: Multi-core recommended

## TTS Engine Selection

The backend now supports multiple TTS engines that can be selected at runtime:

### Available Engines

1. **Kokoro TTS** (default)
   - High-quality neural TTS
   - Multiple voices per language
   - No pitch control
   - ~500MB model size

2. **Piper TTS**
   - Fast, lightweight neural TTS
   - Pitch shifting support (-12 to +12 semitones)
   - Multiple voice models
   - Lower resource usage

### Switching Engines

**Via Environment Variable:**
```bash
export TTS_ENGINE=piper  # or kokoro
uvicorn main:app --reload
```

**Via API:**
```bash
# Get available engines
curl http://localhost:8000/api/tts/engines

# Set active engine
curl -X POST http://localhost:8000/api/tts/engines \
  -H "Content-Type: application/json" \
  -d '{"engine": "piper"}'
```

## API Endpoints

### GET /api/health

Health check endpoint.

**Response:**
```json
{
  "status": "ok"
}
```

**Example:**
```bash
curl http://localhost:8000/api/health
```

### POST /api/tts/roles

Detect distinct roles/speakers in text using AI-powered detection.

**AI-Powered Detection:**
When Ollama is available, the system uses AI (llama3) to intelligently detect speakers in unmarked text. This allows you to input plain narrative text without explicit role markers, and the AI will identify all characters and speakers.

**Request Body:**
```json
{
  "text": "Alice: Hello there!\nBob: Hi Alice, how are you?\nNarrator: They greeted each other warmly."
}
```

**Response:**
```json
{
  "roles": [
    {"id": "Alice", "label": "Alice"},
    {"id": "Bob", "label": "Bob"},
    {"id": "Narrator", "label": "Narrator"}
  ]
}
```

**Detection Methods:**
1. **AI Detection (Primary)**: Uses Ollama with llama3 to intelligently identify speakers in unmarked text
   - Example: "The wizard spoke softly. The knight replied." → Detects "Wizard" and "Knight"
2. **Pattern Matching (Fallback)**: Falls back to pattern-based detection if Ollama unavailable
   - Colon pattern: `RoleName: dialogue text`
   - Bracket pattern: `[RoleName] dialogue text`

**Example with AI Detection:**
```bash
curl -X POST http://localhost:8000/api/tts/roles \
  -H "Content-Type: application/json" \
  -d '{"text":"The wizard gazed into his crystal ball. The knight drew his sword."}'
```

**Example with Pattern Markers:**
```bash
curl -X POST http://localhost:8000/api/tts/roles \
  -H "Content-Type: application/json" \
  -d '{"text":"Alice: Hello!\nBob: Hi there!"}'
```

### GET /api/tts/engines

Get list of available TTS engines and current selection.

**Response:**
```json
{
  "engines": [
    {"name": "kokoro", "active": true},
    {"name": "piper", "active": false}
  ],
  "current": "kokoro"
}
```

**Example:**
```bash
curl http://localhost:8000/api/tts/engines
```

### POST /api/tts/engines

Set the active TTS engine.

**Request Body:**
```json
{
  "engine": "piper"
}
```

**Response:**
```json
{
  "status": "ok",
  "engine": "piper"
}
```

**Example:**
```bash
curl -X POST http://localhost:8000/api/tts/engines \
  -H "Content-Type: application/json" \
  -d '{"engine": "piper"}'
```

### GET /api/tts/voices

Get available voices for all TTS engines.

**Response:**
```json
{
  "voices": {
    "kokoro": ["af_bella", "af_sarah", "am_adam", "am_michael", ...],
    "piper": ["en_US-lessac-medium", "en_US-amy-medium", ...]
  }
}
```

**Example:**
```bash
curl http://localhost:8000/api/tts/voices
```

### POST /api/tts

Generate multi-voice audiobook with AI-powered text segmentation.

**AI-Powered Segmentation:**
When Ollama is available, the system uses AI to intelligently split text and assign each segment to the appropriate speaker. This means you can input plain narrative text, and the AI will automatically:
- Assign dialogue to the correct character
- Assign narrative descriptions to the Narrator
- Handle mixed dialogue and narration seamlessly

**Request Body:**
```json
{
  "text": "Alice: Hello there!\nBob: Hi Alice!",
  "lang": "a",
  "roles": [
    {"id": "Alice", "voice": "af_bella"},
    {"id": "Bob", "voice": "am_adam"}
  ],
  "speed": 1.0
}
```

**Parameters:**
- `text` (required): Text to synthesize (can be plain narrative or marked with role patterns)
- `lang` (required): Language code (a=American English, b=British English, f=French, j=Japanese)
- `roles` (required): Array of role-to-voice mappings
- `speed` (optional): Playback speed (0.5-1.5, default: 1.0)
- `pitch_adjustments` (optional): Array of role-to-pitch mappings (Piper only, -12 to +12 semitones)

**Response:**
- Content-Type: `audio/wav`
- Content-Disposition: `attachment; filename=audiobook.wav`

**Segmentation Methods:**
1. **AI Segmentation (Primary)**: Uses Ollama to intelligently split text and assign speakers
   - Handles unmarked dialogue and narrative
   - Automatically assigns dialogue to characters and descriptions to Narrator
2. **Pattern Segmentation (Fallback)**: Uses explicit role markers when AI unavailable
   - Requires `RoleName:` or `[RoleName]` markers in text
- Content-Disposition: `attachment; filename="audiobook.wav"`
- Body: Audio file bytes

**Example (Kokoro):**
```bash
curl -X POST http://localhost:8000/api/tts \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Alice: Hello!\nBob: Hi!",
    "lang": "a",
    "roles": [
      {"id": "Alice", "voice": "af_bella"},
      {"id": "Bob", "voice": "am_adam"}
    ],
    "speed": 1.0
  }' \
  --output audiobook.wav
```

**Example (Piper with pitch control):**
```bash
curl -X POST http://localhost:8000/api/tts \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Alice: Hello!\nBob: Hi!",
    "lang": "a",
    "roles": [
      {"id": "Alice", "voice": "en_US-amy-medium"},
      {"id": "Bob", "voice": "en_US-lessac-medium"}
    ],
    "speed": 1.0,
    "pitch_adjustments": [
      {"id": "Alice", "pitch": 3},
      {"id": "Bob", "pitch": -2}
    ]
  }' \
  --output audiobook.wav
```

## Kokoro TTS Configuration

### Supported Languages

| Code | Language |
|------|----------|
| `a` | American English |
| `b` | British English |
| `f` | French |
| `j` | Japanese |

### Available Voices

| Voice ID | Gender | Accent | Description |
|----------|--------|--------|-------------|
| `af_bella` | Female | American | Bella |
| `af_sarah` | Female | American | Sarah |
| `am_adam` | Male | American | Adam |
| `am_michael` | Male | American | Michael |
| `bf_emma` | Female | British | Emma |
| `bf_isabella` | Female | British | Isabella |
| `bm_george` | Male | British | George |
| `bm_lewis` | Male | British | Lewis |

### Model Storage

Models are automatically downloaded on first use and cached:
- **Linux/macOS**: `~/.cache/kokoro/`
- **Windows**: `%USERPROFILE%\.cache\kokoro\`

First synthesis may take longer due to model download (~500MB).

## Project Structure

```
backend/
├── services/
│   ├── __init__.py
│   ├── role_detector.py      # Role detection logic
│   └── tts_service.py         # TTS synthesis service
├── main.py                    # FastAPI application & endpoints
├── requirements.txt           # Python dependencies
├── Dockerfile                 # Docker image configuration
├── docker-compose.yml         # Docker Compose setup
├── .env.example               # Environment template
├── .dockerignore              # Docker ignore patterns
├── test_api.py                # API endpoint tests
├── test_role_detection.py     # Role detection tests
├── test_tts_service.py        # TTS service tests
└── README.md                  # This file
```

## Testing

### Run All Tests

```bash
# Activate virtual environment
source venv/bin/activate  # macOS/Linux
# or
venv\Scripts\activate  # Windows

# Run tests
pytest test_*.py -v
```

### Run Specific Tests

```bash
# Test API endpoints
pytest test_api.py -v

# Test role detection
pytest test_role_detection.py -v

# Test TTS service
pytest test_tts_service.py -v
```

### Test Coverage

```bash
pytest --cov=. --cov-report=html test_*.py
```

## Troubleshooting

### espeak-ng not found

**Error:** `OSError: espeak-ng not found`

**Solution:**
```bash
# Install espeak-ng
brew install espeak-ng  # macOS
sudo apt-get install espeak-ng  # Linux

# Verify
espeak-ng --version
```

### Module not found

**Error:** `ModuleNotFoundError: No module named 'kokoro'`

**Solution:**
```bash
source venv/bin/activate
pip install -r requirements.txt
```

### Port already in use

**Error:** `Address already in use`

**Solution:**
```bash
# Find process using port 8000
lsof -i :8000  # macOS/Linux
netstat -ano | findstr :8000  # Windows

# Use different port
uvicorn main:app --port 8001
```

### Model download fails

**Symptoms:** First synthesis times out

**Solution:**
- Check internet connection
- Check firewall settings
- Manually trigger download:
  ```bash
  python -c "from kokoro import KPipeline; KPipeline(lang_code='a')"
  ```

### Out of memory

**Error:** Memory error during synthesis

**Solution:**
- Reduce text length
- Increase system RAM
- For Docker: `docker run -m 4g ...`

## Performance Optimization

### Tips for Faster Synthesis

1. **Use cached pipelines**: The service caches KPipeline instances per language
2. **Shorter texts**: Break very long texts into chapters
3. **Multi-core CPU**: Synthesis benefits from multiple cores
4. **SSD storage**: Faster model loading from SSD

### Monitoring

```bash
# View logs
docker-compose logs -f backend

# Monitor resource usage
docker stats text-to-audiobook-backend

# Check health
curl http://localhost:8000/api/health
```

## Development

### Code Style

```bash
# Format code
black .

# Lint
flake8 .

# Type checking
mypy .
```

### Adding New Voices

To add new Kokoro voices, update the voice validation in `main.py`:

```python
SUPPORTED_VOICES = [
    "af_bella", "af_sarah", "am_adam", "am_michael",
    "bf_emma", "bf_isabella", "bm_george", "bm_lewis",
    # Add new voices here
]
```

### Adding New Languages

To add new languages, update the language validation in `main.py`:

```python
SUPPORTED_LANGUAGES = ["a", "b", "f", "j"]  # Add new codes
```

## Security Considerations

- Input validation on all endpoints
- Text length limits to prevent DoS
- CORS configured for frontend origin
- No sensitive data in logs
- Rate limiting recommended for production

## Production Deployment

### Recommendations

1. **Use environment variables** for configuration
2. **Enable HTTPS** with reverse proxy (nginx, Caddy)
3. **Set up monitoring** (health checks, logs)
4. **Configure rate limiting**
5. **Use production ASGI server** (Uvicorn with workers)
6. **Set resource limits** (memory, CPU)

### Production Command

```bash
uvicorn main:app \
  --host 0.0.0.0 \
  --port 8000 \
  --workers 4 \
  --log-level info \
  --no-access-log
```

## Additional Resources

- [Kokoro TTS GitHub](https://github.com/hexgrad/kokoro)
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
- [Main Project README](../README.md)
- [Design Document](../.kiro/specs/text-to-audiobook/design.md)

## License

MIT