audiobook/backend

Text to Audiobook Backend

FastAPI backend service for converting text to multi-voice audiobooks using Kokoro TTS (hexgrad/Kokoro-82M).

Overview

The backend provides REST API endpoints for:

• Role/speaker detection in text
• Multi-voice audio synthesis
• Audio streaming and download

Prerequisites

• Python 3.10 or higher
• espeak-ng (system dependency for grapheme-to-phoneme conversion)
• Piper TTS (optional, for alternative TTS engine with pitch control)
• Docker (optional, for containerized deployment)

Local Development Setup

1. Install espeak-ng

macOS:

brew install espeak-ng

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install espeak-ng

Linux (Fedora/RHEL):

sudo dnf install espeak-ng

Windows: Download from espeak-ng releases

Verify installation:

espeak-ng --version

2. Python Environment Setup

# Create virtual environment
python -m venv venv

# Activate virtual environment
# macOS/Linux:
source venv/bin/activate
# Windows:
venv\Scripts\activate

# Upgrade pip
pip install --upgrade pip

# Install dependencies
pip install -r requirements.txt

3. Piper TTS Setup (Optional)

For an alternative TTS engine with pitch control support:

Installation:

# Run the installation script
./install_piper.sh

Or install manually:

• Visit Piper releases
• Download the appropriate binary for your platform
• Extract and place in your PATH

Verify installation:

piper --version

Download Voice Models:

Piper requires manual voice model download. See PIPER_SETUP.md for detailed instructions.

Quick steps:

1. Visit https://github.com/rhasspy/piper/releases/tag/v1.2.0
2. Download voice models (e.g., en_US-lessac-medium.onnx and .onnx.json)
3. Place in backend/piper_models/ directory

Benefits of Piper TTS:

• ✨ Pitch shifting support (-12 to +12 semitones per role)
• ✨ Fast, lightweight neural TTS
• ✨ Multiple voice models available
• ✨ Lower resource usage than Kokoro

4. Ollama Setup (Optional but Recommended)

For AI-powered speaker detection and text segmentation, install Ollama:

Installation:

• Visit ollama.com and download for your platform
• Or use package managers:

  # macOS
  brew install ollama
  
  # Linux
  curl -fsSL https://ollama.com/install.sh | sh
  

Pull the required model:

ollama pull llama3

Start Ollama service:

ollama serve

Verify Ollama is running:

curl http://localhost:11434/api/tags

Benefits of using Ollama:

• ✨ Intelligently detects speakers in unmarked text
• ✨ Automatically segments dialogue and assigns to correct characters
• ✨ Handles narrative text without explicit role markers
• ✨ Falls back to pattern matching if unavailable

5. Environment Configuration

cp .env.example .env

Edit .env if needed:

PORT=8000
LOG_LEVEL=info
OLLAMA_BASE_URL=http://localhost:11434  # Optional: Ollama API URL
TTS_ENGINE=kokoro  # Options: kokoro, piper (default: kokoro)

6. Run the Server

uvicorn main:app --reload --host 0.0.0.0 --port 8000

The server will start on http://localhost:8000

Available URLs:

• API Base: http://localhost:8000
• Interactive API Docs: http://localhost:8000/docs
• Alternative API Docs: http://localhost:8000/redoc
• Health Check: http://localhost:8000/api/health

Docker Deployment

Using Docker Compose (Recommended)

# Build and start
docker-compose up -d

# View logs
docker-compose logs -f backend

# Stop
docker-compose down

# Rebuild after code changes
docker-compose up -d --build

Using Docker Directly

# Build image
docker build -t text-to-audiobook-backend .

# Run container
docker run -d \
  --name text-to-audiobook-backend \
  -p 8000:8000 \
  -e PORT=8000 \
  -e LOG_LEVEL=info \
  text-to-audiobook-backend

# View logs
docker logs -f text-to-audiobook-backend

# Stop and remove
docker stop text-to-audiobook-backend
docker rm text-to-audiobook-backend

Docker Configuration

The Dockerfile includes:

• Python 3.10 slim base image
• Automatic espeak-ng installation
• Python dependencies from requirements.txt
• Health check monitoring
• Uvicorn ASGI server

Resource Requirements:

• Memory: 2GB minimum (4GB recommended)
• Storage: ~500MB for Kokoro models
• CPU: Multi-core recommended

TTS Engine Selection

The backend now supports multiple TTS engines that can be selected at runtime:

Available Engines

1. Kokoro TTS (default)

   • High-quality neural TTS
   • Multiple voices per language
   • No pitch control
   • ~500MB model size

2. Piper TTS

   • Fast, lightweight neural TTS
   • Pitch shifting support (-12 to +12 semitones)
   • Multiple voice models
   • Lower resource usage

Switching Engines

Via Environment Variable:

export TTS_ENGINE=piper  # or kokoro
uvicorn main:app --reload

Via API:

# Get available engines
curl http://localhost:8000/api/tts/engines

# Set active engine
curl -X POST http://localhost:8000/api/tts/engines \
  -H "Content-Type: application/json" \
  -d '{"engine": "piper"}'

API Endpoints

GET /api/health

Health check endpoint.

Response:

{
  "status": "ok"
}

Example:

curl http://localhost:8000/api/health

POST /api/tts/roles

Detect distinct roles/speakers in text using AI-powered detection.

AI-Powered Detection: When Ollama is available, the system uses AI (llama3) to intelligently detect speakers in unmarked text. This allows you to input plain narrative text without explicit role markers, and the AI will identify all characters and speakers.

Request Body:

{
  "text": "Alice: Hello there!\nBob: Hi Alice, how are you?\nNarrator: They greeted each other warmly."
}

Response:

{
  "roles": [
    {"id": "Alice", "label": "Alice"},
    {"id": "Bob", "label": "Bob"},
    {"id": "Narrator", "label": "Narrator"}
  ]
}

Detection Methods:

1. AI Detection (Primary): Uses Ollama with llama3 to intelligently identify speakers in unmarked text
   • Example: "The wizard spoke softly. The knight replied." → Detects "Wizard" and "Knight"
2. Pattern Matching (Fallback): Falls back to pattern-based detection if Ollama unavailable
   • Colon pattern: RoleName: dialogue text
   • Bracket pattern: [RoleName] dialogue text

Example with AI Detection:

curl -X POST http://localhost:8000/api/tts/roles \
  -H "Content-Type: application/json" \
  -d '{"text":"The wizard gazed into his crystal ball. The knight drew his sword."}'

Example with Pattern Markers:

curl -X POST http://localhost:8000/api/tts/roles \
  -H "Content-Type: application/json" \
  -d '{"text":"Alice: Hello!\nBob: Hi there!"}'

GET /api/tts/engines

Get list of available TTS engines and current selection.

Response:

{
  "engines": [
    {"name": "kokoro", "active": true},
    {"name": "piper", "active": false}
  ],
  "current": "kokoro"
}

Example:

curl http://localhost:8000/api/tts/engines

POST /api/tts/engines

Set the active TTS engine.

Request Body:

{
  "engine": "piper"
}

Response:

{
  "status": "ok",
  "engine": "piper"
}

Example:

curl -X POST http://localhost:8000/api/tts/engines \
  -H "Content-Type: application/json" \
  -d '{"engine": "piper"}'

GET /api/tts/voices

Get available voices for all TTS engines.

Response:

{
  "voices": {
    "kokoro": ["af_bella", "af_sarah", "am_adam", "am_michael", ...],
    "piper": ["en_US-lessac-medium", "en_US-amy-medium", ...]
  }
}

Example:

curl http://localhost:8000/api/tts/voices

POST /api/tts

Generate multi-voice audiobook with AI-powered text segmentation.

AI-Powered Segmentation: When Ollama is available, the system uses AI to intelligently split text and assign each segment to the appropriate speaker. This means you can input plain narrative text, and the AI will automatically:

• Assign dialogue to the correct character
• Assign narrative descriptions to the Narrator
• Handle mixed dialogue and narration seamlessly

Request Body:

{
  "text": "Alice: Hello there!\nBob: Hi Alice!",
  "lang": "a",
  "roles": [
    {"id": "Alice", "voice": "af_bella"},
    {"id": "Bob", "voice": "am_adam"}
  ],
  "speed": 1.0
}

Parameters:

• text (required): Text to synthesize (can be plain narrative or marked with role patterns)
• lang (required): Language code (a=American English, b=British English, f=French, j=Japanese)
• roles (required): Array of role-to-voice mappings
• speed (optional): Playback speed (0.5-1.5, default: 1.0)
• pitch_adjustments (optional): Array of role-to-pitch mappings (Piper only, -12 to +12 semitones)

Response:

• Content-Type: audio/wav
• Content-Disposition: attachment; filename=audiobook.wav

Segmentation Methods:

1. AI Segmentation (Primary): Uses Ollama to intelligently split text and assign speakers
   • Handles unmarked dialogue and narrative
   • Automatically assigns dialogue to characters and descriptions to Narrator
2. Pattern Segmentation (Fallback): Uses explicit role markers when AI unavailable
   • Requires RoleName: or [RoleName] markers in text

• Content-Disposition: attachment; filename="audiobook.wav"
• Body: Audio file bytes

Example (Kokoro):

curl -X POST http://localhost:8000/api/tts \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Alice: Hello!\nBob: Hi!",
    "lang": "a",
    "roles": [
      {"id": "Alice", "voice": "af_bella"},
      {"id": "Bob", "voice": "am_adam"}
    ],
    "speed": 1.0
  }' \
  --output audiobook.wav

Example (Piper with pitch control):

curl -X POST http://localhost:8000/api/tts \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Alice: Hello!\nBob: Hi!",
    "lang": "a",
    "roles": [
      {"id": "Alice", "voice": "en_US-amy-medium"},
      {"id": "Bob", "voice": "en_US-lessac-medium"}
    ],
    "speed": 1.0,
    "pitch_adjustments": [
      {"id": "Alice", "pitch": 3},
      {"id": "Bob", "pitch": -2}
    ]
  }' \
  --output audiobook.wav

Kokoro TTS Configuration

Supported Languages

Code	Language
a	American English
b	British English
f	French
j	Japanese

Available Voices

Voice ID	Gender	Accent	Description
af_bella	Female	American	Bella
af_sarah	Female	American	Sarah
am_adam	Male	American	Adam
am_michael	Male	American	Michael
bf_emma	Female	British	Emma
bf_isabella	Female	British	Isabella
bm_george	Male	British	George
bm_lewis	Male	British	Lewis

Model Storage

Models are automatically downloaded on first use and cached:

• Linux/macOS: ~/.cache/kokoro/
• Windows: %USERPROFILE%\.cache\kokoro\

First synthesis may take longer due to model download (~500MB).

Project Structure

backend/
├── services/
│   ├── __init__.py
│   ├── role_detector.py      # Role detection logic
│   └── tts_service.py         # TTS synthesis service
├── main.py                    # FastAPI application & endpoints
├── requirements.txt           # Python dependencies
├── Dockerfile                 # Docker image configuration
├── docker-compose.yml         # Docker Compose setup
├── .env.example               # Environment template
├── .dockerignore              # Docker ignore patterns
├── test_api.py                # API endpoint tests
├── test_role_detection.py     # Role detection tests
├── test_tts_service.py        # TTS service tests
└── README.md                  # This file

Testing

Run All Tests

# Activate virtual environment
source venv/bin/activate  # macOS/Linux
# or
venv\Scripts\activate  # Windows

# Run tests
pytest test_*.py -v

Run Specific Tests

# Test API endpoints
pytest test_api.py -v

# Test role detection
pytest test_role_detection.py -v

# Test TTS service
pytest test_tts_service.py -v

Test Coverage

pytest --cov=. --cov-report=html test_*.py

Troubleshooting

espeak-ng not found

Error: OSError: espeak-ng not found

Solution:

# Install espeak-ng
brew install espeak-ng  # macOS
sudo apt-get install espeak-ng  # Linux

# Verify
espeak-ng --version

Module not found

Error: ModuleNotFoundError: No module named 'kokoro'

Solution:

source venv/bin/activate
pip install -r requirements.txt

Port already in use

Error: Address already in use

Solution:

# Find process using port 8000
lsof -i :8000  # macOS/Linux
netstat -ano | findstr :8000  # Windows

# Use different port
uvicorn main:app --port 8001

Model download fails

Symptoms: First synthesis times out

Solution:

• Check internet connection
• Check firewall settings
• Manually trigger download:

  python -c "from kokoro import KPipeline; KPipeline(lang_code='a')"
  

Out of memory

Error: Memory error during synthesis

Solution:

• Reduce text length
• Increase system RAM
• For Docker: docker run -m 4g ...

Performance Optimization

Tips for Faster Synthesis

1. Use cached pipelines: The service caches KPipeline instances per language
2. Shorter texts: Break very long texts into chapters
3. Multi-core CPU: Synthesis benefits from multiple cores
4. SSD storage: Faster model loading from SSD

Monitoring

# View logs
docker-compose logs -f backend

# Monitor resource usage
docker stats text-to-audiobook-backend

# Check health
curl http://localhost:8000/api/health

Development

Code Style

# Format code
black .

# Lint
flake8 .

# Type checking
mypy .

Adding New Voices

To add new Kokoro voices, update the voice validation in main.py:

SUPPORTED_VOICES = [
    "af_bella", "af_sarah", "am_adam", "am_michael",
    "bf_emma", "bf_isabella", "bm_george", "bm_lewis",
    # Add new voices here
]

Adding New Languages

To add new languages, update the language validation in main.py:

SUPPORTED_LANGUAGES = ["a", "b", "f", "j"]  # Add new codes

Security Considerations

• Input validation on all endpoints
• Text length limits to prevent DoS
• CORS configured for frontend origin
• No sensitive data in logs
• Rate limiting recommended for production

Production Deployment

Recommendations

1. Use environment variables for configuration
2. Enable HTTPS with reverse proxy (nginx, Caddy)
3. Set up monitoring (health checks, logs)
4. Configure rate limiting
5. Use production ASGI server (Uvicorn with workers)
6. Set resource limits (memory, CPU)

Production Command

uvicorn main:app \
  --host 0.0.0.0 \
  --port 8000 \
  --workers 4 \
  --log-level info \
  --no-access-log

Additional Resources

• Kokoro TTS GitHub
• FastAPI Documentation
• Main Project README
• Design Document

License

MIT