audiobook/.kiro/specs/text-to-audiobook/design.md

Design Document: Text to Audiobook

Overview

The Text to Audiobook system is a full-stack application that converts long-form text into multi-voice audiobooks using Kokoro TTS. The architecture follows a client-server model with a React TypeScript frontend and a Python FastAPI backend running in Docker. The system automatically detects speaker roles in text, allows voice assignment per role, and generates high-quality audio using the Kokoro-82M model.

The application workflow consists of three main phases:

1. Text Input & Role Detection: User provides text, system identifies distinct speakers
2. Voice Assignment: User maps each detected role to a specific Kokoro voice
3. Audio Generation & Playback: System synthesizes multi-voice audio and provides playback/download

Architecture

High-Level Architecture

┌─────────────────────────────────────────────────────────────┐
│                         Browser                              │
│  ┌───────────────────────────────────────────────────────┐  │
│  │         React Frontend (TypeScript + Vite)            │  │
│  │  - Text Input Component                               │  │
│  │  - Role Assignment Component                          │  │
│  │  - Audio Player Component                             │  │
│  │  - API Client (tts.ts)                                │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                            │
                            │ HTTP/REST
                            │
┌─────────────────────────────────────────────────────────────┐
│                    Docker Container                          │
│  ┌───────────────────────────────────────────────────────┐  │
│  │         Python Backend (FastAPI)                      │  │
│  │  ┌─────────────────────────────────────────────────┐  │  │
│  │  │  API Layer                                      │  │  │
│  │  │  - /api/health                                  │  │  │
│  │  │  - /api/tts/roles                               │  │  │
│  │  │  - /api/tts                                     │  │  │
│  │  └─────────────────────────────────────────────────┘  │  │
│  │  ┌─────────────────────────────────────────────────┐  │  │
│  │  │  Role Detection Service                         │  │  │
│  │  │  - Pattern matching for role markers            │  │  │
│  │  │  - Role extraction and deduplication            │  │  │
│  │  └─────────────────────────────────────────────────┘  │  │
│  │  ┌─────────────────────────────────────────────────┐  │  │
│  │  │  TTS Service                                    │  │  │
│  │  │  - Kokoro KPipeline management                  │  │  │
│  │  │  - Text segmentation                            │  │  │
│  │  │  - Voice synthesis per role                     │  │  │
│  │  │  - Audio concatenation                          │  │  │
│  │  └─────────────────────────────────────────────────┘  │  │
│  │  ┌─────────────────────────────────────────────────┐  │  │
│  │  │  Kokoro TTS Engine                              │  │  │
│  │  │  - hexgrad/Kokoro-82M model                     │  │  │
│  │  │  - espeak-ng for G2P                            │  │  │
│  │  │  - soundfile for audio I/O                      │  │  │
│  │  └─────────────────────────────────────────────────┘  │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

Technology Stack

Frontend:

• React 18+ with TypeScript
• Vite for build tooling
• Native fetch API for HTTP requests
• HTML5 Audio API for playback
• CSS/Tailwind for responsive styling

Backend:

• Python 3.10+
• FastAPI for REST API
• Uvicorn as ASGI server
• kokoro library (hexgrad/Kokoro-82M)
• soundfile for audio processing
• espeak-ng for grapheme-to-phoneme conversion
• httpx for Ollama API communication
• Ollama with llama3:8B for AI-based speaker detection

Infrastructure:

• Docker for containerization
• Docker Compose for orchestration (optional)

Components and Interfaces

Frontend Components

1. App Component (App.tsx)

Main application container managing the overall workflow state.

State:

• text: string - User input text
• roles: Role[] - Detected roles from backend
• voiceAssignments: Map<string, string> - Role ID to Voice ID mapping
• selectedLanguage: string - Language code (default: 'a')
• speed: number - Playback speed (default: 1.0)
• audioUrl: string | null - Generated audio blob URL
• isDetectingRoles: boolean - Loading state for role detection
• isGenerating: boolean - Loading state for audio generation
• error: string | null - Error message if any

Methods:

• handleTextChange(text: string) - Updates text state
• handleDetectRoles() - Calls API to detect roles
• handleVoiceAssignment(roleId: string, voiceId: string) - Updates voice mapping
• handleGenerate() - Calls API to generate audiobook
• handleDownload() - Triggers download of generated audio

2. TextInputSection Component

Displays the text input area and "Next: Assign voices" button.

Props:

• text: string
• onChange: (text: string) => void
• onNext: () => void
• isLoading: boolean
• disabled: boolean

3. VoiceAssignmentSection Component

Displays role-to-voice mapping interface with language and speed controls.

Props:

• roles: Role[]
• voiceAssignments: Map<string, string>
• onVoiceChange: (roleId: string, voiceId: string) => void
• language: string
• onLanguageChange: (lang: string) => void
• speed: number
• onSpeedChange: (speed: number) => void
• onGenerate: () => void
• isGenerating: boolean
• allRolesAssigned: boolean

4. AudioPlayerSection Component

Displays audio player and download button after generation.

Props:

• audioUrl: string
• onDownload: () => void

5. API Client (src/api/tts.ts)

Interfaces:

export interface Role {
  id: string;
  label: string;
}

export interface RoleWithVoice extends Role {
  voice: string;
}

export interface RolesResponse {
  roles: Role[];
}

export interface GenerateRequest {
  text: string;
  lang: string;
  roles: { id: string; voice: string }[];
  speed?: number;
}

Functions:

export async function detectRoles(text: string): Promise<RolesResponse>
export async function generateAudiobook(payload: GenerateRequest): Promise<Blob>
export async function checkHealth(): Promise<{ status: string }>

Backend Components

1. API Layer (main.py)

Endpoints:

@app.get("/api/health")
async def health_check() -> dict:
    """Returns service health status"""
    return {"status": "ok"}

@app.post("/api/tts/roles")
async def detect_roles(request: RoleDetectionRequest) -> RolesResponse:
    """Analyzes text and returns detected roles"""
    pass

@app.post("/api/tts")
async def generate_audiobook(request: TTSRequest) -> StreamingResponse:
    """Generates multi-voice audiobook and streams audio"""
    pass

Request/Response Models:

class RoleDetectionRequest(BaseModel):
    text: str

class RoleInfo(BaseModel):
    id: str
    label: str

class RolesResponse(BaseModel):
    roles: List[RoleInfo]

class RoleVoiceMapping(BaseModel):
    id: str
    voice: str

class TTSRequest(BaseModel):
    text: str
    lang: str
    roles: List[RoleVoiceMapping]
    speed: Optional[float] = 1.0

2. Role Detection Service (services/role_detector.py)

Class: RoleDetector

Methods:

def detect_roles(self, text: str) -> List[RoleInfo]:
    """
    Analyzes text to identify distinct speaker roles.
    First attempts AI-based detection using Ollama, then falls back to pattern matching.
    Supports patterns:
    - "RoleName:" at line start
    - "[RoleName]" preceding dialogue
    Returns unique roles or default "Narrator" if none found.
    """
    pass

def _detect_with_ollama(self, text: str) -> Set[str]:
    """
    Uses Ollama with llama3:8B to intelligently detect speakers.
    Returns set of speaker names or empty set if Ollama unavailable.
    """
    pass

def _extract_colon_pattern(self, text: str) -> Set[str]:
    """Extracts roles from "Name:" pattern"""
    pass

def _extract_bracket_pattern(self, text: str) -> Set[str]:
    """Extracts roles from "[Name]" pattern"""
    pass

3. Ollama Integration Service (services/ollama_service.py)

Class: OllamaService

Attributes:

• base_url: str - Ollama API base URL (from environment)
• model: str = "llama3:8b" - Model to use for speaker detection
• timeout: int = 30 - Request timeout in seconds

Methods:

def __init__(self, base_url: str = "http://localhost:11434"):
    """Initializes Ollama service with base URL"""
    pass

async def detect_speakers(self, text: str) -> List[str]:
    """
    Sends text to Ollama for speaker detection.
    Returns list of detected speaker names.
    Raises exception if Ollama is unavailable or request fails.
    """
    pass

def _build_prompt(self, text: str) -> str:
    """
    Constructs prompt for speaker detection.
    Instructs model to return JSON array of speaker names.
    """
    pass

def _parse_response(self, response: str) -> List[str]:
    """
    Parses Ollama response to extract speaker names.
    Handles both JSON and plain text responses.
    """
    pass

async def is_available(self) -> bool:
    """
    Checks if Ollama service is available.
    Returns True if service responds to health check.
    """
    pass

4. TTS Service (services/tts_service.py)

Class: TTSService

Attributes:

• pipelines: Dict[str, KPipeline] - Cached pipelines per language
• default_voice: str = "af_bella"

Methods:

def __init__(self):
    """Initializes TTS service"""
    pass

def get_pipeline(self, lang_code: str) -> KPipeline:
    """
    Returns or creates KPipeline for given language.
    Caches pipelines to avoid reloading.
    """
    pass

def segment_text(self, text: str, role_mappings: Dict[str, str]) -> List[Tuple[str, str]]:
    """
    Segments text into (role_id, text_content) tuples.
    Identifies role markers and assigns segments accordingly.
    """
    pass

def synthesize_segment(self, pipeline: KPipeline, text: str, voice: str, speed: float) -> np.ndarray:
    """
    Synthesizes a single text segment with specified voice.
    Returns audio samples as numpy array.
    """
    pass

def concatenate_audio(self, segments: List[np.ndarray]) -> np.ndarray:
    """
    Concatenates multiple audio segments into single array.
    Ensures consistent sample rate (24kHz).
    """
    pass

async def generate_audiobook(
    self,
    text: str,
    lang: str,
    role_voice_map: Dict[str, str],
    speed: float = 1.0
) -> bytes:
    """
    Main orchestration method.
    Segments text, synthesizes each segment, concatenates, and returns audio bytes.
    """
    pass

Data Models

Frontend Data Models

Role:

interface Role {
  id: string;        // Unique identifier (e.g., "Narrator", "Alice")
  label: string;     // Display name for UI
}

Voice Option:

interface VoiceOption {
  id: string;        // Kokoro voice ID (e.g., "af_bella")
  label: string;     // Display name (e.g., "Bella (Female, American)")
  gender: 'male' | 'female';
  accent: string;    // e.g., "American", "British"
}

Language Option:

interface LanguageOption {
  code: string;      // Kokoro language code (e.g., "a", "b", "f", "j")
  label: string;     // Display name (e.g., "American English")
}

Application State:

interface AppState {
  phase: 'input' | 'assignment' | 'generated';
  text: string;
  roles: Role[];
  voiceAssignments: Map<string, string>;
  language: string;
  speed: number;
  audioUrl: string | null;
  isLoading: boolean;
  error: string | null;
}

Backend Data Models

Text Segment:

@dataclass
class TextSegment:
    role_id: str
    text: str
    voice: str

Audio Chunk:

@dataclass
class AudioChunk:
    samples: np.ndarray
    sample_rate: int = 24000

Constants

Supported Voices:

const KOKORO_VOICES: VoiceOption[] = [
  { id: 'af_bella', label: 'Bella (Female, American)', gender: 'female', accent: 'American' },
  { id: 'af_sarah', label: 'Sarah (Female, American)', gender: 'female', accent: 'American' },
  { id: 'am_adam', label: 'Adam (Male, American)', gender: 'male', accent: 'American' },
  { id: 'am_michael', label: 'Michael (Male, American)', gender: 'male', accent: 'American' },
  { id: 'bf_emma', label: 'Emma (Female, British)', gender: 'female', accent: 'British' },
  { id: 'bf_isabella', label: 'Isabella (Female, British)', gender: 'female', accent: 'British' },
  { id: 'bm_george', label: 'George (Male, British)', gender: 'male', accent: 'British' },
  { id: 'bm_lewis', label: 'Lewis (Male, British)', gender: 'male', accent: 'British' },
];

Supported Languages:

const LANGUAGES: LanguageOption[] = [
  { code: 'a', label: 'American English' },
  { code: 'b', label: 'British English' },
  { code: 'f', label: 'French' },
  { code: 'j', label: 'Japanese' },
];

Role Detection Patterns:

ROLE_PATTERNS = {
    'colon': r'^([A-Z][a-zA-Z\s]+):\s',      # "Alice: Hello there"
    'bracket': r'\[([A-Z][a-zA-Z\s]+)\]',    # "[Narrator] The story begins"
}

Correctness Properties

A property is a characteristic or behavior that should hold true across all valid executions of a system—essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.

Property 1: Text persistence invariant

For any text entered by the user, the text content should remain unchanged as the application transitions between input, assignment, and generated phases. Validates: Requirements 1.5

Property 2: Role extraction from colon pattern

For any text containing lines starting with "RoleName:", the role detection should extract RoleName as a distinct role. Validates: Requirements 2.2

Property 3: Role extraction from bracket pattern

For any text containing "[RoleName]" patterns, the role detection should extract RoleName as a distinct role. Validates: Requirements 2.3

Property 4: Role deduplication

For any text with multiple instances of the same role marker, the role detection should return each unique role exactly once. Validates: Requirements 2.5

Property 5: Role response structure

For any detected roles, each role object in the response should contain both id and label fields. Validates: Requirements 2.6

Property 6: Voice assignment storage

For any role and voice selection, the frontend should correctly store and retrieve the role-to-voice mapping. Validates: Requirements 3.4

Property 7: Generate button disabled when incomplete

For any state where not all roles have assigned voices, the "Generate Audiobook" button should be disabled. Validates: Requirements 3.5

Property 8: Generate button enabled when complete

For any state where all roles have assigned voices, the "Generate Audiobook" button should be enabled. Validates: Requirements 3.6

Property 9: Language selection storage

For any language selection, the frontend should store the selected language code for the generation request. Validates: Requirements 4.4

Property 10: Speed range constraints

For any speed slider interaction, the value should be constrained between 0.5x and 1.5x. Validates: Requirements 5.2

Property 11: Speed value storage

For any speed adjustment, the frontend should store the speed value for the generation request. Validates: Requirements 5.4

Property 12: Generation request completeness

For any valid application state, the generation request should contain all required fields: text, language code, role-to-voice mappings, and speed. Validates: Requirements 6.1

Property 13: Voice assignment per segment

For any text segment with a role marker, the backend should use the Kokoro voice assigned to that role for synthesis. Validates: Requirements 6.3

Property 14: Text chunking for long content

For any long text input, the backend should split it into manageable chunks per paragraph or sentence. Validates: Requirements 6.5

Property 15: Audio concatenation at 24 kHz

For any set of synthesized chunks, the backend should concatenate them into a single audio stream at 24 kHz sample rate. Validates: Requirements 6.6

Property 16: Audio response content type

For any successful synthesis, the HTTP response should include Content-Type header set to audio/mpeg or audio/wav. Validates: Requirements 6.7

Property 17: Audio response filename

For any audio response, the HTTP response should include Content-Disposition header with filename "audiobook.mp3" or "audiobook.wav". Validates: Requirements 6.8

Property 18: Blob URL creation

For any audio response from the backend, the frontend should convert it to a Blob and create a valid object URL. Validates: Requirements 7.1

Property 19: Download functionality

For any generated audio blob, clicking the download button should trigger file save with the correct filename. Validates: Requirements 7.5

Property 20: Loading state during role detection

For any role detection request, the frontend should display a loading state indicator until the response is received. Validates: Requirements 8.1

Property 21: Loading state during generation

For any audiobook generation request, the frontend should display loading or progress feedback until completion. Validates: Requirements 8.2

Property 22: Long text warning

For any text input exceeding a threshold length, the frontend should display a message indicating "Generation may take several minutes for very long content". Validates: Requirements 8.3

Property 23: Error message display

For any failed API request, the frontend should display a clear error message to the user. Validates: Requirements 8.4

Property 24: Role response structure validation

For any response from /api/tts/roles, the JSON should contain a roles array with objects having id and label fields. Validates: Requirements 9.3

Property 25: Request validation

For any request to /api/tts missing required fields (text, lang, or roles), the backend should reject it with an appropriate error. Validates: Requirements 9.5

Property 26: Pipeline initialization per language

For any supported language code (a, b, f, j), the backend should successfully load a KPipeline for that language. Validates: Requirements 10.3

Property 27: Audio sample rate

For any synthesized audio, the output should have a sample rate of 24 kHz. Validates: Requirements 10.4

Property 28: Ollama speaker detection

For any text without explicit role markers, when Ollama is available, the backend should attempt AI-based speaker detection before falling back to pattern matching. Validates: Requirements 14.1

Property 29: Ollama JSON response parsing

For any valid JSON response from Ollama containing speaker names, the backend should successfully parse and extract the speaker list. Validates: Requirements 14.3

Property 30: Ollama fallback behavior

For any text, when Ollama is unavailable or fails, the backend should fall back to pattern-based role detection without error. Validates: Requirements 14.4

Property 31: Role deduplication across detection methods

For any text where both AI detection and pattern detection find roles, the backend should merge results and return each unique role exactly once. Validates: Requirements 14.5

Error Handling

Frontend Error Handling

Network Errors:

• Catch fetch errors and display user-friendly messages
• Implement retry logic for transient failures (optional)
• Show specific error messages for timeout, connection refused, etc.

Validation Errors:

• Validate text is non-empty before API calls
• Validate all roles have assigned voices before generation
• Show inline validation messages near relevant controls

API Error Responses:

• Parse error responses from backend (4xx, 5xx)
• Display error details to user
• Log errors to console for debugging

Audio Playback Errors:

• Handle cases where audio blob is invalid
• Show error if browser doesn't support audio format
• Provide fallback message if playback fails

Backend Error Handling

Request Validation:

• Validate all required fields are present
• Return 400 Bad Request with descriptive error message
• Validate language codes and voice IDs are supported

TTS Processing Errors:

• Catch exceptions during pipeline initialization
• Handle synthesis failures gracefully
• Return 500 Internal Server Error with error details
• Log errors for debugging

Ollama Integration Errors:

• Handle Ollama service unavailable gracefully
• Fall back to pattern-based detection on Ollama failure
• Log Ollama errors for debugging without failing request
• Handle malformed JSON responses from Ollama
• Implement timeout for Ollama requests (30 seconds)

Resource Management:

• Handle out-of-memory scenarios for very long text
• Implement timeouts for long-running synthesis
• Clean up temporary audio files/buffers

Model Loading Errors:

• Catch errors when loading Kokoro models
• Return clear error if models are missing
• Provide helpful error messages for setup issues

Error Response Format

class ErrorResponse(BaseModel):
    error: str
    message: str
    details: Optional[dict] = None

Example:

{
  "error": "ValidationError",
  "message": "Missing required field: lang",
  "details": {
    "field": "lang",
    "required": true
  }
}

Testing Strategy

Unit Testing

Frontend Unit Tests:

• Test individual components in isolation
• Mock API calls using jest/vitest mocks
• Test state management logic
• Test utility functions (e.g., blob URL creation)
• Verify UI elements render correctly
• Test form validation logic

Backend Unit Tests:

• Test role detection patterns independently
• Test text segmentation logic
• Test audio concatenation
• Test API endpoint request/response handling
• Mock Kokoro TTS calls to test logic without actual synthesis

Example Unit Tests:

• Empty text validation prevents API call
• Health endpoint returns correct status
• Role detector extracts "Alice:" pattern correctly
• Role detector extracts "[Narrator]" pattern correctly
• Default "Narrator" role returned when no patterns found
• Voice assignment updates state correctly
• Download button triggers file save

Property-Based Testing

Property-based tests will verify universal properties across many randomly generated inputs using a PBT library.

Frontend Property Tests (using fast-check for TypeScript):

We will use fast-check as the property-based testing library for the React TypeScript frontend.

Backend Property Tests (using Hypothesis for Python):

We will use Hypothesis as the property-based testing library for the Python FastAPI backend.

Configuration:

• Each property-based test should run a minimum of 100 iterations
• Tests should generate diverse inputs to explore edge cases
• Each test must include a comment referencing the design property it validates

Property Test Examples:

Frontend:

• Property 1: Text persistence - generate random text, verify it persists across phase transitions
• Property 6: Voice assignment storage - generate random role/voice pairs, verify storage/retrieval
• Property 10: Speed range - generate random speed values, verify clamping to 0.5-1.5 range

Backend:

• Property 2: Colon pattern extraction - generate random text with "Name:" patterns, verify extraction
• Property 3: Bracket pattern extraction - generate random text with "[Name]" patterns, verify extraction
• Property 4: Role deduplication - generate text with duplicate roles, verify uniqueness
• Property 14: Text chunking - generate long text, verify chunks are reasonable size
• Property 27: Audio sample rate - generate random text, verify output is 24 kHz

Integration Testing

End-to-End Flow Tests:

• Test complete workflow: text input → role detection → voice assignment → generation → playback
• Test with sample texts containing multiple roles
• Verify audio file is generated and playable
• Test error scenarios (network failures, invalid inputs)

API Integration Tests:

• Test all API endpoints with real requests
• Verify request/response formats match specifications
• Test error responses for invalid inputs
• Test with different languages and voices

Manual Testing Checklist

• Enter text and verify role detection works
• Assign voices to all roles
• Generate audiobook and verify audio plays
• Download audiobook and verify file is valid
• Test with very long text (multiple pages)
• Test with text containing no role markers
• Test with different languages (English, French, Japanese)
• Test on mobile devices for responsiveness
• Test error scenarios (empty text, network failure)
• Verify loading states appear during processing

Implementation Notes

Frontend Implementation

State Management:

• Use React hooks (useState, useEffect) for local state
• Consider useReducer for complex state transitions
• Store voice assignments in a Map for efficient lookups

API Client:

• Use native fetch API
• Implement error handling and timeout logic
• Use environment variable for base URL configuration

Audio Handling:

• Use Blob API for audio data
• Create object URLs with URL.createObjectURL()
• Revoke object URLs when component unmounts to prevent memory leaks
• Use HTML5 audio element with controls attribute

Styling:

• Use Tailwind CSS for responsive design
• Ensure mobile-friendly touch targets (min 44x44px)
• Use loading spinners for async operations
• Implement accessible form controls

Backend Implementation

Kokoro Integration:

• Initialize KPipeline instances lazily (on first use)
• Cache pipelines per language to avoid reloading
• Use appropriate voice IDs from Kokoro documentation
• Handle model download/loading errors gracefully

Text Processing:

• Use regex for role pattern matching
• Split text on paragraph boundaries (\n\n) or sentence boundaries
• Preserve whitespace and formatting where appropriate
• Handle edge cases (empty lines, special characters)

Audio Processing:

• Use soundfile library for audio I/O
• Concatenate numpy arrays for audio chunks
• Ensure consistent sample rate (24 kHz) across all chunks
• Convert to MP3 or WAV format for streaming

Performance Optimization:

• Consider async processing for long texts
• Implement streaming response for large audio files
• Use connection pooling for database (if needed)
• Monitor memory usage for very long texts

Docker Configuration:

• Use multi-stage build to reduce image size
• Install system dependencies (espeak-ng) in base layer
• Copy only necessary files to container
• Use .dockerignore to exclude unnecessary files
• Set appropriate environment variables
• Use health check endpoint for container orchestration

Security Considerations

Input Validation:

• Sanitize user input to prevent injection attacks
• Limit text length to prevent DoS (e.g., max 1MB)
• Validate language codes and voice IDs against whitelist
• Rate limit API endpoints to prevent abuse

CORS Configuration:

• Configure CORS to allow frontend origin
• Restrict allowed methods and headers
• Don't expose sensitive headers

Error Messages:

• Don't expose internal implementation details in errors
• Log detailed errors server-side only
• Return generic error messages to client

Deployment Considerations

Environment Variables:

• Frontend: VITE_API_BASE_URL
• Backend: PORT, MODEL_PATH (optional), LOG_LEVEL

Resource Requirements:

• Backend: Minimum 2GB RAM for model loading
• Storage: ~500MB for Kokoro models
• CPU: Multi-core recommended for faster synthesis

Monitoring:

• Log API requests and response times
• Monitor memory usage during synthesis
• Track error rates and types
• Set up health check monitoring

Future Enhancements

Potential Improvements:

• Add support for more languages and voices
• Implement progress tracking for long generations
• Add ability to preview individual role voices
• Support for SSML markup for advanced control
• Batch processing for multiple texts
• User accounts and saved voice preferences
• Audio editing (trim, adjust volume per role)
• Export to different audio formats (FLAC, OGG)
• Background processing with job queue
• WebSocket for real-time progress updates