audiobook/.kiro/specs/text-to-audiobook/design.md

# Design Document: Text to Audiobook

## Overview

The Text to Audiobook system is a full-stack application that converts long-form text into multi-voice audiobooks using Kokoro TTS. The architecture follows a client-server model with a React TypeScript frontend and a Python FastAPI backend running in Docker. The system automatically detects speaker roles in text, allows voice assignment per role, and generates high-quality audio using the Kokoro-82M model.

The application workflow consists of three main phases:
1. **Text Input & Role Detection**: User provides text, system identifies distinct speakers
2. **Voice Assignment**: User maps each detected role to a specific Kokoro voice
3. **Audio Generation & Playback**: System synthesizes multi-voice audio and provides playback/download

## Architecture

### High-Level Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                         Browser                              │
│  ┌───────────────────────────────────────────────────────┐  │
│  │         React Frontend (TypeScript + Vite)            │  │
│  │  - Text Input Component                               │  │
│  │  - Role Assignment Component                          │  │
│  │  - Audio Player Component                             │  │
│  │  - API Client (tts.ts)                                │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                            │
                            │ HTTP/REST
                            │
┌─────────────────────────────────────────────────────────────┐
│                    Docker Container                          │
│  ┌───────────────────────────────────────────────────────┐  │
│  │         Python Backend (FastAPI)                      │  │
│  │  ┌─────────────────────────────────────────────────┐  │  │
│  │  │  API Layer                                      │  │  │
│  │  │  - /api/health                                  │  │  │
│  │  │  - /api/tts/roles                               │  │  │
│  │  │  - /api/tts                                     │  │  │
│  │  └─────────────────────────────────────────────────┘  │  │
│  │  ┌─────────────────────────────────────────────────┐  │  │
│  │  │  Role Detection Service                         │  │  │
│  │  │  - Pattern matching for role markers            │  │  │
│  │  │  - Role extraction and deduplication            │  │  │
│  │  └─────────────────────────────────────────────────┘  │  │
│  │  ┌─────────────────────────────────────────────────┐  │  │
│  │  │  TTS Service                                    │  │  │
│  │  │  - Kokoro KPipeline management                  │  │  │
│  │  │  - Text segmentation                            │  │  │
│  │  │  - Voice synthesis per role                     │  │  │
│  │  │  - Audio concatenation                          │  │  │
│  │  └─────────────────────────────────────────────────┘  │  │
│  │  ┌─────────────────────────────────────────────────┐  │  │
│  │  │  Kokoro TTS Engine                              │  │  │
│  │  │  - hexgrad/Kokoro-82M model                     │  │  │
│  │  │  - espeak-ng for G2P                            │  │  │
│  │  │  - soundfile for audio I/O                      │  │  │
│  │  └─────────────────────────────────────────────────┘  │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
```

### Technology Stack

**Frontend:**
- React 18+ with TypeScript
- Vite for build tooling
- Native fetch API for HTTP requests
- HTML5 Audio API for playback
- CSS/Tailwind for responsive styling

**Backend:**
- Python 3.10+
- FastAPI for REST API
- Uvicorn as ASGI server
- kokoro library (hexgrad/Kokoro-82M)
- soundfile for audio processing
- espeak-ng for grapheme-to-phoneme conversion

**Infrastructure:**
- Docker for containerization
- Docker Compose for orchestration (optional)

## Components and Interfaces

### Frontend Components

#### 1. App Component (`App.tsx`)
Main application container managing the overall workflow state.

**State:**
- `text: string` - User input text
- `roles: Role[]` - Detected roles from backend
- `voiceAssignments: Map<string, string>` - Role ID to Voice ID mapping
- `selectedLanguage: string` - Language code (default: 'a')
- `speed: number` - Playback speed (default: 1.0)
- `audioUrl: string | null` - Generated audio blob URL
- `isDetectingRoles: boolean` - Loading state for role detection
- `isGenerating: boolean` - Loading state for audio generation
- `error: string | null` - Error message if any

**Methods:**
- `handleTextChange(text: string)` - Updates text state
- `handleDetectRoles()` - Calls API to detect roles
- `handleVoiceAssignment(roleId: string, voiceId: string)` - Updates voice mapping
- `handleGenerate()` - Calls API to generate audiobook
- `handleDownload()` - Triggers download of generated audio

#### 2. TextInputSection Component
Displays the text input area and "Next: Assign voices" button.

**Props:**
- `text: string`
- `onChange: (text: string) => void`
- `onNext: () => void`
- `isLoading: boolean`
- `disabled: boolean`

#### 3. VoiceAssignmentSection Component
Displays role-to-voice mapping interface with language and speed controls.

**Props:**
- `roles: Role[]`
- `voiceAssignments: Map<string, string>`
- `onVoiceChange: (roleId: string, voiceId: string) => void`
- `language: string`
- `onLanguageChange: (lang: string) => void`
- `speed: number`
- `onSpeedChange: (speed: number) => void`
- `onGenerate: () => void`
- `isGenerating: boolean`
- `allRolesAssigned: boolean`

#### 4. AudioPlayerSection Component
Displays audio player and download button after generation.

**Props:**
- `audioUrl: string`
- `onDownload: () => void`

#### 5. API Client (`src/api/tts.ts`)

**Interfaces:**
```typescript
export interface Role {
  id: string;
  label: string;
}

export interface RoleWithVoice extends Role {
  voice: string;
}

export interface RolesResponse {
  roles: Role[];
}

export interface GenerateRequest {
  text: string;
  lang: string;
  roles: { id: string; voice: string }[];
  speed?: number;
}
```

**Functions:**
```typescript
export async function detectRoles(text: string): Promise<RolesResponse>
export async function generateAudiobook(payload: GenerateRequest): Promise<Blob>
export async function checkHealth(): Promise<{ status: string }>
```

### Backend Components

#### 1. API Layer (`main.py`)

**Endpoints:**

```python
@app.get("/api/health")
async def health_check() -> dict:
    """Returns service health status"""
    return {"status": "ok"}

@app.post("/api/tts/roles")
async def detect_roles(request: RoleDetectionRequest) -> RolesResponse:
    """Analyzes text and returns detected roles"""
    pass

@app.post("/api/tts")
async def generate_audiobook(request: TTSRequest) -> StreamingResponse:
    """Generates multi-voice audiobook and streams audio"""
    pass
```

**Request/Response Models:**
```python
class RoleDetectionRequest(BaseModel):
    text: str

class RoleInfo(BaseModel):
    id: str
    label: str

class RolesResponse(BaseModel):
    roles: List[RoleInfo]

class RoleVoiceMapping(BaseModel):
    id: str
    voice: str

class TTSRequest(BaseModel):
    text: str
    lang: str
    roles: List[RoleVoiceMapping]
    speed: Optional[float] = 1.0
```

#### 2. Role Detection Service (`services/role_detector.py`)

**Class: RoleDetector**

**Methods:**
```python
def detect_roles(self, text: str) -> List[RoleInfo]:
    """
    Analyzes text to identify distinct speaker roles.
    Supports patterns:
    - "RoleName:" at line start
    - "[RoleName]" preceding dialogue
    Returns unique roles or default "Narrator" if none found.
    """
    pass

def _extract_colon_pattern(self, text: str) -> Set[str]:
    """Extracts roles from "Name:" pattern"""
    pass

def _extract_bracket_pattern(self, text: str) -> Set[str]:
    """Extracts roles from "[Name]" pattern"""
    pass
```

#### 3. TTS Service (`services/tts_service.py`)

**Class: TTSService**

**Attributes:**
- `pipelines: Dict[str, KPipeline]` - Cached pipelines per language
- `default_voice: str = "af_bella"`

**Methods:**
```python
def __init__(self):
    """Initializes TTS service"""
    pass

def get_pipeline(self, lang_code: str) -> KPipeline:
    """
    Returns or creates KPipeline for given language.
    Caches pipelines to avoid reloading.
    """
    pass

def segment_text(self, text: str, role_mappings: Dict[str, str]) -> List[Tuple[str, str]]:
    """
    Segments text into (role_id, text_content) tuples.
    Identifies role markers and assigns segments accordingly.
    """
    pass

def synthesize_segment(self, pipeline: KPipeline, text: str, voice: str, speed: float) -> np.ndarray:
    """
    Synthesizes a single text segment with specified voice.
    Returns audio samples as numpy array.
    """
    pass

def concatenate_audio(self, segments: List[np.ndarray]) -> np.ndarray:
    """
    Concatenates multiple audio segments into single array.
    Ensures consistent sample rate (24kHz).
    """
    pass

async def generate_audiobook(
    self,
    text: str,
    lang: str,
    role_voice_map: Dict[str, str],
    speed: float = 1.0
) -> bytes:
    """
    Main orchestration method.
    Segments text, synthesizes each segment, concatenates, and returns audio bytes.
    """
    pass
```

## Data Models

### Frontend Data Models

**Role:**
```typescript
interface Role {
  id: string;        // Unique identifier (e.g., "Narrator", "Alice")
  label: string;     // Display name for UI
}
```

**Voice Option:**
```typescript
interface VoiceOption {
  id: string;        // Kokoro voice ID (e.g., "af_bella")
  label: string;     // Display name (e.g., "Bella (Female, American)")
  gender: 'male' | 'female';
  accent: string;    // e.g., "American", "British"
}
```

**Language Option:**
```typescript
interface LanguageOption {
  code: string;      // Kokoro language code (e.g., "a", "b", "f", "j")
  label: string;     // Display name (e.g., "American English")
}
```

**Application State:**
```typescript
interface AppState {
  phase: 'input' | 'assignment' | 'generated';
  text: string;
  roles: Role[];
  voiceAssignments: Map<string, string>;
  language: string;
  speed: number;
  audioUrl: string | null;
  isLoading: boolean;
  error: string | null;
}
```

### Backend Data Models

**Text Segment:**
```python
@dataclass
class TextSegment:
    role_id: str
    text: str
    voice: str
```

**Audio Chunk:**
```python
@dataclass
class AudioChunk:
    samples: np.ndarray
    sample_rate: int = 24000
```

### Constants

**Supported Voices:**
```typescript
const KOKORO_VOICES: VoiceOption[] = [
  { id: 'af_bella', label: 'Bella (Female, American)', gender: 'female', accent: 'American' },
  { id: 'af_sarah', label: 'Sarah (Female, American)', gender: 'female', accent: 'American' },
  { id: 'am_adam', label: 'Adam (Male, American)', gender: 'male', accent: 'American' },
  { id: 'am_michael', label: 'Michael (Male, American)', gender: 'male', accent: 'American' },
  { id: 'bf_emma', label: 'Emma (Female, British)', gender: 'female', accent: 'British' },
  { id: 'bf_isabella', label: 'Isabella (Female, British)', gender: 'female', accent: 'British' },
  { id: 'bm_george', label: 'George (Male, British)', gender: 'male', accent: 'British' },
  { id: 'bm_lewis', label: 'Lewis (Male, British)', gender: 'male', accent: 'British' },
];
```

**Supported Languages:**
```typescript
const LANGUAGES: LanguageOption[] = [
  { code: 'a', label: 'American English' },
  { code: 'b', label: 'British English' },
  { code: 'f', label: 'French' },
  { code: 'j', label: 'Japanese' },
];
```

**Role Detection Patterns:**
```python
ROLE_PATTERNS = {
    'colon': r'^([A-Z][a-zA-Z\s]+):\s',      # "Alice: Hello there"
    'bracket': r'\[([A-Z][a-zA-Z\s]+)\]',    # "[Narrator] The story begins"
}
```


## Correctness Properties

*A property is a characteristic or behavior that should hold true across all valid executions of a system—essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*

### Property 1: Text persistence invariant
*For any* text entered by the user, the text content should remain unchanged as the application transitions between input, assignment, and generated phases.
**Validates: Requirements 1.5**

### Property 2: Role extraction from colon pattern
*For any* text containing lines starting with "RoleName:", the role detection should extract RoleName as a distinct role.
**Validates: Requirements 2.2**

### Property 3: Role extraction from bracket pattern
*For any* text containing "[RoleName]" patterns, the role detection should extract RoleName as a distinct role.
**Validates: Requirements 2.3**

### Property 4: Role deduplication
*For any* text with multiple instances of the same role marker, the role detection should return each unique role exactly once.
**Validates: Requirements 2.5**

### Property 5: Role response structure
*For any* detected roles, each role object in the response should contain both id and label fields.
**Validates: Requirements 2.6**

### Property 6: Voice assignment storage
*For any* role and voice selection, the frontend should correctly store and retrieve the role-to-voice mapping.
**Validates: Requirements 3.4**

### Property 7: Generate button disabled when incomplete
*For any* state where not all roles have assigned voices, the "Generate Audiobook" button should be disabled.
**Validates: Requirements 3.5**

### Property 8: Generate button enabled when complete
*For any* state where all roles have assigned voices, the "Generate Audiobook" button should be enabled.
**Validates: Requirements 3.6**

### Property 9: Language selection storage
*For any* language selection, the frontend should store the selected language code for the generation request.
**Validates: Requirements 4.4**

### Property 10: Speed range constraints
*For any* speed slider interaction, the value should be constrained between 0.5x and 1.5x.
**Validates: Requirements 5.2**

### Property 11: Speed value storage
*For any* speed adjustment, the frontend should store the speed value for the generation request.
**Validates: Requirements 5.4**

### Property 12: Generation request completeness
*For any* valid application state, the generation request should contain all required fields: text, language code, role-to-voice mappings, and speed.
**Validates: Requirements 6.1**

### Property 13: Voice assignment per segment
*For any* text segment with a role marker, the backend should use the Kokoro voice assigned to that role for synthesis.
**Validates: Requirements 6.3**

### Property 14: Text chunking for long content
*For any* long text input, the backend should split it into manageable chunks per paragraph or sentence.
**Validates: Requirements 6.5**

### Property 15: Audio concatenation at 24 kHz
*For any* set of synthesized chunks, the backend should concatenate them into a single audio stream at 24 kHz sample rate.
**Validates: Requirements 6.6**

### Property 16: Audio response content type
*For any* successful synthesis, the HTTP response should include Content-Type header set to audio/mpeg or audio/wav.
**Validates: Requirements 6.7**

### Property 17: Audio response filename
*For any* audio response, the HTTP response should include Content-Disposition header with filename "audiobook.mp3" or "audiobook.wav".
**Validates: Requirements 6.8**

### Property 18: Blob URL creation
*For any* audio response from the backend, the frontend should convert it to a Blob and create a valid object URL.
**Validates: Requirements 7.1**

### Property 19: Download functionality
*For any* generated audio blob, clicking the download button should trigger file save with the correct filename.
**Validates: Requirements 7.5**

### Property 20: Loading state during role detection
*For any* role detection request, the frontend should display a loading state indicator until the response is received.
**Validates: Requirements 8.1**

### Property 21: Loading state during generation
*For any* audiobook generation request, the frontend should display loading or progress feedback until completion.
**Validates: Requirements 8.2**

### Property 22: Long text warning
*For any* text input exceeding a threshold length, the frontend should display a message indicating "Generation may take several minutes for very long content".
**Validates: Requirements 8.3**

### Property 23: Error message display
*For any* failed API request, the frontend should display a clear error message to the user.
**Validates: Requirements 8.4**

### Property 24: Role response structure validation
*For any* response from /api/tts/roles, the JSON should contain a roles array with objects having id and label fields.
**Validates: Requirements 9.3**

### Property 25: Request validation
*For any* request to /api/tts missing required fields (text, lang, or roles), the backend should reject it with an appropriate error.
**Validates: Requirements 9.5**

### Property 26: Pipeline initialization per language
*For any* supported language code (a, b, f, j), the backend should successfully load a KPipeline for that language.
**Validates: Requirements 10.3**

### Property 27: Audio sample rate
*For any* synthesized audio, the output should have a sample rate of 24 kHz.
**Validates: Requirements 10.4**

## Error Handling

### Frontend Error Handling

**Network Errors:**
- Catch fetch errors and display user-friendly messages
- Implement retry logic for transient failures (optional)
- Show specific error messages for timeout, connection refused, etc.

**Validation Errors:**
- Validate text is non-empty before API calls
- Validate all roles have assigned voices before generation
- Show inline validation messages near relevant controls

**API Error Responses:**
- Parse error responses from backend (4xx, 5xx)
- Display error details to user
- Log errors to console for debugging

**Audio Playback Errors:**
- Handle cases where audio blob is invalid
- Show error if browser doesn't support audio format
- Provide fallback message if playback fails

### Backend Error Handling

**Request Validation:**
- Validate all required fields are present
- Return 400 Bad Request with descriptive error message
- Validate language codes and voice IDs are supported

**TTS Processing Errors:**
- Catch exceptions during pipeline initialization
- Handle synthesis failures gracefully
- Return 500 Internal Server Error with error details
- Log errors for debugging

**Resource Management:**
- Handle out-of-memory scenarios for very long text
- Implement timeouts for long-running synthesis
- Clean up temporary audio files/buffers

**Model Loading Errors:**
- Catch errors when loading Kokoro models
- Return clear error if models are missing
- Provide helpful error messages for setup issues

### Error Response Format

```python
class ErrorResponse(BaseModel):
    error: str
    message: str
    details: Optional[dict] = None
```

Example:
```json
{
  "error": "ValidationError",
  "message": "Missing required field: lang",
  "details": {
    "field": "lang",
    "required": true
  }
}
```

## Testing Strategy

### Unit Testing

**Frontend Unit Tests:**
- Test individual components in isolation
- Mock API calls using jest/vitest mocks
- Test state management logic
- Test utility functions (e.g., blob URL creation)
- Verify UI elements render correctly
- Test form validation logic

**Backend Unit Tests:**
- Test role detection patterns independently
- Test text segmentation logic
- Test audio concatenation
- Test API endpoint request/response handling
- Mock Kokoro TTS calls to test logic without actual synthesis

**Example Unit Tests:**
- Empty text validation prevents API call
- Health endpoint returns correct status
- Role detector extracts "Alice:" pattern correctly
- Role detector extracts "[Narrator]" pattern correctly
- Default "Narrator" role returned when no patterns found
- Voice assignment updates state correctly
- Download button triggers file save

### Property-Based Testing

Property-based tests will verify universal properties across many randomly generated inputs using a PBT library.

**Frontend Property Tests (using fast-check for TypeScript):**

We will use **fast-check** as the property-based testing library for the React TypeScript frontend.

**Backend Property Tests (using Hypothesis for Python):**

We will use **Hypothesis** as the property-based testing library for the Python FastAPI backend.

**Configuration:**
- Each property-based test should run a minimum of 100 iterations
- Tests should generate diverse inputs to explore edge cases
- Each test must include a comment referencing the design property it validates

**Property Test Examples:**

*Frontend:*
- Property 1: Text persistence - generate random text, verify it persists across phase transitions
- Property 6: Voice assignment storage - generate random role/voice pairs, verify storage/retrieval
- Property 10: Speed range - generate random speed values, verify clamping to 0.5-1.5 range

*Backend:*
- Property 2: Colon pattern extraction - generate random text with "Name:" patterns, verify extraction
- Property 3: Bracket pattern extraction - generate random text with "[Name]" patterns, verify extraction
- Property 4: Role deduplication - generate text with duplicate roles, verify uniqueness
- Property 14: Text chunking - generate long text, verify chunks are reasonable size
- Property 27: Audio sample rate - generate random text, verify output is 24 kHz

### Integration Testing

**End-to-End Flow Tests:**
- Test complete workflow: text input → role detection → voice assignment → generation → playback
- Test with sample texts containing multiple roles
- Verify audio file is generated and playable
- Test error scenarios (network failures, invalid inputs)

**API Integration Tests:**
- Test all API endpoints with real requests
- Verify request/response formats match specifications
- Test error responses for invalid inputs
- Test with different languages and voices

### Manual Testing Checklist

- [ ] Enter text and verify role detection works
- [ ] Assign voices to all roles
- [ ] Generate audiobook and verify audio plays
- [ ] Download audiobook and verify file is valid
- [ ] Test with very long text (multiple pages)
- [ ] Test with text containing no role markers
- [ ] Test with different languages (English, French, Japanese)
- [ ] Test on mobile devices for responsiveness
- [ ] Test error scenarios (empty text, network failure)
- [ ] Verify loading states appear during processing

## Implementation Notes

### Frontend Implementation

**State Management:**
- Use React hooks (useState, useEffect) for local state
- Consider useReducer for complex state transitions
- Store voice assignments in a Map for efficient lookups

**API Client:**
- Use native fetch API
- Implement error handling and timeout logic
- Use environment variable for base URL configuration

**Audio Handling:**
- Use Blob API for audio data
- Create object URLs with URL.createObjectURL()
- Revoke object URLs when component unmounts to prevent memory leaks
- Use HTML5 audio element with controls attribute

**Styling:**
- Use Tailwind CSS for responsive design
- Ensure mobile-friendly touch targets (min 44x44px)
- Use loading spinners for async operations
- Implement accessible form controls

### Backend Implementation

**Kokoro Integration:**
- Initialize KPipeline instances lazily (on first use)
- Cache pipelines per language to avoid reloading
- Use appropriate voice IDs from Kokoro documentation
- Handle model download/loading errors gracefully

**Text Processing:**
- Use regex for role pattern matching
- Split text on paragraph boundaries (\n\n) or sentence boundaries
- Preserve whitespace and formatting where appropriate
- Handle edge cases (empty lines, special characters)

**Audio Processing:**
- Use soundfile library for audio I/O
- Concatenate numpy arrays for audio chunks
- Ensure consistent sample rate (24 kHz) across all chunks
- Convert to MP3 or WAV format for streaming

**Performance Optimization:**
- Consider async processing for long texts
- Implement streaming response for large audio files
- Use connection pooling for database (if needed)
- Monitor memory usage for very long texts

**Docker Configuration:**
- Use multi-stage build to reduce image size
- Install system dependencies (espeak-ng) in base layer
- Copy only necessary files to container
- Use .dockerignore to exclude unnecessary files
- Set appropriate environment variables
- Use health check endpoint for container orchestration

### Security Considerations

**Input Validation:**
- Sanitize user input to prevent injection attacks
- Limit text length to prevent DoS (e.g., max 1MB)
- Validate language codes and voice IDs against whitelist
- Rate limit API endpoints to prevent abuse

**CORS Configuration:**
- Configure CORS to allow frontend origin
- Restrict allowed methods and headers
- Don't expose sensitive headers

**Error Messages:**
- Don't expose internal implementation details in errors
- Log detailed errors server-side only
- Return generic error messages to client

### Deployment Considerations

**Environment Variables:**
- Frontend: VITE_API_BASE_URL
- Backend: PORT, MODEL_PATH (optional), LOG_LEVEL

**Resource Requirements:**
- Backend: Minimum 2GB RAM for model loading
- Storage: ~500MB for Kokoro models
- CPU: Multi-core recommended for faster synthesis

**Monitoring:**
- Log API requests and response times
- Monitor memory usage during synthesis
- Track error rates and types
- Set up health check monitoring

## Future Enhancements

**Potential Improvements:**
- Add support for more languages and voices
- Implement progress tracking for long generations
- Add ability to preview individual role voices
- Support for SSML markup for advanced control
- Batch processing for multiple texts
- User accounts and saved voice preferences
- Audio editing (trim, adjust volume per role)
- Export to different audio formats (FLAC, OGG)
- Background processing with job queue
- WebSocket for real-time progress updates