- Added Requirement 14 for intelligent speaker detection using Ollama/llama3:8B - Updated design document with OllamaService component and integration details - Added 4 new correctness properties for Ollama functionality - Added tasks 15-17 for implementing Ollama integration - Updated technology stack to include httpx and Ollama - Added error handling guidelines for Ollama integration - Configured fallback to pattern-based detection when Ollama unavailable This enhancement allows users to paste raw dialogue text without manual role markers. The system will use AI to intelligently detect speakers, making the tool more user-friendly.
850 lines
30 KiB
Markdown
850 lines
30 KiB
Markdown
# Design Document: Text to Audiobook
|
|
|
|
## Overview
|
|
|
|
The Text to Audiobook system is a full-stack application that converts long-form text into multi-voice audiobooks using Kokoro TTS. The architecture follows a client-server model with a React TypeScript frontend and a Python FastAPI backend running in Docker. The system automatically detects speaker roles in text, allows voice assignment per role, and generates high-quality audio using the Kokoro-82M model.
|
|
|
|
The application workflow consists of three main phases:
|
|
1. **Text Input & Role Detection**: User provides text, system identifies distinct speakers
|
|
2. **Voice Assignment**: User maps each detected role to a specific Kokoro voice
|
|
3. **Audio Generation & Playback**: System synthesizes multi-voice audio and provides playback/download
|
|
|
|
## Architecture
|
|
|
|
### High-Level Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Browser │
|
|
│ ┌───────────────────────────────────────────────────────┐ │
|
|
│ │ React Frontend (TypeScript + Vite) │ │
|
|
│ │ - Text Input Component │ │
|
|
│ │ - Role Assignment Component │ │
|
|
│ │ - Audio Player Component │ │
|
|
│ │ - API Client (tts.ts) │ │
|
|
│ └───────────────────────────────────────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
│
|
|
│ HTTP/REST
|
|
│
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Docker Container │
|
|
│ ┌───────────────────────────────────────────────────────┐ │
|
|
│ │ Python Backend (FastAPI) │ │
|
|
│ │ ┌─────────────────────────────────────────────────┐ │ │
|
|
│ │ │ API Layer │ │ │
|
|
│ │ │ - /api/health │ │ │
|
|
│ │ │ - /api/tts/roles │ │ │
|
|
│ │ │ - /api/tts │ │ │
|
|
│ │ └─────────────────────────────────────────────────┘ │ │
|
|
│ │ ┌─────────────────────────────────────────────────┐ │ │
|
|
│ │ │ Role Detection Service │ │ │
|
|
│ │ │ - Pattern matching for role markers │ │ │
|
|
│ │ │ - Role extraction and deduplication │ │ │
|
|
│ │ └─────────────────────────────────────────────────┘ │ │
|
|
│ │ ┌─────────────────────────────────────────────────┐ │ │
|
|
│ │ │ TTS Service │ │ │
|
|
│ │ │ - Kokoro KPipeline management │ │ │
|
|
│ │ │ - Text segmentation │ │ │
|
|
│ │ │ - Voice synthesis per role │ │ │
|
|
│ │ │ - Audio concatenation │ │ │
|
|
│ │ └─────────────────────────────────────────────────┘ │ │
|
|
│ │ ┌─────────────────────────────────────────────────┐ │ │
|
|
│ │ │ Kokoro TTS Engine │ │ │
|
|
│ │ │ - hexgrad/Kokoro-82M model │ │ │
|
|
│ │ │ - espeak-ng for G2P │ │ │
|
|
│ │ │ - soundfile for audio I/O │ │ │
|
|
│ │ └─────────────────────────────────────────────────┘ │ │
|
|
│ └───────────────────────────────────────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Technology Stack
|
|
|
|
**Frontend:**
|
|
- React 18+ with TypeScript
|
|
- Vite for build tooling
|
|
- Native fetch API for HTTP requests
|
|
- HTML5 Audio API for playback
|
|
- CSS/Tailwind for responsive styling
|
|
|
|
**Backend:**
|
|
- Python 3.10+
|
|
- FastAPI for REST API
|
|
- Uvicorn as ASGI server
|
|
- kokoro library (hexgrad/Kokoro-82M)
|
|
- soundfile for audio processing
|
|
- espeak-ng for grapheme-to-phoneme conversion
|
|
- httpx for Ollama API communication
|
|
- Ollama with llama3:8B for AI-based speaker detection
|
|
|
|
**Infrastructure:**
|
|
- Docker for containerization
|
|
- Docker Compose for orchestration (optional)
|
|
|
|
## Components and Interfaces
|
|
|
|
### Frontend Components
|
|
|
|
#### 1. App Component (`App.tsx`)
|
|
Main application container managing the overall workflow state.
|
|
|
|
**State:**
|
|
- `text: string` - User input text
|
|
- `roles: Role[]` - Detected roles from backend
|
|
- `voiceAssignments: Map<string, string>` - Role ID to Voice ID mapping
|
|
- `selectedLanguage: string` - Language code (default: 'a')
|
|
- `speed: number` - Playback speed (default: 1.0)
|
|
- `audioUrl: string | null` - Generated audio blob URL
|
|
- `isDetectingRoles: boolean` - Loading state for role detection
|
|
- `isGenerating: boolean` - Loading state for audio generation
|
|
- `error: string | null` - Error message if any
|
|
|
|
**Methods:**
|
|
- `handleTextChange(text: string)` - Updates text state
|
|
- `handleDetectRoles()` - Calls API to detect roles
|
|
- `handleVoiceAssignment(roleId: string, voiceId: string)` - Updates voice mapping
|
|
- `handleGenerate()` - Calls API to generate audiobook
|
|
- `handleDownload()` - Triggers download of generated audio
|
|
|
|
#### 2. TextInputSection Component
|
|
Displays the text input area and "Next: Assign voices" button.
|
|
|
|
**Props:**
|
|
- `text: string`
|
|
- `onChange: (text: string) => void`
|
|
- `onNext: () => void`
|
|
- `isLoading: boolean`
|
|
- `disabled: boolean`
|
|
|
|
#### 3. VoiceAssignmentSection Component
|
|
Displays role-to-voice mapping interface with language and speed controls.
|
|
|
|
**Props:**
|
|
- `roles: Role[]`
|
|
- `voiceAssignments: Map<string, string>`
|
|
- `onVoiceChange: (roleId: string, voiceId: string) => void`
|
|
- `language: string`
|
|
- `onLanguageChange: (lang: string) => void`
|
|
- `speed: number`
|
|
- `onSpeedChange: (speed: number) => void`
|
|
- `onGenerate: () => void`
|
|
- `isGenerating: boolean`
|
|
- `allRolesAssigned: boolean`
|
|
|
|
#### 4. AudioPlayerSection Component
|
|
Displays audio player and download button after generation.
|
|
|
|
**Props:**
|
|
- `audioUrl: string`
|
|
- `onDownload: () => void`
|
|
|
|
#### 5. API Client (`src/api/tts.ts`)
|
|
|
|
**Interfaces:**
|
|
```typescript
|
|
export interface Role {
|
|
id: string;
|
|
label: string;
|
|
}
|
|
|
|
export interface RoleWithVoice extends Role {
|
|
voice: string;
|
|
}
|
|
|
|
export interface RolesResponse {
|
|
roles: Role[];
|
|
}
|
|
|
|
export interface GenerateRequest {
|
|
text: string;
|
|
lang: string;
|
|
roles: { id: string; voice: string }[];
|
|
speed?: number;
|
|
}
|
|
```
|
|
|
|
**Functions:**
|
|
```typescript
|
|
export async function detectRoles(text: string): Promise<RolesResponse>
|
|
export async function generateAudiobook(payload: GenerateRequest): Promise<Blob>
|
|
export async function checkHealth(): Promise<{ status: string }>
|
|
```
|
|
|
|
### Backend Components
|
|
|
|
#### 1. API Layer (`main.py`)
|
|
|
|
**Endpoints:**
|
|
|
|
```python
|
|
@app.get("/api/health")
|
|
async def health_check() -> dict:
|
|
"""Returns service health status"""
|
|
return {"status": "ok"}
|
|
|
|
@app.post("/api/tts/roles")
|
|
async def detect_roles(request: RoleDetectionRequest) -> RolesResponse:
|
|
"""Analyzes text and returns detected roles"""
|
|
pass
|
|
|
|
@app.post("/api/tts")
|
|
async def generate_audiobook(request: TTSRequest) -> StreamingResponse:
|
|
"""Generates multi-voice audiobook and streams audio"""
|
|
pass
|
|
```
|
|
|
|
**Request/Response Models:**
|
|
```python
|
|
class RoleDetectionRequest(BaseModel):
|
|
text: str
|
|
|
|
class RoleInfo(BaseModel):
|
|
id: str
|
|
label: str
|
|
|
|
class RolesResponse(BaseModel):
|
|
roles: List[RoleInfo]
|
|
|
|
class RoleVoiceMapping(BaseModel):
|
|
id: str
|
|
voice: str
|
|
|
|
class TTSRequest(BaseModel):
|
|
text: str
|
|
lang: str
|
|
roles: List[RoleVoiceMapping]
|
|
speed: Optional[float] = 1.0
|
|
```
|
|
|
|
#### 2. Role Detection Service (`services/role_detector.py`)
|
|
|
|
**Class: RoleDetector**
|
|
|
|
**Methods:**
|
|
```python
|
|
def detect_roles(self, text: str) -> List[RoleInfo]:
|
|
"""
|
|
Analyzes text to identify distinct speaker roles.
|
|
First attempts AI-based detection using Ollama, then falls back to pattern matching.
|
|
Supports patterns:
|
|
- "RoleName:" at line start
|
|
- "[RoleName]" preceding dialogue
|
|
Returns unique roles or default "Narrator" if none found.
|
|
"""
|
|
pass
|
|
|
|
def _detect_with_ollama(self, text: str) -> Set[str]:
|
|
"""
|
|
Uses Ollama with llama3:8B to intelligently detect speakers.
|
|
Returns set of speaker names or empty set if Ollama unavailable.
|
|
"""
|
|
pass
|
|
|
|
def _extract_colon_pattern(self, text: str) -> Set[str]:
|
|
"""Extracts roles from "Name:" pattern"""
|
|
pass
|
|
|
|
def _extract_bracket_pattern(self, text: str) -> Set[str]:
|
|
"""Extracts roles from "[Name]" pattern"""
|
|
pass
|
|
```
|
|
|
|
#### 3. Ollama Integration Service (`services/ollama_service.py`)
|
|
|
|
**Class: OllamaService**
|
|
|
|
**Attributes:**
|
|
- `base_url: str` - Ollama API base URL (from environment)
|
|
- `model: str = "llama3:8b"` - Model to use for speaker detection
|
|
- `timeout: int = 30` - Request timeout in seconds
|
|
|
|
**Methods:**
|
|
```python
|
|
def __init__(self, base_url: str = "http://localhost:11434"):
|
|
"""Initializes Ollama service with base URL"""
|
|
pass
|
|
|
|
async def detect_speakers(self, text: str) -> List[str]:
|
|
"""
|
|
Sends text to Ollama for speaker detection.
|
|
Returns list of detected speaker names.
|
|
Raises exception if Ollama is unavailable or request fails.
|
|
"""
|
|
pass
|
|
|
|
def _build_prompt(self, text: str) -> str:
|
|
"""
|
|
Constructs prompt for speaker detection.
|
|
Instructs model to return JSON array of speaker names.
|
|
"""
|
|
pass
|
|
|
|
def _parse_response(self, response: str) -> List[str]:
|
|
"""
|
|
Parses Ollama response to extract speaker names.
|
|
Handles both JSON and plain text responses.
|
|
"""
|
|
pass
|
|
|
|
async def is_available(self) -> bool:
|
|
"""
|
|
Checks if Ollama service is available.
|
|
Returns True if service responds to health check.
|
|
"""
|
|
pass
|
|
```
|
|
|
|
#### 4. TTS Service (`services/tts_service.py`)
|
|
|
|
**Class: TTSService**
|
|
|
|
**Attributes:**
|
|
- `pipelines: Dict[str, KPipeline]` - Cached pipelines per language
|
|
- `default_voice: str = "af_bella"`
|
|
|
|
**Methods:**
|
|
```python
|
|
def __init__(self):
|
|
"""Initializes TTS service"""
|
|
pass
|
|
|
|
def get_pipeline(self, lang_code: str) -> KPipeline:
|
|
"""
|
|
Returns or creates KPipeline for given language.
|
|
Caches pipelines to avoid reloading.
|
|
"""
|
|
pass
|
|
|
|
def segment_text(self, text: str, role_mappings: Dict[str, str]) -> List[Tuple[str, str]]:
|
|
"""
|
|
Segments text into (role_id, text_content) tuples.
|
|
Identifies role markers and assigns segments accordingly.
|
|
"""
|
|
pass
|
|
|
|
def synthesize_segment(self, pipeline: KPipeline, text: str, voice: str, speed: float) -> np.ndarray:
|
|
"""
|
|
Synthesizes a single text segment with specified voice.
|
|
Returns audio samples as numpy array.
|
|
"""
|
|
pass
|
|
|
|
def concatenate_audio(self, segments: List[np.ndarray]) -> np.ndarray:
|
|
"""
|
|
Concatenates multiple audio segments into single array.
|
|
Ensures consistent sample rate (24kHz).
|
|
"""
|
|
pass
|
|
|
|
async def generate_audiobook(
|
|
self,
|
|
text: str,
|
|
lang: str,
|
|
role_voice_map: Dict[str, str],
|
|
speed: float = 1.0
|
|
) -> bytes:
|
|
"""
|
|
Main orchestration method.
|
|
Segments text, synthesizes each segment, concatenates, and returns audio bytes.
|
|
"""
|
|
pass
|
|
```
|
|
|
|
## Data Models
|
|
|
|
### Frontend Data Models
|
|
|
|
**Role:**
|
|
```typescript
|
|
interface Role {
|
|
id: string; // Unique identifier (e.g., "Narrator", "Alice")
|
|
label: string; // Display name for UI
|
|
}
|
|
```
|
|
|
|
**Voice Option:**
|
|
```typescript
|
|
interface VoiceOption {
|
|
id: string; // Kokoro voice ID (e.g., "af_bella")
|
|
label: string; // Display name (e.g., "Bella (Female, American)")
|
|
gender: 'male' | 'female';
|
|
accent: string; // e.g., "American", "British"
|
|
}
|
|
```
|
|
|
|
**Language Option:**
|
|
```typescript
|
|
interface LanguageOption {
|
|
code: string; // Kokoro language code (e.g., "a", "b", "f", "j")
|
|
label: string; // Display name (e.g., "American English")
|
|
}
|
|
```
|
|
|
|
**Application State:**
|
|
```typescript
|
|
interface AppState {
|
|
phase: 'input' | 'assignment' | 'generated';
|
|
text: string;
|
|
roles: Role[];
|
|
voiceAssignments: Map<string, string>;
|
|
language: string;
|
|
speed: number;
|
|
audioUrl: string | null;
|
|
isLoading: boolean;
|
|
error: string | null;
|
|
}
|
|
```
|
|
|
|
### Backend Data Models
|
|
|
|
**Text Segment:**
|
|
```python
|
|
@dataclass
|
|
class TextSegment:
|
|
role_id: str
|
|
text: str
|
|
voice: str
|
|
```
|
|
|
|
**Audio Chunk:**
|
|
```python
|
|
@dataclass
|
|
class AudioChunk:
|
|
samples: np.ndarray
|
|
sample_rate: int = 24000
|
|
```
|
|
|
|
### Constants
|
|
|
|
**Supported Voices:**
|
|
```typescript
|
|
const KOKORO_VOICES: VoiceOption[] = [
|
|
{ id: 'af_bella', label: 'Bella (Female, American)', gender: 'female', accent: 'American' },
|
|
{ id: 'af_sarah', label: 'Sarah (Female, American)', gender: 'female', accent: 'American' },
|
|
{ id: 'am_adam', label: 'Adam (Male, American)', gender: 'male', accent: 'American' },
|
|
{ id: 'am_michael', label: 'Michael (Male, American)', gender: 'male', accent: 'American' },
|
|
{ id: 'bf_emma', label: 'Emma (Female, British)', gender: 'female', accent: 'British' },
|
|
{ id: 'bf_isabella', label: 'Isabella (Female, British)', gender: 'female', accent: 'British' },
|
|
{ id: 'bm_george', label: 'George (Male, British)', gender: 'male', accent: 'British' },
|
|
{ id: 'bm_lewis', label: 'Lewis (Male, British)', gender: 'male', accent: 'British' },
|
|
];
|
|
```
|
|
|
|
**Supported Languages:**
|
|
```typescript
|
|
const LANGUAGES: LanguageOption[] = [
|
|
{ code: 'a', label: 'American English' },
|
|
{ code: 'b', label: 'British English' },
|
|
{ code: 'f', label: 'French' },
|
|
{ code: 'j', label: 'Japanese' },
|
|
];
|
|
```
|
|
|
|
**Role Detection Patterns:**
|
|
```python
|
|
ROLE_PATTERNS = {
|
|
'colon': r'^([A-Z][a-zA-Z\s]+):\s', # "Alice: Hello there"
|
|
'bracket': r'\[([A-Z][a-zA-Z\s]+)\]', # "[Narrator] The story begins"
|
|
}
|
|
```
|
|
|
|
|
|
## Correctness Properties
|
|
|
|
*A property is a characteristic or behavior that should hold true across all valid executions of a system—essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
|
|
|
|
### Property 1: Text persistence invariant
|
|
*For any* text entered by the user, the text content should remain unchanged as the application transitions between input, assignment, and generated phases.
|
|
**Validates: Requirements 1.5**
|
|
|
|
### Property 2: Role extraction from colon pattern
|
|
*For any* text containing lines starting with "RoleName:", the role detection should extract RoleName as a distinct role.
|
|
**Validates: Requirements 2.2**
|
|
|
|
### Property 3: Role extraction from bracket pattern
|
|
*For any* text containing "[RoleName]" patterns, the role detection should extract RoleName as a distinct role.
|
|
**Validates: Requirements 2.3**
|
|
|
|
### Property 4: Role deduplication
|
|
*For any* text with multiple instances of the same role marker, the role detection should return each unique role exactly once.
|
|
**Validates: Requirements 2.5**
|
|
|
|
### Property 5: Role response structure
|
|
*For any* detected roles, each role object in the response should contain both id and label fields.
|
|
**Validates: Requirements 2.6**
|
|
|
|
### Property 6: Voice assignment storage
|
|
*For any* role and voice selection, the frontend should correctly store and retrieve the role-to-voice mapping.
|
|
**Validates: Requirements 3.4**
|
|
|
|
### Property 7: Generate button disabled when incomplete
|
|
*For any* state where not all roles have assigned voices, the "Generate Audiobook" button should be disabled.
|
|
**Validates: Requirements 3.5**
|
|
|
|
### Property 8: Generate button enabled when complete
|
|
*For any* state where all roles have assigned voices, the "Generate Audiobook" button should be enabled.
|
|
**Validates: Requirements 3.6**
|
|
|
|
### Property 9: Language selection storage
|
|
*For any* language selection, the frontend should store the selected language code for the generation request.
|
|
**Validates: Requirements 4.4**
|
|
|
|
### Property 10: Speed range constraints
|
|
*For any* speed slider interaction, the value should be constrained between 0.5x and 1.5x.
|
|
**Validates: Requirements 5.2**
|
|
|
|
### Property 11: Speed value storage
|
|
*For any* speed adjustment, the frontend should store the speed value for the generation request.
|
|
**Validates: Requirements 5.4**
|
|
|
|
### Property 12: Generation request completeness
|
|
*For any* valid application state, the generation request should contain all required fields: text, language code, role-to-voice mappings, and speed.
|
|
**Validates: Requirements 6.1**
|
|
|
|
### Property 13: Voice assignment per segment
|
|
*For any* text segment with a role marker, the backend should use the Kokoro voice assigned to that role for synthesis.
|
|
**Validates: Requirements 6.3**
|
|
|
|
### Property 14: Text chunking for long content
|
|
*For any* long text input, the backend should split it into manageable chunks per paragraph or sentence.
|
|
**Validates: Requirements 6.5**
|
|
|
|
### Property 15: Audio concatenation at 24 kHz
|
|
*For any* set of synthesized chunks, the backend should concatenate them into a single audio stream at 24 kHz sample rate.
|
|
**Validates: Requirements 6.6**
|
|
|
|
### Property 16: Audio response content type
|
|
*For any* successful synthesis, the HTTP response should include Content-Type header set to audio/mpeg or audio/wav.
|
|
**Validates: Requirements 6.7**
|
|
|
|
### Property 17: Audio response filename
|
|
*For any* audio response, the HTTP response should include Content-Disposition header with filename "audiobook.mp3" or "audiobook.wav".
|
|
**Validates: Requirements 6.8**
|
|
|
|
### Property 18: Blob URL creation
|
|
*For any* audio response from the backend, the frontend should convert it to a Blob and create a valid object URL.
|
|
**Validates: Requirements 7.1**
|
|
|
|
### Property 19: Download functionality
|
|
*For any* generated audio blob, clicking the download button should trigger file save with the correct filename.
|
|
**Validates: Requirements 7.5**
|
|
|
|
### Property 20: Loading state during role detection
|
|
*For any* role detection request, the frontend should display a loading state indicator until the response is received.
|
|
**Validates: Requirements 8.1**
|
|
|
|
### Property 21: Loading state during generation
|
|
*For any* audiobook generation request, the frontend should display loading or progress feedback until completion.
|
|
**Validates: Requirements 8.2**
|
|
|
|
### Property 22: Long text warning
|
|
*For any* text input exceeding a threshold length, the frontend should display a message indicating "Generation may take several minutes for very long content".
|
|
**Validates: Requirements 8.3**
|
|
|
|
### Property 23: Error message display
|
|
*For any* failed API request, the frontend should display a clear error message to the user.
|
|
**Validates: Requirements 8.4**
|
|
|
|
### Property 24: Role response structure validation
|
|
*For any* response from /api/tts/roles, the JSON should contain a roles array with objects having id and label fields.
|
|
**Validates: Requirements 9.3**
|
|
|
|
### Property 25: Request validation
|
|
*For any* request to /api/tts missing required fields (text, lang, or roles), the backend should reject it with an appropriate error.
|
|
**Validates: Requirements 9.5**
|
|
|
|
### Property 26: Pipeline initialization per language
|
|
*For any* supported language code (a, b, f, j), the backend should successfully load a KPipeline for that language.
|
|
**Validates: Requirements 10.3**
|
|
|
|
### Property 27: Audio sample rate
|
|
*For any* synthesized audio, the output should have a sample rate of 24 kHz.
|
|
**Validates: Requirements 10.4**
|
|
|
|
### Property 28: Ollama speaker detection
|
|
*For any* text without explicit role markers, when Ollama is available, the backend should attempt AI-based speaker detection before falling back to pattern matching.
|
|
**Validates: Requirements 14.1**
|
|
|
|
### Property 29: Ollama JSON response parsing
|
|
*For any* valid JSON response from Ollama containing speaker names, the backend should successfully parse and extract the speaker list.
|
|
**Validates: Requirements 14.3**
|
|
|
|
### Property 30: Ollama fallback behavior
|
|
*For any* text, when Ollama is unavailable or fails, the backend should fall back to pattern-based role detection without error.
|
|
**Validates: Requirements 14.4**
|
|
|
|
### Property 31: Role deduplication across detection methods
|
|
*For any* text where both AI detection and pattern detection find roles, the backend should merge results and return each unique role exactly once.
|
|
**Validates: Requirements 14.5**
|
|
|
|
## Error Handling
|
|
|
|
### Frontend Error Handling
|
|
|
|
**Network Errors:**
|
|
- Catch fetch errors and display user-friendly messages
|
|
- Implement retry logic for transient failures (optional)
|
|
- Show specific error messages for timeout, connection refused, etc.
|
|
|
|
**Validation Errors:**
|
|
- Validate text is non-empty before API calls
|
|
- Validate all roles have assigned voices before generation
|
|
- Show inline validation messages near relevant controls
|
|
|
|
**API Error Responses:**
|
|
- Parse error responses from backend (4xx, 5xx)
|
|
- Display error details to user
|
|
- Log errors to console for debugging
|
|
|
|
**Audio Playback Errors:**
|
|
- Handle cases where audio blob is invalid
|
|
- Show error if browser doesn't support audio format
|
|
- Provide fallback message if playback fails
|
|
|
|
### Backend Error Handling
|
|
|
|
**Request Validation:**
|
|
- Validate all required fields are present
|
|
- Return 400 Bad Request with descriptive error message
|
|
- Validate language codes and voice IDs are supported
|
|
|
|
**TTS Processing Errors:**
|
|
- Catch exceptions during pipeline initialization
|
|
- Handle synthesis failures gracefully
|
|
- Return 500 Internal Server Error with error details
|
|
- Log errors for debugging
|
|
|
|
**Ollama Integration Errors:**
|
|
- Handle Ollama service unavailable gracefully
|
|
- Fall back to pattern-based detection on Ollama failure
|
|
- Log Ollama errors for debugging without failing request
|
|
- Handle malformed JSON responses from Ollama
|
|
- Implement timeout for Ollama requests (30 seconds)
|
|
|
|
**Resource Management:**
|
|
- Handle out-of-memory scenarios for very long text
|
|
- Implement timeouts for long-running synthesis
|
|
- Clean up temporary audio files/buffers
|
|
|
|
**Model Loading Errors:**
|
|
- Catch errors when loading Kokoro models
|
|
- Return clear error if models are missing
|
|
- Provide helpful error messages for setup issues
|
|
|
|
### Error Response Format
|
|
|
|
```python
|
|
class ErrorResponse(BaseModel):
|
|
error: str
|
|
message: str
|
|
details: Optional[dict] = None
|
|
```
|
|
|
|
Example:
|
|
```json
|
|
{
|
|
"error": "ValidationError",
|
|
"message": "Missing required field: lang",
|
|
"details": {
|
|
"field": "lang",
|
|
"required": true
|
|
}
|
|
}
|
|
```
|
|
|
|
## Testing Strategy
|
|
|
|
### Unit Testing
|
|
|
|
**Frontend Unit Tests:**
|
|
- Test individual components in isolation
|
|
- Mock API calls using jest/vitest mocks
|
|
- Test state management logic
|
|
- Test utility functions (e.g., blob URL creation)
|
|
- Verify UI elements render correctly
|
|
- Test form validation logic
|
|
|
|
**Backend Unit Tests:**
|
|
- Test role detection patterns independently
|
|
- Test text segmentation logic
|
|
- Test audio concatenation
|
|
- Test API endpoint request/response handling
|
|
- Mock Kokoro TTS calls to test logic without actual synthesis
|
|
|
|
**Example Unit Tests:**
|
|
- Empty text validation prevents API call
|
|
- Health endpoint returns correct status
|
|
- Role detector extracts "Alice:" pattern correctly
|
|
- Role detector extracts "[Narrator]" pattern correctly
|
|
- Default "Narrator" role returned when no patterns found
|
|
- Voice assignment updates state correctly
|
|
- Download button triggers file save
|
|
|
|
### Property-Based Testing
|
|
|
|
Property-based tests will verify universal properties across many randomly generated inputs using a PBT library.
|
|
|
|
**Frontend Property Tests (using fast-check for TypeScript):**
|
|
|
|
We will use **fast-check** as the property-based testing library for the React TypeScript frontend.
|
|
|
|
**Backend Property Tests (using Hypothesis for Python):**
|
|
|
|
We will use **Hypothesis** as the property-based testing library for the Python FastAPI backend.
|
|
|
|
**Configuration:**
|
|
- Each property-based test should run a minimum of 100 iterations
|
|
- Tests should generate diverse inputs to explore edge cases
|
|
- Each test must include a comment referencing the design property it validates
|
|
|
|
**Property Test Examples:**
|
|
|
|
*Frontend:*
|
|
- Property 1: Text persistence - generate random text, verify it persists across phase transitions
|
|
- Property 6: Voice assignment storage - generate random role/voice pairs, verify storage/retrieval
|
|
- Property 10: Speed range - generate random speed values, verify clamping to 0.5-1.5 range
|
|
|
|
*Backend:*
|
|
- Property 2: Colon pattern extraction - generate random text with "Name:" patterns, verify extraction
|
|
- Property 3: Bracket pattern extraction - generate random text with "[Name]" patterns, verify extraction
|
|
- Property 4: Role deduplication - generate text with duplicate roles, verify uniqueness
|
|
- Property 14: Text chunking - generate long text, verify chunks are reasonable size
|
|
- Property 27: Audio sample rate - generate random text, verify output is 24 kHz
|
|
|
|
### Integration Testing
|
|
|
|
**End-to-End Flow Tests:**
|
|
- Test complete workflow: text input → role detection → voice assignment → generation → playback
|
|
- Test with sample texts containing multiple roles
|
|
- Verify audio file is generated and playable
|
|
- Test error scenarios (network failures, invalid inputs)
|
|
|
|
**API Integration Tests:**
|
|
- Test all API endpoints with real requests
|
|
- Verify request/response formats match specifications
|
|
- Test error responses for invalid inputs
|
|
- Test with different languages and voices
|
|
|
|
### Manual Testing Checklist
|
|
|
|
- [ ] Enter text and verify role detection works
|
|
- [ ] Assign voices to all roles
|
|
- [ ] Generate audiobook and verify audio plays
|
|
- [ ] Download audiobook and verify file is valid
|
|
- [ ] Test with very long text (multiple pages)
|
|
- [ ] Test with text containing no role markers
|
|
- [ ] Test with different languages (English, French, Japanese)
|
|
- [ ] Test on mobile devices for responsiveness
|
|
- [ ] Test error scenarios (empty text, network failure)
|
|
- [ ] Verify loading states appear during processing
|
|
|
|
## Implementation Notes
|
|
|
|
### Frontend Implementation
|
|
|
|
**State Management:**
|
|
- Use React hooks (useState, useEffect) for local state
|
|
- Consider useReducer for complex state transitions
|
|
- Store voice assignments in a Map for efficient lookups
|
|
|
|
**API Client:**
|
|
- Use native fetch API
|
|
- Implement error handling and timeout logic
|
|
- Use environment variable for base URL configuration
|
|
|
|
**Audio Handling:**
|
|
- Use Blob API for audio data
|
|
- Create object URLs with URL.createObjectURL()
|
|
- Revoke object URLs when component unmounts to prevent memory leaks
|
|
- Use HTML5 audio element with controls attribute
|
|
|
|
**Styling:**
|
|
- Use Tailwind CSS for responsive design
|
|
- Ensure mobile-friendly touch targets (min 44x44px)
|
|
- Use loading spinners for async operations
|
|
- Implement accessible form controls
|
|
|
|
### Backend Implementation
|
|
|
|
**Kokoro Integration:**
|
|
- Initialize KPipeline instances lazily (on first use)
|
|
- Cache pipelines per language to avoid reloading
|
|
- Use appropriate voice IDs from Kokoro documentation
|
|
- Handle model download/loading errors gracefully
|
|
|
|
**Text Processing:**
|
|
- Use regex for role pattern matching
|
|
- Split text on paragraph boundaries (\n\n) or sentence boundaries
|
|
- Preserve whitespace and formatting where appropriate
|
|
- Handle edge cases (empty lines, special characters)
|
|
|
|
**Audio Processing:**
|
|
- Use soundfile library for audio I/O
|
|
- Concatenate numpy arrays for audio chunks
|
|
- Ensure consistent sample rate (24 kHz) across all chunks
|
|
- Convert to MP3 or WAV format for streaming
|
|
|
|
**Performance Optimization:**
|
|
- Consider async processing for long texts
|
|
- Implement streaming response for large audio files
|
|
- Use connection pooling for database (if needed)
|
|
- Monitor memory usage for very long texts
|
|
|
|
**Docker Configuration:**
|
|
- Use multi-stage build to reduce image size
|
|
- Install system dependencies (espeak-ng) in base layer
|
|
- Copy only necessary files to container
|
|
- Use .dockerignore to exclude unnecessary files
|
|
- Set appropriate environment variables
|
|
- Use health check endpoint for container orchestration
|
|
|
|
### Security Considerations
|
|
|
|
**Input Validation:**
|
|
- Sanitize user input to prevent injection attacks
|
|
- Limit text length to prevent DoS (e.g., max 1MB)
|
|
- Validate language codes and voice IDs against whitelist
|
|
- Rate limit API endpoints to prevent abuse
|
|
|
|
**CORS Configuration:**
|
|
- Configure CORS to allow frontend origin
|
|
- Restrict allowed methods and headers
|
|
- Don't expose sensitive headers
|
|
|
|
**Error Messages:**
|
|
- Don't expose internal implementation details in errors
|
|
- Log detailed errors server-side only
|
|
- Return generic error messages to client
|
|
|
|
### Deployment Considerations
|
|
|
|
**Environment Variables:**
|
|
- Frontend: VITE_API_BASE_URL
|
|
- Backend: PORT, MODEL_PATH (optional), LOG_LEVEL
|
|
|
|
**Resource Requirements:**
|
|
- Backend: Minimum 2GB RAM for model loading
|
|
- Storage: ~500MB for Kokoro models
|
|
- CPU: Multi-core recommended for faster synthesis
|
|
|
|
**Monitoring:**
|
|
- Log API requests and response times
|
|
- Monitor memory usage during synthesis
|
|
- Track error rates and types
|
|
- Set up health check monitoring
|
|
|
|
## Future Enhancements
|
|
|
|
**Potential Improvements:**
|
|
- Add support for more languages and voices
|
|
- Implement progress tracking for long generations
|
|
- Add ability to preview individual role voices
|
|
- Support for SSML markup for advanced control
|
|
- Batch processing for multiple texts
|
|
- User accounts and saved voice preferences
|
|
- Audio editing (trim, adjust volume per role)
|
|
- Export to different audio formats (FLAC, OGG)
|
|
- Background processing with job queue
|
|
- WebSocket for real-time progress updates
|