Task 1: Set up project structure and dependencies
- Created backend directory with FastAPI project structure - Created frontend directory with React + TypeScript + Vite - Configured Python virtual environment and installed dependencies (fastapi, uvicorn, kokoro, soundfile, python-multipart) - Verified espeak-ng system dependency is installed - Configured Node.js project with TypeScript, React, and Vite - Set up Tailwind CSS for styling - Set up environment variable configuration for both projects - Created comprehensive README documentation Requirements: 9.6, 10.2, 11.1
This commit is contained in:
771
.kiro/specs/text-to-audiobook/design.md
Normal file
771
.kiro/specs/text-to-audiobook/design.md
Normal file
@@ -0,0 +1,771 @@
|
||||
# Design Document: Text to Audiobook
|
||||
|
||||
## Overview
|
||||
|
||||
The Text to Audiobook system is a full-stack application that converts long-form text into multi-voice audiobooks using Kokoro TTS. The architecture follows a client-server model with a React TypeScript frontend and a Python FastAPI backend running in Docker. The system automatically detects speaker roles in text, allows voice assignment per role, and generates high-quality audio using the Kokoro-82M model.
|
||||
|
||||
The application workflow consists of three main phases:
|
||||
1. **Text Input & Role Detection**: User provides text, system identifies distinct speakers
|
||||
2. **Voice Assignment**: User maps each detected role to a specific Kokoro voice
|
||||
3. **Audio Generation & Playback**: System synthesizes multi-voice audio and provides playback/download
|
||||
|
||||
## Architecture
|
||||
|
||||
### High-Level Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Browser │
|
||||
│ ┌───────────────────────────────────────────────────────┐ │
|
||||
│ │ React Frontend (TypeScript + Vite) │ │
|
||||
│ │ - Text Input Component │ │
|
||||
│ │ - Role Assignment Component │ │
|
||||
│ │ - Audio Player Component │ │
|
||||
│ │ - API Client (tts.ts) │ │
|
||||
│ └───────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ HTTP/REST
|
||||
│
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Docker Container │
|
||||
│ ┌───────────────────────────────────────────────────────┐ │
|
||||
│ │ Python Backend (FastAPI) │ │
|
||||
│ │ ┌─────────────────────────────────────────────────┐ │ │
|
||||
│ │ │ API Layer │ │ │
|
||||
│ │ │ - /api/health │ │ │
|
||||
│ │ │ - /api/tts/roles │ │ │
|
||||
│ │ │ - /api/tts │ │ │
|
||||
│ │ └─────────────────────────────────────────────────┘ │ │
|
||||
│ │ ┌─────────────────────────────────────────────────┐ │ │
|
||||
│ │ │ Role Detection Service │ │ │
|
||||
│ │ │ - Pattern matching for role markers │ │ │
|
||||
│ │ │ - Role extraction and deduplication │ │ │
|
||||
│ │ └─────────────────────────────────────────────────┘ │ │
|
||||
│ │ ┌─────────────────────────────────────────────────┐ │ │
|
||||
│ │ │ TTS Service │ │ │
|
||||
│ │ │ - Kokoro KPipeline management │ │ │
|
||||
│ │ │ - Text segmentation │ │ │
|
||||
│ │ │ - Voice synthesis per role │ │ │
|
||||
│ │ │ - Audio concatenation │ │ │
|
||||
│ │ └─────────────────────────────────────────────────┘ │ │
|
||||
│ │ ┌─────────────────────────────────────────────────┐ │ │
|
||||
│ │ │ Kokoro TTS Engine │ │ │
|
||||
│ │ │ - hexgrad/Kokoro-82M model │ │ │
|
||||
│ │ │ - espeak-ng for G2P │ │ │
|
||||
│ │ │ - soundfile for audio I/O │ │ │
|
||||
│ │ └─────────────────────────────────────────────────┘ │ │
|
||||
│ └───────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Technology Stack
|
||||
|
||||
**Frontend:**
|
||||
- React 18+ with TypeScript
|
||||
- Vite for build tooling
|
||||
- Native fetch API for HTTP requests
|
||||
- HTML5 Audio API for playback
|
||||
- CSS/Tailwind for responsive styling
|
||||
|
||||
**Backend:**
|
||||
- Python 3.10+
|
||||
- FastAPI for REST API
|
||||
- Uvicorn as ASGI server
|
||||
- kokoro library (hexgrad/Kokoro-82M)
|
||||
- soundfile for audio processing
|
||||
- espeak-ng for grapheme-to-phoneme conversion
|
||||
|
||||
**Infrastructure:**
|
||||
- Docker for containerization
|
||||
- Docker Compose for orchestration (optional)
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
### Frontend Components
|
||||
|
||||
#### 1. App Component (`App.tsx`)
|
||||
Main application container managing the overall workflow state.
|
||||
|
||||
**State:**
|
||||
- `text: string` - User input text
|
||||
- `roles: Role[]` - Detected roles from backend
|
||||
- `voiceAssignments: Map<string, string>` - Role ID to Voice ID mapping
|
||||
- `selectedLanguage: string` - Language code (default: 'a')
|
||||
- `speed: number` - Playback speed (default: 1.0)
|
||||
- `audioUrl: string | null` - Generated audio blob URL
|
||||
- `isDetectingRoles: boolean` - Loading state for role detection
|
||||
- `isGenerating: boolean` - Loading state for audio generation
|
||||
- `error: string | null` - Error message if any
|
||||
|
||||
**Methods:**
|
||||
- `handleTextChange(text: string)` - Updates text state
|
||||
- `handleDetectRoles()` - Calls API to detect roles
|
||||
- `handleVoiceAssignment(roleId: string, voiceId: string)` - Updates voice mapping
|
||||
- `handleGenerate()` - Calls API to generate audiobook
|
||||
- `handleDownload()` - Triggers download of generated audio
|
||||
|
||||
#### 2. TextInputSection Component
|
||||
Displays the text input area and "Next: Assign voices" button.
|
||||
|
||||
**Props:**
|
||||
- `text: string`
|
||||
- `onChange: (text: string) => void`
|
||||
- `onNext: () => void`
|
||||
- `isLoading: boolean`
|
||||
- `disabled: boolean`
|
||||
|
||||
#### 3. VoiceAssignmentSection Component
|
||||
Displays role-to-voice mapping interface with language and speed controls.
|
||||
|
||||
**Props:**
|
||||
- `roles: Role[]`
|
||||
- `voiceAssignments: Map<string, string>`
|
||||
- `onVoiceChange: (roleId: string, voiceId: string) => void`
|
||||
- `language: string`
|
||||
- `onLanguageChange: (lang: string) => void`
|
||||
- `speed: number`
|
||||
- `onSpeedChange: (speed: number) => void`
|
||||
- `onGenerate: () => void`
|
||||
- `isGenerating: boolean`
|
||||
- `allRolesAssigned: boolean`
|
||||
|
||||
#### 4. AudioPlayerSection Component
|
||||
Displays audio player and download button after generation.
|
||||
|
||||
**Props:**
|
||||
- `audioUrl: string`
|
||||
- `onDownload: () => void`
|
||||
|
||||
#### 5. API Client (`src/api/tts.ts`)
|
||||
|
||||
**Interfaces:**
|
||||
```typescript
|
||||
export interface Role {
|
||||
id: string;
|
||||
label: string;
|
||||
}
|
||||
|
||||
export interface RoleWithVoice extends Role {
|
||||
voice: string;
|
||||
}
|
||||
|
||||
export interface RolesResponse {
|
||||
roles: Role[];
|
||||
}
|
||||
|
||||
export interface GenerateRequest {
|
||||
text: string;
|
||||
lang: string;
|
||||
roles: { id: string; voice: string }[];
|
||||
speed?: number;
|
||||
}
|
||||
```
|
||||
|
||||
**Functions:**
|
||||
```typescript
|
||||
export async function detectRoles(text: string): Promise<RolesResponse>
|
||||
export async function generateAudiobook(payload: GenerateRequest): Promise<Blob>
|
||||
export async function checkHealth(): Promise<{ status: string }>
|
||||
```
|
||||
|
||||
### Backend Components
|
||||
|
||||
#### 1. API Layer (`main.py`)
|
||||
|
||||
**Endpoints:**
|
||||
|
||||
```python
|
||||
@app.get("/api/health")
|
||||
async def health_check() -> dict:
|
||||
"""Returns service health status"""
|
||||
return {"status": "ok"}
|
||||
|
||||
@app.post("/api/tts/roles")
|
||||
async def detect_roles(request: RoleDetectionRequest) -> RolesResponse:
|
||||
"""Analyzes text and returns detected roles"""
|
||||
pass
|
||||
|
||||
@app.post("/api/tts")
|
||||
async def generate_audiobook(request: TTSRequest) -> StreamingResponse:
|
||||
"""Generates multi-voice audiobook and streams audio"""
|
||||
pass
|
||||
```
|
||||
|
||||
**Request/Response Models:**
|
||||
```python
|
||||
class RoleDetectionRequest(BaseModel):
|
||||
text: str
|
||||
|
||||
class RoleInfo(BaseModel):
|
||||
id: str
|
||||
label: str
|
||||
|
||||
class RolesResponse(BaseModel):
|
||||
roles: List[RoleInfo]
|
||||
|
||||
class RoleVoiceMapping(BaseModel):
|
||||
id: str
|
||||
voice: str
|
||||
|
||||
class TTSRequest(BaseModel):
|
||||
text: str
|
||||
lang: str
|
||||
roles: List[RoleVoiceMapping]
|
||||
speed: Optional[float] = 1.0
|
||||
```
|
||||
|
||||
#### 2. Role Detection Service (`services/role_detector.py`)
|
||||
|
||||
**Class: RoleDetector**
|
||||
|
||||
**Methods:**
|
||||
```python
|
||||
def detect_roles(self, text: str) -> List[RoleInfo]:
|
||||
"""
|
||||
Analyzes text to identify distinct speaker roles.
|
||||
Supports patterns:
|
||||
- "RoleName:" at line start
|
||||
- "[RoleName]" preceding dialogue
|
||||
Returns unique roles or default "Narrator" if none found.
|
||||
"""
|
||||
pass
|
||||
|
||||
def _extract_colon_pattern(self, text: str) -> Set[str]:
|
||||
"""Extracts roles from "Name:" pattern"""
|
||||
pass
|
||||
|
||||
def _extract_bracket_pattern(self, text: str) -> Set[str]:
|
||||
"""Extracts roles from "[Name]" pattern"""
|
||||
pass
|
||||
```
|
||||
|
||||
#### 3. TTS Service (`services/tts_service.py`)
|
||||
|
||||
**Class: TTSService**
|
||||
|
||||
**Attributes:**
|
||||
- `pipelines: Dict[str, KPipeline]` - Cached pipelines per language
|
||||
- `default_voice: str = "af_bella"`
|
||||
|
||||
**Methods:**
|
||||
```python
|
||||
def __init__(self):
|
||||
"""Initializes TTS service"""
|
||||
pass
|
||||
|
||||
def get_pipeline(self, lang_code: str) -> KPipeline:
|
||||
"""
|
||||
Returns or creates KPipeline for given language.
|
||||
Caches pipelines to avoid reloading.
|
||||
"""
|
||||
pass
|
||||
|
||||
def segment_text(self, text: str, role_mappings: Dict[str, str]) -> List[Tuple[str, str]]:
|
||||
"""
|
||||
Segments text into (role_id, text_content) tuples.
|
||||
Identifies role markers and assigns segments accordingly.
|
||||
"""
|
||||
pass
|
||||
|
||||
def synthesize_segment(self, pipeline: KPipeline, text: str, voice: str, speed: float) -> np.ndarray:
|
||||
"""
|
||||
Synthesizes a single text segment with specified voice.
|
||||
Returns audio samples as numpy array.
|
||||
"""
|
||||
pass
|
||||
|
||||
def concatenate_audio(self, segments: List[np.ndarray]) -> np.ndarray:
|
||||
"""
|
||||
Concatenates multiple audio segments into single array.
|
||||
Ensures consistent sample rate (24kHz).
|
||||
"""
|
||||
pass
|
||||
|
||||
async def generate_audiobook(
|
||||
self,
|
||||
text: str,
|
||||
lang: str,
|
||||
role_voice_map: Dict[str, str],
|
||||
speed: float = 1.0
|
||||
) -> bytes:
|
||||
"""
|
||||
Main orchestration method.
|
||||
Segments text, synthesizes each segment, concatenates, and returns audio bytes.
|
||||
"""
|
||||
pass
|
||||
```
|
||||
|
||||
## Data Models
|
||||
|
||||
### Frontend Data Models
|
||||
|
||||
**Role:**
|
||||
```typescript
|
||||
interface Role {
|
||||
id: string; // Unique identifier (e.g., "Narrator", "Alice")
|
||||
label: string; // Display name for UI
|
||||
}
|
||||
```
|
||||
|
||||
**Voice Option:**
|
||||
```typescript
|
||||
interface VoiceOption {
|
||||
id: string; // Kokoro voice ID (e.g., "af_bella")
|
||||
label: string; // Display name (e.g., "Bella (Female, American)")
|
||||
gender: 'male' | 'female';
|
||||
accent: string; // e.g., "American", "British"
|
||||
}
|
||||
```
|
||||
|
||||
**Language Option:**
|
||||
```typescript
|
||||
interface LanguageOption {
|
||||
code: string; // Kokoro language code (e.g., "a", "b", "f", "j")
|
||||
label: string; // Display name (e.g., "American English")
|
||||
}
|
||||
```
|
||||
|
||||
**Application State:**
|
||||
```typescript
|
||||
interface AppState {
|
||||
phase: 'input' | 'assignment' | 'generated';
|
||||
text: string;
|
||||
roles: Role[];
|
||||
voiceAssignments: Map<string, string>;
|
||||
language: string;
|
||||
speed: number;
|
||||
audioUrl: string | null;
|
||||
isLoading: boolean;
|
||||
error: string | null;
|
||||
}
|
||||
```
|
||||
|
||||
### Backend Data Models
|
||||
|
||||
**Text Segment:**
|
||||
```python
|
||||
@dataclass
|
||||
class TextSegment:
|
||||
role_id: str
|
||||
text: str
|
||||
voice: str
|
||||
```
|
||||
|
||||
**Audio Chunk:**
|
||||
```python
|
||||
@dataclass
|
||||
class AudioChunk:
|
||||
samples: np.ndarray
|
||||
sample_rate: int = 24000
|
||||
```
|
||||
|
||||
### Constants
|
||||
|
||||
**Supported Voices:**
|
||||
```typescript
|
||||
const KOKORO_VOICES: VoiceOption[] = [
|
||||
{ id: 'af_bella', label: 'Bella (Female, American)', gender: 'female', accent: 'American' },
|
||||
{ id: 'af_sarah', label: 'Sarah (Female, American)', gender: 'female', accent: 'American' },
|
||||
{ id: 'am_adam', label: 'Adam (Male, American)', gender: 'male', accent: 'American' },
|
||||
{ id: 'am_michael', label: 'Michael (Male, American)', gender: 'male', accent: 'American' },
|
||||
{ id: 'bf_emma', label: 'Emma (Female, British)', gender: 'female', accent: 'British' },
|
||||
{ id: 'bf_isabella', label: 'Isabella (Female, British)', gender: 'female', accent: 'British' },
|
||||
{ id: 'bm_george', label: 'George (Male, British)', gender: 'male', accent: 'British' },
|
||||
{ id: 'bm_lewis', label: 'Lewis (Male, British)', gender: 'male', accent: 'British' },
|
||||
];
|
||||
```
|
||||
|
||||
**Supported Languages:**
|
||||
```typescript
|
||||
const LANGUAGES: LanguageOption[] = [
|
||||
{ code: 'a', label: 'American English' },
|
||||
{ code: 'b', label: 'British English' },
|
||||
{ code: 'f', label: 'French' },
|
||||
{ code: 'j', label: 'Japanese' },
|
||||
];
|
||||
```
|
||||
|
||||
**Role Detection Patterns:**
|
||||
```python
|
||||
ROLE_PATTERNS = {
|
||||
'colon': r'^([A-Z][a-zA-Z\s]+):\s', # "Alice: Hello there"
|
||||
'bracket': r'\[([A-Z][a-zA-Z\s]+)\]', # "[Narrator] The story begins"
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## Correctness Properties
|
||||
|
||||
*A property is a characteristic or behavior that should hold true across all valid executions of a system—essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
|
||||
|
||||
### Property 1: Text persistence invariant
|
||||
*For any* text entered by the user, the text content should remain unchanged as the application transitions between input, assignment, and generated phases.
|
||||
**Validates: Requirements 1.5**
|
||||
|
||||
### Property 2: Role extraction from colon pattern
|
||||
*For any* text containing lines starting with "RoleName:", the role detection should extract RoleName as a distinct role.
|
||||
**Validates: Requirements 2.2**
|
||||
|
||||
### Property 3: Role extraction from bracket pattern
|
||||
*For any* text containing "[RoleName]" patterns, the role detection should extract RoleName as a distinct role.
|
||||
**Validates: Requirements 2.3**
|
||||
|
||||
### Property 4: Role deduplication
|
||||
*For any* text with multiple instances of the same role marker, the role detection should return each unique role exactly once.
|
||||
**Validates: Requirements 2.5**
|
||||
|
||||
### Property 5: Role response structure
|
||||
*For any* detected roles, each role object in the response should contain both id and label fields.
|
||||
**Validates: Requirements 2.6**
|
||||
|
||||
### Property 6: Voice assignment storage
|
||||
*For any* role and voice selection, the frontend should correctly store and retrieve the role-to-voice mapping.
|
||||
**Validates: Requirements 3.4**
|
||||
|
||||
### Property 7: Generate button disabled when incomplete
|
||||
*For any* state where not all roles have assigned voices, the "Generate Audiobook" button should be disabled.
|
||||
**Validates: Requirements 3.5**
|
||||
|
||||
### Property 8: Generate button enabled when complete
|
||||
*For any* state where all roles have assigned voices, the "Generate Audiobook" button should be enabled.
|
||||
**Validates: Requirements 3.6**
|
||||
|
||||
### Property 9: Language selection storage
|
||||
*For any* language selection, the frontend should store the selected language code for the generation request.
|
||||
**Validates: Requirements 4.4**
|
||||
|
||||
### Property 10: Speed range constraints
|
||||
*For any* speed slider interaction, the value should be constrained between 0.5x and 1.5x.
|
||||
**Validates: Requirements 5.2**
|
||||
|
||||
### Property 11: Speed value storage
|
||||
*For any* speed adjustment, the frontend should store the speed value for the generation request.
|
||||
**Validates: Requirements 5.4**
|
||||
|
||||
### Property 12: Generation request completeness
|
||||
*For any* valid application state, the generation request should contain all required fields: text, language code, role-to-voice mappings, and speed.
|
||||
**Validates: Requirements 6.1**
|
||||
|
||||
### Property 13: Voice assignment per segment
|
||||
*For any* text segment with a role marker, the backend should use the Kokoro voice assigned to that role for synthesis.
|
||||
**Validates: Requirements 6.3**
|
||||
|
||||
### Property 14: Text chunking for long content
|
||||
*For any* long text input, the backend should split it into manageable chunks per paragraph or sentence.
|
||||
**Validates: Requirements 6.5**
|
||||
|
||||
### Property 15: Audio concatenation at 24 kHz
|
||||
*For any* set of synthesized chunks, the backend should concatenate them into a single audio stream at 24 kHz sample rate.
|
||||
**Validates: Requirements 6.6**
|
||||
|
||||
### Property 16: Audio response content type
|
||||
*For any* successful synthesis, the HTTP response should include Content-Type header set to audio/mpeg or audio/wav.
|
||||
**Validates: Requirements 6.7**
|
||||
|
||||
### Property 17: Audio response filename
|
||||
*For any* audio response, the HTTP response should include Content-Disposition header with filename "audiobook.mp3" or "audiobook.wav".
|
||||
**Validates: Requirements 6.8**
|
||||
|
||||
### Property 18: Blob URL creation
|
||||
*For any* audio response from the backend, the frontend should convert it to a Blob and create a valid object URL.
|
||||
**Validates: Requirements 7.1**
|
||||
|
||||
### Property 19: Download functionality
|
||||
*For any* generated audio blob, clicking the download button should trigger file save with the correct filename.
|
||||
**Validates: Requirements 7.5**
|
||||
|
||||
### Property 20: Loading state during role detection
|
||||
*For any* role detection request, the frontend should display a loading state indicator until the response is received.
|
||||
**Validates: Requirements 8.1**
|
||||
|
||||
### Property 21: Loading state during generation
|
||||
*For any* audiobook generation request, the frontend should display loading or progress feedback until completion.
|
||||
**Validates: Requirements 8.2**
|
||||
|
||||
### Property 22: Long text warning
|
||||
*For any* text input exceeding a threshold length, the frontend should display a message indicating "Generation may take several minutes for very long content".
|
||||
**Validates: Requirements 8.3**
|
||||
|
||||
### Property 23: Error message display
|
||||
*For any* failed API request, the frontend should display a clear error message to the user.
|
||||
**Validates: Requirements 8.4**
|
||||
|
||||
### Property 24: Role response structure validation
|
||||
*For any* response from /api/tts/roles, the JSON should contain a roles array with objects having id and label fields.
|
||||
**Validates: Requirements 9.3**
|
||||
|
||||
### Property 25: Request validation
|
||||
*For any* request to /api/tts missing required fields (text, lang, or roles), the backend should reject it with an appropriate error.
|
||||
**Validates: Requirements 9.5**
|
||||
|
||||
### Property 26: Pipeline initialization per language
|
||||
*For any* supported language code (a, b, f, j), the backend should successfully load a KPipeline for that language.
|
||||
**Validates: Requirements 10.3**
|
||||
|
||||
### Property 27: Audio sample rate
|
||||
*For any* synthesized audio, the output should have a sample rate of 24 kHz.
|
||||
**Validates: Requirements 10.4**
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Frontend Error Handling
|
||||
|
||||
**Network Errors:**
|
||||
- Catch fetch errors and display user-friendly messages
|
||||
- Implement retry logic for transient failures (optional)
|
||||
- Show specific error messages for timeout, connection refused, etc.
|
||||
|
||||
**Validation Errors:**
|
||||
- Validate text is non-empty before API calls
|
||||
- Validate all roles have assigned voices before generation
|
||||
- Show inline validation messages near relevant controls
|
||||
|
||||
**API Error Responses:**
|
||||
- Parse error responses from backend (4xx, 5xx)
|
||||
- Display error details to user
|
||||
- Log errors to console for debugging
|
||||
|
||||
**Audio Playback Errors:**
|
||||
- Handle cases where audio blob is invalid
|
||||
- Show error if browser doesn't support audio format
|
||||
- Provide fallback message if playback fails
|
||||
|
||||
### Backend Error Handling
|
||||
|
||||
**Request Validation:**
|
||||
- Validate all required fields are present
|
||||
- Return 400 Bad Request with descriptive error message
|
||||
- Validate language codes and voice IDs are supported
|
||||
|
||||
**TTS Processing Errors:**
|
||||
- Catch exceptions during pipeline initialization
|
||||
- Handle synthesis failures gracefully
|
||||
- Return 500 Internal Server Error with error details
|
||||
- Log errors for debugging
|
||||
|
||||
**Resource Management:**
|
||||
- Handle out-of-memory scenarios for very long text
|
||||
- Implement timeouts for long-running synthesis
|
||||
- Clean up temporary audio files/buffers
|
||||
|
||||
**Model Loading Errors:**
|
||||
- Catch errors when loading Kokoro models
|
||||
- Return clear error if models are missing
|
||||
- Provide helpful error messages for setup issues
|
||||
|
||||
### Error Response Format
|
||||
|
||||
```python
|
||||
class ErrorResponse(BaseModel):
|
||||
error: str
|
||||
message: str
|
||||
details: Optional[dict] = None
|
||||
```
|
||||
|
||||
Example:
|
||||
```json
|
||||
{
|
||||
"error": "ValidationError",
|
||||
"message": "Missing required field: lang",
|
||||
"details": {
|
||||
"field": "lang",
|
||||
"required": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Testing
|
||||
|
||||
**Frontend Unit Tests:**
|
||||
- Test individual components in isolation
|
||||
- Mock API calls using jest/vitest mocks
|
||||
- Test state management logic
|
||||
- Test utility functions (e.g., blob URL creation)
|
||||
- Verify UI elements render correctly
|
||||
- Test form validation logic
|
||||
|
||||
**Backend Unit Tests:**
|
||||
- Test role detection patterns independently
|
||||
- Test text segmentation logic
|
||||
- Test audio concatenation
|
||||
- Test API endpoint request/response handling
|
||||
- Mock Kokoro TTS calls to test logic without actual synthesis
|
||||
|
||||
**Example Unit Tests:**
|
||||
- Empty text validation prevents API call
|
||||
- Health endpoint returns correct status
|
||||
- Role detector extracts "Alice:" pattern correctly
|
||||
- Role detector extracts "[Narrator]" pattern correctly
|
||||
- Default "Narrator" role returned when no patterns found
|
||||
- Voice assignment updates state correctly
|
||||
- Download button triggers file save
|
||||
|
||||
### Property-Based Testing
|
||||
|
||||
Property-based tests will verify universal properties across many randomly generated inputs using a PBT library.
|
||||
|
||||
**Frontend Property Tests (using fast-check for TypeScript):**
|
||||
|
||||
We will use **fast-check** as the property-based testing library for the React TypeScript frontend.
|
||||
|
||||
**Backend Property Tests (using Hypothesis for Python):**
|
||||
|
||||
We will use **Hypothesis** as the property-based testing library for the Python FastAPI backend.
|
||||
|
||||
**Configuration:**
|
||||
- Each property-based test should run a minimum of 100 iterations
|
||||
- Tests should generate diverse inputs to explore edge cases
|
||||
- Each test must include a comment referencing the design property it validates
|
||||
|
||||
**Property Test Examples:**
|
||||
|
||||
*Frontend:*
|
||||
- Property 1: Text persistence - generate random text, verify it persists across phase transitions
|
||||
- Property 6: Voice assignment storage - generate random role/voice pairs, verify storage/retrieval
|
||||
- Property 10: Speed range - generate random speed values, verify clamping to 0.5-1.5 range
|
||||
|
||||
*Backend:*
|
||||
- Property 2: Colon pattern extraction - generate random text with "Name:" patterns, verify extraction
|
||||
- Property 3: Bracket pattern extraction - generate random text with "[Name]" patterns, verify extraction
|
||||
- Property 4: Role deduplication - generate text with duplicate roles, verify uniqueness
|
||||
- Property 14: Text chunking - generate long text, verify chunks are reasonable size
|
||||
- Property 27: Audio sample rate - generate random text, verify output is 24 kHz
|
||||
|
||||
### Integration Testing
|
||||
|
||||
**End-to-End Flow Tests:**
|
||||
- Test complete workflow: text input → role detection → voice assignment → generation → playback
|
||||
- Test with sample texts containing multiple roles
|
||||
- Verify audio file is generated and playable
|
||||
- Test error scenarios (network failures, invalid inputs)
|
||||
|
||||
**API Integration Tests:**
|
||||
- Test all API endpoints with real requests
|
||||
- Verify request/response formats match specifications
|
||||
- Test error responses for invalid inputs
|
||||
- Test with different languages and voices
|
||||
|
||||
### Manual Testing Checklist
|
||||
|
||||
- [ ] Enter text and verify role detection works
|
||||
- [ ] Assign voices to all roles
|
||||
- [ ] Generate audiobook and verify audio plays
|
||||
- [ ] Download audiobook and verify file is valid
|
||||
- [ ] Test with very long text (multiple pages)
|
||||
- [ ] Test with text containing no role markers
|
||||
- [ ] Test with different languages (English, French, Japanese)
|
||||
- [ ] Test on mobile devices for responsiveness
|
||||
- [ ] Test error scenarios (empty text, network failure)
|
||||
- [ ] Verify loading states appear during processing
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Frontend Implementation
|
||||
|
||||
**State Management:**
|
||||
- Use React hooks (useState, useEffect) for local state
|
||||
- Consider useReducer for complex state transitions
|
||||
- Store voice assignments in a Map for efficient lookups
|
||||
|
||||
**API Client:**
|
||||
- Use native fetch API
|
||||
- Implement error handling and timeout logic
|
||||
- Use environment variable for base URL configuration
|
||||
|
||||
**Audio Handling:**
|
||||
- Use Blob API for audio data
|
||||
- Create object URLs with URL.createObjectURL()
|
||||
- Revoke object URLs when component unmounts to prevent memory leaks
|
||||
- Use HTML5 audio element with controls attribute
|
||||
|
||||
**Styling:**
|
||||
- Use Tailwind CSS for responsive design
|
||||
- Ensure mobile-friendly touch targets (min 44x44px)
|
||||
- Use loading spinners for async operations
|
||||
- Implement accessible form controls
|
||||
|
||||
### Backend Implementation
|
||||
|
||||
**Kokoro Integration:**
|
||||
- Initialize KPipeline instances lazily (on first use)
|
||||
- Cache pipelines per language to avoid reloading
|
||||
- Use appropriate voice IDs from Kokoro documentation
|
||||
- Handle model download/loading errors gracefully
|
||||
|
||||
**Text Processing:**
|
||||
- Use regex for role pattern matching
|
||||
- Split text on paragraph boundaries (\n\n) or sentence boundaries
|
||||
- Preserve whitespace and formatting where appropriate
|
||||
- Handle edge cases (empty lines, special characters)
|
||||
|
||||
**Audio Processing:**
|
||||
- Use soundfile library for audio I/O
|
||||
- Concatenate numpy arrays for audio chunks
|
||||
- Ensure consistent sample rate (24 kHz) across all chunks
|
||||
- Convert to MP3 or WAV format for streaming
|
||||
|
||||
**Performance Optimization:**
|
||||
- Consider async processing for long texts
|
||||
- Implement streaming response for large audio files
|
||||
- Use connection pooling for database (if needed)
|
||||
- Monitor memory usage for very long texts
|
||||
|
||||
**Docker Configuration:**
|
||||
- Use multi-stage build to reduce image size
|
||||
- Install system dependencies (espeak-ng) in base layer
|
||||
- Copy only necessary files to container
|
||||
- Use .dockerignore to exclude unnecessary files
|
||||
- Set appropriate environment variables
|
||||
- Use health check endpoint for container orchestration
|
||||
|
||||
### Security Considerations
|
||||
|
||||
**Input Validation:**
|
||||
- Sanitize user input to prevent injection attacks
|
||||
- Limit text length to prevent DoS (e.g., max 1MB)
|
||||
- Validate language codes and voice IDs against whitelist
|
||||
- Rate limit API endpoints to prevent abuse
|
||||
|
||||
**CORS Configuration:**
|
||||
- Configure CORS to allow frontend origin
|
||||
- Restrict allowed methods and headers
|
||||
- Don't expose sensitive headers
|
||||
|
||||
**Error Messages:**
|
||||
- Don't expose internal implementation details in errors
|
||||
- Log detailed errors server-side only
|
||||
- Return generic error messages to client
|
||||
|
||||
### Deployment Considerations
|
||||
|
||||
**Environment Variables:**
|
||||
- Frontend: VITE_API_BASE_URL
|
||||
- Backend: PORT, MODEL_PATH (optional), LOG_LEVEL
|
||||
|
||||
**Resource Requirements:**
|
||||
- Backend: Minimum 2GB RAM for model loading
|
||||
- Storage: ~500MB for Kokoro models
|
||||
- CPU: Multi-core recommended for faster synthesis
|
||||
|
||||
**Monitoring:**
|
||||
- Log API requests and response times
|
||||
- Monitor memory usage during synthesis
|
||||
- Track error rates and types
|
||||
- Set up health check monitoring
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
**Potential Improvements:**
|
||||
- Add support for more languages and voices
|
||||
- Implement progress tracking for long generations
|
||||
- Add ability to preview individual role voices
|
||||
- Support for SSML markup for advanced control
|
||||
- Batch processing for multiple texts
|
||||
- User accounts and saved voice preferences
|
||||
- Audio editing (trim, adjust volume per role)
|
||||
- Export to different audio formats (FLAC, OGG)
|
||||
- Background processing with job queue
|
||||
- WebSocket for real-time progress updates
|
||||
Reference in New Issue
Block a user