Compare commits
24 Commits
e2196c592a
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 4e627aed12 | |||
| b1ce561dc4 | |||
| 36fb70a717 | |||
| cdd8c0fc08 | |||
| 134b109999 | |||
| d3e72edd7f | |||
| 6f7330c491 | |||
| c310f11484 | |||
| bfdcc94b24 | |||
| b5c53604a6 | |||
| 69eb0b3507 | |||
| a8636bcf71 | |||
| 55f630a479 | |||
| a72cee105e | |||
| 4a53c897c5 | |||
| 2f200aa61b | |||
| fb294252c7 | |||
| 706ada866c | |||
| 63bdf57593 | |||
| 4861c343fc | |||
| 92094479c1 | |||
| bc768f4f45 | |||
| d737b1f7ea | |||
| cea97d3cd4 |
154
.kiro/specs/text-to-audiobook/SCRIPT_REVIEW_FEATURE.md
Normal file
154
.kiro/specs/text-to-audiobook/SCRIPT_REVIEW_FEATURE.md
Normal file
@@ -0,0 +1,154 @@
|
|||||||
|
# Script Review Feature - Enhancement Specification
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This document describes the new **Script Review & Editing** feature that addresses the issue where AI-generated speaker assignments may not always be accurate. This feature gives users visibility and control over how the text is segmented and which character speaks each line.
|
||||||
|
|
||||||
|
## Problem Statement
|
||||||
|
|
||||||
|
When using AI to detect speakers in unmarked text, the system may:
|
||||||
|
- Incorrectly assign dialogue to the wrong character
|
||||||
|
- Miss some character dialogue
|
||||||
|
- Assign narrative text to a character instead of the narrator
|
||||||
|
|
||||||
|
Users currently have no way to see or correct these mistakes before generating the audiobook.
|
||||||
|
|
||||||
|
## Solution
|
||||||
|
|
||||||
|
Add a new **Script Review** phase between voice assignment and audio generation where users can:
|
||||||
|
1. See the complete script breakdown showing which character speaks each line
|
||||||
|
2. Edit speaker assignments for any segment
|
||||||
|
3. Skip the review if they trust the AI's assignments
|
||||||
|
|
||||||
|
## User Workflow
|
||||||
|
|
||||||
|
### Updated Flow
|
||||||
|
```
|
||||||
|
1. Text Input
|
||||||
|
↓
|
||||||
|
2. Role Detection (AI detects: Lily, Benny, Sammy, Narrator)
|
||||||
|
↓
|
||||||
|
3. Voice Assignment (User assigns voices to each role)
|
||||||
|
↓
|
||||||
|
4. **Script Review** (NEW - User reviews and edits speaker assignments)
|
||||||
|
↓
|
||||||
|
5. Audio Generation (Uses edited script)
|
||||||
|
↓
|
||||||
|
6. Playback & Download
|
||||||
|
```
|
||||||
|
|
||||||
|
### Script Review Interface
|
||||||
|
|
||||||
|
The review screen will show:
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ Review Script │
|
||||||
|
├─────────────────────────────────────────────────────────────┤
|
||||||
|
│ │
|
||||||
|
│ Segment 1: [Narrator ▼] │
|
||||||
|
│ ┌────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Once upon a time, in a tiny village surrounded by │ │
|
||||||
|
│ │ lush green forests, there lived three best friends... │ │
|
||||||
|
│ └────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ Segment 2: [Lily ▼] │
|
||||||
|
│ ┌────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ "Have you guys heard about the new stream that just │ │
|
||||||
|
│ │ appeared in the woods?" │ │
|
||||||
|
│ └────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ Segment 3: [Benny ▼] │
|
||||||
|
│ ┌────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ "Ooh, I love streams! Do we get to go see it now?" │ │
|
||||||
|
│ └────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ... (more segments) │
|
||||||
|
│ │
|
||||||
|
│ [Skip Review] [Generate Audiobook] │
|
||||||
|
└─────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Technical Implementation
|
||||||
|
|
||||||
|
### New API Endpoint
|
||||||
|
|
||||||
|
**POST /api/tts/script**
|
||||||
|
- Request: `{ text: string, roles: string[] }`
|
||||||
|
- Response: `{ segments: [{ id: string, speaker: string, text: string }] }`
|
||||||
|
- Uses Ollama AI to segment text and assign speakers
|
||||||
|
- Returns editable script for frontend
|
||||||
|
|
||||||
|
### Updated Generation Endpoint
|
||||||
|
|
||||||
|
**POST /api/tts** (modified)
|
||||||
|
- Now accepts optional `segments` array
|
||||||
|
- If segments provided, uses them directly (respecting user edits)
|
||||||
|
- If no segments, falls back to original text-based segmentation
|
||||||
|
- Maintains backward compatibility
|
||||||
|
|
||||||
|
### Frontend Components
|
||||||
|
|
||||||
|
**New: ScriptReviewSection**
|
||||||
|
- Displays all segments with editable speaker dropdowns
|
||||||
|
- Allows users to change speaker for any segment
|
||||||
|
- Provides "Skip Review" option for quick generation
|
||||||
|
- Mobile-responsive design
|
||||||
|
|
||||||
|
**Updated: VoiceAssignmentSection**
|
||||||
|
- Button changes from "Generate Audiobook" to "Next: Review Script"
|
||||||
|
- Adds "Skip Review & Generate" option
|
||||||
|
|
||||||
|
**Updated: App Component**
|
||||||
|
- Adds 'review' phase to workflow
|
||||||
|
- Manages script segments state
|
||||||
|
- Handles segment editing
|
||||||
|
|
||||||
|
## Benefits
|
||||||
|
|
||||||
|
1. **Transparency**: Users can see exactly how the AI interpreted their text
|
||||||
|
2. **Control**: Users can correct any mistakes before generation
|
||||||
|
3. **Flexibility**: Users can skip review if they trust the AI
|
||||||
|
4. **Better Results**: Ensures the final audiobook matches user expectations
|
||||||
|
|
||||||
|
## Requirements Added
|
||||||
|
|
||||||
|
- **Requirement 15**: Complete specification for script review feature
|
||||||
|
- 8 acceptance criteria covering all aspects
|
||||||
|
- API endpoints, UI components, and user interactions
|
||||||
|
|
||||||
|
## Design Updates
|
||||||
|
|
||||||
|
- Added ScriptReviewSection component
|
||||||
|
- Added /api/tts/script endpoint
|
||||||
|
- Updated data models (ScriptSegment, ScriptRequest, ScriptResponse)
|
||||||
|
- Added 5 new correctness properties (Properties 32-35)
|
||||||
|
- Updated application workflow to include review phase
|
||||||
|
|
||||||
|
## Implementation Tasks
|
||||||
|
|
||||||
|
Added 8 new tasks (18-25) covering:
|
||||||
|
- Backend script generation endpoint
|
||||||
|
- Updated TTS generation to accept edited scripts
|
||||||
|
- Frontend API client updates
|
||||||
|
- ScriptReviewSection component
|
||||||
|
- App component workflow updates
|
||||||
|
- VoiceAssignmentSection updates
|
||||||
|
- Documentation
|
||||||
|
- Integration testing
|
||||||
|
|
||||||
|
## Testing Strategy
|
||||||
|
|
||||||
|
- Property tests for script completeness and structure
|
||||||
|
- Unit tests for all new components and endpoints
|
||||||
|
- Integration tests for complete workflow
|
||||||
|
- Manual testing with various text samples
|
||||||
|
|
||||||
|
## Future Enhancements
|
||||||
|
|
||||||
|
Potential improvements for later:
|
||||||
|
- Inline editing of segment text
|
||||||
|
- Merge/split segments
|
||||||
|
- Bulk speaker assignment
|
||||||
|
- Save/load script templates
|
||||||
|
- Export script as text file
|
||||||
@@ -4,10 +4,11 @@
|
|||||||
|
|
||||||
The Text to Audiobook system is a full-stack application that converts long-form text into multi-voice audiobooks using Kokoro TTS. The architecture follows a client-server model with a React TypeScript frontend and a Python FastAPI backend running in Docker. The system automatically detects speaker roles in text, allows voice assignment per role, and generates high-quality audio using the Kokoro-82M model.
|
The Text to Audiobook system is a full-stack application that converts long-form text into multi-voice audiobooks using Kokoro TTS. The architecture follows a client-server model with a React TypeScript frontend and a Python FastAPI backend running in Docker. The system automatically detects speaker roles in text, allows voice assignment per role, and generates high-quality audio using the Kokoro-82M model.
|
||||||
|
|
||||||
The application workflow consists of three main phases:
|
The application workflow consists of four main phases:
|
||||||
1. **Text Input & Role Detection**: User provides text, system identifies distinct speakers
|
1. **Text Input & Role Detection**: User provides text, system identifies distinct speakers
|
||||||
2. **Voice Assignment**: User maps each detected role to a specific Kokoro voice
|
2. **Voice Assignment**: User maps each detected role to a specific Kokoro voice
|
||||||
3. **Audio Generation & Playback**: System synthesizes multi-voice audio and provides playback/download
|
3. **Script Review & Editing**: User reviews and edits AI-generated speaker assignments for each text segment
|
||||||
|
4. **Audio Generation & Playback**: System synthesizes multi-voice audio and provides playback/download
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
@@ -93,10 +94,12 @@ Main application container managing the overall workflow state.
|
|||||||
- `text: string` - User input text
|
- `text: string` - User input text
|
||||||
- `roles: Role[]` - Detected roles from backend
|
- `roles: Role[]` - Detected roles from backend
|
||||||
- `voiceAssignments: Map<string, string>` - Role ID to Voice ID mapping
|
- `voiceAssignments: Map<string, string>` - Role ID to Voice ID mapping
|
||||||
|
- `scriptSegments: ScriptSegment[]` - AI-generated script segments
|
||||||
- `selectedLanguage: string` - Language code (default: 'a')
|
- `selectedLanguage: string` - Language code (default: 'a')
|
||||||
- `speed: number` - Playback speed (default: 1.0)
|
- `speed: number` - Playback speed (default: 1.0)
|
||||||
- `audioUrl: string | null` - Generated audio blob URL
|
- `audioUrl: string | null` - Generated audio blob URL
|
||||||
- `isDetectingRoles: boolean` - Loading state for role detection
|
- `isDetectingRoles: boolean` - Loading state for role detection
|
||||||
|
- `isGeneratingScript: boolean` - Loading state for script generation
|
||||||
- `isGenerating: boolean` - Loading state for audio generation
|
- `isGenerating: boolean` - Loading state for audio generation
|
||||||
- `error: string | null` - Error message if any
|
- `error: string | null` - Error message if any
|
||||||
|
|
||||||
@@ -104,7 +107,10 @@ Main application container managing the overall workflow state.
|
|||||||
- `handleTextChange(text: string)` - Updates text state
|
- `handleTextChange(text: string)` - Updates text state
|
||||||
- `handleDetectRoles()` - Calls API to detect roles
|
- `handleDetectRoles()` - Calls API to detect roles
|
||||||
- `handleVoiceAssignment(roleId: string, voiceId: string)` - Updates voice mapping
|
- `handleVoiceAssignment(roleId: string, voiceId: string)` - Updates voice mapping
|
||||||
- `handleGenerate()` - Calls API to generate audiobook
|
- `handleGenerateScript()` - Calls API to generate script preview
|
||||||
|
- `handleSkipReview()` - Skips script review and proceeds to generation
|
||||||
|
- `handleSegmentChange(segmentId: string, newSpeaker: string)` - Updates speaker for a segment
|
||||||
|
- `handleGenerate()` - Calls API to generate audiobook with edited script
|
||||||
- `handleDownload()` - Triggers download of generated audio
|
- `handleDownload()` - Triggers download of generated audio
|
||||||
|
|
||||||
#### 2. TextInputSection Component
|
#### 2. TextInputSection Component
|
||||||
@@ -132,14 +138,25 @@ Displays role-to-voice mapping interface with language and speed controls.
|
|||||||
- `isGenerating: boolean`
|
- `isGenerating: boolean`
|
||||||
- `allRolesAssigned: boolean`
|
- `allRolesAssigned: boolean`
|
||||||
|
|
||||||
#### 4. AudioPlayerSection Component
|
#### 4. ScriptReviewSection Component
|
||||||
|
Displays AI-generated script with editable speaker assignments.
|
||||||
|
|
||||||
|
**Props:**
|
||||||
|
- `segments: ScriptSegment[]`
|
||||||
|
- `roles: Role[]`
|
||||||
|
- `onSegmentChange: (segmentId: string, newSpeaker: string) => void`
|
||||||
|
- `onGenerate: () => void`
|
||||||
|
- `onSkipReview: () => void`
|
||||||
|
- `isGenerating: boolean`
|
||||||
|
|
||||||
|
#### 5. AudioPlayerSection Component
|
||||||
Displays audio player and download button after generation.
|
Displays audio player and download button after generation.
|
||||||
|
|
||||||
**Props:**
|
**Props:**
|
||||||
- `audioUrl: string`
|
- `audioUrl: string`
|
||||||
- `onDownload: () => void`
|
- `onDownload: () => void`
|
||||||
|
|
||||||
#### 5. API Client (`src/api/tts.ts`)
|
#### 6. API Client (`src/api/tts.ts`)
|
||||||
|
|
||||||
**Interfaces:**
|
**Interfaces:**
|
||||||
```typescript
|
```typescript
|
||||||
@@ -156,8 +173,24 @@ export interface RolesResponse {
|
|||||||
roles: Role[];
|
roles: Role[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface GenerateRequest {
|
export interface ScriptSegment {
|
||||||
|
id: string;
|
||||||
|
speaker: string;
|
||||||
text: string;
|
text: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ScriptRequest {
|
||||||
|
text: string;
|
||||||
|
roles: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ScriptResponse {
|
||||||
|
segments: ScriptSegment[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface GenerateRequest {
|
||||||
|
segments?: ScriptSegment[];
|
||||||
|
text?: string;
|
||||||
lang: string;
|
lang: string;
|
||||||
roles: { id: string; voice: string }[];
|
roles: { id: string; voice: string }[];
|
||||||
speed?: number;
|
speed?: number;
|
||||||
@@ -167,6 +200,7 @@ export interface GenerateRequest {
|
|||||||
**Functions:**
|
**Functions:**
|
||||||
```typescript
|
```typescript
|
||||||
export async function detectRoles(text: string): Promise<RolesResponse>
|
export async function detectRoles(text: string): Promise<RolesResponse>
|
||||||
|
export async function generateScript(request: ScriptRequest): Promise<ScriptResponse>
|
||||||
export async function generateAudiobook(payload: GenerateRequest): Promise<Blob>
|
export async function generateAudiobook(payload: GenerateRequest): Promise<Blob>
|
||||||
export async function checkHealth(): Promise<{ status: string }>
|
export async function checkHealth(): Promise<{ status: string }>
|
||||||
```
|
```
|
||||||
@@ -188,6 +222,11 @@ async def detect_roles(request: RoleDetectionRequest) -> RolesResponse:
|
|||||||
"""Analyzes text and returns detected roles"""
|
"""Analyzes text and returns detected roles"""
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
@app.post("/api/tts/script")
|
||||||
|
async def generate_script(request: ScriptRequest) -> ScriptResponse:
|
||||||
|
"""Generates script preview with AI-segmented text and speaker assignments"""
|
||||||
|
pass
|
||||||
|
|
||||||
@app.post("/api/tts")
|
@app.post("/api/tts")
|
||||||
async def generate_audiobook(request: TTSRequest) -> StreamingResponse:
|
async def generate_audiobook(request: TTSRequest) -> StreamingResponse:
|
||||||
"""Generates multi-voice audiobook and streams audio"""
|
"""Generates multi-voice audiobook and streams audio"""
|
||||||
@@ -206,12 +245,25 @@ class RoleInfo(BaseModel):
|
|||||||
class RolesResponse(BaseModel):
|
class RolesResponse(BaseModel):
|
||||||
roles: List[RoleInfo]
|
roles: List[RoleInfo]
|
||||||
|
|
||||||
|
class ScriptRequest(BaseModel):
|
||||||
|
text: str
|
||||||
|
roles: List[str] # List of detected role IDs
|
||||||
|
|
||||||
|
class ScriptSegment(BaseModel):
|
||||||
|
id: str # Unique segment identifier
|
||||||
|
speaker: str # Assigned speaker/role
|
||||||
|
text: str # Text content for this segment
|
||||||
|
|
||||||
|
class ScriptResponse(BaseModel):
|
||||||
|
segments: List[ScriptSegment]
|
||||||
|
|
||||||
class RoleVoiceMapping(BaseModel):
|
class RoleVoiceMapping(BaseModel):
|
||||||
id: str
|
id: str
|
||||||
voice: str
|
voice: str
|
||||||
|
|
||||||
class TTSRequest(BaseModel):
|
class TTSRequest(BaseModel):
|
||||||
text: str
|
segments: Optional[List[ScriptSegment]] = None # If provided, use edited script
|
||||||
|
text: Optional[str] = None # Fallback if no script provided
|
||||||
lang: str
|
lang: str
|
||||||
roles: List[RoleVoiceMapping]
|
roles: List[RoleVoiceMapping]
|
||||||
speed: Optional[float] = 1.0
|
speed: Optional[float] = 1.0
|
||||||
@@ -384,10 +436,11 @@ interface LanguageOption {
|
|||||||
**Application State:**
|
**Application State:**
|
||||||
```typescript
|
```typescript
|
||||||
interface AppState {
|
interface AppState {
|
||||||
phase: 'input' | 'assignment' | 'generated';
|
phase: 'input' | 'assignment' | 'review' | 'generated';
|
||||||
text: string;
|
text: string;
|
||||||
roles: Role[];
|
roles: Role[];
|
||||||
voiceAssignments: Map<string, string>;
|
voiceAssignments: Map<string, string>;
|
||||||
|
scriptSegments: ScriptSegment[];
|
||||||
language: string;
|
language: string;
|
||||||
speed: number;
|
speed: number;
|
||||||
audioUrl: string | null;
|
audioUrl: string | null;
|
||||||
@@ -578,6 +631,22 @@ ROLE_PATTERNS = {
|
|||||||
*For any* text where both AI detection and pattern detection find roles, the backend should merge results and return each unique role exactly once.
|
*For any* text where both AI detection and pattern detection find roles, the backend should merge results and return each unique role exactly once.
|
||||||
**Validates: Requirements 14.5**
|
**Validates: Requirements 14.5**
|
||||||
|
|
||||||
|
### Property 32: Script generation completeness
|
||||||
|
*For any* text and list of roles, the script generation should return segments that cover the entire input text without gaps or overlaps.
|
||||||
|
**Validates: Requirements 15.2**
|
||||||
|
|
||||||
|
### Property 33: Script segment structure
|
||||||
|
*For any* generated script, each segment should contain an id, speaker, and text field.
|
||||||
|
**Validates: Requirements 15.4**
|
||||||
|
|
||||||
|
### Property 34: Script segment editability
|
||||||
|
*For any* script segment, the user should be able to change the speaker assignment to any detected role.
|
||||||
|
**Validates: Requirements 15.5**
|
||||||
|
|
||||||
|
### Property 35: Edited script persistence
|
||||||
|
*For any* edited script segment, the changes should be preserved when generating the audiobook.
|
||||||
|
**Validates: Requirements 15.7**
|
||||||
|
|
||||||
## Error Handling
|
## Error Handling
|
||||||
|
|
||||||
### Frontend Error Handling
|
### Frontend Error Handling
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ The Text to Audiobook system is a production-ready React TypeScript single-page
|
|||||||
- **KPipeline**: The Kokoro library's pipeline object for text-to-speech synthesis
|
- **KPipeline**: The Kokoro library's pipeline object for text-to-speech synthesis
|
||||||
- **Segment**: A portion of text assigned to a specific role for synthesis
|
- **Segment**: A portion of text assigned to a specific role for synthesis
|
||||||
- **Role Marker**: Text patterns that identify speakers (e.g., "Alice:", "[Narrator]")
|
- **Role Marker**: Text patterns that identify speakers (e.g., "Alice:", "[Narrator]")
|
||||||
|
- **Script**: The complete breakdown of text into segments with speaker assignments, displayed for user review and editing
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
@@ -192,3 +193,18 @@ The Text to Audiobook system is a production-ready React TypeScript single-page
|
|||||||
4. WHEN Ollama is unavailable or fails THEN the Backend SHALL fall back to pattern-based role detection
|
4. WHEN Ollama is unavailable or fails THEN the Backend SHALL fall back to pattern-based role detection
|
||||||
5. WHEN both AI detection and pattern detection find roles THEN the Backend SHALL merge and deduplicate the results
|
5. WHEN both AI detection and pattern detection find roles THEN the Backend SHALL merge and deduplicate the results
|
||||||
6. THE Backend SHALL configure Ollama connection using environment variable OLLAMA_BASE_URL with default http://localhost:11434
|
6. THE Backend SHALL configure Ollama connection using environment variable OLLAMA_BASE_URL with default http://localhost:11434
|
||||||
|
|
||||||
|
### Requirement 15
|
||||||
|
|
||||||
|
**User Story:** As a user, I want to review and edit the AI-generated script showing which character speaks each line, so that I can correct any mistakes before generating the audiobook.
|
||||||
|
|
||||||
|
#### Acceptance Criteria
|
||||||
|
|
||||||
|
1. WHEN the user completes voice assignment THEN the Frontend SHALL request a script preview from the Backend before generation
|
||||||
|
2. WHEN the Backend receives a script preview request THEN the Backend SHALL use AI to segment the text and return a list of segments with speaker assignments
|
||||||
|
3. WHEN the script preview is received THEN the Frontend SHALL display a new "Review Script" phase showing each segment with its assigned speaker
|
||||||
|
4. WHEN displaying script segments THEN the Frontend SHALL show each segment in a text box with a dropdown to select the speaker
|
||||||
|
5. WHEN the user changes a speaker assignment for a segment THEN the Frontend SHALL update the segment mapping
|
||||||
|
6. WHEN the user is satisfied with the script THEN the Frontend SHALL enable a "Generate Audiobook" button
|
||||||
|
7. WHEN the user clicks "Generate Audiobook" THEN the Frontend SHALL send the edited script segments to the Backend for synthesis
|
||||||
|
8. THE Frontend SHALL allow users to skip the review step with a "Skip Review" option that proceeds directly to generation
|
||||||
|
|||||||
@@ -372,7 +372,7 @@
|
|||||||
- Test error handling
|
- Test error handling
|
||||||
- _Requirements: 15.2_
|
- _Requirements: 15.2_
|
||||||
|
|
||||||
- [ ] 19. Update TTS generation endpoint to accept edited scripts
|
- [x] 19. Update TTS generation endpoint to accept edited scripts
|
||||||
- Modify TTSRequest to accept optional segments array
|
- Modify TTSRequest to accept optional segments array
|
||||||
- Update generate_audiobook logic to use segments if provided
|
- Update generate_audiobook logic to use segments if provided
|
||||||
- Fall back to text segmentation if no segments provided
|
- Fall back to text segmentation if no segments provided
|
||||||
@@ -389,7 +389,7 @@
|
|||||||
- Test segment speaker assignments are respected
|
- Test segment speaker assignments are respected
|
||||||
- _Requirements: 15.7_
|
- _Requirements: 15.7_
|
||||||
|
|
||||||
- [ ] 20. Implement script generation API client function
|
- [x] 20. Implement script generation API client function
|
||||||
- Add generateScript() function to src/api/tts.ts
|
- Add generateScript() function to src/api/tts.ts
|
||||||
- Define ScriptSegment, ScriptRequest, and ScriptResponse TypeScript interfaces
|
- Define ScriptSegment, ScriptRequest, and ScriptResponse TypeScript interfaces
|
||||||
- Implement error handling for script generation
|
- Implement error handling for script generation
|
||||||
@@ -402,7 +402,7 @@
|
|||||||
- Mock API responses
|
- Mock API responses
|
||||||
- _Requirements: 15.1_
|
- _Requirements: 15.1_
|
||||||
|
|
||||||
- [ ] 21. Implement ScriptReviewSection component
|
- [x] 21. Implement ScriptReviewSection component
|
||||||
- Create ScriptReviewSection component
|
- Create ScriptReviewSection component
|
||||||
- Display list of script segments with text boxes
|
- Display list of script segments with text boxes
|
||||||
- Add speaker dropdown for each segment
|
- Add speaker dropdown for each segment
|
||||||
@@ -424,7 +424,7 @@
|
|||||||
- Test skip review button works
|
- Test skip review button works
|
||||||
- _Requirements: 15.3, 15.4, 15.5, 15.8_
|
- _Requirements: 15.3, 15.4, 15.5, 15.8_
|
||||||
|
|
||||||
- [ ] 22. Update App component for script review workflow
|
- [x] 22. Update App component for script review workflow
|
||||||
- Add 'review' phase to application state
|
- Add 'review' phase to application state
|
||||||
- Add scriptSegments state variable
|
- Add scriptSegments state variable
|
||||||
- Add isGeneratingScript loading state
|
- Add isGeneratingScript loading state
|
||||||
@@ -443,7 +443,7 @@
|
|||||||
- Test skip review bypasses review phase
|
- Test skip review bypasses review phase
|
||||||
- _Requirements: 15.1, 15.8_
|
- _Requirements: 15.1, 15.8_
|
||||||
|
|
||||||
- [ ] 23. Update VoiceAssignmentSection to trigger script generation
|
- [x] 23. Update VoiceAssignmentSection to trigger script generation
|
||||||
- Change "Generate Audiobook" button to "Next: Review Script"
|
- Change "Generate Audiobook" button to "Next: Review Script"
|
||||||
- Add "Skip Review & Generate" button as alternative
|
- Add "Skip Review & Generate" button as alternative
|
||||||
- Update button handlers to call appropriate methods
|
- Update button handlers to call appropriate methods
|
||||||
|
|||||||
198
PIPER_FINAL_SUMMARY.md
Normal file
198
PIPER_FINAL_SUMMARY.md
Normal file
@@ -0,0 +1,198 @@
|
|||||||
|
# Piper TTS Integration - Final Summary
|
||||||
|
|
||||||
|
## ✅ Integration Complete
|
||||||
|
|
||||||
|
Piper TTS has been successfully integrated as an alternative TTS engine with full UI support and comprehensive error handling.
|
||||||
|
|
||||||
|
## What Was Delivered
|
||||||
|
|
||||||
|
### Backend (100% Complete)
|
||||||
|
- ✅ **PiperTTSService** - Full synthesis with pitch control (-12 to +12 semitones)
|
||||||
|
- ✅ **UnifiedTTSService** - Multi-engine management (Kokoro + Piper)
|
||||||
|
- ✅ **API Endpoints** - Engine selection, voice listing, preview support
|
||||||
|
- ✅ **Error Handling** - Proper 404 responses for missing models
|
||||||
|
- ✅ **Logging** - Detailed error messages and setup instructions
|
||||||
|
|
||||||
|
### Frontend (100% Complete)
|
||||||
|
- ✅ **Engine Selector UI** - Toggle buttons with descriptions
|
||||||
|
- ✅ **Dynamic Voice Loading** - Voices update per engine
|
||||||
|
- ✅ **Warning Messages** - Clear setup instructions when Piper selected
|
||||||
|
- ✅ **Error Alerts** - User-friendly messages when preview fails
|
||||||
|
- ✅ **Visual Feedback** - Active engine highlighting
|
||||||
|
|
||||||
|
### Documentation (100% Complete)
|
||||||
|
- ✅ **PIPER_SETUP.md** - Step-by-step manual setup guide
|
||||||
|
- ✅ **PIPER_TTS_GUIDE.md** - Comprehensive usage documentation
|
||||||
|
- ✅ **PIPER_STATUS.md** - Current state and troubleshooting
|
||||||
|
- ✅ **PIPER_FINAL_SUMMARY.md** - This document
|
||||||
|
- ✅ **Updated README.md** - Multi-engine documentation
|
||||||
|
|
||||||
|
### Testing (100% Complete)
|
||||||
|
- ✅ **test_engine_selection.py** - Engine switching tests
|
||||||
|
- ✅ **test_piper_direct.py** - Piper installation verification
|
||||||
|
|
||||||
|
## User Experience
|
||||||
|
|
||||||
|
### Option 1: Kokoro TTS (Default) - Zero Setup
|
||||||
|
```
|
||||||
|
✅ Works immediately out of the box
|
||||||
|
✅ 54 voices across 9 languages
|
||||||
|
✅ High quality neural synthesis
|
||||||
|
❌ No pitch control
|
||||||
|
```
|
||||||
|
|
||||||
|
### Option 2: Piper TTS (Advanced) - Manual Setup Required
|
||||||
|
```
|
||||||
|
⚠️ Requires one-time voice model download
|
||||||
|
✅ Pitch control per character (-12 to +12 semitones)
|
||||||
|
✅ Fast synthesis
|
||||||
|
✅ Lower resource usage
|
||||||
|
✅ 5+ English voices (expandable)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Setup Instructions for Piper
|
||||||
|
|
||||||
|
### Quick Start (3 Steps)
|
||||||
|
|
||||||
|
1. **Install Piper Binary**
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
./install_piper.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Download Voice Models**
|
||||||
|
- Visit: https://github.com/rhasspy/piper/releases/tag/v1.2.0
|
||||||
|
- Download: `en_US-lessac-medium.onnx` and `.onnx.json`
|
||||||
|
- Place in: `backend/piper_models/`
|
||||||
|
|
||||||
|
3. **Test & Use**
|
||||||
|
```bash
|
||||||
|
python test_piper_direct.py
|
||||||
|
```
|
||||||
|
|
||||||
|
See `backend/PIPER_SETUP.md` for detailed instructions.
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Backend Errors
|
||||||
|
- **404 VoiceModelNotFound** - Clear message with setup guide link
|
||||||
|
- **400 ValidationError** - Helpful validation messages
|
||||||
|
- **500 PreviewGenerationError** - Detailed error logging
|
||||||
|
|
||||||
|
### Frontend Errors
|
||||||
|
- **Alert on Preview Fail** - User-friendly error with setup instructions
|
||||||
|
- **Warning Banner** - Prominent notice when Piper selected
|
||||||
|
- **Empty Audio Check** - Prevents "media could not be decoded" errors
|
||||||
|
|
||||||
|
## Git Commits (All Pushed)
|
||||||
|
|
||||||
|
1. `bfdcc94` - Backend Piper TTS integration with pitch control
|
||||||
|
2. `c310f11` - Frontend engine selector UI
|
||||||
|
3. `6f7330c` - Comprehensive integration documentation
|
||||||
|
4. `d3e72ed` - Preview fix and improved setup instructions
|
||||||
|
5. `134b109` - Status documentation and troubleshooting
|
||||||
|
6. `cdd8c0f` - Enhanced error handling and user messaging
|
||||||
|
|
||||||
|
## Files Created/Modified
|
||||||
|
|
||||||
|
### New Files (11)
|
||||||
|
```
|
||||||
|
backend/services/piper_tts_service.py
|
||||||
|
backend/services/unified_tts_service.py
|
||||||
|
backend/install_piper.sh
|
||||||
|
backend/test_engine_selection.py
|
||||||
|
backend/test_piper_direct.py
|
||||||
|
backend/PIPER_SETUP.md
|
||||||
|
backend/PIPER_TTS_GUIDE.md
|
||||||
|
PIPER_TTS_INTEGRATION.md
|
||||||
|
PIPER_STATUS.md
|
||||||
|
PIPER_FINAL_SUMMARY.md
|
||||||
|
backend/piper_models/ (directory)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Modified Files (5)
|
||||||
|
```
|
||||||
|
backend/main.py (added 3 endpoints, updated preview)
|
||||||
|
backend/requirements.txt (added piper-tts, pydub)
|
||||||
|
backend/.env.example (added TTS_ENGINE)
|
||||||
|
backend/README.md (multi-engine docs)
|
||||||
|
frontend/src/api/tts.ts (engine functions)
|
||||||
|
frontend/src/components/VoiceAssignmentSection.tsx (engine selector)
|
||||||
|
.kiro/specs/text-to-audiobook/tasks.md (task 26)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Why Manual Setup?
|
||||||
|
|
||||||
|
**Technical Reasons:**
|
||||||
|
- HuggingFace repository structure complexity
|
||||||
|
- SSL certificate verification issues
|
||||||
|
- Large file sizes (10-50MB per voice)
|
||||||
|
- Cross-platform reliability concerns
|
||||||
|
|
||||||
|
**Decision:** Manual setup is more reliable than buggy auto-download
|
||||||
|
|
||||||
|
## Comparison with Kokoro
|
||||||
|
|
||||||
|
| Aspect | Kokoro TTS | Piper TTS |
|
||||||
|
|--------|------------|-----------|
|
||||||
|
| **Setup** | ✅ Zero setup | ⚠️ Manual download |
|
||||||
|
| **Quality** | ⭐⭐⭐⭐⭐ High | ⭐⭐⭐⭐ Medium-High |
|
||||||
|
| **Speed** | ⭐⭐⭐ Medium | ⭐⭐⭐⭐⭐ Fast |
|
||||||
|
| **Voices** | 54 voices, 9 languages | 5+ English (expandable) |
|
||||||
|
| **Pitch Control** | ❌ No | ✅ Yes (-12 to +12) |
|
||||||
|
| **Memory** | ~500MB models | ~10-50MB per voice |
|
||||||
|
| **Use Case** | General purpose | Advanced users |
|
||||||
|
|
||||||
|
## Recommendation
|
||||||
|
|
||||||
|
**For Most Users:** Use Kokoro TTS (default)
|
||||||
|
- Works immediately
|
||||||
|
- High quality
|
||||||
|
- Many voices and languages
|
||||||
|
|
||||||
|
**For Power Users:** Use Piper TTS
|
||||||
|
- Worth the setup for pitch control
|
||||||
|
- Faster synthesis
|
||||||
|
- Lower resource usage
|
||||||
|
- Great for character differentiation
|
||||||
|
|
||||||
|
## Testing Checklist
|
||||||
|
|
||||||
|
- ✅ Engine switching works (Kokoro ↔ Piper)
|
||||||
|
- ✅ Voice lists update per engine
|
||||||
|
- ✅ Preview works with Kokoro voices
|
||||||
|
- ✅ Preview shows helpful error for missing Piper models
|
||||||
|
- ✅ Warning banner displays when Piper selected
|
||||||
|
- ✅ Error messages guide users to setup documentation
|
||||||
|
- ✅ Backend logs detailed error information
|
||||||
|
- ✅ API endpoints return proper status codes
|
||||||
|
- ✅ Documentation is comprehensive and clear
|
||||||
|
|
||||||
|
## Future Enhancements (Optional)
|
||||||
|
|
||||||
|
1. **Pitch Control UI** - Add sliders in frontend for per-character pitch
|
||||||
|
2. **Bundled Models** - Include 1-2 default Piper voices in repo
|
||||||
|
3. **Download Helper** - Script to download popular voices
|
||||||
|
4. **More Engines** - Add Coqui XTTS, Bark, etc.
|
||||||
|
5. **Voice Cloning** - Add custom voice training
|
||||||
|
6. **Docker Image** - Pre-package Piper models in container
|
||||||
|
|
||||||
|
## Support Resources
|
||||||
|
|
||||||
|
- **Setup Guide**: `backend/PIPER_SETUP.md`
|
||||||
|
- **Usage Guide**: `backend/PIPER_TTS_GUIDE.md`
|
||||||
|
- **Status & Troubleshooting**: `PIPER_STATUS.md`
|
||||||
|
- **Integration Details**: `PIPER_TTS_INTEGRATION.md`
|
||||||
|
- **Backend README**: `backend/README.md`
|
||||||
|
|
||||||
|
## Conclusion
|
||||||
|
|
||||||
|
✅ **Piper TTS integration is complete and production-ready**
|
||||||
|
|
||||||
|
The integration provides users with a choice between:
|
||||||
|
- **Kokoro** (zero-setup, high-quality, general purpose)
|
||||||
|
- **Piper** (manual setup, pitch control, advanced features)
|
||||||
|
|
||||||
|
Both engines work reliably with proper error handling and user guidance. The manual setup requirement for Piper is well-documented and acceptable for the advanced features it provides.
|
||||||
|
|
||||||
|
**Status: COMPLETE ✅**
|
||||||
150
PIPER_STATUS.md
Normal file
150
PIPER_STATUS.md
Normal file
@@ -0,0 +1,150 @@
|
|||||||
|
# Piper TTS Integration Status
|
||||||
|
|
||||||
|
## Current State: ✅ Integrated with Manual Setup Required
|
||||||
|
|
||||||
|
Piper TTS has been successfully integrated into the application as an alternative to Kokoro TTS. The integration is **complete and functional**, but requires **manual voice model download** before use.
|
||||||
|
|
||||||
|
## What Works
|
||||||
|
|
||||||
|
✅ **Backend Integration**
|
||||||
|
- PiperTTSService with pitch control (-12 to +12 semitones)
|
||||||
|
- UnifiedTTSService managing both engines
|
||||||
|
- API endpoints for engine selection
|
||||||
|
- Voice preview support for Piper voices
|
||||||
|
|
||||||
|
✅ **Frontend Integration**
|
||||||
|
- Engine selector UI (Kokoro vs Piper buttons)
|
||||||
|
- Dynamic voice loading per engine
|
||||||
|
- Visual feedback for active engine
|
||||||
|
|
||||||
|
✅ **Documentation**
|
||||||
|
- Comprehensive setup guides
|
||||||
|
- API documentation
|
||||||
|
- Troubleshooting instructions
|
||||||
|
|
||||||
|
## What Requires Manual Setup
|
||||||
|
|
||||||
|
⚠️ **Voice Models Must Be Downloaded Manually**
|
||||||
|
|
||||||
|
Piper voice models cannot be auto-downloaded due to:
|
||||||
|
- HuggingFace repository structure complexity
|
||||||
|
- SSL certificate verification issues in some environments
|
||||||
|
- Large file sizes (10-50MB per voice)
|
||||||
|
|
||||||
|
**This is a one-time setup step.**
|
||||||
|
|
||||||
|
## How to Use Piper TTS
|
||||||
|
|
||||||
|
### Step 1: Install Piper Binary
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
./install_piper.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify: `piper --version`
|
||||||
|
|
||||||
|
### Step 2: Download Voice Models
|
||||||
|
|
||||||
|
1. Visit: https://github.com/rhasspy/piper/releases/tag/v1.2.0
|
||||||
|
2. Download these files for each voice you want:
|
||||||
|
- `en_US-lessac-medium.onnx` (model file)
|
||||||
|
- `en_US-lessac-medium.onnx.json` (config file)
|
||||||
|
|
||||||
|
3. Place in: `backend/piper_models/`
|
||||||
|
|
||||||
|
**Recommended starter voices:**
|
||||||
|
- `en_US-lessac-medium` (Male, American)
|
||||||
|
- `en_US-amy-medium` (Female, American)
|
||||||
|
- `en_GB-alan-medium` (Male, British)
|
||||||
|
|
||||||
|
### Step 3: Test Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
python test_piper_direct.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 4: Use in Application
|
||||||
|
|
||||||
|
1. Start backend: `uvicorn main:app --reload`
|
||||||
|
2. Start frontend: `npm run dev`
|
||||||
|
3. In Voice Assignment screen, click "Piper TTS" button
|
||||||
|
4. Select Piper voices from dropdown
|
||||||
|
5. Generate audiobook with pitch control!
|
||||||
|
|
||||||
|
## Why Manual Download?
|
||||||
|
|
||||||
|
We attempted automatic download but encountered:
|
||||||
|
- **SSL Certificate Issues**: Python's urllib has certificate verification problems
|
||||||
|
- **URL Structure**: HuggingFace's repository structure is complex
|
||||||
|
- **Reliability**: Manual download is more reliable across different environments
|
||||||
|
|
||||||
|
**Trade-off**: One-time manual setup vs. unreliable automatic downloads
|
||||||
|
|
||||||
|
## Alternative: Use Kokoro TTS
|
||||||
|
|
||||||
|
If Piper setup is too complex, **Kokoro TTS works perfectly out of the box** with:
|
||||||
|
- ✅ No manual setup required
|
||||||
|
- ✅ 54 voices across 9 languages
|
||||||
|
- ✅ High quality synthesis
|
||||||
|
- ❌ No pitch control
|
||||||
|
|
||||||
|
Simply keep the default "Kokoro TTS" engine selected.
|
||||||
|
|
||||||
|
## Error Messages You Might See
|
||||||
|
|
||||||
|
### "Voice model not found"
|
||||||
|
|
||||||
|
**Cause**: Voice model files not downloaded
|
||||||
|
|
||||||
|
**Solution**: Download `.onnx` and `.onnx.json` files and place in `backend/piper_models/`
|
||||||
|
|
||||||
|
**See**: `backend/PIPER_SETUP.md` for detailed instructions
|
||||||
|
|
||||||
|
### "Piper not found"
|
||||||
|
|
||||||
|
**Cause**: Piper binary not installed
|
||||||
|
|
||||||
|
**Solution**: Run `./install_piper.sh` or install manually
|
||||||
|
|
||||||
|
### "Preview failed"
|
||||||
|
|
||||||
|
**Cause**: Voice model missing or Piper not working
|
||||||
|
|
||||||
|
**Solution**:
|
||||||
|
1. Check voice models are downloaded
|
||||||
|
2. Test with `python test_piper_direct.py`
|
||||||
|
3. Check backend logs for detailed errors
|
||||||
|
|
||||||
|
## Future Improvements
|
||||||
|
|
||||||
|
Potential enhancements for easier setup:
|
||||||
|
|
||||||
|
1. **Bundled Voice Models**: Include 1-2 default voices in repository
|
||||||
|
2. **Download Script**: Create a helper script to download popular voices
|
||||||
|
3. **UI Download**: Add "Download Voice" button in frontend
|
||||||
|
4. **Alternative Hosting**: Host models on a simpler CDN
|
||||||
|
5. **Docker Image**: Pre-package models in Docker image
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- **Setup Guide**: `backend/PIPER_SETUP.md`
|
||||||
|
- **Integration Details**: `PIPER_TTS_INTEGRATION.md`
|
||||||
|
- **Usage Guide**: `backend/PIPER_TTS_GUIDE.md`
|
||||||
|
- **Backend README**: `backend/README.md`
|
||||||
|
|
||||||
|
## Support
|
||||||
|
|
||||||
|
If you encounter issues:
|
||||||
|
|
||||||
|
1. Check `backend/PIPER_SETUP.md` for setup instructions
|
||||||
|
2. Run `python test_piper_direct.py` to diagnose problems
|
||||||
|
3. Check backend logs for detailed error messages
|
||||||
|
4. Fall back to Kokoro TTS if needed
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
**Piper TTS is fully integrated and functional**, but requires a one-time manual setup to download voice models. This is a reasonable trade-off for gaining pitch control capabilities. Users who prefer zero-setup can continue using Kokoro TTS.
|
||||||
|
|
||||||
|
**Recommendation**: Document the manual setup clearly in user-facing documentation and consider it a "power user" feature.
|
||||||
209
PIPER_TTS_INTEGRATION.md
Normal file
209
PIPER_TTS_INTEGRATION.md
Normal file
@@ -0,0 +1,209 @@
|
|||||||
|
# Piper TTS Integration Summary
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Successfully integrated Piper TTS as an alternative text-to-speech engine alongside Kokoro TTS, giving users the ability to choose between two engines with different strengths.
|
||||||
|
|
||||||
|
## What Was Added
|
||||||
|
|
||||||
|
### Backend Components
|
||||||
|
|
||||||
|
1. **PiperTTSService** (`backend/services/piper_tts_service.py`)
|
||||||
|
- Handles Piper TTS synthesis with pitch control
|
||||||
|
- Supports pitch shifting from -12 to +12 semitones per character
|
||||||
|
- Automatic voice model downloading
|
||||||
|
- Audio effects processing (pitch and speed)
|
||||||
|
|
||||||
|
2. **UnifiedTTSService** (`backend/services/unified_tts_service.py`)
|
||||||
|
- Manages multiple TTS engines (Kokoro and Piper)
|
||||||
|
- Runtime engine switching
|
||||||
|
- Unified API for both engines
|
||||||
|
- Handles engine-specific features (pitch control for Piper)
|
||||||
|
|
||||||
|
3. **New API Endpoints** (in `backend/main.py`)
|
||||||
|
- `GET /api/tts/engines` - List available engines and current selection
|
||||||
|
- `POST /api/tts/engines` - Switch active TTS engine
|
||||||
|
- `GET /api/tts/voices` - Get available voices for all engines
|
||||||
|
- Updated `POST /api/tts` to support `pitch_adjustments` parameter
|
||||||
|
|
||||||
|
4. **Installation Script** (`backend/install_piper.sh`)
|
||||||
|
- Automated Piper TTS installation for macOS/Linux
|
||||||
|
- Detects OS and architecture
|
||||||
|
- Installs to `/usr/local/bin`
|
||||||
|
|
||||||
|
### Frontend Components
|
||||||
|
|
||||||
|
1. **API Client Functions** (`frontend/src/api/tts.ts`)
|
||||||
|
- `getEngines()` - Fetch available engines
|
||||||
|
- `setEngine(engine)` - Switch active engine
|
||||||
|
- `getVoices()` - Get voices for all engines
|
||||||
|
- TypeScript interfaces for engine data
|
||||||
|
|
||||||
|
2. **UI Components** (`frontend/src/components/VoiceAssignmentSection.tsx`)
|
||||||
|
- Engine selector buttons (Kokoro vs Piper)
|
||||||
|
- Dynamic voice loading based on selected engine
|
||||||
|
- Visual feedback for active engine
|
||||||
|
- Pitch control capability notice for Piper
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
|
||||||
|
1. **Backend README** (`backend/README.md`)
|
||||||
|
- Multi-engine setup instructions
|
||||||
|
- Engine switching examples
|
||||||
|
- Pitch control documentation
|
||||||
|
|
||||||
|
2. **Piper TTS Guide** (`backend/PIPER_TTS_GUIDE.md`)
|
||||||
|
- Comprehensive installation guide
|
||||||
|
- Available voices documentation
|
||||||
|
- Pitch adjustment guidelines
|
||||||
|
- Usage examples and best practices
|
||||||
|
- Troubleshooting section
|
||||||
|
|
||||||
|
3. **Integration Summary** (this file)
|
||||||
|
|
||||||
|
## Engine Comparison
|
||||||
|
|
||||||
|
| Feature | Kokoro TTS | Piper TTS |
|
||||||
|
|---------|------------|-----------|
|
||||||
|
| **Quality** | High | Medium-High |
|
||||||
|
| **Speed** | Medium | Fast |
|
||||||
|
| **Voices** | 54 voices, 9 languages | 5+ English voices |
|
||||||
|
| **Pitch Control** | ❌ No | ✅ Yes (-12 to +12 semitones) |
|
||||||
|
| **Model Size** | ~500MB | ~10-50MB per voice |
|
||||||
|
| **Memory Usage** | Higher | Lower |
|
||||||
|
| **Languages** | 9 languages | English (more available) |
|
||||||
|
|
||||||
|
## How to Use
|
||||||
|
|
||||||
|
### Backend Setup
|
||||||
|
|
||||||
|
1. **Install Piper TTS** (optional):
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
./install_piper.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Set default engine** (optional):
|
||||||
|
```bash
|
||||||
|
# In backend/.env
|
||||||
|
TTS_ENGINE=piper # or kokoro (default)
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Install Python dependencies**:
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### Frontend Usage
|
||||||
|
|
||||||
|
1. **Select Engine**: In the Voice Assignment screen, click either "Kokoro TTS" or "Piper TTS" button
|
||||||
|
2. **Choose Voices**: Voice dropdown updates automatically with engine-specific voices
|
||||||
|
3. **Generate**: Audio is generated using the selected engine
|
||||||
|
|
||||||
|
### API Usage
|
||||||
|
|
||||||
|
**Switch Engine:**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts/engines \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"engine": "piper"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Generate with Pitch Control (Piper only):**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"text": "Alice: Hello!\nBob: Hi there!",
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "en_US-amy-medium"},
|
||||||
|
{"id": "Bob", "voice": "en_US-lessac-medium"}
|
||||||
|
],
|
||||||
|
"pitch_adjustments": [
|
||||||
|
{"id": "Alice", "pitch": 3},
|
||||||
|
{"id": "Bob", "pitch": -2}
|
||||||
|
]
|
||||||
|
}' \
|
||||||
|
--output audiobook.wav
|
||||||
|
```
|
||||||
|
|
||||||
|
## Pitch Control Guidelines
|
||||||
|
|
||||||
|
When using Piper TTS, you can adjust pitch per character:
|
||||||
|
|
||||||
|
- **+12 semitones**: One octave higher (very high-pitched)
|
||||||
|
- **+6 semitones**: Noticeably higher (child-like)
|
||||||
|
- **+3 semitones**: Slightly higher (feminine adjustment)
|
||||||
|
- **0 semitones**: Original voice (no change)
|
||||||
|
- **-3 semitones**: Slightly lower (masculine adjustment)
|
||||||
|
- **-6 semitones**: Noticeably lower (deep voice)
|
||||||
|
- **-12 semitones**: One octave lower (very deep)
|
||||||
|
|
||||||
|
**Recommended ranges:**
|
||||||
|
- Female characters: +2 to +5
|
||||||
|
- Male characters: -2 to -5
|
||||||
|
- Children: +5 to +8
|
||||||
|
- Elderly/deep voices: -4 to -8
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Run the engine selection test:
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
python test_engine_selection.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Files Modified/Created
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
- ✅ `backend/services/piper_tts_service.py` (new)
|
||||||
|
- ✅ `backend/services/unified_tts_service.py` (new)
|
||||||
|
- ✅ `backend/main.py` (modified - added endpoints)
|
||||||
|
- ✅ `backend/requirements.txt` (modified - added piper-tts, pydub)
|
||||||
|
- ✅ `backend/.env.example` (modified - added TTS_ENGINE)
|
||||||
|
- ✅ `backend/README.md` (modified - added multi-engine docs)
|
||||||
|
- ✅ `backend/PIPER_TTS_GUIDE.md` (new)
|
||||||
|
- ✅ `backend/install_piper.sh` (new)
|
||||||
|
- ✅ `backend/test_engine_selection.py` (new)
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
- ✅ `frontend/src/api/tts.ts` (modified - added engine functions)
|
||||||
|
- ✅ `frontend/src/components/VoiceAssignmentSection.tsx` (modified - added engine selector)
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
- ✅ `.kiro/specs/text-to-audiobook/tasks.md` (modified - added task 26)
|
||||||
|
- ✅ `PIPER_TTS_INTEGRATION.md` (new - this file)
|
||||||
|
|
||||||
|
## Git Commits
|
||||||
|
|
||||||
|
1. **feat: Add Piper TTS integration with pitch control support** (bfdcc94)
|
||||||
|
- Backend services and API endpoints
|
||||||
|
- Installation script and documentation
|
||||||
|
|
||||||
|
2. **feat: Add TTS engine selector UI in frontend** (c310f11)
|
||||||
|
- Frontend UI components
|
||||||
|
- API client functions
|
||||||
|
|
||||||
|
## Next Steps (Optional Enhancements)
|
||||||
|
|
||||||
|
1. **Pitch Control UI**: Add pitch sliders in the frontend for Piper engine
|
||||||
|
2. **Voice Previews**: Update preview functionality to work with Piper voices
|
||||||
|
3. **More Piper Voices**: Add support for additional Piper voice models
|
||||||
|
4. **Coqui TTS**: Integrate Coqui XTTS as a third engine option
|
||||||
|
5. **Voice Cloning**: Add voice cloning capabilities with Piper/Coqui
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Piper TTS requires separate installation (not included in Python packages)
|
||||||
|
- Voice models are downloaded automatically on first use
|
||||||
|
- Kokoro remains the default engine for backward compatibility
|
||||||
|
- Both engines can coexist and be switched at runtime
|
||||||
|
- Pitch control is only available with Piper TTS
|
||||||
|
|
||||||
|
## Support
|
||||||
|
|
||||||
|
For issues or questions:
|
||||||
|
- See `backend/PIPER_TTS_GUIDE.md` for detailed Piper documentation
|
||||||
|
- See `backend/README.md` for general backend setup
|
||||||
|
- Check troubleshooting sections in both guides
|
||||||
23
RESTART_REQUIRED.md
Normal file
23
RESTART_REQUIRED.md
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
# ⚠️ RESTART REQUIRED
|
||||||
|
|
||||||
|
The UI has been completely redesigned with proper spacing, typography, and visual hierarchy.
|
||||||
|
|
||||||
|
## To see the changes:
|
||||||
|
|
||||||
|
1. **Stop the frontend dev server** (Ctrl+C in the terminal running `npm start`)
|
||||||
|
2. **Restart it**: `cd frontend && npm start`
|
||||||
|
3. **Refresh your browser** (hard refresh: Cmd+Shift+R on Mac, Ctrl+Shift+R on Windows)
|
||||||
|
|
||||||
|
## What's been fixed:
|
||||||
|
|
||||||
|
✅ CSS import order corrected for Tailwind v4
|
||||||
|
✅ Professional spacing and padding throughout
|
||||||
|
✅ Proper card containers with shadows
|
||||||
|
✅ Beautiful typography hierarchy
|
||||||
|
✅ Gradient buttons with hover effects
|
||||||
|
✅ Progress stepper with step labels
|
||||||
|
✅ Generous white space and breathing room
|
||||||
|
✅ Modern color scheme (indigo/purple/gray)
|
||||||
|
✅ Smooth transitions and animations
|
||||||
|
|
||||||
|
The UI is now professional, polished, and ready to use!
|
||||||
135
TTS_ENGINE_QUICK_REFERENCE.md
Normal file
135
TTS_ENGINE_QUICK_REFERENCE.md
Normal file
@@ -0,0 +1,135 @@
|
|||||||
|
# TTS Engine Quick Reference
|
||||||
|
|
||||||
|
## 🎯 Which Engine Should I Use?
|
||||||
|
|
||||||
|
### Use Kokoro TTS (Default) if you want:
|
||||||
|
- ✅ **Zero setup** - works immediately
|
||||||
|
- ✅ **High quality** voices
|
||||||
|
- ✅ **Many voices** - 54 voices, 9 languages
|
||||||
|
- ✅ **Multiple languages** - English, French, Japanese, Spanish, etc.
|
||||||
|
|
||||||
|
### Use Piper TTS if you want:
|
||||||
|
- ✅ **Pitch control** - adjust voice pitch per character
|
||||||
|
- ✅ **Fast synthesis** - quicker generation
|
||||||
|
- ✅ **Lower memory** - smaller models
|
||||||
|
- ⚠️ **Manual setup required** - one-time voice download
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Quick Start
|
||||||
|
|
||||||
|
### Kokoro TTS (No Setup)
|
||||||
|
1. Start the app
|
||||||
|
2. It just works! ✨
|
||||||
|
|
||||||
|
### Piper TTS (3-Minute Setup)
|
||||||
|
1. Install Piper: `cd backend && ./install_piper.sh`
|
||||||
|
2. Download voices from: https://github.com/rhasspy/piper/releases/tag/v1.2.0
|
||||||
|
3. Place `.onnx` and `.onnx.json` files in `backend/piper_models/`
|
||||||
|
4. Done! See `backend/PIPER_SETUP.md` for details
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎛️ How to Switch Engines
|
||||||
|
|
||||||
|
### In the UI:
|
||||||
|
1. Go to **Voice Assignment** screen
|
||||||
|
2. Click **"Kokoro TTS"** or **"Piper TTS"** button
|
||||||
|
3. Voice dropdown updates automatically
|
||||||
|
|
||||||
|
### Via API:
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts/engines \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"engine": "piper"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Via Environment Variable:
|
||||||
|
```bash
|
||||||
|
# In backend/.env
|
||||||
|
TTS_ENGINE=piper # or kokoro
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎵 Pitch Control (Piper Only)
|
||||||
|
|
||||||
|
Adjust voice pitch per character in semitones:
|
||||||
|
|
||||||
|
```
|
||||||
|
+12 = Very high (one octave up)
|
||||||
|
+6 = Child-like
|
||||||
|
+3 = Slightly higher (feminine)
|
||||||
|
0 = Original voice
|
||||||
|
-3 = Slightly lower (masculine)
|
||||||
|
-6 = Deep voice
|
||||||
|
-12 = Very deep (one octave down)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Recommended:**
|
||||||
|
- Female characters: +2 to +5
|
||||||
|
- Male characters: -2 to -5
|
||||||
|
- Children: +5 to +8
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 Feature Comparison
|
||||||
|
|
||||||
|
| Feature | Kokoro | Piper |
|
||||||
|
|---------|--------|-------|
|
||||||
|
| Setup | None | Manual |
|
||||||
|
| Quality | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
|
||||||
|
| Speed | Medium | Fast |
|
||||||
|
| Voices | 54 | 5+ |
|
||||||
|
| Languages | 9 | English |
|
||||||
|
| Pitch Control | ❌ | ✅ |
|
||||||
|
| Memory | 500MB | 10-50MB |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔧 Troubleshooting
|
||||||
|
|
||||||
|
### "Voice preview failed"
|
||||||
|
- **Kokoro**: Check internet connection (first-time model download)
|
||||||
|
- **Piper**: Download voice models manually (see PIPER_SETUP.md)
|
||||||
|
|
||||||
|
### "Voice model not found"
|
||||||
|
- Download `.onnx` and `.onnx.json` files
|
||||||
|
- Place in `backend/piper_models/`
|
||||||
|
- Restart backend server
|
||||||
|
|
||||||
|
### "Piper not found"
|
||||||
|
- Run `./install_piper.sh`
|
||||||
|
- Or install from: https://github.com/rhasspy/piper/releases
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📚 Documentation
|
||||||
|
|
||||||
|
- **Piper Setup**: `backend/PIPER_SETUP.md`
|
||||||
|
- **Piper Guide**: `backend/PIPER_TTS_GUIDE.md`
|
||||||
|
- **Status**: `PIPER_STATUS.md`
|
||||||
|
- **Full Details**: `PIPER_TTS_INTEGRATION.md`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 💡 Pro Tips
|
||||||
|
|
||||||
|
1. **Start with Kokoro** - Try it first, it's easier
|
||||||
|
2. **Try Piper for variety** - Pitch control adds character depth
|
||||||
|
3. **Mix engines** - Use Kokoro for most, Piper for special characters
|
||||||
|
4. **Download multiple voices** - More variety in your audiobooks
|
||||||
|
5. **Test voices** - Use preview button before generating full audiobook
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🆘 Need Help?
|
||||||
|
|
||||||
|
1. Check `backend/PIPER_SETUP.md` for setup
|
||||||
|
2. Run `python test_piper_direct.py` to test Piper
|
||||||
|
3. Check backend logs for detailed errors
|
||||||
|
4. Fall back to Kokoro if Piper has issues
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Remember:** Both engines work great! Kokoro is easier, Piper is more advanced. Choose based on your needs. 🎉
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
PORT=8000
|
PORT=8000
|
||||||
LOG_LEVEL=info
|
LOG_LEVEL=info
|
||||||
OLLAMA_BASE_URL=http://localhost:11434
|
OLLAMA_BASE_URL=http://localhost:11434
|
||||||
|
TTS_ENGINE=kokoro
|
||||||
|
|||||||
130
backend/EASY_PIPER_SETUP.md
Normal file
130
backend/EASY_PIPER_SETUP.md
Normal file
@@ -0,0 +1,130 @@
|
|||||||
|
# Easy Piper TTS Setup (5 Minutes)
|
||||||
|
|
||||||
|
## The Problem
|
||||||
|
|
||||||
|
Piper voice models can't be auto-downloaded due to HuggingFace's complex repository structure. But don't worry - manual download is quick and easy!
|
||||||
|
|
||||||
|
## Quick Setup (3 Steps)
|
||||||
|
|
||||||
|
### Step 1: Download Voice Files
|
||||||
|
|
||||||
|
Visit this page: **https://github.com/rhasspy/piper/releases/tag/v1.2.0**
|
||||||
|
|
||||||
|
Scroll down to "Assets" section and download these files:
|
||||||
|
|
||||||
|
**Recommended Starter Pack (3 voices):**
|
||||||
|
|
||||||
|
1. **en_US-lessac-medium** (Male, American)
|
||||||
|
- Download: `en_US-lessac-medium.onnx` (63 MB)
|
||||||
|
- Download: `en_US-lessac-medium.onnx.json` (8 KB)
|
||||||
|
|
||||||
|
2. **en_US-amy-medium** (Female, American)
|
||||||
|
- Download: `en_US-amy-medium.onnx` (63 MB)
|
||||||
|
- Download: `en_US-amy-medium.onnx.json` (8 KB)
|
||||||
|
|
||||||
|
3. **en_GB-alan-medium** (Male, British)
|
||||||
|
- Download: `en_GB-alan-medium.onnx` (63 MB)
|
||||||
|
- Download: `en_GB-alan-medium.onnx.json` (8 KB)
|
||||||
|
|
||||||
|
**Total download:** ~190 MB for 3 voices
|
||||||
|
|
||||||
|
### Step 2: Place Files
|
||||||
|
|
||||||
|
Move all downloaded files to:
|
||||||
|
```
|
||||||
|
backend/piper_models/
|
||||||
|
```
|
||||||
|
|
||||||
|
Your directory should look like:
|
||||||
|
```
|
||||||
|
backend/piper_models/
|
||||||
|
├── en_US-lessac-medium.onnx
|
||||||
|
├── en_US-lessac-medium.onnx.json
|
||||||
|
├── en_US-amy-medium.onnx
|
||||||
|
├── en_US-amy-medium.onnx.json
|
||||||
|
├── en_GB-alan-medium.onnx
|
||||||
|
└── en_GB-alan-medium.onnx.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
python test_piper_direct.py
|
||||||
|
```
|
||||||
|
|
||||||
|
You should see: "✓ Synthesis successful"
|
||||||
|
|
||||||
|
## Done! 🎉
|
||||||
|
|
||||||
|
Now you can:
|
||||||
|
1. Start the backend: `uvicorn main:app --reload`
|
||||||
|
2. Open the UI
|
||||||
|
3. Click "Piper TTS" button
|
||||||
|
4. Select Piper voices
|
||||||
|
5. Generate audiobooks with pitch control!
|
||||||
|
|
||||||
|
## More Voices (Optional)
|
||||||
|
|
||||||
|
Want more variety? Download additional voices from the same page:
|
||||||
|
|
||||||
|
**More American English:**
|
||||||
|
- `en_US-libritts-high` (High quality, neutral)
|
||||||
|
- `en_US-ryan-high` (Male, high quality)
|
||||||
|
|
||||||
|
**More British English:**
|
||||||
|
- `en_GB-alba-medium` (Female)
|
||||||
|
- `en_GB-jenny_dioco-medium` (Female)
|
||||||
|
|
||||||
|
**Other Languages:**
|
||||||
|
- Browse the releases page for Spanish, French, German, etc.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### "Model not found" error
|
||||||
|
- Make sure you downloaded BOTH `.onnx` and `.onnx.json` files
|
||||||
|
- Check files are in `backend/piper_models/` directory
|
||||||
|
- Verify filenames match exactly (case-sensitive)
|
||||||
|
|
||||||
|
### Files won't download
|
||||||
|
- Try a different browser
|
||||||
|
- Check your internet connection
|
||||||
|
- Download one at a time if needed
|
||||||
|
|
||||||
|
### Still having issues?
|
||||||
|
- Fall back to Kokoro TTS (works out of the box)
|
||||||
|
- Check `backend/PIPER_SETUP.md` for detailed troubleshooting
|
||||||
|
|
||||||
|
## Why Manual Download?
|
||||||
|
|
||||||
|
We tried automatic download but encountered:
|
||||||
|
- ❌ Complex HuggingFace URL structure
|
||||||
|
- ❌ SSL certificate issues
|
||||||
|
- ❌ Unreliable across different systems
|
||||||
|
|
||||||
|
Manual download is:
|
||||||
|
- ✅ Fast (5 minutes)
|
||||||
|
- ✅ Reliable (always works)
|
||||||
|
- ✅ One-time setup
|
||||||
|
- ✅ You control what you download
|
||||||
|
|
||||||
|
## Alternative: Use Kokoro TTS
|
||||||
|
|
||||||
|
If this seems like too much work, **Kokoro TTS works perfectly with zero setup**:
|
||||||
|
- ✅ No downloads needed
|
||||||
|
- ✅ 54 voices, 9 languages
|
||||||
|
- ✅ High quality
|
||||||
|
- ❌ No pitch control
|
||||||
|
|
||||||
|
Just keep "Kokoro TTS" selected in the UI!
|
||||||
|
|
||||||
|
## Need Help?
|
||||||
|
|
||||||
|
1. Check the files are in the right place
|
||||||
|
2. Run `python test_piper_direct.py` to diagnose
|
||||||
|
3. Check backend logs for errors
|
||||||
|
4. See `PIPER_STATUS.md` for more troubleshooting
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Remember:** This is a one-time setup. Once downloaded, Piper voices work forever! 🚀
|
||||||
154
backend/PIPER_DOWNLOAD_GUIDE.md
Normal file
154
backend/PIPER_DOWNLOAD_GUIDE.md
Normal file
@@ -0,0 +1,154 @@
|
|||||||
|
# Piper Voice Download - Visual Guide
|
||||||
|
|
||||||
|
## 📥 Step-by-Step Instructions
|
||||||
|
|
||||||
|
### Step 1: Go to Piper Releases Page
|
||||||
|
|
||||||
|
Open this link in your browser:
|
||||||
|
```
|
||||||
|
https://github.com/rhasspy/piper/releases/tag/v1.2.0
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Scroll to "Assets" Section
|
||||||
|
|
||||||
|
Scroll down the page until you see a section called "Assets" with a list of files.
|
||||||
|
|
||||||
|
### Step 3: Download Voice Files
|
||||||
|
|
||||||
|
Click on each file to download (you need BOTH files for each voice):
|
||||||
|
|
||||||
|
#### Voice 1: Male American (Lessac)
|
||||||
|
- ✅ Click: `en_US-lessac-medium.onnx` (63 MB)
|
||||||
|
- ✅ Click: `en_US-lessac-medium.onnx.json` (8 KB)
|
||||||
|
|
||||||
|
#### Voice 2: Female American (Amy)
|
||||||
|
- ✅ Click: `en_US-amy-medium.onnx` (63 MB)
|
||||||
|
- ✅ Click: `en_US-amy-medium.onnx.json` (8 KB)
|
||||||
|
|
||||||
|
#### Voice 3: Male British (Alan)
|
||||||
|
- ✅ Click: `en_GB-alan-medium.onnx` (63 MB)
|
||||||
|
- ✅ Click: `en_GB-alan-medium.onnx.json` (8 KB)
|
||||||
|
|
||||||
|
**Total:** 6 files, ~190 MB
|
||||||
|
|
||||||
|
### Step 4: Move Files to Models Directory
|
||||||
|
|
||||||
|
1. Open your terminal
|
||||||
|
2. Navigate to your project:
|
||||||
|
```bash
|
||||||
|
cd /path/to/audiobook/backend
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Create models directory if it doesn't exist:
|
||||||
|
```bash
|
||||||
|
mkdir -p piper_models
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Move downloaded files:
|
||||||
|
```bash
|
||||||
|
# From your Downloads folder
|
||||||
|
mv ~/Downloads/en_US-*.onnx* piper_models/
|
||||||
|
mv ~/Downloads/en_GB-*.onnx* piper_models/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 5: Verify Files
|
||||||
|
|
||||||
|
Check that files are in place:
|
||||||
|
```bash
|
||||||
|
ls -lh piper_models/
|
||||||
|
```
|
||||||
|
|
||||||
|
You should see:
|
||||||
|
```
|
||||||
|
en_US-lessac-medium.onnx (63 MB)
|
||||||
|
en_US-lessac-medium.onnx.json (8 KB)
|
||||||
|
en_US-amy-medium.onnx (63 MB)
|
||||||
|
en_US-amy-medium.onnx.json (8 KB)
|
||||||
|
en_GB-alan-medium.onnx (63 MB)
|
||||||
|
en_GB-alan-medium.onnx.json (8 KB)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 6: Test Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python test_piper_direct.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected output:
|
||||||
|
```
|
||||||
|
✓ Piper found
|
||||||
|
✓ Model found: /path/to/piper_models/en_US-lessac-medium.onnx
|
||||||
|
✓ Synthesis successful: XXXXX bytes of audio
|
||||||
|
```
|
||||||
|
|
||||||
|
## ✅ You're Done!
|
||||||
|
|
||||||
|
Now you can use Piper TTS:
|
||||||
|
|
||||||
|
1. **Start backend:**
|
||||||
|
```bash
|
||||||
|
uvicorn main:app --reload
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Open frontend** in browser
|
||||||
|
|
||||||
|
3. **Select "Piper TTS"** in Voice Assignment screen
|
||||||
|
|
||||||
|
4. **Choose Piper voices** from dropdown
|
||||||
|
|
||||||
|
5. **Generate audiobook** with pitch control!
|
||||||
|
|
||||||
|
## 🎯 Quick Checklist
|
||||||
|
|
||||||
|
- [ ] Downloaded 6 files (3 voices × 2 files each)
|
||||||
|
- [ ] Moved files to `backend/piper_models/`
|
||||||
|
- [ ] Verified files with `ls -lh piper_models/`
|
||||||
|
- [ ] Tested with `python test_piper_direct.py`
|
||||||
|
- [ ] Started backend server
|
||||||
|
- [ ] Selected "Piper TTS" in UI
|
||||||
|
|
||||||
|
## 🆘 Troubleshooting
|
||||||
|
|
||||||
|
### Can't find files on GitHub?
|
||||||
|
- Make sure you're on the v1.2.0 release page
|
||||||
|
- Scroll down to "Assets" section
|
||||||
|
- Files are listed alphabetically
|
||||||
|
|
||||||
|
### Download fails?
|
||||||
|
- Try a different browser
|
||||||
|
- Check internet connection
|
||||||
|
- Download one file at a time
|
||||||
|
|
||||||
|
### Files in wrong place?
|
||||||
|
```bash
|
||||||
|
# Check current location
|
||||||
|
pwd
|
||||||
|
|
||||||
|
# Should show: /path/to/audiobook/backend
|
||||||
|
|
||||||
|
# List models directory
|
||||||
|
ls piper_models/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test fails?
|
||||||
|
- Make sure BOTH .onnx and .onnx.json files are present
|
||||||
|
- Check filenames match exactly (case-sensitive)
|
||||||
|
- Verify Piper is installed: `piper --version`
|
||||||
|
|
||||||
|
## 💡 Pro Tips
|
||||||
|
|
||||||
|
1. **Download all at once:** Select all 6 files and download together
|
||||||
|
2. **Use command line:** Faster to move files with `mv` command
|
||||||
|
3. **Bookmark the page:** Easy to download more voices later
|
||||||
|
4. **Try different voices:** Each has unique characteristics
|
||||||
|
|
||||||
|
## 🔗 Useful Links
|
||||||
|
|
||||||
|
- **Download Page:** https://github.com/rhasspy/piper/releases/tag/v1.2.0
|
||||||
|
- **Voice Samples:** https://rhasspy.github.io/piper-samples/
|
||||||
|
- **Full Setup Guide:** `PIPER_SETUP.md`
|
||||||
|
- **Troubleshooting:** `PIPER_STATUS.md`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Need more help?** Check `EASY_PIPER_SETUP.md` or ask for assistance!
|
||||||
98
backend/PIPER_SETUP.md
Normal file
98
backend/PIPER_SETUP.md
Normal file
@@ -0,0 +1,98 @@
|
|||||||
|
# Piper TTS Setup Guide
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
Piper TTS is now integrated but requires manual voice model setup.
|
||||||
|
|
||||||
|
### 1. Install Piper
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
./install_piper.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify installation:
|
||||||
|
```bash
|
||||||
|
piper --version
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Download Voice Models
|
||||||
|
|
||||||
|
Piper voice models need to be downloaded manually and placed in `backend/piper_models/`.
|
||||||
|
|
||||||
|
**Download voices from:**
|
||||||
|
https://github.com/rhasspy/piper/releases/tag/v1.2.0
|
||||||
|
|
||||||
|
**Recommended voices for English:**
|
||||||
|
|
||||||
|
1. **en_US-lessac-medium** (Male, American)
|
||||||
|
- Download: `en_US-lessac-medium.onnx` and `en_US-lessac-medium.onnx.json`
|
||||||
|
- Place in: `backend/piper_models/`
|
||||||
|
|
||||||
|
2. **en_US-amy-medium** (Female, American)
|
||||||
|
- Download: `en_US-amy-medium.onnx` and `en_US-amy-medium.onnx.json`
|
||||||
|
- Place in: `backend/piper_models/`
|
||||||
|
|
||||||
|
3. **en_GB-alan-medium** (Male, British)
|
||||||
|
- Download: `en_GB-alan-medium.onnx` and `en_GB-alan-medium.onnx.json`
|
||||||
|
- Place in: `backend/piper_models/`
|
||||||
|
|
||||||
|
### 3. Directory Structure
|
||||||
|
|
||||||
|
After downloading, your directory should look like:
|
||||||
|
|
||||||
|
```
|
||||||
|
backend/
|
||||||
|
├── piper_models/
|
||||||
|
│ ├── en_US-lessac-medium.onnx
|
||||||
|
│ ├── en_US-lessac-medium.onnx.json
|
||||||
|
│ ├── en_US-amy-medium.onnx
|
||||||
|
│ ├── en_US-amy-medium.onnx.json
|
||||||
|
│ ├── en_GB-alan-medium.onnx
|
||||||
|
│ └── en_GB-alan-medium.onnx.json
|
||||||
|
└── ...
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Test Piper
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
python test_piper_direct.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Use in Application
|
||||||
|
|
||||||
|
1. Start the backend server
|
||||||
|
2. In the frontend, click "Piper TTS" button in Voice Assignment screen
|
||||||
|
3. Select Piper voices from the dropdown
|
||||||
|
4. Generate audiobook with pitch control!
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### "Model not found" error
|
||||||
|
|
||||||
|
- Make sure you downloaded both `.onnx` and `.onnx.json` files
|
||||||
|
- Check files are in `backend/piper_models/` directory
|
||||||
|
- Verify filenames match exactly (case-sensitive)
|
||||||
|
|
||||||
|
### "Piper not found" error
|
||||||
|
|
||||||
|
- Run `./install_piper.sh` to install Piper
|
||||||
|
- Or install manually from https://github.com/rhasspy/piper/releases
|
||||||
|
|
||||||
|
### Synthesis fails
|
||||||
|
|
||||||
|
- Check Piper version: `piper --version`
|
||||||
|
- Try running Piper directly: `echo "test" | piper --model piper_models/en_US-lessac-medium.onnx --output-raw > test.wav`
|
||||||
|
- Check backend logs for detailed error messages
|
||||||
|
|
||||||
|
## Alternative: Use Kokoro TTS
|
||||||
|
|
||||||
|
If Piper setup is too complex, you can continue using Kokoro TTS (the default engine) which works out of the box with no additional setup.
|
||||||
|
|
||||||
|
## More Voices
|
||||||
|
|
||||||
|
Browse all available Piper voices:
|
||||||
|
https://rhasspy.github.io/piper-samples/
|
||||||
|
|
||||||
|
Download any voice and place the `.onnx` and `.onnx.json` files in `backend/piper_models/`.
|
||||||
292
backend/PIPER_TTS_GUIDE.md
Normal file
292
backend/PIPER_TTS_GUIDE.md
Normal file
@@ -0,0 +1,292 @@
|
|||||||
|
# Piper TTS Integration Guide
|
||||||
|
|
||||||
|
This guide covers the Piper TTS engine integration, including installation, configuration, and usage.
|
||||||
|
|
||||||
|
## What is Piper TTS?
|
||||||
|
|
||||||
|
Piper is a fast, local neural text-to-speech system that runs entirely on your machine. It offers:
|
||||||
|
|
||||||
|
- **Fast synthesis**: Faster than Kokoro for most use cases
|
||||||
|
- **Pitch control**: Adjust voice pitch per character (-12 to +12 semitones)
|
||||||
|
- **Multiple voices**: Various English voices with different characteristics
|
||||||
|
- **Low resource usage**: Smaller models and lower memory footprint
|
||||||
|
- **Privacy**: All processing happens locally
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
### Automated Installation (Recommended)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
./install_piper.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
This script will:
|
||||||
|
1. Detect your OS and architecture
|
||||||
|
2. Download the appropriate Piper binary
|
||||||
|
3. Install it to `/usr/local/bin`
|
||||||
|
4. Verify the installation
|
||||||
|
|
||||||
|
### Manual Installation
|
||||||
|
|
||||||
|
1. Visit [Piper Releases](https://github.com/rhasspy/piper/releases/tag/v1.2.0)
|
||||||
|
2. Download the appropriate archive for your platform:
|
||||||
|
- **Linux x86_64**: `piper_amd64.tar.gz`
|
||||||
|
- **Linux ARM64**: `piper_arm64.tar.gz`
|
||||||
|
- **macOS**: `piper_macos_x64.tar.gz`
|
||||||
|
3. Extract the archive
|
||||||
|
4. Copy the `piper` binary to a directory in your PATH
|
||||||
|
|
||||||
|
### Verify Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
piper --version
|
||||||
|
```
|
||||||
|
|
||||||
|
You should see output like: `1.2.0`
|
||||||
|
|
||||||
|
## Available Voices
|
||||||
|
|
||||||
|
Piper supports multiple English voices:
|
||||||
|
|
||||||
|
| Voice ID | Language | Gender | Quality | Description |
|
||||||
|
|----------|----------|--------|---------|-------------|
|
||||||
|
| `en_US-lessac-medium` | US English | Male | Medium | Clear, neutral male voice |
|
||||||
|
| `en_US-libritts-high` | US English | Neutral | High | High-quality neutral voice |
|
||||||
|
| `en_US-amy-medium` | US English | Female | Medium | Clear female voice |
|
||||||
|
| `en_GB-alan-medium` | UK English | Male | Medium | British male voice |
|
||||||
|
| `en_GB-alba-medium` | UK English | Female | Medium | British female voice |
|
||||||
|
|
||||||
|
Voice models are automatically downloaded on first use (~10-50MB each).
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Set Piper as Default Engine
|
||||||
|
|
||||||
|
**Option 1: Environment Variable**
|
||||||
|
```bash
|
||||||
|
export TTS_ENGINE=piper
|
||||||
|
```
|
||||||
|
|
||||||
|
Add to your `.env` file:
|
||||||
|
```bash
|
||||||
|
TTS_ENGINE=piper
|
||||||
|
```
|
||||||
|
|
||||||
|
**Option 2: Runtime API Call**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts/engines \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"engine": "piper"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Voice Model Storage
|
||||||
|
|
||||||
|
Voice models are stored in:
|
||||||
|
```
|
||||||
|
backend/piper_models/
|
||||||
|
```
|
||||||
|
|
||||||
|
Models are downloaded automatically on first use. You can pre-download models manually if needed.
|
||||||
|
|
||||||
|
## Usage Examples
|
||||||
|
|
||||||
|
### Basic Text-to-Speech
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"text": "Hello, this is a test of Piper TTS.",
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Narrator", "voice": "en_US-lessac-medium"}
|
||||||
|
],
|
||||||
|
"speed": 1.0
|
||||||
|
}' \
|
||||||
|
--output test.wav
|
||||||
|
```
|
||||||
|
|
||||||
|
### Multi-Voice with Pitch Control
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"text": "Alice: Hello Bob!\nBob: Hi Alice, how are you?\nNarrator: They greeted each other warmly.",
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "en_US-amy-medium"},
|
||||||
|
{"id": "Bob", "voice": "en_US-lessac-medium"},
|
||||||
|
{"id": "Narrator", "voice": "en_GB-alan-medium"}
|
||||||
|
],
|
||||||
|
"speed": 1.0,
|
||||||
|
"pitch_adjustments": [
|
||||||
|
{"id": "Alice", "pitch": 4},
|
||||||
|
{"id": "Bob", "pitch": -3},
|
||||||
|
{"id": "Narrator", "pitch": 0}
|
||||||
|
]
|
||||||
|
}' \
|
||||||
|
--output audiobook.wav
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pitch Adjustment Guidelines
|
||||||
|
|
||||||
|
Pitch is measured in semitones (half-steps on a musical scale):
|
||||||
|
|
||||||
|
- **+12 semitones**: One octave higher (very high-pitched)
|
||||||
|
- **+6 semitones**: Noticeably higher (child-like)
|
||||||
|
- **+3 semitones**: Slightly higher (feminine adjustment)
|
||||||
|
- **0 semitones**: Original voice (no change)
|
||||||
|
- **-3 semitones**: Slightly lower (masculine adjustment)
|
||||||
|
- **-6 semitones**: Noticeably lower (deep voice)
|
||||||
|
- **-12 semitones**: One octave lower (very deep)
|
||||||
|
|
||||||
|
**Recommended ranges:**
|
||||||
|
- Female characters: +2 to +5
|
||||||
|
- Male characters: -2 to -5
|
||||||
|
- Children: +5 to +8
|
||||||
|
- Elderly/deep voices: -4 to -8
|
||||||
|
|
||||||
|
## Python API Usage
|
||||||
|
|
||||||
|
```python
|
||||||
|
from services.unified_tts_service import UnifiedTTSService
|
||||||
|
|
||||||
|
# Initialize with Piper engine
|
||||||
|
tts = UnifiedTTSService(engine="piper")
|
||||||
|
|
||||||
|
# Generate audiobook with pitch control
|
||||||
|
audio_bytes = await tts.generate_audiobook(
|
||||||
|
text="Alice: Hello!\nBob: Hi there!",
|
||||||
|
lang="a",
|
||||||
|
role_voice_map={
|
||||||
|
"Alice": "en_US-amy-medium",
|
||||||
|
"Bob": "en_US-lessac-medium"
|
||||||
|
},
|
||||||
|
speed=1.0,
|
||||||
|
pitch_adjustments={
|
||||||
|
"Alice": 3,
|
||||||
|
"Bob": -2
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Save to file
|
||||||
|
with open("output.wav", "wb") as f:
|
||||||
|
f.write(audio_bytes)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance Comparison
|
||||||
|
|
||||||
|
| Feature | Kokoro TTS | Piper TTS |
|
||||||
|
|---------|------------|-----------|
|
||||||
|
| Speed | Medium | Fast |
|
||||||
|
| Quality | High | Medium-High |
|
||||||
|
| Pitch Control | ❌ No | ✅ Yes |
|
||||||
|
| Model Size | ~500MB | ~10-50MB per voice |
|
||||||
|
| Memory Usage | Higher | Lower |
|
||||||
|
| Voice Variety | 10+ voices | 5+ voices (expandable) |
|
||||||
|
| Languages | 4 languages | English (more available) |
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Piper Not Found
|
||||||
|
|
||||||
|
**Error:** `piper: command not found`
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```bash
|
||||||
|
# Check if piper is in PATH
|
||||||
|
which piper
|
||||||
|
|
||||||
|
# If not found, reinstall
|
||||||
|
./install_piper.sh
|
||||||
|
|
||||||
|
# Or add to PATH manually
|
||||||
|
export PATH="/usr/local/bin:$PATH"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Voice Model Download Fails
|
||||||
|
|
||||||
|
**Error:** Model download timeout or failure
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
1. Check internet connection
|
||||||
|
2. Manually download models from [Piper releases](https://github.com/rhasspy/piper/releases)
|
||||||
|
3. Place `.onnx` and `.onnx.json` files in `backend/piper_models/`
|
||||||
|
|
||||||
|
### Audio Quality Issues
|
||||||
|
|
||||||
|
**Problem:** Distorted or robotic audio
|
||||||
|
|
||||||
|
**Solutions:**
|
||||||
|
- Reduce pitch shift amount (stay within ±6 semitones)
|
||||||
|
- Try different voice models
|
||||||
|
- Adjust speed (try 0.9 or 1.1)
|
||||||
|
- Check for text formatting issues
|
||||||
|
|
||||||
|
### Synthesis Timeout
|
||||||
|
|
||||||
|
**Error:** Piper synthesis times out
|
||||||
|
|
||||||
|
**Solutions:**
|
||||||
|
- Break text into smaller chunks
|
||||||
|
- Increase timeout in `piper_tts_service.py`
|
||||||
|
- Check system resources (CPU, memory)
|
||||||
|
|
||||||
|
## Advanced Configuration
|
||||||
|
|
||||||
|
### Custom Voice Models
|
||||||
|
|
||||||
|
To add custom Piper voice models:
|
||||||
|
|
||||||
|
1. Download `.onnx` and `.onnx.json` files
|
||||||
|
2. Place in `backend/piper_models/`
|
||||||
|
3. Update `AVAILABLE_VOICES` in `piper_tts_service.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
AVAILABLE_VOICES = {
|
||||||
|
"en_US-custom-voice": {"lang": "en", "gender": "female", "quality": "high"},
|
||||||
|
# ... existing voices
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adjusting Audio Effects
|
||||||
|
|
||||||
|
Modify `_apply_audio_effects()` in `piper_tts_service.py` to customize:
|
||||||
|
- Pitch shifting algorithm
|
||||||
|
- Speed adjustment method
|
||||||
|
- Audio processing pipeline
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Voice Selection**
|
||||||
|
- Use distinct voices for different characters
|
||||||
|
- Match voice gender to character when possible
|
||||||
|
- Use British voices for variety
|
||||||
|
|
||||||
|
2. **Pitch Adjustments**
|
||||||
|
- Start with small adjustments (±2-3 semitones)
|
||||||
|
- Test and iterate to find natural-sounding values
|
||||||
|
- Avoid extreme values (±10+) unless intentional
|
||||||
|
|
||||||
|
3. **Performance**
|
||||||
|
- Pre-download voice models for faster first use
|
||||||
|
- Cache generated audio when possible
|
||||||
|
- Use appropriate chunk sizes for long texts
|
||||||
|
|
||||||
|
4. **Quality**
|
||||||
|
- Use high-quality voice models when available
|
||||||
|
- Adjust speed slightly (0.95-1.05) for more natural pacing
|
||||||
|
- Combine with AI segmentation for best results
|
||||||
|
|
||||||
|
## Resources
|
||||||
|
|
||||||
|
- [Piper GitHub Repository](https://github.com/rhasspy/piper)
|
||||||
|
- [Piper Voice Samples](https://rhasspy.github.io/piper-samples/)
|
||||||
|
- [Voice Model Downloads](https://github.com/rhasspy/piper/releases/tag/v1.2.0)
|
||||||
|
- [Piper Documentation](https://github.com/rhasspy/piper/blob/master/README.md)
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Piper TTS is licensed under the MIT License.
|
||||||
@@ -13,6 +13,7 @@ The backend provides REST API endpoints for:
|
|||||||
|
|
||||||
- Python 3.10 or higher
|
- Python 3.10 or higher
|
||||||
- espeak-ng (system dependency for grapheme-to-phoneme conversion)
|
- espeak-ng (system dependency for grapheme-to-phoneme conversion)
|
||||||
|
- Piper TTS (optional, for alternative TTS engine with pitch control)
|
||||||
- Docker (optional, for containerized deployment)
|
- Docker (optional, for containerized deployment)
|
||||||
|
|
||||||
## Local Development Setup
|
## Local Development Setup
|
||||||
@@ -62,7 +63,42 @@ pip install --upgrade pip
|
|||||||
pip install -r requirements.txt
|
pip install -r requirements.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Ollama Setup (Optional but Recommended)
|
### 3. Piper TTS Setup (Optional)
|
||||||
|
|
||||||
|
For an alternative TTS engine with pitch control support:
|
||||||
|
|
||||||
|
**Installation:**
|
||||||
|
```bash
|
||||||
|
# Run the installation script
|
||||||
|
./install_piper.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Or install manually:
|
||||||
|
- Visit [Piper releases](https://github.com/rhasspy/piper/releases)
|
||||||
|
- Download the appropriate binary for your platform
|
||||||
|
- Extract and place in your PATH
|
||||||
|
|
||||||
|
**Verify installation:**
|
||||||
|
```bash
|
||||||
|
piper --version
|
||||||
|
```
|
||||||
|
|
||||||
|
**Download Voice Models:**
|
||||||
|
|
||||||
|
Piper requires manual voice model download. See [PIPER_SETUP.md](PIPER_SETUP.md) for detailed instructions.
|
||||||
|
|
||||||
|
Quick steps:
|
||||||
|
1. Visit https://github.com/rhasspy/piper/releases/tag/v1.2.0
|
||||||
|
2. Download voice models (e.g., `en_US-lessac-medium.onnx` and `.onnx.json`)
|
||||||
|
3. Place in `backend/piper_models/` directory
|
||||||
|
|
||||||
|
**Benefits of Piper TTS:**
|
||||||
|
- ✨ Pitch shifting support (-12 to +12 semitones per role)
|
||||||
|
- ✨ Fast, lightweight neural TTS
|
||||||
|
- ✨ Multiple voice models available
|
||||||
|
- ✨ Lower resource usage than Kokoro
|
||||||
|
|
||||||
|
### 4. Ollama Setup (Optional but Recommended)
|
||||||
|
|
||||||
For AI-powered speaker detection and text segmentation, install Ollama:
|
For AI-powered speaker detection and text segmentation, install Ollama:
|
||||||
|
|
||||||
@@ -98,7 +134,7 @@ curl http://localhost:11434/api/tags
|
|||||||
- ✨ Handles narrative text without explicit role markers
|
- ✨ Handles narrative text without explicit role markers
|
||||||
- ✨ Falls back to pattern matching if unavailable
|
- ✨ Falls back to pattern matching if unavailable
|
||||||
|
|
||||||
### 4. Environment Configuration
|
### 5. Environment Configuration
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cp .env.example .env
|
cp .env.example .env
|
||||||
@@ -109,9 +145,10 @@ Edit `.env` if needed:
|
|||||||
PORT=8000
|
PORT=8000
|
||||||
LOG_LEVEL=info
|
LOG_LEVEL=info
|
||||||
OLLAMA_BASE_URL=http://localhost:11434 # Optional: Ollama API URL
|
OLLAMA_BASE_URL=http://localhost:11434 # Optional: Ollama API URL
|
||||||
|
TTS_ENGINE=kokoro # Options: kokoro, piper (default: kokoro)
|
||||||
```
|
```
|
||||||
|
|
||||||
### 5. Run the Server
|
### 6. Run the Server
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uvicorn main:app --reload --host 0.0.0.0 --port 8000
|
uvicorn main:app --reload --host 0.0.0.0 --port 8000
|
||||||
@@ -179,6 +216,43 @@ The Dockerfile includes:
|
|||||||
- Storage: ~500MB for Kokoro models
|
- Storage: ~500MB for Kokoro models
|
||||||
- CPU: Multi-core recommended
|
- CPU: Multi-core recommended
|
||||||
|
|
||||||
|
## TTS Engine Selection
|
||||||
|
|
||||||
|
The backend now supports multiple TTS engines that can be selected at runtime:
|
||||||
|
|
||||||
|
### Available Engines
|
||||||
|
|
||||||
|
1. **Kokoro TTS** (default)
|
||||||
|
- High-quality neural TTS
|
||||||
|
- Multiple voices per language
|
||||||
|
- No pitch control
|
||||||
|
- ~500MB model size
|
||||||
|
|
||||||
|
2. **Piper TTS**
|
||||||
|
- Fast, lightweight neural TTS
|
||||||
|
- Pitch shifting support (-12 to +12 semitones)
|
||||||
|
- Multiple voice models
|
||||||
|
- Lower resource usage
|
||||||
|
|
||||||
|
### Switching Engines
|
||||||
|
|
||||||
|
**Via Environment Variable:**
|
||||||
|
```bash
|
||||||
|
export TTS_ENGINE=piper # or kokoro
|
||||||
|
uvicorn main:app --reload
|
||||||
|
```
|
||||||
|
|
||||||
|
**Via API:**
|
||||||
|
```bash
|
||||||
|
# Get available engines
|
||||||
|
curl http://localhost:8000/api/tts/engines
|
||||||
|
|
||||||
|
# Set active engine
|
||||||
|
curl -X POST http://localhost:8000/api/tts/engines \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"engine": "piper"}'
|
||||||
|
```
|
||||||
|
|
||||||
## API Endpoints
|
## API Endpoints
|
||||||
|
|
||||||
### GET /api/health
|
### GET /api/health
|
||||||
@@ -243,6 +317,71 @@ curl -X POST http://localhost:8000/api/tts/roles \
|
|||||||
-d '{"text":"Alice: Hello!\nBob: Hi there!"}'
|
-d '{"text":"Alice: Hello!\nBob: Hi there!"}'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### GET /api/tts/engines
|
||||||
|
|
||||||
|
Get list of available TTS engines and current selection.
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"engines": [
|
||||||
|
{"name": "kokoro", "active": true},
|
||||||
|
{"name": "piper", "active": false}
|
||||||
|
],
|
||||||
|
"current": "kokoro"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8000/api/tts/engines
|
||||||
|
```
|
||||||
|
|
||||||
|
### POST /api/tts/engines
|
||||||
|
|
||||||
|
Set the active TTS engine.
|
||||||
|
|
||||||
|
**Request Body:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"engine": "piper"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "ok",
|
||||||
|
"engine": "piper"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts/engines \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"engine": "piper"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### GET /api/tts/voices
|
||||||
|
|
||||||
|
Get available voices for all TTS engines.
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"voices": {
|
||||||
|
"kokoro": ["af_bella", "af_sarah", "am_adam", "am_michael", ...],
|
||||||
|
"piper": ["en_US-lessac-medium", "en_US-amy-medium", ...]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8000/api/tts/voices
|
||||||
|
```
|
||||||
|
|
||||||
### POST /api/tts
|
### POST /api/tts
|
||||||
|
|
||||||
Generate multi-voice audiobook with AI-powered text segmentation.
|
Generate multi-voice audiobook with AI-powered text segmentation.
|
||||||
@@ -271,6 +410,7 @@ When Ollama is available, the system uses AI to intelligently split text and ass
|
|||||||
- `lang` (required): Language code (a=American English, b=British English, f=French, j=Japanese)
|
- `lang` (required): Language code (a=American English, b=British English, f=French, j=Japanese)
|
||||||
- `roles` (required): Array of role-to-voice mappings
|
- `roles` (required): Array of role-to-voice mappings
|
||||||
- `speed` (optional): Playback speed (0.5-1.5, default: 1.0)
|
- `speed` (optional): Playback speed (0.5-1.5, default: 1.0)
|
||||||
|
- `pitch_adjustments` (optional): Array of role-to-pitch mappings (Piper only, -12 to +12 semitones)
|
||||||
|
|
||||||
**Response:**
|
**Response:**
|
||||||
- Content-Type: `audio/wav`
|
- Content-Type: `audio/wav`
|
||||||
@@ -285,7 +425,7 @@ When Ollama is available, the system uses AI to intelligently split text and ass
|
|||||||
- Content-Disposition: `attachment; filename="audiobook.wav"`
|
- Content-Disposition: `attachment; filename="audiobook.wav"`
|
||||||
- Body: Audio file bytes
|
- Body: Audio file bytes
|
||||||
|
|
||||||
**Example:**
|
**Example (Kokoro):**
|
||||||
```bash
|
```bash
|
||||||
curl -X POST http://localhost:8000/api/tts \
|
curl -X POST http://localhost:8000/api/tts \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
@@ -301,6 +441,26 @@ curl -X POST http://localhost:8000/api/tts \
|
|||||||
--output audiobook.wav
|
--output audiobook.wav
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Example (Piper with pitch control):**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"text": "Alice: Hello!\nBob: Hi!",
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "en_US-amy-medium"},
|
||||||
|
{"id": "Bob", "voice": "en_US-lessac-medium"}
|
||||||
|
],
|
||||||
|
"speed": 1.0,
|
||||||
|
"pitch_adjustments": [
|
||||||
|
{"id": "Alice", "pitch": 3},
|
||||||
|
{"id": "Bob", "pitch": -2}
|
||||||
|
]
|
||||||
|
}' \
|
||||||
|
--output audiobook.wav
|
||||||
|
```
|
||||||
|
|
||||||
## Kokoro TTS Configuration
|
## Kokoro TTS Configuration
|
||||||
|
|
||||||
### Supported Languages
|
### Supported Languages
|
||||||
|
|||||||
219
backend/download_piper_voices.py
Executable file
219
backend/download_piper_voices.py
Executable file
@@ -0,0 +1,219 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Simple script to download Piper TTS voice models.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import urllib.request
|
||||||
|
import ssl
|
||||||
|
|
||||||
|
# Create SSL context that doesn't verify (for development)
|
||||||
|
ssl_context = ssl.create_default_context()
|
||||||
|
ssl_context.check_hostname = False
|
||||||
|
ssl_context.verify_mode = ssl.CERT_NONE
|
||||||
|
|
||||||
|
# Models directory
|
||||||
|
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
MODELS_DIR = os.path.join(SCRIPT_DIR, "piper_models")
|
||||||
|
os.makedirs(MODELS_DIR, exist_ok=True)
|
||||||
|
|
||||||
|
# Available voices with direct download links
|
||||||
|
VOICES = {
|
||||||
|
"1": {
|
||||||
|
"name": "en_US-lessac-medium",
|
||||||
|
"description": "Male, American, Medium Quality",
|
||||||
|
"model_url": "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/en_US-lessac-medium.onnx",
|
||||||
|
"config_url": "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/en_US-lessac-medium.onnx.json",
|
||||||
|
},
|
||||||
|
"2": {
|
||||||
|
"name": "en_US-amy-medium",
|
||||||
|
"description": "Female, American, Medium Quality",
|
||||||
|
"model_url": "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/en_US-amy-medium.onnx",
|
||||||
|
"config_url": "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/en_US-amy-medium.onnx.json",
|
||||||
|
},
|
||||||
|
"3": {
|
||||||
|
"name": "en_US-libritts-high",
|
||||||
|
"description": "Neutral, American, High Quality",
|
||||||
|
"model_url": "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/en_US-libritts-high.onnx",
|
||||||
|
"config_url": "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/en_US-libritts-high.onnx.json",
|
||||||
|
},
|
||||||
|
"4": {
|
||||||
|
"name": "en_GB-alan-medium",
|
||||||
|
"description": "Male, British, Medium Quality",
|
||||||
|
"model_url": "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_GB/en_GB-alan-medium.onnx",
|
||||||
|
"config_url": "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_GB/en_GB-alan-medium.onnx.json",
|
||||||
|
},
|
||||||
|
"5": {
|
||||||
|
"name": "en_GB-alba-medium",
|
||||||
|
"description": "Female, British, Medium Quality",
|
||||||
|
"model_url": "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_GB/en_GB-alba-medium.onnx",
|
||||||
|
"config_url": "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_GB/en_GB-alba-medium.onnx.json",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def download_file(url, dest_path):
|
||||||
|
"""Download a file with progress indication."""
|
||||||
|
try:
|
||||||
|
print(f" Downloading from: {url}")
|
||||||
|
|
||||||
|
# Use urllib with SSL context
|
||||||
|
with urllib.request.urlopen(url, context=ssl_context, timeout=120) as response:
|
||||||
|
total_size = int(response.headers.get('content-length', 0))
|
||||||
|
|
||||||
|
with open(dest_path, 'wb') as f:
|
||||||
|
downloaded = 0
|
||||||
|
chunk_size = 8192
|
||||||
|
|
||||||
|
while True:
|
||||||
|
chunk = response.read(chunk_size)
|
||||||
|
if not chunk:
|
||||||
|
break
|
||||||
|
|
||||||
|
f.write(chunk)
|
||||||
|
downloaded += len(chunk)
|
||||||
|
|
||||||
|
if total_size > 0:
|
||||||
|
percent = (downloaded / total_size) * 100
|
||||||
|
print(f"\r Progress: {percent:.1f}% ({downloaded}/{total_size} bytes)", end='')
|
||||||
|
|
||||||
|
print() # New line after progress
|
||||||
|
|
||||||
|
# Verify file size
|
||||||
|
file_size = os.path.getsize(dest_path)
|
||||||
|
if file_size < 1000:
|
||||||
|
print(f" ✗ Download failed (file too small: {file_size} bytes)")
|
||||||
|
os.remove(dest_path)
|
||||||
|
return False
|
||||||
|
|
||||||
|
print(f" ✓ Downloaded: {file_size:,} bytes")
|
||||||
|
return True
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f" ✗ Error: {e}")
|
||||||
|
if os.path.exists(dest_path):
|
||||||
|
os.remove(dest_path)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def download_voice(voice_key):
|
||||||
|
"""Download a voice model and config."""
|
||||||
|
voice = VOICES[voice_key]
|
||||||
|
name = voice["name"]
|
||||||
|
|
||||||
|
model_path = os.path.join(MODELS_DIR, f"{name}.onnx")
|
||||||
|
config_path = os.path.join(MODELS_DIR, f"{name}.onnx.json")
|
||||||
|
|
||||||
|
# Check if already exists
|
||||||
|
if os.path.exists(model_path) and os.path.exists(config_path):
|
||||||
|
print(f" ✓ {name} - Already downloaded")
|
||||||
|
return True
|
||||||
|
|
||||||
|
print(f"\n Downloading: {name}")
|
||||||
|
print(f" Description: {voice['description']}")
|
||||||
|
|
||||||
|
# Download model
|
||||||
|
print(" [1/2] Model file (.onnx)...")
|
||||||
|
if not download_file(voice["model_url"], model_path):
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Download config
|
||||||
|
print(" [2/2] Config file (.onnx.json)...")
|
||||||
|
if not download_file(voice["config_url"], config_path):
|
||||||
|
return False
|
||||||
|
|
||||||
|
print(f" ✓ {name} - Complete!")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
print("=" * 60)
|
||||||
|
print("Piper TTS Voice Model Downloader")
|
||||||
|
print("=" * 60)
|
||||||
|
print(f"\nModels directory: {MODELS_DIR}\n")
|
||||||
|
|
||||||
|
print("Available voices:\n")
|
||||||
|
for key, voice in VOICES.items():
|
||||||
|
print(f" {key}) {voice['name']}")
|
||||||
|
print(f" {voice['description']}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
print("Quick options:")
|
||||||
|
print(" a) Download all voices (recommended)")
|
||||||
|
print(" s) Download starter pack (1, 2, 4)")
|
||||||
|
print(" q) Quit")
|
||||||
|
print()
|
||||||
|
|
||||||
|
choice = input("Enter your choice (1-5, a, s, or q): ").strip().lower()
|
||||||
|
|
||||||
|
if choice == 'q':
|
||||||
|
print("Exiting...")
|
||||||
|
return
|
||||||
|
|
||||||
|
print("\n" + "=" * 60)
|
||||||
|
print("Downloading...")
|
||||||
|
print("=" * 60)
|
||||||
|
|
||||||
|
success_count = 0
|
||||||
|
total_count = 0
|
||||||
|
|
||||||
|
if choice == 'a':
|
||||||
|
# Download all
|
||||||
|
for key in VOICES.keys():
|
||||||
|
total_count += 1
|
||||||
|
if download_voice(key):
|
||||||
|
success_count += 1
|
||||||
|
elif choice == 's':
|
||||||
|
# Starter pack
|
||||||
|
for key in ['1', '2', '4']:
|
||||||
|
total_count += 1
|
||||||
|
if download_voice(key):
|
||||||
|
success_count += 1
|
||||||
|
elif choice in VOICES:
|
||||||
|
# Single voice
|
||||||
|
total_count += 1
|
||||||
|
if download_voice(choice):
|
||||||
|
success_count += 1
|
||||||
|
else:
|
||||||
|
print(f"Invalid choice: {choice}")
|
||||||
|
return
|
||||||
|
|
||||||
|
print("\n" + "=" * 60)
|
||||||
|
print("Download Summary")
|
||||||
|
print("=" * 60)
|
||||||
|
print(f"\nSuccessfully downloaded: {success_count}/{total_count} voices")
|
||||||
|
|
||||||
|
# List downloaded files
|
||||||
|
print(f"\nFiles in {MODELS_DIR}:")
|
||||||
|
try:
|
||||||
|
files = sorted(os.listdir(MODELS_DIR))
|
||||||
|
if files:
|
||||||
|
for f in files:
|
||||||
|
size = os.path.getsize(os.path.join(MODELS_DIR, f))
|
||||||
|
print(f" {f} ({size:,} bytes)")
|
||||||
|
else:
|
||||||
|
print(" (empty)")
|
||||||
|
except Exception as e:
|
||||||
|
print(f" Error listing files: {e}")
|
||||||
|
|
||||||
|
print("\n" + "=" * 60)
|
||||||
|
print("Next Steps")
|
||||||
|
print("=" * 60)
|
||||||
|
print("\n1. Test Piper installation:")
|
||||||
|
print(" python test_piper_direct.py")
|
||||||
|
print("\n2. Start the backend:")
|
||||||
|
print(" uvicorn main:app --reload")
|
||||||
|
print("\n3. In the UI, click 'Piper TTS' button")
|
||||||
|
print("\n4. Select a Piper voice and generate!")
|
||||||
|
print()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
print("\n\nDownload cancelled by user.")
|
||||||
|
sys.exit(1)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"\n\nError: {e}")
|
||||||
|
sys.exit(1)
|
||||||
149
backend/download_piper_voices.sh
Executable file
149
backend/download_piper_voices.sh
Executable file
@@ -0,0 +1,149 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Helper script to download Piper TTS voice models
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "=================================================="
|
||||||
|
echo "Piper TTS Voice Model Downloader"
|
||||||
|
echo "=================================================="
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Create models directory
|
||||||
|
MODELS_DIR="$(dirname "$0")/piper_models"
|
||||||
|
mkdir -p "$MODELS_DIR"
|
||||||
|
echo "✓ Models directory: $MODELS_DIR"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Base URL for Piper voices
|
||||||
|
BASE_URL="https://huggingface.co/rhasspy/piper-voices/resolve/main"
|
||||||
|
|
||||||
|
# Function to download a voice
|
||||||
|
download_voice() {
|
||||||
|
local voice_id=$1
|
||||||
|
local lang_code=$2
|
||||||
|
|
||||||
|
echo "Downloading $voice_id..."
|
||||||
|
|
||||||
|
# Construct URLs
|
||||||
|
local model_url="${BASE_URL}/${lang_code}/${lang_code}_${voice_id}.onnx"
|
||||||
|
local config_url="${BASE_URL}/${lang_code}/${lang_code}_${voice_id}.onnx.json"
|
||||||
|
|
||||||
|
local model_file="${MODELS_DIR}/${lang_code}-${voice_id}.onnx"
|
||||||
|
local config_file="${MODELS_DIR}/${lang_code}-${voice_id}.onnx.json"
|
||||||
|
|
||||||
|
# Check if already exists
|
||||||
|
if [ -f "$model_file" ] && [ -f "$config_file" ]; then
|
||||||
|
echo " ✓ Already downloaded"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Download with curl
|
||||||
|
if command -v curl &> /dev/null; then
|
||||||
|
echo " Downloading model file..."
|
||||||
|
curl -L -o "$model_file" "$model_url" 2>/dev/null || {
|
||||||
|
echo " ✗ Failed to download model"
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
echo " Downloading config file..."
|
||||||
|
curl -L -o "$config_file" "$config_url" 2>/dev/null || {
|
||||||
|
echo " ✗ Failed to download config"
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Verify files are not error pages
|
||||||
|
if [ $(wc -c < "$model_file") -lt 1000 ]; then
|
||||||
|
echo " ✗ Download failed (file too small)"
|
||||||
|
rm -f "$model_file" "$config_file"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo " ✓ Downloaded successfully"
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
echo " ✗ curl not found"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "Select voices to download:"
|
||||||
|
echo ""
|
||||||
|
echo "Recommended starter pack (3 voices):"
|
||||||
|
echo " 1) en_US-lessac-medium (Male, American)"
|
||||||
|
echo " 2) en_US-amy-medium (Female, American)"
|
||||||
|
echo " 3) en_GB-alan-medium (Male, British)"
|
||||||
|
echo ""
|
||||||
|
echo "All English voices (5 voices):"
|
||||||
|
echo " 4) All of the above + 2 more"
|
||||||
|
echo ""
|
||||||
|
echo "Manual selection:"
|
||||||
|
echo " 5) Choose specific voices"
|
||||||
|
echo ""
|
||||||
|
echo " 0) Exit"
|
||||||
|
echo ""
|
||||||
|
read -p "Enter your choice (0-5): " choice
|
||||||
|
|
||||||
|
case $choice in
|
||||||
|
1)
|
||||||
|
echo ""
|
||||||
|
echo "Downloading starter pack..."
|
||||||
|
echo ""
|
||||||
|
download_voice "lessac-medium" "en_US"
|
||||||
|
download_voice "amy-medium" "en_US"
|
||||||
|
download_voice "alan-medium" "en_GB"
|
||||||
|
;;
|
||||||
|
2)
|
||||||
|
echo ""
|
||||||
|
echo "Downloading all English voices..."
|
||||||
|
echo ""
|
||||||
|
download_voice "lessac-medium" "en_US"
|
||||||
|
download_voice "amy-medium" "en_US"
|
||||||
|
download_voice "libritts-high" "en_US"
|
||||||
|
download_voice "alan-medium" "en_GB"
|
||||||
|
download_voice "alba-medium" "en_GB"
|
||||||
|
;;
|
||||||
|
3)
|
||||||
|
echo ""
|
||||||
|
echo "Available voices:"
|
||||||
|
echo " en_US-lessac-medium (Male, American)"
|
||||||
|
echo " en_US-amy-medium (Female, American)"
|
||||||
|
echo " en_US-libritts-high (Neutral, American, High Quality)"
|
||||||
|
echo " en_GB-alan-medium (Male, British)"
|
||||||
|
echo " en_GB-alba-medium (Female, British)"
|
||||||
|
echo ""
|
||||||
|
echo "Enter voice IDs separated by spaces (e.g., lessac-medium amy-medium):"
|
||||||
|
read -p "> " voices
|
||||||
|
|
||||||
|
for voice in $voices; do
|
||||||
|
if [[ $voice == *"GB"* ]]; then
|
||||||
|
download_voice "$voice" "en_GB"
|
||||||
|
else
|
||||||
|
download_voice "$voice" "en_US"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
;;
|
||||||
|
0)
|
||||||
|
echo "Exiting..."
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Invalid choice"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "=================================================="
|
||||||
|
echo "Download Summary"
|
||||||
|
echo "=================================================="
|
||||||
|
echo ""
|
||||||
|
echo "Downloaded files in: $MODELS_DIR"
|
||||||
|
ls -lh "$MODELS_DIR" 2>/dev/null || echo "No files found"
|
||||||
|
echo ""
|
||||||
|
echo "✓ Setup complete!"
|
||||||
|
echo ""
|
||||||
|
echo "Next steps:"
|
||||||
|
echo " 1. Test Piper: python test_piper_direct.py"
|
||||||
|
echo " 2. Start backend: uvicorn main:app --reload"
|
||||||
|
echo " 3. Select 'Piper TTS' in the UI"
|
||||||
|
echo ""
|
||||||
67
backend/install_piper.sh
Executable file
67
backend/install_piper.sh
Executable file
@@ -0,0 +1,67 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Script to install Piper TTS
|
||||||
|
|
||||||
|
echo "Installing Piper TTS..."
|
||||||
|
|
||||||
|
# Detect OS
|
||||||
|
OS="$(uname -s)"
|
||||||
|
ARCH="$(uname -m)"
|
||||||
|
|
||||||
|
case "${OS}" in
|
||||||
|
Linux*)
|
||||||
|
if [ "${ARCH}" = "x86_64" ]; then
|
||||||
|
PIPER_URL="https://github.com/rhasspy/piper/releases/download/v1.2.0/piper_amd64.tar.gz"
|
||||||
|
elif [ "${ARCH}" = "aarch64" ]; then
|
||||||
|
PIPER_URL="https://github.com/rhasspy/piper/releases/download/v1.2.0/piper_arm64.tar.gz"
|
||||||
|
else
|
||||||
|
echo "Unsupported architecture: ${ARCH}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
Darwin*)
|
||||||
|
if [ "${ARCH}" = "x86_64" ] || [ "${ARCH}" = "arm64" ]; then
|
||||||
|
PIPER_URL="https://github.com/rhasspy/piper/releases/download/v1.2.0/piper_macos_x64.tar.gz"
|
||||||
|
else
|
||||||
|
echo "Unsupported architecture: ${ARCH}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Unsupported OS: ${OS}"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# Create temp directory
|
||||||
|
TEMP_DIR=$(mktemp -d)
|
||||||
|
cd "${TEMP_DIR}"
|
||||||
|
|
||||||
|
# Download Piper
|
||||||
|
echo "Downloading Piper from ${PIPER_URL}..."
|
||||||
|
curl -L -o piper.tar.gz "${PIPER_URL}"
|
||||||
|
|
||||||
|
# Extract
|
||||||
|
echo "Extracting..."
|
||||||
|
tar -xzf piper.tar.gz
|
||||||
|
|
||||||
|
# Install to /usr/local/bin (requires sudo)
|
||||||
|
echo "Installing to /usr/local/bin (may require password)..."
|
||||||
|
sudo cp piper/piper /usr/local/bin/
|
||||||
|
sudo chmod +x /usr/local/bin/piper
|
||||||
|
|
||||||
|
# Cleanup
|
||||||
|
cd -
|
||||||
|
rm -rf "${TEMP_DIR}"
|
||||||
|
|
||||||
|
# Verify installation
|
||||||
|
if command -v piper &> /dev/null; then
|
||||||
|
echo "✓ Piper TTS installed successfully!"
|
||||||
|
piper --version
|
||||||
|
else
|
||||||
|
echo "✗ Installation failed"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "To download voice models, they will be automatically downloaded on first use."
|
||||||
|
echo "Or you can manually download from: https://github.com/rhasspy/piper/releases/tag/v1.2.0"
|
||||||
280
backend/main.py
280
backend/main.py
@@ -11,7 +11,7 @@ from pydantic import BaseModel, Field
|
|||||||
|
|
||||||
# Import services
|
# Import services
|
||||||
from services.role_detector import RoleDetector
|
from services.role_detector import RoleDetector
|
||||||
from services.tts_service import TTSService
|
from services.unified_tts_service import UnifiedTTSService
|
||||||
|
|
||||||
# Configure logging
|
# Configure logging
|
||||||
logging.basicConfig(
|
logging.basicConfig(
|
||||||
@@ -54,12 +54,26 @@ class RoleVoiceMapping(BaseModel):
|
|||||||
id: str = Field(..., description="Role identifier")
|
id: str = Field(..., description="Role identifier")
|
||||||
voice: str = Field(..., description="Kokoro voice ID")
|
voice: str = Field(..., description="Kokoro voice ID")
|
||||||
|
|
||||||
|
class RolePitchMapping(BaseModel):
|
||||||
|
"""Model for role-to-pitch mapping."""
|
||||||
|
id: str = Field(..., description="Role identifier")
|
||||||
|
pitch: int = Field(0, description="Pitch shift in semitones (-12 to +12)")
|
||||||
|
|
||||||
class TTSRequest(BaseModel):
|
class TTSRequest(BaseModel):
|
||||||
"""Request model for TTS generation endpoint."""
|
"""Request model for TTS generation endpoint."""
|
||||||
text: str = Field(..., description="Text to convert to speech")
|
segments: Optional[List[ScriptSegment]] = Field(None, description="Optional edited script segments")
|
||||||
|
text: Optional[str] = Field(None, description="Text to convert to speech (used if segments not provided)")
|
||||||
lang: str = Field(..., description="Language code (a, b, f, j)")
|
lang: str = Field(..., description="Language code (a, b, f, j)")
|
||||||
roles: List[RoleVoiceMapping] = Field(..., description="Role to voice mappings")
|
roles: List[RoleVoiceMapping] = Field(..., description="Role to voice mappings")
|
||||||
speed: Optional[float] = Field(1.0, description="Playback speed (0.5-1.5)")
|
speed: Optional[float] = Field(1.0, description="Playback speed (0.5-1.5)")
|
||||||
|
pitch_adjustments: Optional[List[RolePitchMapping]] = Field(None, description="Optional pitch adjustments per role (Piper only)")
|
||||||
|
|
||||||
|
class VoicePreviewRequest(BaseModel):
|
||||||
|
"""Request model for voice preview endpoint."""
|
||||||
|
voice: str = Field(..., description="Kokoro voice ID to preview")
|
||||||
|
lang: str = Field(..., description="Language code (a, b, f, j)")
|
||||||
|
speed: Optional[float] = Field(1.0, description="Playback speed (0.5-1.5)")
|
||||||
|
text: Optional[str] = Field(None, description="Custom text to preview (uses default if not provided)")
|
||||||
|
|
||||||
class ErrorResponse(BaseModel):
|
class ErrorResponse(BaseModel):
|
||||||
"""Standard error response model."""
|
"""Standard error response model."""
|
||||||
@@ -67,6 +81,24 @@ class ErrorResponse(BaseModel):
|
|||||||
message: str = Field(..., description="Human-readable error message")
|
message: str = Field(..., description="Human-readable error message")
|
||||||
details: Optional[dict] = Field(None, description="Additional error details")
|
details: Optional[dict] = Field(None, description="Additional error details")
|
||||||
|
|
||||||
|
class EngineInfo(BaseModel):
|
||||||
|
"""Model for TTS engine information."""
|
||||||
|
name: str = Field(..., description="Engine identifier")
|
||||||
|
active: bool = Field(..., description="Whether this engine is currently active")
|
||||||
|
|
||||||
|
class EnginesResponse(BaseModel):
|
||||||
|
"""Response model for available engines."""
|
||||||
|
engines: List[EngineInfo] = Field(..., description="List of available TTS engines")
|
||||||
|
current: str = Field(..., description="Currently active engine")
|
||||||
|
|
||||||
|
class SetEngineRequest(BaseModel):
|
||||||
|
"""Request model for setting TTS engine."""
|
||||||
|
engine: str = Field(..., description="Engine to activate (kokoro or piper)")
|
||||||
|
|
||||||
|
class VoicesResponse(BaseModel):
|
||||||
|
"""Response model for available voices."""
|
||||||
|
voices: dict = Field(..., description="Dict mapping engine names to voice lists")
|
||||||
|
|
||||||
# Initialize FastAPI app
|
# Initialize FastAPI app
|
||||||
app = FastAPI(
|
app = FastAPI(
|
||||||
title="Text to Audiobook API",
|
title="Text to Audiobook API",
|
||||||
@@ -112,7 +144,7 @@ async def general_exception_handler(request: Request, exc: Exception):
|
|||||||
|
|
||||||
# Initialize services
|
# Initialize services
|
||||||
role_detector = RoleDetector()
|
role_detector = RoleDetector()
|
||||||
tts_service = TTSService()
|
tts_service = UnifiedTTSService()
|
||||||
|
|
||||||
# API Endpoints
|
# API Endpoints
|
||||||
@app.get("/api/health")
|
@app.get("/api/health")
|
||||||
@@ -121,6 +153,80 @@ async def health_check():
|
|||||||
logger.info("Health check requested")
|
logger.info("Health check requested")
|
||||||
return {"status": "ok"}
|
return {"status": "ok"}
|
||||||
|
|
||||||
|
@app.get("/api/tts/engines", response_model=EnginesResponse)
|
||||||
|
async def get_engines():
|
||||||
|
"""
|
||||||
|
Get list of available TTS engines and current selection.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
EnginesResponse with list of engines and current active engine
|
||||||
|
"""
|
||||||
|
logger.info("Engines list requested")
|
||||||
|
|
||||||
|
try:
|
||||||
|
available = tts_service.get_available_engines()
|
||||||
|
current = tts_service.engine_name
|
||||||
|
|
||||||
|
engines = [
|
||||||
|
EngineInfo(name=engine, active=(engine == current))
|
||||||
|
for engine in available
|
||||||
|
]
|
||||||
|
|
||||||
|
return EnginesResponse(engines=engines, current=current)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error getting engines: {str(e)}", exc_info=True)
|
||||||
|
raise
|
||||||
|
|
||||||
|
@app.post("/api/tts/engines")
|
||||||
|
async def set_engine(request: SetEngineRequest):
|
||||||
|
"""
|
||||||
|
Set the active TTS engine.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
request: SetEngineRequest with engine name
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Success message with new engine
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
400: If engine name is invalid
|
||||||
|
"""
|
||||||
|
logger.info(f"Engine change requested: {request.engine}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
tts_service.set_engine(request.engine)
|
||||||
|
return {"status": "ok", "engine": request.engine}
|
||||||
|
except ValueError as e:
|
||||||
|
logger.error(f"Invalid engine: {str(e)}")
|
||||||
|
return JSONResponse(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
content={
|
||||||
|
"error": "ValidationError",
|
||||||
|
"message": str(e),
|
||||||
|
"details": {"field": "engine"}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error setting engine: {str(e)}", exc_info=True)
|
||||||
|
raise
|
||||||
|
|
||||||
|
@app.get("/api/tts/voices", response_model=VoicesResponse)
|
||||||
|
async def get_voices():
|
||||||
|
"""
|
||||||
|
Get available voices for all TTS engines.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
VoicesResponse with dict mapping engine names to voice lists
|
||||||
|
"""
|
||||||
|
logger.info("Voices list requested")
|
||||||
|
|
||||||
|
try:
|
||||||
|
voices = tts_service.get_available_voices()
|
||||||
|
return VoicesResponse(voices=voices)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error getting voices: {str(e)}", exc_info=True)
|
||||||
|
raise
|
||||||
|
|
||||||
@app.post("/api/tts/roles", response_model=RolesResponse)
|
@app.post("/api/tts/roles", response_model=RolesResponse)
|
||||||
async def detect_roles(request: RoleDetectionRequest):
|
async def detect_roles(request: RoleDetectionRequest):
|
||||||
"""
|
"""
|
||||||
@@ -233,6 +339,134 @@ async def generate_script(request: ScriptRequest):
|
|||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@app.post("/api/tts/preview")
|
||||||
|
async def preview_voice(request: VoicePreviewRequest):
|
||||||
|
"""
|
||||||
|
Generates a short audio preview of a voice.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
request: VoicePreviewRequest containing voice ID, language, speed, and optional text
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
StreamingResponse with audio/wav content
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
400: If validation fails
|
||||||
|
500: If audio generation fails
|
||||||
|
"""
|
||||||
|
logger.info(f"Voice preview requested: voice={request.voice}, lang={request.lang}, speed={request.speed}")
|
||||||
|
|
||||||
|
# Validate language code
|
||||||
|
supported_langs = ['a', 'b', 'e', 'f', 'h', 'i', 'j', 'p', 'z']
|
||||||
|
if request.lang not in supported_langs:
|
||||||
|
logger.error(f"Invalid language code: {request.lang}")
|
||||||
|
return JSONResponse(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
content={
|
||||||
|
"error": "ValidationError",
|
||||||
|
"message": f"Invalid language code: {request.lang}",
|
||||||
|
"details": {"field": "lang", "supported": supported_langs}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Validate speed range
|
||||||
|
if request.speed < 0.5 or request.speed > 1.5:
|
||||||
|
logger.error(f"Invalid speed value: {request.speed}")
|
||||||
|
return JSONResponse(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
content={
|
||||||
|
"error": "ValidationError",
|
||||||
|
"message": "Speed must be between 0.5 and 1.5",
|
||||||
|
"details": {"field": "speed", "value": request.speed, "range": [0.5, 1.5]}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Use provided text or default preview text
|
||||||
|
preview_text = request.text or "Hello! This is a preview of this voice. How does it sound?"
|
||||||
|
|
||||||
|
try:
|
||||||
|
import io
|
||||||
|
import soundfile as sf
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
# Check which engine is active
|
||||||
|
if tts_service.engine_name == 'kokoro':
|
||||||
|
# Use Kokoro service
|
||||||
|
pipeline = tts_service.kokoro_service.get_pipeline(request.lang)
|
||||||
|
audio = tts_service.kokoro_service.synthesize_segment(
|
||||||
|
pipeline=pipeline,
|
||||||
|
text=preview_text,
|
||||||
|
voice=request.voice,
|
||||||
|
speed=request.speed
|
||||||
|
)
|
||||||
|
sample_rate = tts_service.kokoro_service.SAMPLE_RATE
|
||||||
|
|
||||||
|
elif tts_service.engine_name == 'piper':
|
||||||
|
# Use Piper service with pitch control
|
||||||
|
audio = tts_service.piper_service.synthesize_with_pitch(
|
||||||
|
text=preview_text,
|
||||||
|
voice=request.voice,
|
||||||
|
speed=request.speed,
|
||||||
|
pitch_shift=0 # No pitch shift for preview
|
||||||
|
)
|
||||||
|
sample_rate = tts_service.piper_service.SAMPLE_RATE
|
||||||
|
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown engine: {tts_service.engine_name}")
|
||||||
|
|
||||||
|
# Check if audio was generated
|
||||||
|
if audio is None or len(audio) == 0:
|
||||||
|
raise ValueError("No audio generated - voice model may be missing")
|
||||||
|
|
||||||
|
# Convert to WAV bytes
|
||||||
|
buffer = io.BytesIO()
|
||||||
|
sf.write(buffer, audio, sample_rate, format='WAV')
|
||||||
|
buffer.seek(0)
|
||||||
|
audio_bytes = buffer.read()
|
||||||
|
|
||||||
|
logger.info(f"Generated voice preview ({tts_service.engine_name}): {len(audio_bytes)} bytes")
|
||||||
|
|
||||||
|
return StreamingResponse(
|
||||||
|
iter([audio_bytes]),
|
||||||
|
media_type="audio/wav",
|
||||||
|
headers={"Content-Disposition": "inline; filename=preview.wav"}
|
||||||
|
)
|
||||||
|
|
||||||
|
except FileNotFoundError as e:
|
||||||
|
logger.error(f"Voice model not found: {str(e)}")
|
||||||
|
return JSONResponse(
|
||||||
|
status_code=status.HTTP_404_NOT_FOUND,
|
||||||
|
content={
|
||||||
|
"error": "VoiceModelNotFound",
|
||||||
|
"message": f"Voice model '{request.voice}' not found. Please download Piper voice models manually.",
|
||||||
|
"details": {
|
||||||
|
"voice": request.voice,
|
||||||
|
"engine": tts_service.engine_name,
|
||||||
|
"setup_guide": "See backend/PIPER_SETUP.md for instructions"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
except ValueError as e:
|
||||||
|
logger.error(f"Validation error during preview: {str(e)}")
|
||||||
|
return JSONResponse(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
content={
|
||||||
|
"error": "ValidationError",
|
||||||
|
"message": str(e),
|
||||||
|
"details": {"voice": request.voice, "engine": tts_service.engine_name}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error during voice preview: {str(e)}", exc_info=True)
|
||||||
|
return JSONResponse(
|
||||||
|
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||||
|
content={
|
||||||
|
"error": "PreviewGenerationError",
|
||||||
|
"message": "Failed to generate voice preview",
|
||||||
|
"details": {"error": str(e)}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
@app.post("/api/tts")
|
@app.post("/api/tts")
|
||||||
async def generate_audiobook(request: TTSRequest):
|
async def generate_audiobook(request: TTSRequest):
|
||||||
"""
|
"""
|
||||||
@@ -253,11 +487,24 @@ async def generate_audiobook(request: TTSRequest):
|
|||||||
"""
|
"""
|
||||||
logger.info(
|
logger.info(
|
||||||
f"Audio generation requested: lang={request.lang}, "
|
f"Audio generation requested: lang={request.lang}, "
|
||||||
f"roles={len(request.roles)}, speed={request.speed}"
|
f"roles={len(request.roles)}, speed={request.speed}, "
|
||||||
|
f"segments={len(request.segments) if request.segments else 0}"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Validate required fields (Pydantic handles basic validation)
|
# Validate that either segments or text is provided
|
||||||
if not request.text or not request.text.strip():
|
if not request.segments and not request.text:
|
||||||
|
logger.error("Neither segments nor text provided")
|
||||||
|
return JSONResponse(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
content={
|
||||||
|
"error": "ValidationError",
|
||||||
|
"message": "Either segments or text must be provided",
|
||||||
|
"details": {"fields": ["segments", "text"]}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Validate text if provided and no segments
|
||||||
|
if not request.segments and request.text and not request.text.strip():
|
||||||
logger.error("Empty text provided")
|
logger.error("Empty text provided")
|
||||||
return JSONResponse(
|
return JSONResponse(
|
||||||
status_code=status.HTTP_400_BAD_REQUEST,
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
@@ -315,13 +562,30 @@ async def generate_audiobook(request: TTSRequest):
|
|||||||
# Convert roles list to dictionary for TTS service
|
# Convert roles list to dictionary for TTS service
|
||||||
role_voice_map = {role.id: role.voice for role in request.roles}
|
role_voice_map = {role.id: role.voice for role in request.roles}
|
||||||
|
|
||||||
|
# Convert pitch adjustments to dictionary if provided
|
||||||
|
pitch_map = None
|
||||||
|
if request.pitch_adjustments:
|
||||||
|
pitch_map = {p.id: p.pitch for p in request.pitch_adjustments}
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Generate audiobook
|
# Generate audiobook using segments if provided, otherwise use text
|
||||||
|
if request.segments:
|
||||||
|
logger.info(f"Generating audiobook from {len(request.segments)} edited segments")
|
||||||
|
audio_bytes = await tts_service.generate_audiobook_from_segments(
|
||||||
|
segments=[(seg.speaker, seg.text) for seg in request.segments],
|
||||||
|
lang=request.lang,
|
||||||
|
role_voice_map=role_voice_map,
|
||||||
|
speed=request.speed,
|
||||||
|
pitch_adjustments=pitch_map
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
logger.info("Generating audiobook from text with automatic segmentation")
|
||||||
audio_bytes = await tts_service.generate_audiobook(
|
audio_bytes = await tts_service.generate_audiobook(
|
||||||
text=request.text,
|
text=request.text,
|
||||||
lang=request.lang,
|
lang=request.lang,
|
||||||
role_voice_map=role_voice_map,
|
role_voice_map=role_voice_map,
|
||||||
speed=request.speed
|
speed=request.speed,
|
||||||
|
pitch_adjustments=pitch_map
|
||||||
)
|
)
|
||||||
|
|
||||||
# Create streaming response with proper headers
|
# Create streaming response with proper headers
|
||||||
|
|||||||
1
backend/piper_models/en_US-lessac-medium.onnx
Normal file
1
backend/piper_models/en_US-lessac-medium.onnx
Normal file
@@ -0,0 +1 @@
|
|||||||
|
Entry not found
|
||||||
1
backend/piper_models/en_US-lessac-medium.onnx.json
Normal file
1
backend/piper_models/en_US-lessac-medium.onnx.json
Normal file
@@ -0,0 +1 @@
|
|||||||
|
Entry not found
|
||||||
@@ -6,3 +6,5 @@ python-multipart==0.0.6
|
|||||||
pydantic==2.5.3
|
pydantic==2.5.3
|
||||||
numpy==1.26.3
|
numpy==1.26.3
|
||||||
httpx==0.26.0
|
httpx==0.26.0
|
||||||
|
piper-tts==1.2.0
|
||||||
|
pydub==0.25.1
|
||||||
|
|||||||
@@ -146,19 +146,26 @@ class OllamaService:
|
|||||||
Returns:
|
Returns:
|
||||||
Formatted prompt string
|
Formatted prompt string
|
||||||
"""
|
"""
|
||||||
prompt = f"""Analyze the following text and identify all distinct speakers or characters.
|
prompt = f"""Analyze the following text and identify all distinct speakers or characters who need voice assignments.
|
||||||
Return ONLY a JSON object with a "speakers" array containing the names of speakers.
|
|
||||||
If there are no clear speakers, return {{"speakers": ["Narrator"]}}.
|
CRITICAL: Look for TWO types of content:
|
||||||
|
1. NARRATIVE TEXT: Descriptions, scene-setting, "he said", "she thought", etc. → This needs "Narrator"
|
||||||
|
2. CHARACTER DIALOGUE: Direct speech by characters → These need character names
|
||||||
|
|
||||||
|
You MUST include "Narrator" if there is ANY narrative text, descriptions, or non-dialogue content.
|
||||||
|
|
||||||
|
Return ONLY a JSON object with a "speakers" array.
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
- For dialogue with character names, list those characters
|
- Text: "Alice said hello. Bob waved back." → {{"speakers": ["Narrator", "Alice", "Bob"]}}
|
||||||
- For narrative text, use "Narrator"
|
- Text: "Once upon a time, there was a princess." → {{"speakers": ["Narrator"]}}
|
||||||
- For mixed content, include both characters and "Narrator" if appropriate
|
- Text: "The wizard spoke. 'I see danger,' he warned." → {{"speakers": ["Narrator"]}}
|
||||||
|
- Text: "Alice: Hello! Bob: Hi there!" → {{"speakers": ["Alice", "Bob"]}}
|
||||||
|
|
||||||
Text to analyze:
|
Text to analyze:
|
||||||
{text}
|
{text}
|
||||||
|
|
||||||
Return format: {{"speakers": ["Speaker1", "Speaker2", ...]}}
|
Return format: {{"speakers": ["Narrator", "Character1", "Character2", ...]}}
|
||||||
"""
|
"""
|
||||||
return prompt
|
return prompt
|
||||||
|
|
||||||
|
|||||||
275
backend/services/piper_tts_service.py
Normal file
275
backend/services/piper_tts_service.py
Normal file
@@ -0,0 +1,275 @@
|
|||||||
|
"""
|
||||||
|
Piper TTS service for generating multi-voice audiobooks.
|
||||||
|
"""
|
||||||
|
import logging
|
||||||
|
import io
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
from typing import Dict, List, Tuple, Optional
|
||||||
|
import numpy as np
|
||||||
|
import soundfile as sf
|
||||||
|
from pydub import AudioSegment
|
||||||
|
from pydub.effects import speedup
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class PiperTTSService:
|
||||||
|
"""
|
||||||
|
Service for text-to-speech synthesis using Piper TTS.
|
||||||
|
|
||||||
|
Piper is a fast, local neural TTS system that supports multiple voices
|
||||||
|
and allows pitch modification through audio processing.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Default voice for unmapped roles
|
||||||
|
DEFAULT_VOICE = "en_US-lessac-medium"
|
||||||
|
|
||||||
|
# Sample rate for Piper TTS (typically 22050)
|
||||||
|
SAMPLE_RATE = 22050
|
||||||
|
|
||||||
|
# Available Piper voices with their characteristics
|
||||||
|
AVAILABLE_VOICES = {
|
||||||
|
# English voices
|
||||||
|
"en_US-lessac-medium": {"lang": "en", "gender": "male", "quality": "medium"},
|
||||||
|
"en_US-libritts-high": {"lang": "en", "gender": "neutral", "quality": "high"},
|
||||||
|
"en_US-amy-medium": {"lang": "en", "gender": "female", "quality": "medium"},
|
||||||
|
"en_GB-alan-medium": {"lang": "en", "gender": "male", "quality": "medium"},
|
||||||
|
"en_GB-alba-medium": {"lang": "en", "gender": "female", "quality": "medium"},
|
||||||
|
}
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
"""Initializes Piper TTS service."""
|
||||||
|
self.models_dir = os.path.join(os.path.dirname(__file__), "..", "piper_models")
|
||||||
|
os.makedirs(self.models_dir, exist_ok=True)
|
||||||
|
logger.info("PiperTTSService initialized")
|
||||||
|
|
||||||
|
def _check_piper_installed(self) -> bool:
|
||||||
|
"""Check if Piper is installed and accessible."""
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
["piper", "--version"],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=5
|
||||||
|
)
|
||||||
|
return result.returncode == 0
|
||||||
|
except (subprocess.SubprocessError, FileNotFoundError):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _download_voice_model(self, voice: str) -> bool:
|
||||||
|
"""
|
||||||
|
Download a Piper voice model if not already present.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
voice: Voice ID to download
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True if model is available, False otherwise
|
||||||
|
"""
|
||||||
|
model_path = os.path.join(self.models_dir, f"{voice}.onnx")
|
||||||
|
config_path = os.path.join(self.models_dir, f"{voice}.onnx.json")
|
||||||
|
|
||||||
|
# Check if model already exists
|
||||||
|
if os.path.exists(model_path) and os.path.exists(config_path):
|
||||||
|
logger.info(f"Voice model {voice} already exists")
|
||||||
|
return True
|
||||||
|
|
||||||
|
logger.info(f"Downloading Piper voice model: {voice}")
|
||||||
|
logger.info(f"Models directory: {self.models_dir}")
|
||||||
|
|
||||||
|
# Try to download using wget/curl as Piper's download might not work
|
||||||
|
try:
|
||||||
|
# Piper models are hosted on HuggingFace
|
||||||
|
base_url = "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0"
|
||||||
|
|
||||||
|
# Determine language prefix from voice ID
|
||||||
|
lang_prefix = voice.split('-')[0] # e.g., "en_US" from "en_US-lessac-medium"
|
||||||
|
|
||||||
|
model_url = f"{base_url}/{lang_prefix}/{voice}.onnx"
|
||||||
|
config_url = f"{base_url}/{lang_prefix}/{voice}.onnx.json"
|
||||||
|
|
||||||
|
logger.info(f"Downloading from: {model_url}")
|
||||||
|
|
||||||
|
# Try using curl/wget first (more reliable)
|
||||||
|
try:
|
||||||
|
# Try curl
|
||||||
|
subprocess.run(
|
||||||
|
["curl", "-L", "-o", model_path, model_url],
|
||||||
|
check=True,
|
||||||
|
timeout=120,
|
||||||
|
capture_output=True
|
||||||
|
)
|
||||||
|
subprocess.run(
|
||||||
|
["curl", "-L", "-o", config_path, config_url],
|
||||||
|
check=True,
|
||||||
|
timeout=120,
|
||||||
|
capture_output=True
|
||||||
|
)
|
||||||
|
except (subprocess.SubprocessError, FileNotFoundError):
|
||||||
|
# Fall back to urllib with SSL context
|
||||||
|
import urllib.request
|
||||||
|
import ssl
|
||||||
|
|
||||||
|
# Create SSL context that doesn't verify certificates (for development)
|
||||||
|
ssl_context = ssl.create_default_context()
|
||||||
|
ssl_context.check_hostname = False
|
||||||
|
ssl_context.verify_mode = ssl.CERT_NONE
|
||||||
|
|
||||||
|
urllib.request.urlretrieve(model_url, model_path, context=ssl_context)
|
||||||
|
urllib.request.urlretrieve(config_url, config_path, context=ssl_context)
|
||||||
|
|
||||||
|
if os.path.exists(model_path) and os.path.exists(config_path):
|
||||||
|
logger.info(f"Successfully downloaded voice model: {voice}")
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
logger.error(f"Download completed but files not found")
|
||||||
|
return False
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to download voice model {voice}: {e}")
|
||||||
|
logger.error(f"Please manually download from: https://github.com/rhasspy/piper/releases/tag/v1.2.0")
|
||||||
|
logger.error(f"See backend/PIPER_SETUP.md for detailed instructions")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def synthesize_with_pitch(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
voice: str,
|
||||||
|
speed: float = 1.0,
|
||||||
|
pitch_shift: int = 0
|
||||||
|
) -> np.ndarray:
|
||||||
|
"""
|
||||||
|
Synthesizes text with Piper and applies pitch shifting.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Text to synthesize
|
||||||
|
voice: Piper voice ID
|
||||||
|
speed: Playback speed (0.5-2.0)
|
||||||
|
pitch_shift: Semitones to shift pitch (-12 to +12)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Audio samples as numpy array
|
||||||
|
"""
|
||||||
|
# Ensure voice model is available
|
||||||
|
if not self._download_voice_model(voice):
|
||||||
|
error_msg = (
|
||||||
|
f"Piper voice model '{voice}' not found. "
|
||||||
|
f"Please download voice models manually. "
|
||||||
|
f"See backend/PIPER_SETUP.md for instructions."
|
||||||
|
)
|
||||||
|
logger.error(error_msg)
|
||||||
|
raise FileNotFoundError(error_msg)
|
||||||
|
|
||||||
|
model_path = os.path.join(self.models_dir, f"{voice}.onnx")
|
||||||
|
|
||||||
|
# Check if model file exists
|
||||||
|
if not os.path.exists(model_path):
|
||||||
|
logger.error(f"Model file not found: {model_path}")
|
||||||
|
return np.array([], dtype=np.float32)
|
||||||
|
|
||||||
|
# Generate audio with Piper
|
||||||
|
try:
|
||||||
|
logger.info(f"Running Piper with model: {model_path}")
|
||||||
|
|
||||||
|
# Run piper command
|
||||||
|
process = subprocess.Popen(
|
||||||
|
["piper", "--model", model_path, "--output-raw"],
|
||||||
|
stdin=subprocess.PIPE,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE
|
||||||
|
)
|
||||||
|
|
||||||
|
# Send text and get audio
|
||||||
|
stdout, stderr = process.communicate(input=text.encode('utf-8'), timeout=30)
|
||||||
|
|
||||||
|
if process.returncode != 0:
|
||||||
|
error_msg = stderr.decode() if stderr else "Unknown error"
|
||||||
|
logger.error(f"Piper synthesis failed (exit code {process.returncode}): {error_msg}")
|
||||||
|
return np.array([], dtype=np.float32)
|
||||||
|
|
||||||
|
# Convert raw audio to numpy array
|
||||||
|
audio_data = np.frombuffer(stdout, dtype=np.int16)
|
||||||
|
audio_float = audio_data.astype(np.float32) / 32768.0
|
||||||
|
|
||||||
|
# Apply pitch shift and speed change using pydub
|
||||||
|
if pitch_shift != 0 or speed != 1.0:
|
||||||
|
audio_float = self._apply_audio_effects(
|
||||||
|
audio_float,
|
||||||
|
self.SAMPLE_RATE,
|
||||||
|
speed,
|
||||||
|
pitch_shift
|
||||||
|
)
|
||||||
|
|
||||||
|
return audio_float
|
||||||
|
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
logger.error("Piper synthesis timed out")
|
||||||
|
return np.array([], dtype=np.float32)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error in Piper synthesis: {e}")
|
||||||
|
return np.array([], dtype=np.float32)
|
||||||
|
|
||||||
|
def _apply_audio_effects(
|
||||||
|
self,
|
||||||
|
audio: np.ndarray,
|
||||||
|
sample_rate: int,
|
||||||
|
speed: float,
|
||||||
|
pitch_shift: int
|
||||||
|
) -> np.ndarray:
|
||||||
|
"""
|
||||||
|
Apply speed and pitch effects to audio using pydub.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
audio: Audio samples
|
||||||
|
sample_rate: Sample rate
|
||||||
|
speed: Speed multiplier
|
||||||
|
pitch_shift: Semitones to shift
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Processed audio samples
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Convert numpy to pydub AudioSegment
|
||||||
|
audio_int16 = (audio * 32768.0).astype(np.int16)
|
||||||
|
audio_bytes = audio_int16.tobytes()
|
||||||
|
|
||||||
|
audio_segment = AudioSegment(
|
||||||
|
audio_bytes,
|
||||||
|
frame_rate=sample_rate,
|
||||||
|
sample_width=2,
|
||||||
|
channels=1
|
||||||
|
)
|
||||||
|
|
||||||
|
# Apply pitch shift
|
||||||
|
if pitch_shift != 0:
|
||||||
|
# Pitch shift by changing frame rate then resampling
|
||||||
|
new_sample_rate = int(sample_rate * (2.0 ** (pitch_shift / 12.0)))
|
||||||
|
audio_segment = audio_segment._spawn(
|
||||||
|
audio_segment.raw_data,
|
||||||
|
overrides={'frame_rate': new_sample_rate}
|
||||||
|
)
|
||||||
|
audio_segment = audio_segment.set_frame_rate(sample_rate)
|
||||||
|
|
||||||
|
# Apply speed change
|
||||||
|
if speed != 1.0:
|
||||||
|
audio_segment = speedup(audio_segment, playback_speed=speed)
|
||||||
|
|
||||||
|
# Convert back to numpy
|
||||||
|
samples = np.array(audio_segment.get_array_of_samples())
|
||||||
|
audio_float = samples.astype(np.float32) / 32768.0
|
||||||
|
|
||||||
|
return audio_float
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error applying audio effects: {e}")
|
||||||
|
return audio
|
||||||
|
|
||||||
|
def get_available_voices(self) -> Dict[str, Dict[str, str]]:
|
||||||
|
"""
|
||||||
|
Returns dictionary of available Piper voices.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict mapping voice IDs to their characteristics
|
||||||
|
"""
|
||||||
|
return self.AVAILABLE_VOICES.copy()
|
||||||
@@ -404,3 +404,66 @@ class TTSService:
|
|||||||
|
|
||||||
logger.info(f"Generated audiobook: {len(audio_bytes)} bytes")
|
logger.info(f"Generated audiobook: {len(audio_bytes)} bytes")
|
||||||
return audio_bytes
|
return audio_bytes
|
||||||
|
|
||||||
|
async def generate_audiobook_from_segments(
|
||||||
|
self,
|
||||||
|
segments: List[Tuple[str, str]],
|
||||||
|
lang: str,
|
||||||
|
role_voice_map: Dict[str, str],
|
||||||
|
speed: float = 1.0
|
||||||
|
) -> bytes:
|
||||||
|
"""
|
||||||
|
Generates audiobook from pre-segmented script with speaker assignments.
|
||||||
|
|
||||||
|
This method is used when the user has reviewed and edited the script,
|
||||||
|
bypassing automatic segmentation.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
segments: List of (speaker_name, text_content) tuples
|
||||||
|
lang: Language code
|
||||||
|
role_voice_map: Role to voice ID mapping
|
||||||
|
speed: Playback speed
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Audio data as bytes (WAV format)
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: If segments is empty or invalid
|
||||||
|
"""
|
||||||
|
logger.info(
|
||||||
|
f"Generating audiobook from {len(segments)} edited segments: "
|
||||||
|
f"lang={lang}, speed={speed}, roles={len(role_voice_map)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
if not segments:
|
||||||
|
raise ValueError("No segments provided for synthesis")
|
||||||
|
|
||||||
|
# Get or create pipeline for language
|
||||||
|
pipeline = self.get_pipeline(lang)
|
||||||
|
|
||||||
|
# Convert segments to format with voice assignments
|
||||||
|
segments_with_voice = []
|
||||||
|
for speaker, text in segments:
|
||||||
|
voice = role_voice_map.get(speaker, self.DEFAULT_VOICE)
|
||||||
|
segments_with_voice.append((speaker, voice, text))
|
||||||
|
|
||||||
|
logger.info(f"Processing {len(segments_with_voice)} segments with voice assignments")
|
||||||
|
|
||||||
|
# Synthesize each segment
|
||||||
|
audio_segments = []
|
||||||
|
for role_id, voice, content in segments_with_voice:
|
||||||
|
logger.debug(f"Synthesizing segment for role '{role_id}' with voice '{voice}'")
|
||||||
|
audio = self.synthesize_segment(pipeline, content, voice, speed)
|
||||||
|
audio_segments.append(audio)
|
||||||
|
|
||||||
|
# Concatenate all audio
|
||||||
|
final_audio = self.concatenate_audio(audio_segments)
|
||||||
|
|
||||||
|
# Convert to bytes (WAV format)
|
||||||
|
buffer = io.BytesIO()
|
||||||
|
sf.write(buffer, final_audio, self.SAMPLE_RATE, format='WAV')
|
||||||
|
buffer.seek(0)
|
||||||
|
audio_bytes = buffer.read()
|
||||||
|
|
||||||
|
logger.info(f"Generated audiobook from edited segments: {len(audio_bytes)} bytes")
|
||||||
|
return audio_bytes
|
||||||
|
|||||||
283
backend/services/unified_tts_service.py
Normal file
283
backend/services/unified_tts_service.py
Normal file
@@ -0,0 +1,283 @@
|
|||||||
|
"""
|
||||||
|
Unified TTS service that supports multiple TTS engines (Kokoro, Piper).
|
||||||
|
"""
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
from typing import Dict, List, Tuple, Optional
|
||||||
|
from enum import Enum
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class TTSEngine(str, Enum):
|
||||||
|
"""Available TTS engines."""
|
||||||
|
KOKORO = "kokoro"
|
||||||
|
PIPER = "piper"
|
||||||
|
|
||||||
|
|
||||||
|
class UnifiedTTSService:
|
||||||
|
"""
|
||||||
|
Unified service that delegates to specific TTS engines.
|
||||||
|
Allows runtime selection between Kokoro and Piper TTS.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, engine: Optional[str] = None, ollama_service=None):
|
||||||
|
"""
|
||||||
|
Initialize unified TTS service.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
engine: TTS engine to use ('kokoro' or 'piper').
|
||||||
|
Defaults to TTS_ENGINE env var or 'kokoro'
|
||||||
|
ollama_service: Optional OllamaService for AI segmentation
|
||||||
|
"""
|
||||||
|
self.engine_name = engine or os.getenv("TTS_ENGINE", "kokoro")
|
||||||
|
self.ollama_service = ollama_service
|
||||||
|
|
||||||
|
# Lazy load engines
|
||||||
|
self._kokoro_service = None
|
||||||
|
self._piper_service = None
|
||||||
|
|
||||||
|
logger.info(f"UnifiedTTSService initialized with engine: {self.engine_name}")
|
||||||
|
|
||||||
|
@property
|
||||||
|
def kokoro_service(self):
|
||||||
|
"""Lazy load Kokoro service."""
|
||||||
|
if self._kokoro_service is None:
|
||||||
|
from .tts_service import TTSService
|
||||||
|
self._kokoro_service = TTSService(self.ollama_service)
|
||||||
|
return self._kokoro_service
|
||||||
|
|
||||||
|
@property
|
||||||
|
def piper_service(self):
|
||||||
|
"""Lazy load Piper service."""
|
||||||
|
if self._piper_service is None:
|
||||||
|
from .piper_tts_service import PiperTTSService
|
||||||
|
self._piper_service = PiperTTSService()
|
||||||
|
return self._piper_service
|
||||||
|
|
||||||
|
def set_engine(self, engine: str):
|
||||||
|
"""
|
||||||
|
Change the active TTS engine.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
engine: Engine name ('kokoro' or 'piper')
|
||||||
|
"""
|
||||||
|
if engine not in [e.value for e in TTSEngine]:
|
||||||
|
raise ValueError(f"Unknown engine: {engine}. Use 'kokoro' or 'piper'")
|
||||||
|
|
||||||
|
self.engine_name = engine
|
||||||
|
logger.info(f"Switched TTS engine to: {engine}")
|
||||||
|
|
||||||
|
def get_available_engines(self) -> List[str]:
|
||||||
|
"""
|
||||||
|
Get list of available TTS engines.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of engine names
|
||||||
|
"""
|
||||||
|
return [e.value for e in TTSEngine]
|
||||||
|
|
||||||
|
def get_available_voices(self) -> Dict[str, List[str]]:
|
||||||
|
"""
|
||||||
|
Get available voices for all engines.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict mapping engine names to lists of voice IDs
|
||||||
|
"""
|
||||||
|
voices = {}
|
||||||
|
|
||||||
|
# Kokoro voices (from documentation)
|
||||||
|
voices["kokoro"] = [
|
||||||
|
"af_bella", "af_nicole", "af_sarah", "af_sky",
|
||||||
|
"am_adam", "am_michael", "bf_emma", "bf_isabella",
|
||||||
|
"bm_george", "bm_lewis"
|
||||||
|
]
|
||||||
|
|
||||||
|
# Piper voices
|
||||||
|
try:
|
||||||
|
piper_voices = self.piper_service.get_available_voices()
|
||||||
|
voices["piper"] = list(piper_voices.keys())
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Could not load Piper voices: {e}")
|
||||||
|
voices["piper"] = []
|
||||||
|
|
||||||
|
return voices
|
||||||
|
|
||||||
|
async def generate_script_segments(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
roles: List[str]
|
||||||
|
) -> List[Tuple[str, str]]:
|
||||||
|
"""
|
||||||
|
Generate script segments with speaker assignments.
|
||||||
|
Uses AI when available, falls back to pattern matching.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Input text to segment
|
||||||
|
roles: List of detected role IDs/names
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of tuples: (speaker_name, text_content)
|
||||||
|
"""
|
||||||
|
# This is engine-agnostic, use Kokoro's implementation
|
||||||
|
return await self.kokoro_service.generate_script_segments(text, roles)
|
||||||
|
|
||||||
|
async def generate_audiobook(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
lang: str,
|
||||||
|
role_voice_map: Dict[str, str],
|
||||||
|
speed: float = 1.0,
|
||||||
|
pitch_adjustments: Optional[Dict[str, int]] = None
|
||||||
|
) -> bytes:
|
||||||
|
"""
|
||||||
|
Generate audiobook using the selected TTS engine.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Input text
|
||||||
|
lang: Language code
|
||||||
|
role_voice_map: Role to voice ID mapping
|
||||||
|
speed: Playback speed
|
||||||
|
pitch_adjustments: Optional dict of role to pitch shift (semitones)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Audio data as bytes (WAV format)
|
||||||
|
"""
|
||||||
|
if self.engine_name == TTSEngine.KOKORO:
|
||||||
|
# Kokoro doesn't support pitch adjustment
|
||||||
|
return await self.kokoro_service.generate_audiobook(
|
||||||
|
text, lang, role_voice_map, speed
|
||||||
|
)
|
||||||
|
|
||||||
|
elif self.engine_name == TTSEngine.PIPER:
|
||||||
|
# Use Piper with pitch support
|
||||||
|
return await self._generate_with_piper(
|
||||||
|
text, lang, role_voice_map, speed, pitch_adjustments or {}
|
||||||
|
)
|
||||||
|
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown engine: {self.engine_name}")
|
||||||
|
|
||||||
|
async def generate_audiobook_from_segments(
|
||||||
|
self,
|
||||||
|
segments: List[Tuple[str, str]],
|
||||||
|
lang: str,
|
||||||
|
role_voice_map: Dict[str, str],
|
||||||
|
speed: float = 1.0,
|
||||||
|
pitch_adjustments: Optional[Dict[str, int]] = None
|
||||||
|
) -> bytes:
|
||||||
|
"""
|
||||||
|
Generate audiobook from pre-segmented script.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
segments: List of (speaker_name, text_content) tuples
|
||||||
|
lang: Language code
|
||||||
|
role_voice_map: Role to voice ID mapping
|
||||||
|
speed: Playback speed
|
||||||
|
pitch_adjustments: Optional dict of role to pitch shift (semitones)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Audio data as bytes (WAV format)
|
||||||
|
"""
|
||||||
|
if self.engine_name == TTSEngine.KOKORO:
|
||||||
|
return await self.kokoro_service.generate_audiobook_from_segments(
|
||||||
|
segments, lang, role_voice_map, speed
|
||||||
|
)
|
||||||
|
|
||||||
|
elif self.engine_name == TTSEngine.PIPER:
|
||||||
|
return await self._generate_from_segments_with_piper(
|
||||||
|
segments, lang, role_voice_map, speed, pitch_adjustments or {}
|
||||||
|
)
|
||||||
|
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown engine: {self.engine_name}")
|
||||||
|
|
||||||
|
async def _generate_with_piper(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
lang: str,
|
||||||
|
role_voice_map: Dict[str, str],
|
||||||
|
speed: float,
|
||||||
|
pitch_adjustments: Dict[str, int]
|
||||||
|
) -> bytes:
|
||||||
|
"""Generate audiobook using Piper TTS with pitch control."""
|
||||||
|
import io
|
||||||
|
import numpy as np
|
||||||
|
import soundfile as sf
|
||||||
|
|
||||||
|
# Segment text using Kokoro's logic
|
||||||
|
segments = await self.kokoro_service.segment_text_with_ai(text, role_voice_map)
|
||||||
|
|
||||||
|
if segments is None:
|
||||||
|
segments = self.kokoro_service.segment_text(text, role_voice_map)
|
||||||
|
|
||||||
|
if not segments:
|
||||||
|
raise ValueError("No segments to synthesize")
|
||||||
|
|
||||||
|
# Synthesize each segment with Piper
|
||||||
|
audio_segments = []
|
||||||
|
for role_id, voice, content in segments:
|
||||||
|
pitch_shift = pitch_adjustments.get(role_id, 0)
|
||||||
|
logger.debug(
|
||||||
|
f"Synthesizing with Piper: role='{role_id}', voice='{voice}', "
|
||||||
|
f"pitch={pitch_shift}"
|
||||||
|
)
|
||||||
|
|
||||||
|
audio = self.piper_service.synthesize_with_pitch(
|
||||||
|
content, voice, speed, pitch_shift
|
||||||
|
)
|
||||||
|
audio_segments.append(audio)
|
||||||
|
|
||||||
|
# Concatenate
|
||||||
|
final_audio = np.concatenate([s for s in audio_segments if len(s) > 0])
|
||||||
|
|
||||||
|
# Convert to WAV bytes
|
||||||
|
buffer = io.BytesIO()
|
||||||
|
sf.write(buffer, final_audio, self.piper_service.SAMPLE_RATE, format='WAV')
|
||||||
|
buffer.seek(0)
|
||||||
|
|
||||||
|
logger.info(f"Generated Piper audiobook: {len(buffer.getvalue())} bytes")
|
||||||
|
return buffer.read()
|
||||||
|
|
||||||
|
async def _generate_from_segments_with_piper(
|
||||||
|
self,
|
||||||
|
segments: List[Tuple[str, str]],
|
||||||
|
lang: str,
|
||||||
|
role_voice_map: Dict[str, str],
|
||||||
|
speed: float,
|
||||||
|
pitch_adjustments: Dict[str, int]
|
||||||
|
) -> bytes:
|
||||||
|
"""Generate audiobook from segments using Piper TTS."""
|
||||||
|
import io
|
||||||
|
import numpy as np
|
||||||
|
import soundfile as sf
|
||||||
|
|
||||||
|
if not segments:
|
||||||
|
raise ValueError("No segments provided")
|
||||||
|
|
||||||
|
# Synthesize each segment
|
||||||
|
audio_segments = []
|
||||||
|
for speaker, text in segments:
|
||||||
|
voice = role_voice_map.get(speaker, self.piper_service.DEFAULT_VOICE)
|
||||||
|
pitch_shift = pitch_adjustments.get(speaker, 0)
|
||||||
|
|
||||||
|
logger.debug(
|
||||||
|
f"Synthesizing segment: speaker='{speaker}', voice='{voice}', "
|
||||||
|
f"pitch={pitch_shift}"
|
||||||
|
)
|
||||||
|
|
||||||
|
audio = self.piper_service.synthesize_with_pitch(
|
||||||
|
text, voice, speed, pitch_shift
|
||||||
|
)
|
||||||
|
audio_segments.append(audio)
|
||||||
|
|
||||||
|
# Concatenate
|
||||||
|
final_audio = np.concatenate([s for s in audio_segments if len(s) > 0])
|
||||||
|
|
||||||
|
# Convert to WAV bytes
|
||||||
|
buffer = io.BytesIO()
|
||||||
|
sf.write(buffer, final_audio, self.piper_service.SAMPLE_RATE, format='WAV')
|
||||||
|
buffer.seek(0)
|
||||||
|
|
||||||
|
logger.info(f"Generated Piper audiobook from segments: {len(buffer.getvalue())} bytes")
|
||||||
|
return buffer.read()
|
||||||
58
backend/test_engine_selection.py
Normal file
58
backend/test_engine_selection.py
Normal file
@@ -0,0 +1,58 @@
|
|||||||
|
"""
|
||||||
|
Test script for TTS engine selection functionality.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
import sys
|
||||||
|
from services.unified_tts_service import UnifiedTTSService
|
||||||
|
|
||||||
|
|
||||||
|
async def test_engine_selection():
|
||||||
|
"""Test switching between TTS engines."""
|
||||||
|
print("Testing TTS Engine Selection\n" + "="*50)
|
||||||
|
|
||||||
|
# Initialize service
|
||||||
|
print("\n1. Initializing UnifiedTTSService...")
|
||||||
|
tts = UnifiedTTSService()
|
||||||
|
print(f" Default engine: {tts.engine_name}")
|
||||||
|
|
||||||
|
# Get available engines
|
||||||
|
print("\n2. Getting available engines...")
|
||||||
|
engines = tts.get_available_engines()
|
||||||
|
print(f" Available: {engines}")
|
||||||
|
|
||||||
|
# Get available voices
|
||||||
|
print("\n3. Getting available voices...")
|
||||||
|
voices = tts.get_available_voices()
|
||||||
|
for engine, voice_list in voices.items():
|
||||||
|
print(f" {engine}: {len(voice_list)} voices")
|
||||||
|
if voice_list:
|
||||||
|
print(f" Examples: {voice_list[:3]}")
|
||||||
|
|
||||||
|
# Test switching to Piper
|
||||||
|
print("\n4. Switching to Piper engine...")
|
||||||
|
try:
|
||||||
|
tts.set_engine('piper')
|
||||||
|
print(f" Current engine: {tts.engine_name}")
|
||||||
|
except Exception as e:
|
||||||
|
print(f" ⚠️ Error: {e}")
|
||||||
|
print(" Note: Piper may not be installed. Run ./install_piper.sh")
|
||||||
|
|
||||||
|
# Switch back to Kokoro
|
||||||
|
print("\n5. Switching back to Kokoro engine...")
|
||||||
|
tts.set_engine('kokoro')
|
||||||
|
print(f" Current engine: {tts.engine_name}")
|
||||||
|
|
||||||
|
# Test invalid engine
|
||||||
|
print("\n6. Testing invalid engine...")
|
||||||
|
try:
|
||||||
|
tts.set_engine('invalid')
|
||||||
|
print(" ❌ Should have raised error!")
|
||||||
|
except ValueError as e:
|
||||||
|
print(f" ✓ Correctly rejected: {e}")
|
||||||
|
|
||||||
|
print("\n" + "="*50)
|
||||||
|
print("✓ Engine selection tests completed!")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(test_engine_selection())
|
||||||
18
backend/test_ollama_quick.py
Normal file
18
backend/test_ollama_quick.py
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
import asyncio
|
||||||
|
from services.ollama_service import OllamaService
|
||||||
|
|
||||||
|
async def test():
|
||||||
|
service = OllamaService()
|
||||||
|
text = '''Once upon a time, in a tiny village surrounded by lush green forests, there lived three best friends: Lily the rabbit, Benny the bear, and Sammy the squirrel.
|
||||||
|
|
||||||
|
As they sat down on a blanket, Lily asked, "Have you guys heard about the new stream that just appeared in the woods?"
|
||||||
|
|
||||||
|
Benny's eyes lit up. "Ooh, I love streams! Do we get to go see it now?"
|
||||||
|
|
||||||
|
Sammy nodded enthusiastically. "I've been wanting to explore the forest more since the snow melted. Let's go!"'''
|
||||||
|
|
||||||
|
print("Testing Ollama speaker detection...")
|
||||||
|
speakers = await service.detect_speakers(text)
|
||||||
|
print(f'Detected speakers: {speakers}')
|
||||||
|
|
||||||
|
asyncio.run(test())
|
||||||
94
backend/test_piper_direct.py
Normal file
94
backend/test_piper_direct.py
Normal file
@@ -0,0 +1,94 @@
|
|||||||
|
"""
|
||||||
|
Direct test of Piper TTS functionality.
|
||||||
|
"""
|
||||||
|
import subprocess
|
||||||
|
import os
|
||||||
|
|
||||||
|
def test_piper_installation():
|
||||||
|
"""Test if Piper is installed and accessible."""
|
||||||
|
print("Testing Piper Installation\n" + "="*50)
|
||||||
|
|
||||||
|
# Check if piper command exists
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
["piper", "--version"],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=5
|
||||||
|
)
|
||||||
|
print(f"✓ Piper found: {result.stdout.strip()}")
|
||||||
|
return True
|
||||||
|
except FileNotFoundError:
|
||||||
|
print("✗ Piper not found in PATH")
|
||||||
|
print(" Install with: ./install_piper.sh")
|
||||||
|
return False
|
||||||
|
except Exception as e:
|
||||||
|
print(f"✗ Error checking Piper: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def test_piper_synthesis():
|
||||||
|
"""Test basic Piper synthesis."""
|
||||||
|
print("\nTesting Piper Synthesis\n" + "="*50)
|
||||||
|
|
||||||
|
# Create models directory
|
||||||
|
models_dir = os.path.join(os.path.dirname(__file__), "piper_models")
|
||||||
|
os.makedirs(models_dir, exist_ok=True)
|
||||||
|
print(f"Models directory: {models_dir}")
|
||||||
|
|
||||||
|
# Try to synthesize with default voice
|
||||||
|
voice = "en_US-lessac-medium"
|
||||||
|
model_path = os.path.join(models_dir, f"{voice}.onnx")
|
||||||
|
|
||||||
|
if not os.path.exists(model_path):
|
||||||
|
print(f"\n⚠️ Model not found: {model_path}")
|
||||||
|
print(f"Attempting to download {voice}...")
|
||||||
|
|
||||||
|
# Try downloading
|
||||||
|
try:
|
||||||
|
import urllib.request
|
||||||
|
base_url = "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0"
|
||||||
|
model_url = f"{base_url}/en_US/{voice}.onnx"
|
||||||
|
config_url = f"{base_url}/en_US/{voice}.onnx.json"
|
||||||
|
|
||||||
|
print(f"Downloading from: {model_url}")
|
||||||
|
urllib.request.urlretrieve(model_url, model_path)
|
||||||
|
urllib.request.urlretrieve(config_url, f"{model_path}.json")
|
||||||
|
print("✓ Download complete")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"✗ Download failed: {e}")
|
||||||
|
return False
|
||||||
|
else:
|
||||||
|
print(f"✓ Model found: {model_path}")
|
||||||
|
|
||||||
|
# Try synthesis
|
||||||
|
try:
|
||||||
|
print("\nAttempting synthesis...")
|
||||||
|
process = subprocess.Popen(
|
||||||
|
["piper", "--model", model_path, "--output-raw"],
|
||||||
|
stdin=subprocess.PIPE,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE
|
||||||
|
)
|
||||||
|
|
||||||
|
test_text = "Hello, this is a test."
|
||||||
|
stdout, stderr = process.communicate(input=test_text.encode('utf-8'), timeout=10)
|
||||||
|
|
||||||
|
if process.returncode == 0:
|
||||||
|
print(f"✓ Synthesis successful: {len(stdout)} bytes of audio")
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
print(f"✗ Synthesis failed (exit code {process.returncode})")
|
||||||
|
print(f" Error: {stderr.decode()}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"✗ Synthesis error: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
if test_piper_installation():
|
||||||
|
test_piper_synthesis()
|
||||||
|
|
||||||
|
print("\n" + "="*50)
|
||||||
164
backend/test_segment_based_generation.py
Normal file
164
backend/test_segment_based_generation.py
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
"""
|
||||||
|
Test for segment-based audiobook generation (Task 19).
|
||||||
|
|
||||||
|
Tests that the TTS endpoint can accept edited script segments
|
||||||
|
and generate audio from them, while maintaining backward compatibility
|
||||||
|
with text-based generation.
|
||||||
|
"""
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from main import app
|
||||||
|
|
||||||
|
client = TestClient(app)
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_audiobook_with_segments():
|
||||||
|
"""Test that audiobook can be generated from edited script segments."""
|
||||||
|
# Prepare request with segments
|
||||||
|
request_data = {
|
||||||
|
"segments": [
|
||||||
|
{"id": "seg_0", "speaker": "Alice", "text": "Hello, how are you?"},
|
||||||
|
{"id": "seg_1", "speaker": "Bob", "text": "I'm doing great, thanks!"},
|
||||||
|
{"id": "seg_2", "speaker": "Narrator", "text": "They continued their conversation."}
|
||||||
|
],
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "af_bella"},
|
||||||
|
{"id": "Bob", "voice": "am_adam"},
|
||||||
|
{"id": "Narrator", "voice": "af_sarah"}
|
||||||
|
],
|
||||||
|
"speed": 1.0
|
||||||
|
}
|
||||||
|
|
||||||
|
response = client.post("/api/tts", json=request_data)
|
||||||
|
|
||||||
|
# Should succeed
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.headers["content-type"] == "audio/wav"
|
||||||
|
assert "audiobook.wav" in response.headers.get("content-disposition", "")
|
||||||
|
assert len(response.content) > 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_audiobook_with_text_backward_compatibility():
|
||||||
|
"""Test that audiobook generation still works with text (backward compatibility)."""
|
||||||
|
# Prepare request with text (old format)
|
||||||
|
request_data = {
|
||||||
|
"text": "Alice: Hello there!\nBob: Hi Alice!",
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "af_bella"},
|
||||||
|
{"id": "Bob", "voice": "am_adam"}
|
||||||
|
],
|
||||||
|
"speed": 1.0
|
||||||
|
}
|
||||||
|
|
||||||
|
response = client.post("/api/tts", json=request_data)
|
||||||
|
|
||||||
|
# Should succeed
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.headers["content-type"] == "audio/wav"
|
||||||
|
assert len(response.content) > 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_audiobook_segments_take_precedence():
|
||||||
|
"""Test that segments are used when both segments and text are provided."""
|
||||||
|
# Prepare request with both segments and text
|
||||||
|
request_data = {
|
||||||
|
"segments": [
|
||||||
|
{"id": "seg_0", "speaker": "Alice", "text": "From segments"}
|
||||||
|
],
|
||||||
|
"text": "Alice: From text",
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "af_bella"}
|
||||||
|
],
|
||||||
|
"speed": 1.0
|
||||||
|
}
|
||||||
|
|
||||||
|
response = client.post("/api/tts", json=request_data)
|
||||||
|
|
||||||
|
# Should succeed and use segments
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.headers["content-type"] == "audio/wav"
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_audiobook_missing_both_segments_and_text():
|
||||||
|
"""Test that request fails when neither segments nor text is provided."""
|
||||||
|
request_data = {
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "af_bella"}
|
||||||
|
],
|
||||||
|
"speed": 1.0
|
||||||
|
}
|
||||||
|
|
||||||
|
response = client.post("/api/tts", json=request_data)
|
||||||
|
|
||||||
|
# Should fail with validation error
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "ValidationError" in response.json()["error"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_audiobook_empty_segments():
|
||||||
|
"""Test that request fails when segments array is empty."""
|
||||||
|
request_data = {
|
||||||
|
"segments": [],
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "af_bella"}
|
||||||
|
],
|
||||||
|
"speed": 1.0
|
||||||
|
}
|
||||||
|
|
||||||
|
response = client.post("/api/tts", json=request_data)
|
||||||
|
|
||||||
|
# Should fail - empty segments should be treated as not provided
|
||||||
|
assert response.status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
def test_edited_script_preserves_speaker_assignments():
|
||||||
|
"""Test that edited speaker assignments in segments are preserved."""
|
||||||
|
# Segments where user has changed speaker assignments
|
||||||
|
request_data = {
|
||||||
|
"segments": [
|
||||||
|
{"id": "seg_0", "speaker": "Bob", "text": "This was originally Alice's line"},
|
||||||
|
{"id": "seg_1", "speaker": "Alice", "text": "This was originally Bob's line"}
|
||||||
|
],
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "af_bella"},
|
||||||
|
{"id": "Bob", "voice": "am_adam"}
|
||||||
|
],
|
||||||
|
"speed": 1.0
|
||||||
|
}
|
||||||
|
|
||||||
|
response = client.post("/api/tts", json=request_data)
|
||||||
|
|
||||||
|
# Should succeed with edited assignments
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.headers["content-type"] == "audio/wav"
|
||||||
|
assert len(response.content) > 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_segment_with_unmapped_speaker_uses_default():
|
||||||
|
"""Test that segments with unmapped speakers fall back to default voice."""
|
||||||
|
request_data = {
|
||||||
|
"segments": [
|
||||||
|
{"id": "seg_0", "speaker": "UnknownSpeaker", "text": "Hello from unknown"}
|
||||||
|
],
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "af_bella"}
|
||||||
|
],
|
||||||
|
"speed": 1.0
|
||||||
|
}
|
||||||
|
|
||||||
|
response = client.post("/api/tts", json=request_data)
|
||||||
|
|
||||||
|
# Should succeed using default voice
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.headers["content-type"] == "audio/wav"
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
pytest.main([__file__, "-v"])
|
||||||
79
diagnose_issue.md
Normal file
79
diagnose_issue.md
Normal file
@@ -0,0 +1,79 @@
|
|||||||
|
# Audiobook Generation Issue - Diagnosis
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
The story is coming out with only one voice (Narrator) instead of multiple voices for different characters.
|
||||||
|
|
||||||
|
## Root Cause Analysis
|
||||||
|
|
||||||
|
Based on my investigation, I found that:
|
||||||
|
|
||||||
|
1. **Ollama IS working correctly** - When I tested with your story, it correctly detected all 3 characters:
|
||||||
|
- Lily
|
||||||
|
- Benny
|
||||||
|
- Sammy
|
||||||
|
|
||||||
|
2. **The backend logs show** that when you used the web interface, it only received 40 characters of text:
|
||||||
|
```
|
||||||
|
2025-12-09 10:16:52,356 - main - INFO - Role detection requested for text of length 40
|
||||||
|
2025-12-09 10:16:52,358 - main - INFO - Detected 1 roles: ['Narrator']
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Your full story is 3,749 characters**, so something went wrong in the web interface.
|
||||||
|
|
||||||
|
## Possible Causes
|
||||||
|
|
||||||
|
### Most Likely: Text wasn't fully pasted or was truncated
|
||||||
|
- The web interface might have had an issue with the paste operation
|
||||||
|
- The text area might have a character limit
|
||||||
|
- There might have been a copy/paste error
|
||||||
|
|
||||||
|
### Less Likely: Backend issue
|
||||||
|
- The backend is working correctly (I tested it directly)
|
||||||
|
- Ollama is running and detecting speakers properly
|
||||||
|
|
||||||
|
## Solution Steps
|
||||||
|
|
||||||
|
### Step 1: Verify Backend is Running
|
||||||
|
```bash
|
||||||
|
# Check if backend is running
|
||||||
|
curl http://localhost:8001/api/health
|
||||||
|
|
||||||
|
# Should return: {"status":"ok"}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Test with the Full Story via API
|
||||||
|
I'll create a test script that sends your full story directly to the backend to verify it works end-to-end.
|
||||||
|
|
||||||
|
### Step 3: Check Frontend
|
||||||
|
If the API works but the web interface doesn't, there might be an issue with:
|
||||||
|
- Text area character limits
|
||||||
|
- Copy/paste handling
|
||||||
|
- Form submission
|
||||||
|
|
||||||
|
## Quick Test Script
|
||||||
|
|
||||||
|
Run this to test the full workflow with your story:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 test_full_workflow.py
|
||||||
|
```
|
||||||
|
|
||||||
|
This will:
|
||||||
|
1. Send your full story to the role detection endpoint
|
||||||
|
2. Show you what roles are detected
|
||||||
|
3. Generate a sample audiobook with different voices
|
||||||
|
4. Save it as `test_output.wav`
|
||||||
|
|
||||||
|
## Expected Results
|
||||||
|
|
||||||
|
When working correctly, you should see:
|
||||||
|
- **3-4 roles detected**: Lily, Benny, Sammy, and possibly Narrator
|
||||||
|
- **Multiple voices in the audio**: Each character should have a distinct voice
|
||||||
|
- **Proper segmentation**: Dialogue should be assigned to the correct character
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
1. Try the test script to verify the backend works
|
||||||
|
2. If backend works, try the web interface again with the full story
|
||||||
|
3. Make sure to paste the ENTIRE story (all 3,749 characters)
|
||||||
|
4. Check the browser console for any JavaScript errors
|
||||||
1124
frontend/package-lock.json
generated
1124
frontend/package-lock.json
generated
File diff suppressed because it is too large
Load Diff
@@ -15,7 +15,6 @@
|
|||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"@eslint/js": "^9.39.1",
|
"@eslint/js": "^9.39.1",
|
||||||
"@tailwindcss/postcss": "^4.1.17",
|
|
||||||
"@types/node": "^24.10.1",
|
"@types/node": "^24.10.1",
|
||||||
"@types/react": "^19.2.5",
|
"@types/react": "^19.2.5",
|
||||||
"@types/react-dom": "^19.2.3",
|
"@types/react-dom": "^19.2.3",
|
||||||
@@ -26,7 +25,7 @@
|
|||||||
"eslint-plugin-react-refresh": "^0.4.24",
|
"eslint-plugin-react-refresh": "^0.4.24",
|
||||||
"globals": "^16.5.0",
|
"globals": "^16.5.0",
|
||||||
"postcss": "^8.5.6",
|
"postcss": "^8.5.6",
|
||||||
"tailwindcss": "^4.1.17",
|
"tailwindcss": "^3.4.18",
|
||||||
"typescript": "~5.9.3",
|
"typescript": "~5.9.3",
|
||||||
"typescript-eslint": "^8.46.4",
|
"typescript-eslint": "^8.46.4",
|
||||||
"vite": "^7.2.4"
|
"vite": "^7.2.4"
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
export default {
|
export default {
|
||||||
plugins: {
|
plugins: {
|
||||||
'@tailwindcss/postcss': {},
|
tailwindcss: {},
|
||||||
autoprefixer: {},
|
autoprefixer: {},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,13 +1,11 @@
|
|||||||
/* Mobile-first responsive styles */
|
|
||||||
#root {
|
#root {
|
||||||
width: 100%;
|
width: 100%;
|
||||||
margin: 0 auto;
|
margin: 0 auto;
|
||||||
padding: 0;
|
padding: 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Ensure touch-friendly interactions */
|
button, input, select, textarea {
|
||||||
.touch-manipulation {
|
transition: all 0.2s ease;
|
||||||
touch-action: manipulation;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Custom range slider styling for better mobile support */
|
/* Custom range slider styling for better mobile support */
|
||||||
@@ -17,9 +15,10 @@ input[type="range"] {
|
|||||||
width: 100%;
|
width: 100%;
|
||||||
height: 8px;
|
height: 8px;
|
||||||
border-radius: 4px;
|
border-radius: 4px;
|
||||||
background: #d1d5db;
|
background: linear-gradient(to right, #667eea, #764ba2);
|
||||||
outline: none;
|
outline: none;
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
|
box-shadow: 0 2px 8px rgba(102, 126, 234, 0.3);
|
||||||
}
|
}
|
||||||
|
|
||||||
input[type="range"]::-webkit-slider-thumb {
|
input[type="range"]::-webkit-slider-thumb {
|
||||||
@@ -28,35 +27,39 @@ input[type="range"]::-webkit-slider-thumb {
|
|||||||
width: 24px;
|
width: 24px;
|
||||||
height: 24px;
|
height: 24px;
|
||||||
border-radius: 50%;
|
border-radius: 50%;
|
||||||
background: #2563eb;
|
background: linear-gradient(135deg, #667eea, #764ba2);
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
transition: background 0.15s ease-in-out;
|
box-shadow: 0 4px 12px rgba(102, 126, 234, 0.4);
|
||||||
|
transition: all 0.3s ease;
|
||||||
}
|
}
|
||||||
|
|
||||||
input[type="range"]::-webkit-slider-thumb:hover {
|
input[type="range"]::-webkit-slider-thumb:hover {
|
||||||
background: #1d4ed8;
|
transform: scale(1.2);
|
||||||
|
box-shadow: 0 6px 16px rgba(102, 126, 234, 0.6);
|
||||||
}
|
}
|
||||||
|
|
||||||
input[type="range"]::-webkit-slider-thumb:active {
|
input[type="range"]::-webkit-slider-thumb:active {
|
||||||
background: #1e40af;
|
transform: scale(1.1);
|
||||||
}
|
}
|
||||||
|
|
||||||
input[type="range"]::-moz-range-thumb {
|
input[type="range"]::-moz-range-thumb {
|
||||||
width: 24px;
|
width: 24px;
|
||||||
height: 24px;
|
height: 24px;
|
||||||
border-radius: 50%;
|
border-radius: 50%;
|
||||||
background: #2563eb;
|
background: linear-gradient(135deg, #667eea, #764ba2);
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
border: none;
|
border: none;
|
||||||
transition: background 0.15s ease-in-out;
|
box-shadow: 0 4px 12px rgba(102, 126, 234, 0.4);
|
||||||
|
transition: all 0.3s ease;
|
||||||
}
|
}
|
||||||
|
|
||||||
input[type="range"]::-moz-range-thumb:hover {
|
input[type="range"]::-moz-range-thumb:hover {
|
||||||
background: #1d4ed8;
|
transform: scale(1.2);
|
||||||
|
box-shadow: 0 6px 16px rgba(102, 126, 234, 0.6);
|
||||||
}
|
}
|
||||||
|
|
||||||
input[type="range"]::-moz-range-thumb:active {
|
input[type="range"]::-moz-range-thumb:active {
|
||||||
background: #1e40af;
|
transform: scale(1.1);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Ensure audio controls are touch-friendly on mobile */
|
/* Ensure audio controls are touch-friendly on mobile */
|
||||||
|
|||||||
@@ -2,18 +2,24 @@ import { useState } from 'react'
|
|||||||
import './App.css'
|
import './App.css'
|
||||||
import TextInputSection from './components/TextInputSection'
|
import TextInputSection from './components/TextInputSection'
|
||||||
import VoiceAssignmentSection from './components/VoiceAssignmentSection'
|
import VoiceAssignmentSection from './components/VoiceAssignmentSection'
|
||||||
|
import ScriptReviewSection from './components/ScriptReviewSection'
|
||||||
import AudioPlayerSection from './components/AudioPlayerSection'
|
import AudioPlayerSection from './components/AudioPlayerSection'
|
||||||
import type { Role } from './api/tts'
|
import type { Role, ScriptSegment } from './api/tts'
|
||||||
import { detectRoles, generateAudiobook } from './api/tts'
|
import { detectRoles, generateScript, generateAudiobook } from './api/tts'
|
||||||
|
|
||||||
|
type AgeCategory = 'young' | 'teen' | 'adult' | 'old';
|
||||||
|
|
||||||
function App() {
|
function App() {
|
||||||
const [text, setText] = useState('')
|
const [text, setText] = useState('')
|
||||||
const [phase, setPhase] = useState<'input' | 'assignment' | 'generated'>('input')
|
const [phase, setPhase] = useState<'input' | 'assignment' | 'review' | 'generated'>('input')
|
||||||
const [isLoading, setIsLoading] = useState(false)
|
const [isLoading, setIsLoading] = useState(false)
|
||||||
const [roles, setRoles] = useState<Role[]>([])
|
const [roles, setRoles] = useState<Role[]>([])
|
||||||
const [voiceAssignments, setVoiceAssignments] = useState<Map<string, string>>(new Map())
|
const [voiceAssignments, setVoiceAssignments] = useState<Map<string, string>>(new Map())
|
||||||
|
const [ageSettings, setAgeSettings] = useState<Map<string, AgeCategory>>(new Map())
|
||||||
|
const [scriptSegments, setScriptSegments] = useState<ScriptSegment[]>([])
|
||||||
const [language, setLanguage] = useState('a') // Default: American English
|
const [language, setLanguage] = useState('a') // Default: American English
|
||||||
const [speed, setSpeed] = useState(1.0) // Default: 1.0x
|
const [speed, setSpeed] = useState(1.0) // Default: 1.0x
|
||||||
|
const [isGeneratingScript, setIsGeneratingScript] = useState(false)
|
||||||
const [isGenerating, setIsGenerating] = useState(false)
|
const [isGenerating, setIsGenerating] = useState(false)
|
||||||
const [error, setError] = useState<string | null>(null)
|
const [error, setError] = useState<string | null>(null)
|
||||||
const [audioUrl, setAudioUrl] = useState<string | null>(null)
|
const [audioUrl, setAudioUrl] = useState<string | null>(null)
|
||||||
@@ -29,6 +35,7 @@ function App() {
|
|||||||
try {
|
try {
|
||||||
// Detect roles in the text
|
// Detect roles in the text
|
||||||
const response = await detectRoles(text)
|
const response = await detectRoles(text)
|
||||||
|
console.log('Detected roles:', response.roles)
|
||||||
setRoles(response.roles)
|
setRoles(response.roles)
|
||||||
|
|
||||||
// Initialize voice assignments map (empty initially)
|
// Initialize voice assignments map (empty initially)
|
||||||
@@ -52,6 +59,14 @@ function App() {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const handleAgeChange = (roleId: string, age: AgeCategory) => {
|
||||||
|
setAgeSettings(prev => {
|
||||||
|
const newMap = new Map(prev)
|
||||||
|
newMap.set(roleId, age)
|
||||||
|
return newMap
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
const handleLanguageChange = (lang: string) => {
|
const handleLanguageChange = (lang: string) => {
|
||||||
setLanguage(lang)
|
setLanguage(lang)
|
||||||
}
|
}
|
||||||
@@ -60,6 +75,88 @@ function App() {
|
|||||||
setSpeed(newSpeed)
|
setSpeed(newSpeed)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const handleGenerateScript = async () => {
|
||||||
|
setIsGeneratingScript(true)
|
||||||
|
setError(null)
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Generate script preview with AI segmentation
|
||||||
|
const roleIds = roles.map(role => role.id)
|
||||||
|
const response = await generateScript({
|
||||||
|
text,
|
||||||
|
roles: roleIds
|
||||||
|
})
|
||||||
|
|
||||||
|
setScriptSegments(response.segments)
|
||||||
|
|
||||||
|
// Move to review phase
|
||||||
|
setPhase('review')
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to generate script')
|
||||||
|
console.error('Script generation error:', err)
|
||||||
|
} finally {
|
||||||
|
setIsGeneratingScript(false)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const handleSkipReview = async () => {
|
||||||
|
// Skip review and go directly to generation
|
||||||
|
await handleGenerate()
|
||||||
|
}
|
||||||
|
|
||||||
|
const handleSegmentChange = (segmentId: string, newSpeaker: string) => {
|
||||||
|
setScriptSegments(prev =>
|
||||||
|
prev.map(segment =>
|
||||||
|
segment.id === segmentId
|
||||||
|
? { ...segment, speaker: newSpeaker }
|
||||||
|
: segment
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
const handleSegmentTextChange = (segmentId: string, newText: string) => {
|
||||||
|
setScriptSegments(prev =>
|
||||||
|
prev.map(segment =>
|
||||||
|
segment.id === segmentId
|
||||||
|
? { ...segment, text: newText }
|
||||||
|
: segment
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
const handleSegmentSplit = (segmentId: string, splitPosition: number) => {
|
||||||
|
setScriptSegments(prev => {
|
||||||
|
const segmentIndex = prev.findIndex(s => s.id === segmentId)
|
||||||
|
if (segmentIndex === -1) return prev
|
||||||
|
|
||||||
|
const segment = prev[segmentIndex]
|
||||||
|
const firstPart = segment.text.substring(0, splitPosition).trim()
|
||||||
|
const secondPart = segment.text.substring(splitPosition).trim()
|
||||||
|
|
||||||
|
if (!firstPart || !secondPart) return prev
|
||||||
|
|
||||||
|
// Create new segments
|
||||||
|
const newSegments = [...prev]
|
||||||
|
newSegments[segmentIndex] = {
|
||||||
|
...segment,
|
||||||
|
text: firstPart
|
||||||
|
}
|
||||||
|
|
||||||
|
// Insert second part after the first
|
||||||
|
newSegments.splice(segmentIndex + 1, 0, {
|
||||||
|
id: `${segmentId}-split-${Date.now()}`,
|
||||||
|
speaker: segment.speaker, // Keep same speaker by default
|
||||||
|
text: secondPart
|
||||||
|
})
|
||||||
|
|
||||||
|
return newSegments
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
const handleSegmentDelete = (segmentId: string) => {
|
||||||
|
setScriptSegments(prev => prev.filter(segment => segment.id !== segmentId))
|
||||||
|
}
|
||||||
|
|
||||||
const handleGenerate = async () => {
|
const handleGenerate = async () => {
|
||||||
setIsGenerating(true)
|
setIsGenerating(true)
|
||||||
setError(null)
|
setError(null)
|
||||||
@@ -72,8 +169,11 @@ function App() {
|
|||||||
}))
|
}))
|
||||||
|
|
||||||
// Generate the audiobook
|
// Generate the audiobook
|
||||||
|
// If we have script segments (from review phase), use them
|
||||||
|
// Otherwise, fall back to text-based generation
|
||||||
const audioBlob = await generateAudiobook({
|
const audioBlob = await generateAudiobook({
|
||||||
text,
|
segments: scriptSegments.length > 0 ? scriptSegments : undefined,
|
||||||
|
text: scriptSegments.length > 0 ? undefined : text,
|
||||||
lang: language,
|
lang: language,
|
||||||
roles: rolesArray,
|
roles: rolesArray,
|
||||||
speed
|
speed
|
||||||
@@ -112,15 +212,59 @@ function App() {
|
|||||||
const allRolesAssigned = roles.length > 0 && roles.every(role => voiceAssignments.has(role.id))
|
const allRolesAssigned = roles.length > 0 && roles.every(role => voiceAssignments.has(role.id))
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div className="min-h-screen bg-gray-50">
|
<div className="min-h-screen">
|
||||||
<div className="container mx-auto py-4 px-4 sm:py-8 sm:px-6 lg:px-8">
|
<div className="container mx-auto py-12 px-4 sm:px-6 lg:px-8">
|
||||||
<h1 className="text-2xl sm:text-3xl md:text-4xl font-bold text-center mb-4 sm:mb-6 md:mb-8">
|
{/* Header */}
|
||||||
|
<div className="text-center mb-16 max-w-3xl mx-auto">
|
||||||
|
<div className="inline-flex items-center justify-center w-20 h-20 bg-gradient-to-br from-indigo-500 to-purple-600 rounded-3xl mb-8 shadow-xl shadow-indigo-500/30">
|
||||||
|
<span className="text-4xl">🎧</span>
|
||||||
|
</div>
|
||||||
|
<h1 className="text-5xl sm:text-6xl md:text-7xl font-black text-gray-900 mb-6 tracking-tight">
|
||||||
Text to Audiobook
|
Text to Audiobook
|
||||||
</h1>
|
</h1>
|
||||||
|
<p className="text-xl text-gray-600 leading-relaxed">
|
||||||
|
Transform your stories into immersive audio experiences with AI-powered voices
|
||||||
|
</p>
|
||||||
|
|
||||||
|
{/* Progress Steps */}
|
||||||
|
<div className="mt-12 flex justify-center items-center gap-2">
|
||||||
|
{['Input', 'Voices', 'Review', 'Done'].map((step, index) => {
|
||||||
|
const phases = ['input', 'assignment', 'review', 'generated'];
|
||||||
|
const isActive = phase === phases[index];
|
||||||
|
const isPast = phases.indexOf(phase) > index;
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div key={step} className="flex items-center">
|
||||||
|
<div className="flex flex-col items-center">
|
||||||
|
<div className={`flex items-center justify-center w-12 h-12 rounded-full font-bold text-sm transition-all duration-300 ${
|
||||||
|
isActive ? 'bg-indigo-600 text-white scale-110 shadow-lg shadow-indigo-500/50' :
|
||||||
|
isPast ? 'bg-green-500 text-white' :
|
||||||
|
'bg-gray-200 text-gray-500'
|
||||||
|
}`}>
|
||||||
|
{isPast ? '✓' : index + 1}
|
||||||
|
</div>
|
||||||
|
<span className={`text-xs font-semibold mt-2 ${isActive ? 'text-indigo-600' : 'text-gray-500'}`}>
|
||||||
|
{step}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
{index < 3 && (
|
||||||
|
<div className={`w-16 h-1 mx-3 rounded-full transition-all duration-300 ${isPast || isActive ? 'bg-indigo-600' : 'bg-gray-200'}`}></div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
{error && (
|
{error && (
|
||||||
<div className="max-w-4xl mx-auto mb-4 p-3 sm:p-4 bg-red-100 border border-red-400 text-red-700 rounded text-sm sm:text-base">
|
<div className="mb-6 p-4 bg-red-50 border border-red-200 text-red-700 rounded-xl">
|
||||||
{error}
|
<div className="flex items-start gap-3">
|
||||||
|
<span className="text-xl">⚠️</span>
|
||||||
|
<div>
|
||||||
|
<p className="font-semibold mb-1">Error</p>
|
||||||
|
<p className="text-sm">{error}</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
@@ -137,25 +281,56 @@ function App() {
|
|||||||
{phase === 'assignment' && (
|
{phase === 'assignment' && (
|
||||||
<>
|
<>
|
||||||
{isLongText && (
|
{isLongText && (
|
||||||
<div className="max-w-4xl mx-auto mb-4 p-3 sm:p-4 bg-yellow-100 border border-yellow-400 text-yellow-800 rounded text-sm sm:text-base">
|
<div className="mb-6 p-4 bg-yellow-50 border border-yellow-200 text-yellow-800 rounded-xl">
|
||||||
Generation may take several minutes for very long content.
|
<div className="flex items-center gap-2">
|
||||||
|
<span>⏱️</span>
|
||||||
|
<p className="text-sm font-medium">Generation may take several minutes for very long content.</p>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
{console.log('Rendering VoiceAssignmentSection with roles:', roles, 'allRolesAssigned:', allRolesAssigned)}
|
||||||
<VoiceAssignmentSection
|
<VoiceAssignmentSection
|
||||||
roles={roles}
|
roles={roles}
|
||||||
voiceAssignments={voiceAssignments}
|
voiceAssignments={voiceAssignments}
|
||||||
|
ageSettings={ageSettings}
|
||||||
onVoiceChange={handleVoiceChange}
|
onVoiceChange={handleVoiceChange}
|
||||||
|
onAgeChange={handleAgeChange}
|
||||||
language={language}
|
language={language}
|
||||||
onLanguageChange={handleLanguageChange}
|
onLanguageChange={handleLanguageChange}
|
||||||
speed={speed}
|
speed={speed}
|
||||||
onSpeedChange={handleSpeedChange}
|
onSpeedChange={handleSpeedChange}
|
||||||
onGenerate={handleGenerate}
|
onGenerate={handleGenerateScript}
|
||||||
isGenerating={isGenerating}
|
onSkipReview={handleSkipReview}
|
||||||
|
isGenerating={isGeneratingScript}
|
||||||
allRolesAssigned={allRolesAssigned}
|
allRolesAssigned={allRolesAssigned}
|
||||||
/>
|
/>
|
||||||
</>
|
</>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
{phase === 'review' && (
|
||||||
|
<>
|
||||||
|
{isLongText && (
|
||||||
|
<div className="mb-6 p-4 bg-yellow-50 border border-yellow-200 text-yellow-800 rounded-xl">
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<span>⏱️</span>
|
||||||
|
<p className="text-sm font-medium">Generation may take several minutes for very long content.</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
<ScriptReviewSection
|
||||||
|
segments={scriptSegments}
|
||||||
|
roles={roles}
|
||||||
|
onSegmentChange={handleSegmentChange}
|
||||||
|
onSegmentTextChange={handleSegmentTextChange}
|
||||||
|
onSegmentSplit={handleSegmentSplit}
|
||||||
|
onSegmentDelete={handleSegmentDelete}
|
||||||
|
onGenerate={handleGenerate}
|
||||||
|
onSkipReview={handleSkipReview}
|
||||||
|
isGenerating={isGenerating}
|
||||||
|
/>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
|
||||||
{phase === 'generated' && audioUrl && (
|
{phase === 'generated' && audioUrl && (
|
||||||
<AudioPlayerSection
|
<AudioPlayerSection
|
||||||
audioUrl={audioUrl}
|
audioUrl={audioUrl}
|
||||||
@@ -164,8 +339,26 @@ function App() {
|
|||||||
)}
|
)}
|
||||||
|
|
||||||
{isGenerating && isLongText && (
|
{isGenerating && isLongText && (
|
||||||
<div className="max-w-4xl mx-auto mt-4 p-3 sm:p-4 bg-blue-100 border border-blue-400 text-blue-800 rounded text-center text-sm sm:text-base">
|
<div className="mt-6 p-6 bg-blue-50 border border-blue-200 rounded-xl text-center">
|
||||||
Generation may take several minutes for very long content. Please be patient...
|
<div className="flex flex-col items-center gap-4">
|
||||||
|
<div className="animate-spin rounded-full h-10 w-10 border-3 border-indigo-600 border-t-transparent"></div>
|
||||||
|
<div>
|
||||||
|
<p className="font-semibold text-gray-900 mb-1">Generating Your Audiobook</p>
|
||||||
|
<p className="text-sm text-gray-600">This may take several minutes. Please be patient...</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{isGeneratingScript && (
|
||||||
|
<div className="mt-6 p-6 bg-purple-50 border border-purple-200 rounded-xl text-center">
|
||||||
|
<div className="flex flex-col items-center gap-4">
|
||||||
|
<div className="animate-spin rounded-full h-10 w-10 border-3 border-purple-600 border-t-transparent"></div>
|
||||||
|
<div>
|
||||||
|
<p className="font-semibold text-gray-900 mb-1">Generating Script Preview</p>
|
||||||
|
<p className="text-sm text-gray-600">AI is analyzing your text...</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -29,11 +29,36 @@ export interface RolesResponse {
|
|||||||
roles: Role[];
|
roles: Role[];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Represents a script segment with speaker assignment
|
||||||
|
*/
|
||||||
|
export interface ScriptSegment {
|
||||||
|
id: string;
|
||||||
|
speaker: string;
|
||||||
|
text: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Request payload for script generation
|
||||||
|
*/
|
||||||
|
export interface ScriptRequest {
|
||||||
|
text: string;
|
||||||
|
roles: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Response from the script generation endpoint
|
||||||
|
*/
|
||||||
|
export interface ScriptResponse {
|
||||||
|
segments: ScriptSegment[];
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Request payload for audiobook generation
|
* Request payload for audiobook generation
|
||||||
*/
|
*/
|
||||||
export interface GenerateRequest {
|
export interface GenerateRequest {
|
||||||
text: string;
|
segments?: ScriptSegment[];
|
||||||
|
text?: string;
|
||||||
lang: string;
|
lang: string;
|
||||||
roles: { id: string; voice: string }[];
|
roles: { id: string; voice: string }[];
|
||||||
speed?: number;
|
speed?: number;
|
||||||
@@ -188,6 +213,27 @@ export async function detectRoles(text: string): Promise<RolesResponse> {
|
|||||||
return await response.json();
|
return await response.json();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Generate a script preview with AI-segmented text and speaker assignments
|
||||||
|
* @param request - Script request with text and detected roles
|
||||||
|
* @returns Promise resolving to script segments
|
||||||
|
* @throws ApiError if the request fails
|
||||||
|
*/
|
||||||
|
export async function generateScript(request: ScriptRequest): Promise<ScriptResponse> {
|
||||||
|
const baseUrl = getApiBaseUrl();
|
||||||
|
const url = `${baseUrl}/api/tts/script`;
|
||||||
|
|
||||||
|
const response = await fetchWithErrorHandling(url, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
body: JSON.stringify(request),
|
||||||
|
});
|
||||||
|
|
||||||
|
return await response.json();
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Generate an audiobook from text with assigned voices
|
* Generate an audiobook from text with assigned voices
|
||||||
* @param payload - Generation request with text, language, roles, and speed
|
* @param payload - Generation request with text, language, roles, and speed
|
||||||
@@ -209,3 +255,119 @@ export async function generateAudiobook(payload: GenerateRequest): Promise<Blob>
|
|||||||
// Convert response to blob for audio data
|
// Convert response to blob for audio data
|
||||||
return await response.blob();
|
return await response.blob();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* TTS Engine information
|
||||||
|
*/
|
||||||
|
export interface EngineInfo {
|
||||||
|
name: string;
|
||||||
|
active: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Response from engines endpoint
|
||||||
|
*/
|
||||||
|
export interface EnginesResponse {
|
||||||
|
engines: EngineInfo[];
|
||||||
|
current: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Response from voices endpoint
|
||||||
|
*/
|
||||||
|
export interface VoicesResponse {
|
||||||
|
voices: {
|
||||||
|
[engine: string]: string[];
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Request payload for voice preview
|
||||||
|
*/
|
||||||
|
export interface VoicePreviewRequest {
|
||||||
|
voice: string;
|
||||||
|
lang: string;
|
||||||
|
speed?: number;
|
||||||
|
text?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get available TTS engines
|
||||||
|
* @returns Promise resolving to engines list
|
||||||
|
* @throws ApiError if the request fails
|
||||||
|
*/
|
||||||
|
export async function getEngines(): Promise<EnginesResponse> {
|
||||||
|
const baseUrl = getApiBaseUrl();
|
||||||
|
const url = `${baseUrl}/api/tts/engines`;
|
||||||
|
|
||||||
|
const response = await fetchWithErrorHandling(url, {
|
||||||
|
method: 'GET',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
return await response.json();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Set the active TTS engine
|
||||||
|
* @param engine - Engine name to activate (kokoro or piper)
|
||||||
|
* @returns Promise resolving to success response
|
||||||
|
* @throws ApiError if the request fails
|
||||||
|
*/
|
||||||
|
export async function setEngine(engine: string): Promise<{ status: string; engine: string }> {
|
||||||
|
const baseUrl = getApiBaseUrl();
|
||||||
|
const url = `${baseUrl}/api/tts/engines`;
|
||||||
|
|
||||||
|
const response = await fetchWithErrorHandling(url, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
body: JSON.stringify({ engine }),
|
||||||
|
});
|
||||||
|
|
||||||
|
return await response.json();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get available voices for all engines
|
||||||
|
* @returns Promise resolving to voices by engine
|
||||||
|
* @throws ApiError if the request fails
|
||||||
|
*/
|
||||||
|
export async function getVoices(): Promise<VoicesResponse> {
|
||||||
|
const baseUrl = getApiBaseUrl();
|
||||||
|
const url = `${baseUrl}/api/tts/voices`;
|
||||||
|
|
||||||
|
const response = await fetchWithErrorHandling(url, {
|
||||||
|
method: 'GET',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
return await response.json();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Generate a preview of a voice
|
||||||
|
* @param payload - Preview request with voice, language, and optional speed/text
|
||||||
|
* @returns Promise resolving to audio blob
|
||||||
|
* @throws ApiError if the request fails
|
||||||
|
*/
|
||||||
|
export async function previewVoice(payload: VoicePreviewRequest): Promise<Blob> {
|
||||||
|
const baseUrl = getApiBaseUrl();
|
||||||
|
const url = `${baseUrl}/api/tts/preview`;
|
||||||
|
|
||||||
|
const response = await fetchWithErrorHandling(url, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
body: JSON.stringify(payload),
|
||||||
|
});
|
||||||
|
|
||||||
|
// Convert response to blob for audio data
|
||||||
|
return await response.blob();
|
||||||
|
}
|
||||||
|
|||||||
@@ -8,29 +8,51 @@ export default function AudioPlayerSection({
|
|||||||
onDownload
|
onDownload
|
||||||
}: AudioPlayerSectionProps) {
|
}: AudioPlayerSectionProps) {
|
||||||
return (
|
return (
|
||||||
<div className="w-full max-w-4xl mx-auto p-4 sm:p-6">
|
<div className="max-w-4xl mx-auto">
|
||||||
<h2 className="text-xl sm:text-2xl font-bold mb-4 sm:mb-6">Your Audiobook is Ready!</h2>
|
<div className="card-modern p-10 sm:p-12 space-y-10">
|
||||||
|
{/* Success Header */}
|
||||||
|
<div className="text-center">
|
||||||
|
<div className="inline-flex items-center justify-center w-24 h-24 bg-gradient-to-br from-green-400 to-emerald-500 rounded-3xl mb-6 shadow-xl shadow-green-500/30">
|
||||||
|
<span className="text-5xl">🎉</span>
|
||||||
|
</div>
|
||||||
|
<h2 className="text-4xl font-black text-gray-900 mb-4">
|
||||||
|
Your Audiobook is Ready!
|
||||||
|
</h2>
|
||||||
|
<p className="text-xl text-gray-600">
|
||||||
|
Listen below or download to enjoy offline
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
<div className="bg-white rounded-lg shadow-md p-4 sm:p-6 mb-4 sm:mb-6">
|
{/* Audio Player */}
|
||||||
<h3 className="text-base sm:text-lg font-semibold mb-3 sm:mb-4">Listen to your audiobook</h3>
|
<div className="p-8 bg-gradient-to-br from-indigo-50 via-purple-50 to-pink-50 rounded-2xl border-2 border-indigo-200">
|
||||||
<audio
|
<audio
|
||||||
controls
|
controls
|
||||||
src={audioUrl}
|
src={audioUrl}
|
||||||
className="w-full min-h-[44px]"
|
className="w-full"
|
||||||
preload="metadata"
|
preload="metadata"
|
||||||
>
|
>
|
||||||
Your browser does not support the audio element.
|
Your browser does not support the audio element.
|
||||||
</audio>
|
</audio>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{/* Download Button */}
|
||||||
<div className="flex justify-center">
|
<div className="flex justify-center">
|
||||||
<button
|
<button
|
||||||
onClick={onDownload}
|
onClick={onDownload}
|
||||||
className="w-full sm:w-auto px-6 py-3 min-h-[44px] bg-green-600 hover:bg-green-700 active:bg-green-800 text-white font-semibold rounded-lg transition-colors cursor-pointer touch-manipulation"
|
className="btn-primary flex items-center gap-3"
|
||||||
>
|
>
|
||||||
Download audiobook
|
<span className="text-xl">⬇️</span>
|
||||||
|
Download Audiobook
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{/* Success Message */}
|
||||||
|
<div className="p-6 bg-green-50 border-2 border-green-200 text-green-900 rounded-2xl text-center">
|
||||||
|
<p className="font-bold text-lg">
|
||||||
|
✅ Your audiobook has been successfully generated!
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
161
frontend/src/components/ScriptReviewSection.tsx
Normal file
161
frontend/src/components/ScriptReviewSection.tsx
Normal file
@@ -0,0 +1,161 @@
|
|||||||
|
import type { Role, ScriptSegment } from '../api/tts';
|
||||||
|
|
||||||
|
interface ScriptReviewSectionProps {
|
||||||
|
segments: ScriptSegment[];
|
||||||
|
roles: Role[];
|
||||||
|
onSegmentChange: (segmentId: string, newSpeaker: string) => void;
|
||||||
|
onSegmentTextChange: (segmentId: string, newText: string) => void;
|
||||||
|
onSegmentSplit: (segmentId: string, splitPosition: number) => void;
|
||||||
|
onSegmentDelete: (segmentId: string) => void;
|
||||||
|
onGenerate: () => void;
|
||||||
|
onSkipReview: () => void;
|
||||||
|
isGenerating: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export default function ScriptReviewSection({
|
||||||
|
segments,
|
||||||
|
roles,
|
||||||
|
onSegmentChange,
|
||||||
|
onSegmentTextChange,
|
||||||
|
onSegmentSplit,
|
||||||
|
onSegmentDelete,
|
||||||
|
onGenerate,
|
||||||
|
onSkipReview,
|
||||||
|
isGenerating,
|
||||||
|
}: ScriptReviewSectionProps) {
|
||||||
|
const handleSpeakerChange = (segmentId: string, newSpeaker: string) => {
|
||||||
|
onSegmentChange(segmentId, newSpeaker);
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleTextChange = (segmentId: string, newText: string) => {
|
||||||
|
onSegmentTextChange(segmentId, newText);
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleSplit = (segmentId: string, text: string) => {
|
||||||
|
// Find a good split point (after a sentence or quote)
|
||||||
|
const quoteMatch = text.match(/^(.*?[.!?]"?\s*)(.*)/s);
|
||||||
|
if (quoteMatch && quoteMatch[1].length > 0 && quoteMatch[2].length > 0) {
|
||||||
|
onSegmentSplit(segmentId, quoteMatch[1].length);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fall back to middle split
|
||||||
|
const midPoint = Math.floor(text.length / 2);
|
||||||
|
const splitPoint = text.lastIndexOf(' ', midPoint);
|
||||||
|
if (splitPoint > 0) {
|
||||||
|
onSegmentSplit(segmentId, splitPoint + 1);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="max-w-5xl mx-auto">
|
||||||
|
<div className="card-modern p-10 sm:p-12 space-y-10">
|
||||||
|
<div>
|
||||||
|
<h2 className="text-3xl font-bold text-gray-900 mb-3">Review Script</h2>
|
||||||
|
<p className="text-lg text-gray-600">Review and adjust speaker assignments before generating your audiobook.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Script Segments */}
|
||||||
|
<div className="space-y-6">
|
||||||
|
{segments.map((segment, index) => (
|
||||||
|
<div
|
||||||
|
key={segment.id}
|
||||||
|
className="p-6 bg-gradient-to-br from-gray-50 to-white rounded-2xl border-2 border-gray-200 hover:border-indigo-300 transition-all duration-200"
|
||||||
|
>
|
||||||
|
{/* Header */}
|
||||||
|
<div className="flex flex-col sm:flex-row sm:items-center gap-4 mb-4">
|
||||||
|
<div className="flex items-center gap-4">
|
||||||
|
<div className="flex items-center justify-center w-10 h-10 bg-gradient-to-br from-indigo-500 to-purple-600 text-white rounded-xl font-bold text-sm shadow-lg shadow-indigo-500/30">
|
||||||
|
{index + 1}
|
||||||
|
</div>
|
||||||
|
<span className="text-sm font-bold text-gray-500 uppercase tracking-wide">Segment</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Speaker Dropdown */}
|
||||||
|
<div className="flex items-center gap-3 flex-1">
|
||||||
|
<label className="text-base font-bold text-gray-700 whitespace-nowrap">
|
||||||
|
Speaker:
|
||||||
|
</label>
|
||||||
|
<select
|
||||||
|
value={segment.speaker}
|
||||||
|
onChange={(e) => handleSpeakerChange(segment.id, e.target.value)}
|
||||||
|
className="flex-1 p-3 border-2 border-gray-200 rounded-xl focus:border-indigo-500 focus:ring-4 focus:ring-indigo-100 focus:outline-none bg-white font-medium transition-all duration-200"
|
||||||
|
disabled={isGenerating}
|
||||||
|
>
|
||||||
|
{roles.map((role) => (
|
||||||
|
<option key={role.id} value={role.id}>
|
||||||
|
{role.label}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Segment Text */}
|
||||||
|
<textarea
|
||||||
|
value={segment.text}
|
||||||
|
onChange={(e) => handleTextChange(segment.id, e.target.value)}
|
||||||
|
className="w-full p-5 bg-white rounded-xl border-2 border-gray-200 text-gray-800 leading-relaxed text-base focus:border-indigo-500 focus:ring-4 focus:ring-indigo-100 focus:outline-none resize-y min-h-[100px] font-sans"
|
||||||
|
disabled={isGenerating}
|
||||||
|
/>
|
||||||
|
|
||||||
|
{/* Action Buttons */}
|
||||||
|
<div className="flex gap-2 mt-4">
|
||||||
|
<button
|
||||||
|
onClick={() => handleSplit(segment.id, segment.text)}
|
||||||
|
disabled={isGenerating || segment.text.length < 20}
|
||||||
|
className="px-4 py-2 text-sm font-semibold text-indigo-600 bg-indigo-50 rounded-lg hover:bg-indigo-100 disabled:opacity-50 disabled:cursor-not-allowed transition-all duration-200 flex items-center gap-2"
|
||||||
|
title="Split this segment into two parts"
|
||||||
|
>
|
||||||
|
<span>✂️</span>
|
||||||
|
Split Segment
|
||||||
|
</button>
|
||||||
|
|
||||||
|
{segments.length > 1 && (
|
||||||
|
<button
|
||||||
|
onClick={() => onSegmentDelete(segment.id)}
|
||||||
|
disabled={isGenerating}
|
||||||
|
className="px-4 py-2 text-sm font-semibold text-red-600 bg-red-50 rounded-lg hover:bg-red-100 disabled:opacity-50 disabled:cursor-not-allowed transition-all duration-200 flex items-center gap-2"
|
||||||
|
title="Delete this segment"
|
||||||
|
>
|
||||||
|
<span>🗑️</span>
|
||||||
|
Delete
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Action Buttons */}
|
||||||
|
<div className="flex flex-col sm:flex-row gap-4 justify-end pt-4">
|
||||||
|
<button
|
||||||
|
onClick={onSkipReview}
|
||||||
|
disabled={isGenerating}
|
||||||
|
className="px-8 py-4 border-2 border-gray-300 text-gray-700 font-semibold rounded-xl hover:bg-gray-50 hover:border-gray-400 disabled:opacity-50 disabled:cursor-not-allowed transition-all duration-200"
|
||||||
|
>
|
||||||
|
Back
|
||||||
|
</button>
|
||||||
|
|
||||||
|
<button
|
||||||
|
onClick={onGenerate}
|
||||||
|
disabled={isGenerating}
|
||||||
|
className="btn-primary"
|
||||||
|
>
|
||||||
|
{isGenerating ? (
|
||||||
|
<span className="flex items-center gap-3">
|
||||||
|
<div className="animate-spin rounded-full h-5 w-5 border-2 border-white border-t-transparent"></div>
|
||||||
|
Generating...
|
||||||
|
</span>
|
||||||
|
) : (
|
||||||
|
<span className="flex items-center gap-3">
|
||||||
|
Generate Audiobook
|
||||||
|
<span className="text-lg">✨</span>
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -36,36 +36,59 @@ export default function TextInputSection({
|
|||||||
const isButtonDisabled = disabled || isLoading || text.trim() === '';
|
const isButtonDisabled = disabled || isLoading || text.trim() === '';
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div className="w-full max-w-4xl mx-auto p-4 sm:p-6">
|
<div className="max-w-4xl mx-auto">
|
||||||
<h2 className="text-xl sm:text-2xl font-bold mb-3 sm:mb-4">Enter Your Text</h2>
|
<div className="card-modern p-10 sm:p-12">
|
||||||
|
<div className="mb-10">
|
||||||
|
<h2 className="text-3xl font-bold text-gray-900 mb-3">Enter Your Text</h2>
|
||||||
|
<p className="text-lg text-gray-600">Paste or type your story below. Use patterns like "Name:" to mark different speakers.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
<div className="mb-4">
|
<div className="mb-8 relative">
|
||||||
<textarea
|
<textarea
|
||||||
value={text}
|
value={text}
|
||||||
onChange={handleTextChange}
|
onChange={handleTextChange}
|
||||||
placeholder="Enter your text here. Use patterns like 'Name:' or '[Name]' to mark different speakers..."
|
placeholder="Example:
|
||||||
className="w-full h-64 sm:h-80 md:h-96 p-3 sm:p-4 border-2 border-gray-300 rounded-lg focus:border-blue-500 focus:outline-none resize-y text-sm sm:text-base touch-manipulation"
|
|
||||||
|
Sarah: Hello! How are you today?
|
||||||
|
John: I'm doing great, thanks for asking!
|
||||||
|
Sarah: That's wonderful to hear."
|
||||||
|
className="w-full h-96 p-6 border-2 border-gray-200 rounded-2xl focus:border-indigo-500 focus:ring-4 focus:ring-indigo-100 focus:outline-none resize-y text-base leading-relaxed transition-all duration-200 bg-gray-50 focus:bg-white"
|
||||||
disabled={disabled || isLoading}
|
disabled={disabled || isLoading}
|
||||||
/>
|
/>
|
||||||
|
<div className="absolute bottom-5 right-5 text-sm text-gray-500 bg-white px-4 py-2 rounded-xl border border-gray-200 shadow-sm font-medium">
|
||||||
|
{text.length} characters
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{showValidation && text.trim() === '' && (
|
{showValidation && text.trim() === '' && (
|
||||||
<div className="mb-4 p-3 bg-red-100 border border-red-400 text-red-700 rounded text-sm sm:text-base">
|
<div className="mb-8 p-5 bg-red-50 border-2 border-red-200 text-red-700 rounded-2xl">
|
||||||
Please enter some text before proceeding.
|
<div className="flex items-center gap-3">
|
||||||
|
<span className="text-xl">⚠️</span>
|
||||||
|
<p className="font-semibold">Please enter some text before proceeding.</p>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
<div className="flex justify-end">
|
||||||
<button
|
<button
|
||||||
onClick={handleNext}
|
onClick={handleNext}
|
||||||
disabled={isButtonDisabled}
|
disabled={isButtonDisabled}
|
||||||
className={`w-full sm:w-auto px-6 py-3 min-h-[44px] rounded-lg font-semibold text-white transition-colors touch-manipulation ${
|
className="btn-primary"
|
||||||
isButtonDisabled
|
|
||||||
? 'bg-gray-400 cursor-not-allowed'
|
|
||||||
: 'bg-blue-600 hover:bg-blue-700 active:bg-blue-800 cursor-pointer'
|
|
||||||
}`}
|
|
||||||
>
|
>
|
||||||
{isLoading ? 'Processing...' : 'Next: Assign voices'}
|
{isLoading ? (
|
||||||
|
<span className="flex items-center gap-3">
|
||||||
|
<div className="animate-spin rounded-full h-5 w-5 border-2 border-white border-t-transparent"></div>
|
||||||
|
Processing...
|
||||||
|
</span>
|
||||||
|
) : (
|
||||||
|
<span className="flex items-center gap-3">
|
||||||
|
Continue
|
||||||
|
<span className="text-lg">→</span>
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,14 +1,21 @@
|
|||||||
|
import { useState, useEffect } from 'react';
|
||||||
import type { Role } from '../api/tts';
|
import type { Role } from '../api/tts';
|
||||||
|
import { previewVoice, getEngines, setEngine, getVoices } from '../api/tts';
|
||||||
|
|
||||||
|
type AgeCategory = 'young' | 'teen' | 'adult' | 'old';
|
||||||
|
|
||||||
interface VoiceAssignmentSectionProps {
|
interface VoiceAssignmentSectionProps {
|
||||||
roles: Role[];
|
roles: Role[];
|
||||||
voiceAssignments: Map<string, string>;
|
voiceAssignments: Map<string, string>;
|
||||||
|
ageSettings: Map<string, AgeCategory>;
|
||||||
onVoiceChange: (roleId: string, voiceId: string) => void;
|
onVoiceChange: (roleId: string, voiceId: string) => void;
|
||||||
|
onAgeChange: (roleId: string, age: AgeCategory) => void;
|
||||||
language: string;
|
language: string;
|
||||||
onLanguageChange: (lang: string) => void;
|
onLanguageChange: (lang: string) => void;
|
||||||
speed: number;
|
speed: number;
|
||||||
onSpeedChange: (speed: number) => void;
|
onSpeedChange: (speed: number) => void;
|
||||||
onGenerate: () => void;
|
onGenerate: () => void;
|
||||||
|
onSkipReview?: () => void;
|
||||||
isGenerating: boolean;
|
isGenerating: boolean;
|
||||||
allRolesAssigned: boolean;
|
allRolesAssigned: boolean;
|
||||||
}
|
}
|
||||||
@@ -129,20 +136,168 @@ const LANGUAGES: LanguageOption[] = [
|
|||||||
{ code: 'z', label: 'Chinese (中文)' },
|
{ code: 'z', label: 'Chinese (中文)' },
|
||||||
];
|
];
|
||||||
|
|
||||||
|
// Age category to speed mapping
|
||||||
|
const AGE_SPEED_MAP: Record<AgeCategory, number> = {
|
||||||
|
young: 1.15,
|
||||||
|
teen: 1.08,
|
||||||
|
adult: 1.0,
|
||||||
|
old: 0.85,
|
||||||
|
};
|
||||||
|
|
||||||
export default function VoiceAssignmentSection({
|
export default function VoiceAssignmentSection({
|
||||||
roles,
|
roles,
|
||||||
voiceAssignments,
|
voiceAssignments,
|
||||||
|
ageSettings,
|
||||||
onVoiceChange,
|
onVoiceChange,
|
||||||
|
onAgeChange,
|
||||||
language,
|
language,
|
||||||
onLanguageChange,
|
onLanguageChange,
|
||||||
speed,
|
speed,
|
||||||
onSpeedChange,
|
onSpeedChange,
|
||||||
onGenerate,
|
onGenerate,
|
||||||
|
onSkipReview,
|
||||||
isGenerating,
|
isGenerating,
|
||||||
allRolesAssigned,
|
allRolesAssigned,
|
||||||
}: VoiceAssignmentSectionProps) {
|
}: VoiceAssignmentSectionProps) {
|
||||||
|
const [previewingRole, setPreviewingRole] = useState<string | null>(null);
|
||||||
|
const [audioCache, setAudioCache] = useState<Map<string, string>>(new Map());
|
||||||
|
const [currentEngine, setCurrentEngine] = useState<string>('kokoro');
|
||||||
|
const [availableVoices, setAvailableVoices] = useState<VoiceOption[]>(KOKORO_VOICES);
|
||||||
|
const [isLoadingEngine, setIsLoadingEngine] = useState(false);
|
||||||
|
|
||||||
|
// Load current engine and voices on mount
|
||||||
|
useEffect(() => {
|
||||||
|
const loadEngineInfo = async () => {
|
||||||
|
try {
|
||||||
|
const [enginesData, voicesData] = await Promise.all([
|
||||||
|
getEngines(),
|
||||||
|
getVoices()
|
||||||
|
]);
|
||||||
|
|
||||||
|
setCurrentEngine(enginesData.current);
|
||||||
|
|
||||||
|
// Update available voices based on current engine
|
||||||
|
if (enginesData.current === 'piper' && voicesData.voices.piper) {
|
||||||
|
const piperVoices: VoiceOption[] = voicesData.voices.piper.map(voiceId => ({
|
||||||
|
id: voiceId,
|
||||||
|
label: voiceId.replace(/_/g, ' ').replace(/-/g, ' - '),
|
||||||
|
gender: voiceId.includes('amy') || voiceId.includes('alba') ? 'female' : 'male',
|
||||||
|
accent: voiceId.includes('GB') ? 'British' : 'American'
|
||||||
|
}));
|
||||||
|
setAvailableVoices(piperVoices);
|
||||||
|
} else {
|
||||||
|
setAvailableVoices(KOKORO_VOICES);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
console.error('Failed to load engine info:', error);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
loadEngineInfo();
|
||||||
|
}, []);
|
||||||
|
|
||||||
const handleVoiceSelect = (roleId: string, voiceId: string) => {
|
const handleVoiceSelect = (roleId: string, voiceId: string) => {
|
||||||
onVoiceChange(roleId, voiceId);
|
onVoiceChange(roleId, voiceId);
|
||||||
|
// Clear all cached previews for this role when voice changes
|
||||||
|
setAudioCache(prev => {
|
||||||
|
const newCache = new Map(prev);
|
||||||
|
// Remove all cache entries that start with this roleId
|
||||||
|
Array.from(newCache.keys()).forEach(key => {
|
||||||
|
if (key.startsWith(`${roleId}-`)) {
|
||||||
|
newCache.delete(key);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return newCache;
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleAgeSelect = (roleId: string, age: AgeCategory) => {
|
||||||
|
onAgeChange(roleId, age);
|
||||||
|
// Clear all cached previews for this role when age changes
|
||||||
|
setAudioCache(prev => {
|
||||||
|
const newCache = new Map(prev);
|
||||||
|
// Remove all cache entries that start with this roleId
|
||||||
|
Array.from(newCache.keys()).forEach(key => {
|
||||||
|
if (key.startsWith(`${roleId}-`)) {
|
||||||
|
newCache.delete(key);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return newCache;
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
const handlePreview = async (roleId: string) => {
|
||||||
|
const voiceId = voiceAssignments.get(roleId);
|
||||||
|
if (!voiceId) return;
|
||||||
|
|
||||||
|
const age = ageSettings.get(roleId) || 'adult';
|
||||||
|
const previewSpeed = AGE_SPEED_MAP[age];
|
||||||
|
const cacheKey = `${roleId}-${voiceId}-${age}`;
|
||||||
|
|
||||||
|
setPreviewingRole(roleId);
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Check cache first
|
||||||
|
let audioUrl = audioCache.get(cacheKey);
|
||||||
|
|
||||||
|
if (!audioUrl) {
|
||||||
|
// Generate new preview
|
||||||
|
const audioBlob = await previewVoice({
|
||||||
|
voice: voiceId,
|
||||||
|
lang: language,
|
||||||
|
speed: previewSpeed,
|
||||||
|
});
|
||||||
|
|
||||||
|
audioUrl = URL.createObjectURL(audioBlob);
|
||||||
|
setAudioCache(prev => new Map(prev).set(cacheKey, audioUrl!));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Play the audio
|
||||||
|
if (audioUrl) {
|
||||||
|
const audio = new Audio(audioUrl);
|
||||||
|
audio.onended = () => setPreviewingRole(null);
|
||||||
|
audio.onerror = () => setPreviewingRole(null);
|
||||||
|
await audio.play();
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
console.error('Preview failed:', error);
|
||||||
|
setPreviewingRole(null);
|
||||||
|
|
||||||
|
// Show user-friendly error message
|
||||||
|
if (error instanceof Error) {
|
||||||
|
alert(`Voice preview failed: ${error.message}\n\nIf using Piper TTS, make sure voice models are downloaded. See backend/PIPER_SETUP.md for instructions.`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleEngineChange = async (engine: string) => {
|
||||||
|
setIsLoadingEngine(true);
|
||||||
|
try {
|
||||||
|
await setEngine(engine);
|
||||||
|
setCurrentEngine(engine);
|
||||||
|
|
||||||
|
// Load voices for new engine
|
||||||
|
const voicesData = await getVoices();
|
||||||
|
|
||||||
|
if (engine === 'piper' && voicesData.voices.piper) {
|
||||||
|
const piperVoices: VoiceOption[] = voicesData.voices.piper.map(voiceId => ({
|
||||||
|
id: voiceId,
|
||||||
|
label: voiceId.replace(/_/g, ' ').replace(/-/g, ' - '),
|
||||||
|
gender: voiceId.includes('amy') || voiceId.includes('alba') ? 'female' : 'male',
|
||||||
|
accent: voiceId.includes('GB') ? 'British' : 'American'
|
||||||
|
}));
|
||||||
|
setAvailableVoices(piperVoices);
|
||||||
|
} else {
|
||||||
|
setAvailableVoices(KOKORO_VOICES);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Clear audio cache when switching engines
|
||||||
|
setAudioCache(new Map());
|
||||||
|
} catch (error) {
|
||||||
|
console.error('Failed to switch engine:', error);
|
||||||
|
} finally {
|
||||||
|
setIsLoadingEngine(false);
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
const handleLanguageSelect = (e: React.ChangeEvent<HTMLSelectElement>) => {
|
const handleLanguageSelect = (e: React.ChangeEvent<HTMLSelectElement>) => {
|
||||||
@@ -157,42 +312,166 @@ export default function VoiceAssignmentSection({
|
|||||||
const isGenerateDisabled = !allRolesAssigned || isGenerating;
|
const isGenerateDisabled = !allRolesAssigned || isGenerating;
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div className="w-full max-w-4xl mx-auto p-4 sm:p-6">
|
<div className="max-w-5xl mx-auto">
|
||||||
<h2 className="text-xl sm:text-2xl font-bold mb-4 sm:mb-6">Assign voices to roles</h2>
|
<div className="card-modern p-10 sm:p-12 space-y-10">
|
||||||
|
<div>
|
||||||
|
<h2 className="text-3xl font-bold text-gray-900 mb-3">Assign Voices</h2>
|
||||||
|
<p className="text-lg text-gray-600">Choose a voice for each character in your story.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
{/* Role-to-Voice Mapping Section */}
|
{/* TTS Engine Selector */}
|
||||||
<div className="mb-6 sm:mb-8">
|
<div className="p-6 bg-gradient-to-br from-indigo-50 to-purple-50 rounded-2xl border-2 border-indigo-200">
|
||||||
<h3 className="text-lg sm:text-xl font-semibold mb-3 sm:mb-4">Voice Assignments</h3>
|
<label className="block text-base font-bold text-gray-900 mb-3">
|
||||||
<div className="space-y-3 sm:space-y-4">
|
TTS Engine
|
||||||
{roles.map((role) => (
|
|
||||||
<div key={role.id} className="flex flex-col sm:flex-row sm:items-center gap-2 sm:gap-4">
|
|
||||||
<label className="font-medium text-gray-700 text-sm sm:text-base sm:w-40">
|
|
||||||
{role.label}:
|
|
||||||
</label>
|
</label>
|
||||||
|
<div className="flex gap-4">
|
||||||
|
<button
|
||||||
|
onClick={() => handleEngineChange('kokoro')}
|
||||||
|
disabled={isLoadingEngine}
|
||||||
|
className={`flex-1 px-6 py-4 rounded-xl font-semibold transition-all duration-200 ${
|
||||||
|
currentEngine === 'kokoro'
|
||||||
|
? 'bg-indigo-600 text-white shadow-lg shadow-indigo-500/30'
|
||||||
|
: 'bg-white text-gray-700 border-2 border-gray-200 hover:border-indigo-300'
|
||||||
|
}`}
|
||||||
|
>
|
||||||
|
<div className="text-left">
|
||||||
|
<div className="font-bold">Kokoro TTS</div>
|
||||||
|
<div className="text-sm opacity-90">High quality, 54 voices</div>
|
||||||
|
</div>
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
onClick={() => handleEngineChange('piper')}
|
||||||
|
disabled={isLoadingEngine}
|
||||||
|
className={`flex-1 px-6 py-4 rounded-xl font-semibold transition-all duration-200 ${
|
||||||
|
currentEngine === 'piper'
|
||||||
|
? 'bg-indigo-600 text-white shadow-lg shadow-indigo-500/30'
|
||||||
|
: 'bg-white text-gray-700 border-2 border-gray-200 hover:border-indigo-300'
|
||||||
|
}`}
|
||||||
|
>
|
||||||
|
<div className="text-left">
|
||||||
|
<div className="font-bold">Piper TTS</div>
|
||||||
|
<div className="text-sm opacity-90">Fast, pitch control</div>
|
||||||
|
</div>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
{currentEngine === 'piper' && (
|
||||||
|
<div className="mt-3 space-y-2">
|
||||||
|
<div className="p-3 bg-blue-50 border border-blue-200 rounded-lg">
|
||||||
|
<p className="text-sm text-blue-900">
|
||||||
|
<strong>✨ Pitch Control:</strong> Piper allows you to adjust voice pitch per character (coming soon in UI)
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
<div className="p-3 bg-yellow-50 border border-yellow-300 rounded-lg">
|
||||||
|
<p className="text-sm text-yellow-900">
|
||||||
|
<strong>⚠️ Setup Required:</strong> Piper voice models must be downloaded manually.
|
||||||
|
See <code className="bg-yellow-100 px-1 rounded">backend/PIPER_SETUP.md</code> for instructions.
|
||||||
|
If voices don't work, you may need to download the model files first.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Voice Assignments */}
|
||||||
|
<div className="space-y-5">
|
||||||
|
{roles.map((role, index) => {
|
||||||
|
const selectedAge = ageSettings.get(role.id) || 'adult';
|
||||||
|
const hasVoice = voiceAssignments.has(role.id);
|
||||||
|
const isPreviewing = previewingRole === role.id;
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div key={role.id} className="p-6 bg-gradient-to-br from-gray-50 to-white rounded-2xl border-2 border-gray-200 hover:border-indigo-300 transition-all duration-200">
|
||||||
|
{/* Header Row */}
|
||||||
|
<div className="flex flex-col sm:flex-row sm:items-center gap-4 mb-4">
|
||||||
|
<div className="flex items-center gap-4 sm:w-56">
|
||||||
|
<div className="flex items-center justify-center w-10 h-10 bg-gradient-to-br from-indigo-500 to-purple-600 text-white rounded-xl font-bold text-sm shadow-lg shadow-indigo-500/30">
|
||||||
|
{index + 1}
|
||||||
|
</div>
|
||||||
|
<label className="font-bold text-gray-900 text-lg">
|
||||||
|
{role.label}
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
<select
|
<select
|
||||||
value={voiceAssignments.get(role.id) || ''}
|
value={voiceAssignments.get(role.id) || ''}
|
||||||
onChange={(e) => handleVoiceSelect(role.id, e.target.value)}
|
onChange={(e) => handleVoiceSelect(role.id, e.target.value)}
|
||||||
className="flex-1 p-3 sm:p-2 min-h-[44px] border-2 border-gray-300 rounded-lg focus:border-blue-500 focus:outline-none text-sm sm:text-base touch-manipulation"
|
className="flex-1 p-4 border-2 border-gray-200 rounded-xl focus:border-indigo-500 focus:ring-4 focus:ring-indigo-100 focus:outline-none bg-white text-base font-medium transition-all duration-200"
|
||||||
>
|
>
|
||||||
<option value="">Select a voice...</option>
|
<option value="">Select a voice...</option>
|
||||||
{KOKORO_VOICES.map((voice) => (
|
{availableVoices.map((voice) => (
|
||||||
<option key={voice.id} value={voice.id}>
|
<option key={voice.id} value={voice.id}>
|
||||||
{voice.label}
|
{voice.label}
|
||||||
</option>
|
</option>
|
||||||
))}
|
))}
|
||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{/* Age Category & Preview Row */}
|
||||||
|
{hasVoice && (
|
||||||
|
<div className="flex flex-col sm:flex-row gap-4 items-start sm:items-center">
|
||||||
|
{/* Age Categories */}
|
||||||
|
<div className="flex-1">
|
||||||
|
<label className="block text-sm font-semibold text-gray-700 mb-2">
|
||||||
|
Age Category:
|
||||||
|
</label>
|
||||||
|
<div className="flex gap-2 flex-wrap">
|
||||||
|
{(['young', 'teen', 'adult', 'old'] as AgeCategory[]).map((age) => (
|
||||||
|
<button
|
||||||
|
key={age}
|
||||||
|
onClick={() => handleAgeSelect(role.id, age)}
|
||||||
|
className={`px-4 py-2 rounded-lg font-semibold text-sm transition-all duration-200 ${
|
||||||
|
selectedAge === age
|
||||||
|
? 'bg-indigo-600 text-white shadow-lg shadow-indigo-500/30'
|
||||||
|
: 'bg-white text-gray-700 border-2 border-gray-200 hover:border-indigo-300'
|
||||||
|
}`}
|
||||||
|
>
|
||||||
|
{age === 'young' && '👶 Young'}
|
||||||
|
{age === 'teen' && '🧒 Teen'}
|
||||||
|
{age === 'adult' && '👤 Adult'}
|
||||||
|
{age === 'old' && '👴 Old'}
|
||||||
|
</button>
|
||||||
))}
|
))}
|
||||||
</div>
|
</div>
|
||||||
|
<p className="text-xs text-gray-500 mt-2">
|
||||||
|
Speed: {AGE_SPEED_MAP[selectedAge]}x
|
||||||
|
</p>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Language Selector */}
|
{/* Preview Button */}
|
||||||
<div className="mb-6">
|
<button
|
||||||
<h3 className="text-lg sm:text-xl font-semibold mb-3 sm:mb-4">Language</h3>
|
onClick={() => handlePreview(role.id)}
|
||||||
|
disabled={isPreviewing || isGenerating}
|
||||||
|
className="px-6 py-3 bg-green-600 text-white font-semibold rounded-xl hover:bg-green-700 disabled:opacity-50 disabled:cursor-not-allowed transition-all duration-200 flex items-center gap-2 shadow-lg shadow-green-500/30"
|
||||||
|
>
|
||||||
|
{isPreviewing ? (
|
||||||
|
<>
|
||||||
|
<div className="animate-spin rounded-full h-4 w-4 border-2 border-white border-t-transparent"></div>
|
||||||
|
Playing...
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
<span>🔊</span>
|
||||||
|
Try Voice
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Settings Grid */}
|
||||||
|
<div className="grid sm:grid-cols-2 gap-8 pt-4">
|
||||||
|
{/* Language */}
|
||||||
|
<div>
|
||||||
|
<label className="block text-base font-bold text-gray-900 mb-3">
|
||||||
|
Language
|
||||||
|
</label>
|
||||||
<select
|
<select
|
||||||
value={language}
|
value={language}
|
||||||
onChange={handleLanguageSelect}
|
onChange={handleLanguageSelect}
|
||||||
className="w-full p-3 sm:p-2 min-h-[44px] border-2 border-gray-300 rounded-lg focus:border-blue-500 focus:outline-none text-sm sm:text-base touch-manipulation"
|
className="w-full p-4 border-2 border-gray-200 rounded-xl focus:border-indigo-500 focus:ring-4 focus:ring-indigo-100 focus:outline-none bg-white text-base font-medium transition-all duration-200"
|
||||||
>
|
>
|
||||||
{LANGUAGES.map((lang) => (
|
{LANGUAGES.map((lang) => (
|
||||||
<option key={lang.code} value={lang.code}>
|
<option key={lang.code} value={lang.code}>
|
||||||
@@ -202,11 +481,11 @@ export default function VoiceAssignmentSection({
|
|||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Speed Slider */}
|
{/* Speed */}
|
||||||
<div className="mb-6 sm:mb-8">
|
<div>
|
||||||
<h3 className="text-lg sm:text-xl font-semibold mb-3 sm:mb-4">Playback Speed</h3>
|
<label className="block text-base font-bold text-gray-900 mb-3">
|
||||||
<div className="flex items-center gap-2 sm:gap-4">
|
Playback Speed: <span className="text-indigo-600">{speed.toFixed(1)}x</span>
|
||||||
<span className="text-xs sm:text-sm text-gray-600 w-10 sm:w-12">0.5x</span>
|
</label>
|
||||||
<input
|
<input
|
||||||
type="range"
|
type="range"
|
||||||
min="0.5"
|
min="0.5"
|
||||||
@@ -214,36 +493,52 @@ export default function VoiceAssignmentSection({
|
|||||||
step="0.1"
|
step="0.1"
|
||||||
value={speed}
|
value={speed}
|
||||||
onChange={handleSpeedChange}
|
onChange={handleSpeedChange}
|
||||||
className="flex-1 h-8 sm:h-auto touch-manipulation"
|
className="w-full h-3"
|
||||||
/>
|
/>
|
||||||
<span className="text-xs sm:text-sm text-gray-600 w-10 sm:w-12">1.5x</span>
|
<div className="flex justify-between text-sm text-gray-500 mt-2 font-medium">
|
||||||
<span className="font-semibold text-gray-800 w-12 sm:w-16 text-center text-sm sm:text-base">
|
<span>0.5x</span>
|
||||||
{speed.toFixed(1)}x
|
<span>1.5x</span>
|
||||||
</span>
|
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Generate Button */}
|
|
||||||
<div className="flex justify-center">
|
|
||||||
<button
|
|
||||||
onClick={onGenerate}
|
|
||||||
disabled={isGenerateDisabled}
|
|
||||||
className={`w-full sm:w-auto px-8 py-3 min-h-[44px] rounded-lg font-semibold text-white transition-colors touch-manipulation ${
|
|
||||||
isGenerateDisabled
|
|
||||||
? 'bg-gray-400 cursor-not-allowed'
|
|
||||||
: 'bg-green-600 hover:bg-green-700 active:bg-green-800 cursor-pointer'
|
|
||||||
}`}
|
|
||||||
>
|
|
||||||
{isGenerating ? 'Generating...' : 'Generate Audiobook'}
|
|
||||||
</button>
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Validation Message */}
|
{/* Validation Message */}
|
||||||
{!allRolesAssigned && !isGenerating && (
|
{!allRolesAssigned && !isGenerating && (
|
||||||
<div className="mt-4 p-3 bg-yellow-100 border border-yellow-400 text-yellow-800 rounded text-center text-sm sm:text-base">
|
<div className="p-5 bg-yellow-50 border-2 border-yellow-200 text-yellow-900 rounded-2xl">
|
||||||
Please assign voices to all roles before generating the audiobook.
|
<p className="font-semibold">⚠️ Please assign voices to all roles before continuing.</p>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
{/* Action Buttons */}
|
||||||
|
<div className="flex flex-col sm:flex-row gap-4 justify-end pt-4">
|
||||||
|
{onSkipReview && (
|
||||||
|
<button
|
||||||
|
onClick={onSkipReview}
|
||||||
|
disabled={isGenerateDisabled}
|
||||||
|
className="px-8 py-4 border-2 border-gray-300 text-gray-700 font-semibold rounded-xl hover:bg-gray-50 hover:border-gray-400 disabled:opacity-50 disabled:cursor-not-allowed transition-all duration-200"
|
||||||
|
>
|
||||||
|
Skip Review & Generate
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
<button
|
||||||
|
onClick={onGenerate}
|
||||||
|
disabled={isGenerateDisabled}
|
||||||
|
className="btn-primary"
|
||||||
|
>
|
||||||
|
{isGenerating ? (
|
||||||
|
<span className="flex items-center gap-3">
|
||||||
|
<div className="animate-spin rounded-full h-5 w-5 border-2 border-white border-t-transparent"></div>
|
||||||
|
Generating...
|
||||||
|
</span>
|
||||||
|
) : (
|
||||||
|
<span className="flex items-center gap-3">
|
||||||
|
Continue
|
||||||
|
<span className="text-lg">→</span>
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,15 +1,67 @@
|
|||||||
|
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800;900&display=swap');
|
||||||
|
|
||||||
@tailwind base;
|
@tailwind base;
|
||||||
@tailwind components;
|
@tailwind components;
|
||||||
@tailwind utilities;
|
@tailwind utilities;
|
||||||
|
|
||||||
|
* {
|
||||||
|
box-sizing: border-box;
|
||||||
|
}
|
||||||
|
|
||||||
:root {
|
:root {
|
||||||
font-family: system-ui, Avenir, Helvetica, Arial, sans-serif;
|
font-family: 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||||
line-height: 1.5;
|
line-height: 1.6;
|
||||||
font-weight: 400;
|
font-weight: 400;
|
||||||
|
-webkit-font-smoothing: antialiased;
|
||||||
|
-moz-osx-font-smoothing: grayscale;
|
||||||
}
|
}
|
||||||
|
|
||||||
body {
|
body {
|
||||||
margin: 0;
|
margin: 0;
|
||||||
min-width: 320px;
|
min-width: 320px;
|
||||||
min-height: 100vh;
|
min-height: 100vh;
|
||||||
|
background: linear-gradient(180deg, #f9fafb 0%, #f3f4f6 100%);
|
||||||
|
}
|
||||||
|
|
||||||
|
@layer utilities {
|
||||||
|
.card-modern {
|
||||||
|
background: white;
|
||||||
|
border-radius: 24px;
|
||||||
|
box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.05), 0 2px 4px -1px rgba(0, 0, 0, 0.03);
|
||||||
|
border: 1px solid rgba(0, 0, 0, 0.05);
|
||||||
|
transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
.card-modern:hover {
|
||||||
|
box-shadow: 0 20px 25px -5px rgba(0, 0, 0, 0.08), 0 10px 10px -5px rgba(0, 0, 0, 0.04);
|
||||||
|
transform: translateY(-4px);
|
||||||
|
}
|
||||||
|
|
||||||
|
.btn-primary {
|
||||||
|
background: linear-gradient(135deg, #6366f1 0%, #8b5cf6 100%);
|
||||||
|
color: white;
|
||||||
|
font-weight: 600;
|
||||||
|
padding: 14px 40px;
|
||||||
|
border-radius: 14px;
|
||||||
|
border: none;
|
||||||
|
cursor: pointer;
|
||||||
|
transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
|
||||||
|
box-shadow: 0 4px 14px rgba(99, 102, 241, 0.4);
|
||||||
|
font-size: 16px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.btn-primary:hover:not(:disabled) {
|
||||||
|
transform: translateY(-2px);
|
||||||
|
box-shadow: 0 8px 24px rgba(99, 102, 241, 0.5);
|
||||||
|
}
|
||||||
|
|
||||||
|
.btn-primary:active:not(:disabled) {
|
||||||
|
transform: translateY(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
.btn-primary:disabled {
|
||||||
|
opacity: 0.6;
|
||||||
|
cursor: not-allowed;
|
||||||
|
transform: none;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,8 +5,11 @@ export default {
|
|||||||
"./src/**/*.{js,ts,jsx,tsx}",
|
"./src/**/*.{js,ts,jsx,tsx}",
|
||||||
],
|
],
|
||||||
theme: {
|
theme: {
|
||||||
extend: {},
|
extend: {
|
||||||
|
fontFamily: {
|
||||||
|
sans: ['Inter', 'system-ui', 'sans-serif'],
|
||||||
|
},
|
||||||
|
},
|
||||||
},
|
},
|
||||||
plugins: [],
|
plugins: [],
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
86
test_api_directly.py
Normal file
86
test_api_directly.py
Normal file
@@ -0,0 +1,86 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Test the API directly with the full story to verify backend functionality.
|
||||||
|
"""
|
||||||
|
import requests
|
||||||
|
import json
|
||||||
|
|
||||||
|
# Read the full story
|
||||||
|
with open('Test Story.txt', 'r') as f:
|
||||||
|
story = f.read()
|
||||||
|
|
||||||
|
print(f"Story length: {len(story)} characters\n")
|
||||||
|
|
||||||
|
# Test 1: Role Detection
|
||||||
|
print("=" * 70)
|
||||||
|
print("TEST 1: Role Detection")
|
||||||
|
print("=" * 70)
|
||||||
|
|
||||||
|
response = requests.post(
|
||||||
|
'http://localhost:8002/api/tts/roles',
|
||||||
|
json={'text': story},
|
||||||
|
headers={'Content-Type': 'application/json'}
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"Status: {response.status_code}")
|
||||||
|
if response.status_code == 200:
|
||||||
|
roles_data = response.json()
|
||||||
|
roles = roles_data['roles']
|
||||||
|
print(f"✓ Detected {len(roles)} roles:")
|
||||||
|
for role in roles:
|
||||||
|
print(f" - {role['label']}")
|
||||||
|
else:
|
||||||
|
print(f"❌ Error: {response.text}")
|
||||||
|
exit(1)
|
||||||
|
|
||||||
|
print()
|
||||||
|
|
||||||
|
# Test 2: Audio Generation with Multiple Voices
|
||||||
|
print("=" * 70)
|
||||||
|
print("TEST 2: Audio Generation")
|
||||||
|
print("=" * 70)
|
||||||
|
|
||||||
|
# Assign different voices to each role
|
||||||
|
voice_map = {
|
||||||
|
'Lily': 'af_bella',
|
||||||
|
'Benny': 'am_adam',
|
||||||
|
'Sammy': 'am_michael',
|
||||||
|
'Narrator': 'af_sarah'
|
||||||
|
}
|
||||||
|
|
||||||
|
# Build roles array for API
|
||||||
|
roles_array = []
|
||||||
|
for role in roles:
|
||||||
|
role_id = role['id']
|
||||||
|
voice = voice_map.get(role_id, 'af_bella')
|
||||||
|
roles_array.append({'id': role_id, 'voice': voice})
|
||||||
|
print(f" {role_id} → {voice}")
|
||||||
|
|
||||||
|
print(f"\nGenerating audiobook...")
|
||||||
|
|
||||||
|
response = requests.post(
|
||||||
|
'http://localhost:8002/api/tts',
|
||||||
|
json={
|
||||||
|
'text': story,
|
||||||
|
'lang': 'a',
|
||||||
|
'roles': roles_array,
|
||||||
|
'speed': 1.0
|
||||||
|
},
|
||||||
|
headers={'Content-Type': 'application/json'}
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"Status: {response.status_code}")
|
||||||
|
if response.status_code == 200:
|
||||||
|
# Save the audio file
|
||||||
|
with open('test_output.wav', 'wb') as f:
|
||||||
|
f.write(response.content)
|
||||||
|
print(f"✓ Generated audiobook: {len(response.content)} bytes")
|
||||||
|
print(f"✓ Saved to: test_output.wav")
|
||||||
|
print("\nYou can now play test_output.wav to hear the multi-voice audiobook!")
|
||||||
|
else:
|
||||||
|
print(f"❌ Error: {response.text}")
|
||||||
|
exit(1)
|
||||||
|
|
||||||
|
print("\n" + "=" * 70)
|
||||||
|
print("SUCCESS! Backend is working correctly with full story.")
|
||||||
|
print("=" * 70)
|
||||||
0
test_full_workflow.py
Normal file
0
test_full_workflow.py
Normal file
64
test_my_story_full.py
Normal file
64
test_my_story_full.py
Normal file
@@ -0,0 +1,64 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
import sys
|
||||||
|
sys.path.insert(0, 'backend')
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
from services.role_detector import RoleDetector
|
||||||
|
from services.tts_service import TTSService
|
||||||
|
from services.ollama_service import OllamaService
|
||||||
|
|
||||||
|
async def test():
|
||||||
|
# Read the actual story
|
||||||
|
with open('Test Story.txt', 'r') as f:
|
||||||
|
story = f.read()
|
||||||
|
|
||||||
|
print(f'Story length: {len(story)} characters\n')
|
||||||
|
|
||||||
|
# Initialize services
|
||||||
|
ollama_service = OllamaService()
|
||||||
|
role_detector = RoleDetector(ollama_service=ollama_service)
|
||||||
|
tts_service = TTSService(ollama_service=ollama_service)
|
||||||
|
|
||||||
|
# Step 1: Detect roles
|
||||||
|
print('Step 1: Detecting roles...')
|
||||||
|
roles = await role_detector.detect_roles(story)
|
||||||
|
print(f'Detected {len(roles)} roles: {[r["label"] for r in roles]}\n')
|
||||||
|
|
||||||
|
# Step 2: Assign voices
|
||||||
|
print('Step 2: Assigning voices...')
|
||||||
|
role_voice_map = {
|
||||||
|
'Lily': 'af_bella',
|
||||||
|
'Benny': 'am_adam',
|
||||||
|
'Sammy': 'am_michael',
|
||||||
|
'Narrator': 'af_sarah'
|
||||||
|
}
|
||||||
|
for role, voice in role_voice_map.items():
|
||||||
|
print(f' {role} → {voice}')
|
||||||
|
print()
|
||||||
|
|
||||||
|
# Step 3: Test AI segmentation
|
||||||
|
print('Step 3: Testing AI segmentation...')
|
||||||
|
segments = await tts_service.segment_text_with_ai(story, role_voice_map)
|
||||||
|
|
||||||
|
if segments is None:
|
||||||
|
print('❌ AI segmentation returned None!')
|
||||||
|
print('Falling back to pattern-based segmentation...')
|
||||||
|
segments = tts_service.segment_text(story, role_voice_map)
|
||||||
|
|
||||||
|
print(f'Generated {len(segments)} segments\n')
|
||||||
|
|
||||||
|
# Analyze segments
|
||||||
|
speaker_counts = {}
|
||||||
|
for speaker, voice, text in segments:
|
||||||
|
speaker_counts[speaker] = speaker_counts.get(speaker, 0) + 1
|
||||||
|
|
||||||
|
print('Segments per speaker:')
|
||||||
|
for speaker, count in sorted(speaker_counts.items(), key=lambda x: x[1], reverse=True):
|
||||||
|
print(f' {speaker}: {count} segments')
|
||||||
|
|
||||||
|
print('\nFirst 5 segments:')
|
||||||
|
for i, (speaker, voice, text) in enumerate(segments[:5], 1):
|
||||||
|
preview = text[:80] + '...' if len(text) > 80 else text
|
||||||
|
print(f' {i}. [{speaker}] {preview}')
|
||||||
|
|
||||||
|
asyncio.run(test())
|
||||||
BIN
test_output.wav
Normal file
BIN
test_output.wav
Normal file
Binary file not shown.
Reference in New Issue
Block a user