diff --git a/.kiro/specs/text-to-audiobook/SCRIPT_REVIEW_FEATURE.md b/.kiro/specs/text-to-audiobook/SCRIPT_REVIEW_FEATURE.md new file mode 100644 index 0000000..83caa0a --- /dev/null +++ b/.kiro/specs/text-to-audiobook/SCRIPT_REVIEW_FEATURE.md @@ -0,0 +1,154 @@ +# Script Review Feature - Enhancement Specification + +## Overview + +This document describes the new **Script Review & Editing** feature that addresses the issue where AI-generated speaker assignments may not always be accurate. This feature gives users visibility and control over how the text is segmented and which character speaks each line. + +## Problem Statement + +When using AI to detect speakers in unmarked text, the system may: +- Incorrectly assign dialogue to the wrong character +- Miss some character dialogue +- Assign narrative text to a character instead of the narrator + +Users currently have no way to see or correct these mistakes before generating the audiobook. + +## Solution + +Add a new **Script Review** phase between voice assignment and audio generation where users can: +1. See the complete script breakdown showing which character speaks each line +2. Edit speaker assignments for any segment +3. Skip the review if they trust the AI's assignments + +## User Workflow + +### Updated Flow +``` +1. Text Input + ↓ +2. Role Detection (AI detects: Lily, Benny, Sammy, Narrator) + ↓ +3. Voice Assignment (User assigns voices to each role) + ↓ +4. **Script Review** (NEW - User reviews and edits speaker assignments) + ↓ +5. Audio Generation (Uses edited script) + ↓ +6. Playback & Download +``` + +### Script Review Interface + +The review screen will show: + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Review Script │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ Segment 1: [Narrator ▼] │ +│ ┌────────────────────────────────────────────────────────┐ │ +│ │ Once upon a time, in a tiny village surrounded by │ │ +│ │ lush green forests, there lived three best friends... │ │ +│ └────────────────────────────────────────────────────────┘ │ +│ │ +│ Segment 2: [Lily ▼] │ +│ ┌────────────────────────────────────────────────────────┐ │ +│ │ "Have you guys heard about the new stream that just │ │ +│ │ appeared in the woods?" │ │ +│ └────────────────────────────────────────────────────────┘ │ +│ │ +│ Segment 3: [Benny ▼] │ +│ ┌────────────────────────────────────────────────────────┐ │ +│ │ "Ooh, I love streams! Do we get to go see it now?" │ │ +│ └────────────────────────────────────────────────────────┘ │ +│ │ +│ ... (more segments) │ +│ │ +│ [Skip Review] [Generate Audiobook] │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Technical Implementation + +### New API Endpoint + +**POST /api/tts/script** +- Request: `{ text: string, roles: string[] }` +- Response: `{ segments: [{ id: string, speaker: string, text: string }] }` +- Uses Ollama AI to segment text and assign speakers +- Returns editable script for frontend + +### Updated Generation Endpoint + +**POST /api/tts** (modified) +- Now accepts optional `segments` array +- If segments provided, uses them directly (respecting user edits) +- If no segments, falls back to original text-based segmentation +- Maintains backward compatibility + +### Frontend Components + +**New: ScriptReviewSection** +- Displays all segments with editable speaker dropdowns +- Allows users to change speaker for any segment +- Provides "Skip Review" option for quick generation +- Mobile-responsive design + +**Updated: VoiceAssignmentSection** +- Button changes from "Generate Audiobook" to "Next: Review Script" +- Adds "Skip Review & Generate" option + +**Updated: App Component** +- Adds 'review' phase to workflow +- Manages script segments state +- Handles segment editing + +## Benefits + +1. **Transparency**: Users can see exactly how the AI interpreted their text +2. **Control**: Users can correct any mistakes before generation +3. **Flexibility**: Users can skip review if they trust the AI +4. **Better Results**: Ensures the final audiobook matches user expectations + +## Requirements Added + +- **Requirement 15**: Complete specification for script review feature + - 8 acceptance criteria covering all aspects + - API endpoints, UI components, and user interactions + +## Design Updates + +- Added ScriptReviewSection component +- Added /api/tts/script endpoint +- Updated data models (ScriptSegment, ScriptRequest, ScriptResponse) +- Added 5 new correctness properties (Properties 32-35) +- Updated application workflow to include review phase + +## Implementation Tasks + +Added 8 new tasks (18-25) covering: +- Backend script generation endpoint +- Updated TTS generation to accept edited scripts +- Frontend API client updates +- ScriptReviewSection component +- App component workflow updates +- VoiceAssignmentSection updates +- Documentation +- Integration testing + +## Testing Strategy + +- Property tests for script completeness and structure +- Unit tests for all new components and endpoints +- Integration tests for complete workflow +- Manual testing with various text samples + +## Future Enhancements + +Potential improvements for later: +- Inline editing of segment text +- Merge/split segments +- Bulk speaker assignment +- Save/load script templates +- Export script as text file diff --git a/.kiro/specs/text-to-audiobook/design.md b/.kiro/specs/text-to-audiobook/design.md index c69467b..d6c632b 100644 --- a/.kiro/specs/text-to-audiobook/design.md +++ b/.kiro/specs/text-to-audiobook/design.md @@ -4,10 +4,11 @@ The Text to Audiobook system is a full-stack application that converts long-form text into multi-voice audiobooks using Kokoro TTS. The architecture follows a client-server model with a React TypeScript frontend and a Python FastAPI backend running in Docker. The system automatically detects speaker roles in text, allows voice assignment per role, and generates high-quality audio using the Kokoro-82M model. -The application workflow consists of three main phases: +The application workflow consists of four main phases: 1. **Text Input & Role Detection**: User provides text, system identifies distinct speakers 2. **Voice Assignment**: User maps each detected role to a specific Kokoro voice -3. **Audio Generation & Playback**: System synthesizes multi-voice audio and provides playback/download +3. **Script Review & Editing**: User reviews and edits AI-generated speaker assignments for each text segment +4. **Audio Generation & Playback**: System synthesizes multi-voice audio and provides playback/download ## Architecture @@ -93,10 +94,12 @@ Main application container managing the overall workflow state. - `text: string` - User input text - `roles: Role[]` - Detected roles from backend - `voiceAssignments: Map` - Role ID to Voice ID mapping +- `scriptSegments: ScriptSegment[]` - AI-generated script segments - `selectedLanguage: string` - Language code (default: 'a') - `speed: number` - Playback speed (default: 1.0) - `audioUrl: string | null` - Generated audio blob URL - `isDetectingRoles: boolean` - Loading state for role detection +- `isGeneratingScript: boolean` - Loading state for script generation - `isGenerating: boolean` - Loading state for audio generation - `error: string | null` - Error message if any @@ -104,7 +107,10 @@ Main application container managing the overall workflow state. - `handleTextChange(text: string)` - Updates text state - `handleDetectRoles()` - Calls API to detect roles - `handleVoiceAssignment(roleId: string, voiceId: string)` - Updates voice mapping -- `handleGenerate()` - Calls API to generate audiobook +- `handleGenerateScript()` - Calls API to generate script preview +- `handleSkipReview()` - Skips script review and proceeds to generation +- `handleSegmentChange(segmentId: string, newSpeaker: string)` - Updates speaker for a segment +- `handleGenerate()` - Calls API to generate audiobook with edited script - `handleDownload()` - Triggers download of generated audio #### 2. TextInputSection Component @@ -132,14 +138,25 @@ Displays role-to-voice mapping interface with language and speed controls. - `isGenerating: boolean` - `allRolesAssigned: boolean` -#### 4. AudioPlayerSection Component +#### 4. ScriptReviewSection Component +Displays AI-generated script with editable speaker assignments. + +**Props:** +- `segments: ScriptSegment[]` +- `roles: Role[]` +- `onSegmentChange: (segmentId: string, newSpeaker: string) => void` +- `onGenerate: () => void` +- `onSkipReview: () => void` +- `isGenerating: boolean` + +#### 5. AudioPlayerSection Component Displays audio player and download button after generation. **Props:** - `audioUrl: string` - `onDownload: () => void` -#### 5. API Client (`src/api/tts.ts`) +#### 6. API Client (`src/api/tts.ts`) **Interfaces:** ```typescript @@ -156,8 +173,24 @@ export interface RolesResponse { roles: Role[]; } -export interface GenerateRequest { +export interface ScriptSegment { + id: string; + speaker: string; text: string; +} + +export interface ScriptRequest { + text: string; + roles: string[]; +} + +export interface ScriptResponse { + segments: ScriptSegment[]; +} + +export interface GenerateRequest { + segments?: ScriptSegment[]; + text?: string; lang: string; roles: { id: string; voice: string }[]; speed?: number; @@ -167,6 +200,7 @@ export interface GenerateRequest { **Functions:** ```typescript export async function detectRoles(text: string): Promise +export async function generateScript(request: ScriptRequest): Promise export async function generateAudiobook(payload: GenerateRequest): Promise export async function checkHealth(): Promise<{ status: string }> ``` @@ -188,6 +222,11 @@ async def detect_roles(request: RoleDetectionRequest) -> RolesResponse: """Analyzes text and returns detected roles""" pass +@app.post("/api/tts/script") +async def generate_script(request: ScriptRequest) -> ScriptResponse: + """Generates script preview with AI-segmented text and speaker assignments""" + pass + @app.post("/api/tts") async def generate_audiobook(request: TTSRequest) -> StreamingResponse: """Generates multi-voice audiobook and streams audio""" @@ -206,12 +245,25 @@ class RoleInfo(BaseModel): class RolesResponse(BaseModel): roles: List[RoleInfo] +class ScriptRequest(BaseModel): + text: str + roles: List[str] # List of detected role IDs + +class ScriptSegment(BaseModel): + id: str # Unique segment identifier + speaker: str # Assigned speaker/role + text: str # Text content for this segment + +class ScriptResponse(BaseModel): + segments: List[ScriptSegment] + class RoleVoiceMapping(BaseModel): id: str voice: str class TTSRequest(BaseModel): - text: str + segments: Optional[List[ScriptSegment]] = None # If provided, use edited script + text: Optional[str] = None # Fallback if no script provided lang: str roles: List[RoleVoiceMapping] speed: Optional[float] = 1.0 @@ -384,10 +436,11 @@ interface LanguageOption { **Application State:** ```typescript interface AppState { - phase: 'input' | 'assignment' | 'generated'; + phase: 'input' | 'assignment' | 'review' | 'generated'; text: string; roles: Role[]; voiceAssignments: Map; + scriptSegments: ScriptSegment[]; language: string; speed: number; audioUrl: string | null; @@ -578,6 +631,22 @@ ROLE_PATTERNS = { *For any* text where both AI detection and pattern detection find roles, the backend should merge results and return each unique role exactly once. **Validates: Requirements 14.5** +### Property 32: Script generation completeness +*For any* text and list of roles, the script generation should return segments that cover the entire input text without gaps or overlaps. +**Validates: Requirements 15.2** + +### Property 33: Script segment structure +*For any* generated script, each segment should contain an id, speaker, and text field. +**Validates: Requirements 15.4** + +### Property 34: Script segment editability +*For any* script segment, the user should be able to change the speaker assignment to any detected role. +**Validates: Requirements 15.5** + +### Property 35: Edited script persistence +*For any* edited script segment, the changes should be preserved when generating the audiobook. +**Validates: Requirements 15.7** + ## Error Handling ### Frontend Error Handling diff --git a/.kiro/specs/text-to-audiobook/requirements.md b/.kiro/specs/text-to-audiobook/requirements.md index 6b3102c..cc6f246 100644 --- a/.kiro/specs/text-to-audiobook/requirements.md +++ b/.kiro/specs/text-to-audiobook/requirements.md @@ -15,6 +15,7 @@ The Text to Audiobook system is a production-ready React TypeScript single-page - **KPipeline**: The Kokoro library's pipeline object for text-to-speech synthesis - **Segment**: A portion of text assigned to a specific role for synthesis - **Role Marker**: Text patterns that identify speakers (e.g., "Alice:", "[Narrator]") +- **Script**: The complete breakdown of text into segments with speaker assignments, displayed for user review and editing ## Requirements @@ -192,3 +193,18 @@ The Text to Audiobook system is a production-ready React TypeScript single-page 4. WHEN Ollama is unavailable or fails THEN the Backend SHALL fall back to pattern-based role detection 5. WHEN both AI detection and pattern detection find roles THEN the Backend SHALL merge and deduplicate the results 6. THE Backend SHALL configure Ollama connection using environment variable OLLAMA_BASE_URL with default http://localhost:11434 + +### Requirement 15 + +**User Story:** As a user, I want to review and edit the AI-generated script showing which character speaks each line, so that I can correct any mistakes before generating the audiobook. + +#### Acceptance Criteria + +1. WHEN the user completes voice assignment THEN the Frontend SHALL request a script preview from the Backend before generation +2. WHEN the Backend receives a script preview request THEN the Backend SHALL use AI to segment the text and return a list of segments with speaker assignments +3. WHEN the script preview is received THEN the Frontend SHALL display a new "Review Script" phase showing each segment with its assigned speaker +4. WHEN displaying script segments THEN the Frontend SHALL show each segment in a text box with a dropdown to select the speaker +5. WHEN the user changes a speaker assignment for a segment THEN the Frontend SHALL update the segment mapping +6. WHEN the user is satisfied with the script THEN the Frontend SHALL enable a "Generate Audiobook" button +7. WHEN the user clicks "Generate Audiobook" THEN the Frontend SHALL send the edited script segments to the Backend for synthesis +8. THE Frontend SHALL allow users to skip the review step with a "Skip Review" option that proceeds directly to generation diff --git a/.kiro/specs/text-to-audiobook/tasks.md b/.kiro/specs/text-to-audiobook/tasks.md index b47a364..0da7be9 100644 --- a/.kiro/specs/text-to-audiobook/tasks.md +++ b/.kiro/specs/text-to-audiobook/tasks.md @@ -443,7 +443,7 @@ - Test skip review bypasses review phase - _Requirements: 15.1, 15.8_ -- [ ] 23. Update VoiceAssignmentSection to trigger script generation +- [x] 23. Update VoiceAssignmentSection to trigger script generation - Change "Generate Audiobook" button to "Next: Review Script" - Add "Skip Review & Generate" button as alternative - Update button handlers to call appropriate methods diff --git a/backend/test_ollama_quick.py b/backend/test_ollama_quick.py new file mode 100644 index 0000000..f6261a5 --- /dev/null +++ b/backend/test_ollama_quick.py @@ -0,0 +1,18 @@ +import asyncio +from services.ollama_service import OllamaService + +async def test(): + service = OllamaService() + text = '''Once upon a time, in a tiny village surrounded by lush green forests, there lived three best friends: Lily the rabbit, Benny the bear, and Sammy the squirrel. + +As they sat down on a blanket, Lily asked, "Have you guys heard about the new stream that just appeared in the woods?" + +Benny's eyes lit up. "Ooh, I love streams! Do we get to go see it now?" + +Sammy nodded enthusiastically. "I've been wanting to explore the forest more since the snow melted. Let's go!"''' + + print("Testing Ollama speaker detection...") + speakers = await service.detect_speakers(text) + print(f'Detected speakers: {speakers}') + +asyncio.run(test()) diff --git a/diagnose_issue.md b/diagnose_issue.md new file mode 100644 index 0000000..539b23e --- /dev/null +++ b/diagnose_issue.md @@ -0,0 +1,79 @@ +# Audiobook Generation Issue - Diagnosis + +## Problem +The story is coming out with only one voice (Narrator) instead of multiple voices for different characters. + +## Root Cause Analysis + +Based on my investigation, I found that: + +1. **Ollama IS working correctly** - When I tested with your story, it correctly detected all 3 characters: + - Lily + - Benny + - Sammy + +2. **The backend logs show** that when you used the web interface, it only received 40 characters of text: + ``` + 2025-12-09 10:16:52,356 - main - INFO - Role detection requested for text of length 40 + 2025-12-09 10:16:52,358 - main - INFO - Detected 1 roles: ['Narrator'] + ``` + +3. **Your full story is 3,749 characters**, so something went wrong in the web interface. + +## Possible Causes + +### Most Likely: Text wasn't fully pasted or was truncated +- The web interface might have had an issue with the paste operation +- The text area might have a character limit +- There might have been a copy/paste error + +### Less Likely: Backend issue +- The backend is working correctly (I tested it directly) +- Ollama is running and detecting speakers properly + +## Solution Steps + +### Step 1: Verify Backend is Running +```bash +# Check if backend is running +curl http://localhost:8001/api/health + +# Should return: {"status":"ok"} +``` + +### Step 2: Test with the Full Story via API +I'll create a test script that sends your full story directly to the backend to verify it works end-to-end. + +### Step 3: Check Frontend +If the API works but the web interface doesn't, there might be an issue with: +- Text area character limits +- Copy/paste handling +- Form submission + +## Quick Test Script + +Run this to test the full workflow with your story: + +```bash +python3 test_full_workflow.py +``` + +This will: +1. Send your full story to the role detection endpoint +2. Show you what roles are detected +3. Generate a sample audiobook with different voices +4. Save it as `test_output.wav` + +## Expected Results + +When working correctly, you should see: +- **3-4 roles detected**: Lily, Benny, Sammy, and possibly Narrator +- **Multiple voices in the audio**: Each character should have a distinct voice +- **Proper segmentation**: Dialogue should be assigned to the correct character + +## Next Steps + +1. Try the test script to verify the backend works +2. If backend works, try the web interface again with the full story +3. Make sure to paste the ENTIRE story (all 3,749 characters) +4. Check the browser console for any JavaScript errors diff --git a/frontend/src/App.css b/frontend/src/App.css index b50dbe9..6f51c51 100644 --- a/frontend/src/App.css +++ b/frontend/src/App.css @@ -10,6 +10,25 @@ touch-action: manipulation; } +/* Smooth animations */ +* { + transition: all 0.2s ease-in-out; +} + +button, input, select, textarea { + transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1); +} + +/* Button hover effects */ +button:not(:disabled):hover { + transform: translateY(-2px); + box-shadow: 0 10px 25px rgba(0, 0, 0, 0.15); +} + +button:not(:disabled):active { + transform: translateY(0); +} + /* Custom range slider styling for better mobile support */ input[type="range"] { -webkit-appearance: none; @@ -17,9 +36,10 @@ input[type="range"] { width: 100%; height: 8px; border-radius: 4px; - background: #d1d5db; + background: linear-gradient(to right, #667eea, #764ba2); outline: none; cursor: pointer; + box-shadow: 0 2px 8px rgba(102, 126, 234, 0.3); } input[type="range"]::-webkit-slider-thumb { @@ -28,35 +48,39 @@ input[type="range"]::-webkit-slider-thumb { width: 24px; height: 24px; border-radius: 50%; - background: #2563eb; + background: linear-gradient(135deg, #667eea, #764ba2); cursor: pointer; - transition: background 0.15s ease-in-out; + box-shadow: 0 4px 12px rgba(102, 126, 234, 0.4); + transition: all 0.3s ease; } input[type="range"]::-webkit-slider-thumb:hover { - background: #1d4ed8; + transform: scale(1.2); + box-shadow: 0 6px 16px rgba(102, 126, 234, 0.6); } input[type="range"]::-webkit-slider-thumb:active { - background: #1e40af; + transform: scale(1.1); } input[type="range"]::-moz-range-thumb { width: 24px; height: 24px; border-radius: 50%; - background: #2563eb; + background: linear-gradient(135deg, #667eea, #764ba2); cursor: pointer; border: none; - transition: background 0.15s ease-in-out; + box-shadow: 0 4px 12px rgba(102, 126, 234, 0.4); + transition: all 0.3s ease; } input[type="range"]::-moz-range-thumb:hover { - background: #1d4ed8; + transform: scale(1.2); + box-shadow: 0 6px 16px rgba(102, 126, 234, 0.6); } input[type="range"]::-moz-range-thumb:active { - background: #1e40af; + transform: scale(1.1); } /* Ensure audio controls are touch-friendly on mobile */ diff --git a/frontend/src/App.tsx b/frontend/src/App.tsx index 925e3fd..4dbb4c2 100644 --- a/frontend/src/App.tsx +++ b/frontend/src/App.tsx @@ -32,6 +32,7 @@ function App() { try { // Detect roles in the text const response = await detectRoles(text) + console.log('Detected roles:', response.roles) setRoles(response.roles) // Initialize voice assignments map (empty initially) @@ -157,15 +158,24 @@ function App() { const allRolesAssigned = roles.length > 0 && roles.every(role => voiceAssignments.has(role.id)) return ( -
+
-

- Text to Audiobook -

+ {/* Header with gradient text and icon */} +
+
+
+ 🎧 +
+
+

+ Text to Audiobook +

+

Transform your stories into immersive audio experiences

+
{error && ( -
- {error} +
+ ⚠️ Error: {error}
)} @@ -186,6 +196,7 @@ function App() { Generation may take several minutes for very long content.
)} + {console.log('Rendering VoiceAssignmentSection with roles:', roles, 'allRolesAssigned:', allRolesAssigned)} - Generation may take several minutes for very long content. Please be patient... +
+
+
+ Generation may take several minutes for very long content. Please be patient... +
)} {isGeneratingScript && ( -
- Generating script preview... This may take a moment. +
+
+
+ Generating script preview... This may take a moment. +
)}
diff --git a/frontend/src/components/AudioPlayerSection.tsx b/frontend/src/components/AudioPlayerSection.tsx index ce81008..89812ff 100644 --- a/frontend/src/components/AudioPlayerSection.tsx +++ b/frontend/src/components/AudioPlayerSection.tsx @@ -8,29 +8,63 @@ export default function AudioPlayerSection({ onDownload }: AudioPlayerSectionProps) { return ( -
-

Your Audiobook is Ready!

+
+ {/* Success Header */} +
+
+
+ 🎉 +
+
+

+ Your Audiobook is Ready! +

+

+ Listen below or download to enjoy offline +

+
-
-

Listen to your audiobook

+ {/* Audio Player Card */} +
+
+
+ 🎧 +
+

Listen Now

+
+
+ {/* Download Button */}
+ + {/* Success Message */} +
+

+ ✅ Your audiobook has been successfully generated! +

+
); } diff --git a/frontend/src/components/ScriptReviewSection.tsx b/frontend/src/components/ScriptReviewSection.tsx index d1ef9b3..bbbe609 100644 --- a/frontend/src/components/ScriptReviewSection.tsx +++ b/frontend/src/components/ScriptReviewSection.tsx @@ -22,35 +22,40 @@ export default function ScriptReviewSection({ }; return ( -
-

Review Script

+
+
+
+ 📋 +
+

Review Script

+
-

+

Review the AI-generated script and adjust speaker assignments as needed.

{/* Script Segments */} -
+
{segments.map((segment, index) => (
{/* Segment Header */}
- - Segment {index + 1} + + #{index + 1} {/* Speaker Dropdown */}
-