Admin/audiobook
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
20e6cd1f9db9f32527b6658ba2cd6f122c33ead7
audiobook/.kiro/specs/text-to-audiobook/tasks.md
Paul Bailey 20e6cd1f9d feat: Add Ollama integration for AI-based speaker detection 
- Added Requirement 14 for intelligent speaker detection using Ollama/llama3:8B
- Updated design document with OllamaService component and integration details
- Added 4 new correctness properties for Ollama functionality
- Added tasks 15-17 for implementing Ollama integration
- Updated technology stack to include httpx and Ollama
- Added error handling guidelines for Ollama integration
- Configured fallback to pattern-based detection when Ollama unavailable

This enhancement allows users to paste raw dialogue text without manual role markers.
The system will use AI to intelligently detect speakers, making the tool more user-friendly.
2025-12-09 10:31:38 +00:00

14 KiB
Raw Blame History

Implementation Plan

  • 1. Set up project structure and dependencies

    • Create backend directory with FastAPI project structure
    • Create frontend directory with React + TypeScript + Vite
    • Configure Python virtual environment and install dependencies (fastapi, uvicorn, kokoro, soundfile, python-multipart)
    • Install espeak-ng system dependency
    • Configure Node.js project with TypeScript, React, and Vite
    • Set up environment variable configuration for both projects
    • Requirements: 9.6, 10.2, 11.1

  • 2. Implement backend API foundation

    • Create FastAPI application with CORS middleware
    • Define Pydantic models for requests and responses (RoleDetectionRequest, RolesResponse, TTSRequest, ErrorResponse)
    • Implement /api/health endpoint
    • Set up error handling middleware
    • Configure logging
    • Requirements: 9.1, 9.2, 9.4

  • * 2.1 Write unit tests for API foundation

    • Test health endpoint returns correct status
    • Test CORS headers are set correctly
    • Test error response format
    • Requirements: 9.1

  • 3. Implement role detection service

    • Create RoleDetector class with pattern matching logic
    • Implement colon pattern extraction ("RoleName:")
    • Implement bracket pattern extraction ("[RoleName]")
    • Implement role deduplication logic
    • Implement fallback to default "Narrator" role
    • Create /api/tts/roles endpoint using RoleDetector
    • Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6

  • * 3.1 Write property test for colon pattern extraction

    • Property 2: Role extraction from colon pattern
    • Validates: Requirements 2.2

  • * 3.2 Write property test for bracket pattern extraction

    • Property 3: Role extraction from bracket pattern
    • Validates: Requirements 2.3

  • * 3.3 Write property test for role deduplication

    • Property 4: Role deduplication
    • Validates: Requirements 2.5

  • * 3.4 Write property test for role response structure

    • Property 5: Role response structure
    • Validates: Requirements 2.6

  • * 3.5 Write unit tests for role detection edge cases

    • Test empty text returns default "Narrator" role
    • Test text with no role markers returns "Narrator"
    • Test mixed patterns (both colon and bracket)
    • Requirements: 2.4

  • 4. Implement TTS service with Kokoro integration

    • Create TTSService class
    • Implement KPipeline initialization and caching per language
    • Implement text segmentation logic to identify role markers and split text
    • Implement segment synthesis with voice assignment
    • Implement audio concatenation at 24 kHz
    • Handle fallback to default voice for unmapped roles
    • Requirements: 10.1, 10.3, 10.4, 6.2, 6.3, 6.4, 6.5, 6.6

  • * 4.1 Write property test for pipeline initialization

    • Property 26: Pipeline initialization per language
    • Validates: Requirements 10.3

  • * 4.2 Write property test for voice assignment per segment

    • Property 13: Voice assignment per segment
    • Validates: Requirements 6.3

  • * 4.3 Write property test for text chunking

    • Property 14: Text chunking for long content
    • Validates: Requirements 6.5

  • * 4.4 Write property test for audio concatenation

    • Property 15: Audio concatenation at 24 kHz
    • Validates: Requirements 6.6

  • * 4.5 Write property test for audio sample rate

    • Property 27: Audio sample rate
    • Validates: Requirements 10.4

  • * 4.6 Write unit tests for TTS service

    • Test default voice fallback for unmapped roles
    • Test synthesis with different language codes (a, b, f, j)
    • Test synthesis with different voice IDs
    • Requirements: 6.4, 10.5, 10.6

  • 5. Implement audio generation endpoint

    • Create /api/tts POST endpoint
    • Validate request fields (text, lang, roles, speed)
    • Integrate TTSService to generate audio
    • Convert audio to bytes (WAV or MP3 format)
    • Stream response with correct Content-Type header
    • Add Content-Disposition header with filename
    • Requirements: 6.1, 6.7, 6.8, 9.5

  • * 5.1 Write property test for request validation

    • Property 25: Request validation
    • Validates: Requirements 9.5

  • * 5.2 Write property test for audio response headers

    • Property 16: Audio response content type
    • Property 17: Audio response filename
    • Validates: Requirements 6.7, 6.8

  • * 5.3 Write unit tests for generation endpoint

    • Test successful generation returns audio
    • Test missing required fields returns 400 error
    • Test invalid language code returns error
    • Test invalid voice ID returns error
    • Requirements: 9.5

  • 6. Create backend Dockerfile and docker-compose

    • Write Dockerfile with Python base image
    • Install system dependencies (espeak-ng)
    • Install Python dependencies
    • Copy backend code
    • Expose port 8000
    • Set CMD to start Uvicorn server
    • Create docker-compose.yml for easy orchestration
    • Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6

  • * 6.1 Test Docker container builds and runs

    • Build Docker image successfully
    • Start container and verify health endpoint responds
    • Requirements: 11.5

  • 7. Set up React frontend project structure

    • Initialize Vite project with React and TypeScript
    • Configure TypeScript with strict mode
    • Set up Tailwind CSS for styling
    • Create component directory structure
    • Configure environment variables (.env file with VITE_API_BASE_URL)
    • Requirements: 9.6, 12.3

  • 8. Implement API client module

    • Create src/api/tts.ts with TypeScript interfaces (Role, RoleWithVoice, RolesResponse, GenerateRequest)
    • Implement detectRoles() function
    • Implement generateAudiobook() function
    • Implement checkHealth() function
    • Add error handling for network failures
    • Use VITE_API_BASE_URL environment variable
    • Requirements: 9.6, 9.7

  • * 8.1 Write unit tests for API client

    • Test API client reads VITE_API_BASE_URL
    • Test error handling for network failures
    • Mock fetch calls and verify request format
    • Requirements: 9.6

  • 9. Implement text input component

    • Create TextInputSection component
    • Add large multiline textarea
    • Add "Next: Assign voices" button
    • Implement text validation (non-empty)
    • Show validation message for empty text
    • Disable button when text is empty
    • Requirements: 1.1, 1.2, 1.3, 1.4

  • * 9.1 Write property test for text acceptance

    • Property 1: Text persistence invariant (partial - input phase)
    • Validates: Requirements 1.2

  • * 9.2 Write unit tests for text input component

    • Test textarea renders correctly
    • Test button is disabled when text is empty
    • Test button is enabled when text is present
    • Test validation message appears for empty text
    • Requirements: 1.1, 1.3, 1.4

  • 10. Implement voice assignment component

    • Create VoiceAssignmentSection component
    • Display "Assign voices to roles" title
    • Render role list with labels and dropdowns
    • Populate dropdowns with Kokoro voice options (af_bella, af_sarah, am_adam, bf_emma, etc.)
    • Implement voice selection handler
    • Add language selector with options (American English, British English, French, Japanese)
    • Add speed slider (0.5x - 1.5x range)
    • Add "Generate Audiobook" button
    • Disable generate button until all roles have voices
    • Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 4.1, 4.2, 4.3, 4.4, 5.1, 5.2, 5.3

  • * 10.1 Write property test for voice assignment storage

    • Property 6: Voice assignment storage
    • Validates: Requirements 3.4

  • * 10.2 Write property test for generate button state

    • Property 7: Generate button disabled when incomplete
    • Property 8: Generate button enabled when complete
    • Validates: Requirements 3.5, 3.6

  • * 10.3 Write property test for language selection

    • Property 9: Language selection storage
    • Validates: Requirements 4.4

  • * 10.4 Write property test for speed range

    • Property 10: Speed range constraints
    • Validates: Requirements 5.2

  • * 10.5 Write property test for speed storage

    • Property 11: Speed value storage
    • Validates: Requirements 5.4

  • * 10.6 Write unit tests for voice assignment component

    • Test section title displays correctly
    • Test dropdowns contain required voices
    • Test language selector contains required options
    • Test default language is American English
    • Test default speed is 1.0x
    • Test speed slider enforces min/max bounds
    • Requirements: 3.1, 3.3, 4.2, 4.3, 5.3

  • 11. Implement audio player component

    • Create AudioPlayerSection component
    • Display HTML5 audio element with controls
    • Add "Download audiobook" button
    • Implement download functionality using blob URL
    • Requirements: 7.2, 7.3, 7.4, 7.5

  • * 11.1 Write property test for blob URL creation

    • Property 18: Blob URL creation
    • Validates: Requirements 7.1

  • * 11.2 Write property test for download functionality

    • Property 19: Download functionality
    • Validates: Requirements 7.5

  • * 11.3 Write unit tests for audio player component

    • Test audio element renders with controls
    • Test download button appears
    • Test download button triggers file save
    • Requirements: 7.2, 7.3, 7.4

  • 12. Implement main App component with workflow orchestration

    • Create App component with state management
    • Implement phase transitions (input → assignment → generated)
    • Wire up text input to role detection
    • Wire up voice assignment to audio generation
    • Implement loading states for async operations
    • Implement error handling and display
    • Add warning message for long text
    • Ensure text persists across phases
    • Requirements: 1.5, 6.1, 8.1, 8.2, 8.3, 8.4

  • * 12.1 Write property test for text persistence

    • Property 1: Text persistence invariant (complete workflow)
    • Validates: Requirements 1.5

  • * 12.2 Write property test for generation request completeness

    • Property 12: Generation request completeness
    • Validates: Requirements 6.1

  • * 12.3 Write property test for loading states

    • Property 20: Loading state during role detection
    • Property 21: Loading state during generation
    • Validates: Requirements 8.1, 8.2

  • * 12.4 Write property test for long text warning

    • Property 22: Long text warning
    • Validates: Requirements 8.3

  • * 12.5 Write property test for error display

    • Property 23: Error message display
    • Validates: Requirements 8.4

  • * 12.6 Write unit tests for App component

    • Test phase transitions work correctly
    • Test error states display messages
    • Test loading states appear during API calls
    • Requirements: 8.1, 8.2, 8.4

  • 13. Add responsive styling and mobile support

    • Apply Tailwind CSS classes for responsive layout
    • Ensure text areas and controls are mobile-friendly
    • Test on different screen sizes
    • Ensure touch targets are appropriately sized
    • Requirements: 12.1, 12.2, 12.3, 12.4

  • 14. Create comprehensive README documentation

    • Document Kokoro model setup and configuration
    • Provide backend setup instructions (local and Docker)
    • Provide frontend setup instructions
    • Include example commands for running both services
    • Add workflow testing steps
    • Include troubleshooting section
    • Requirements: 13.1, 13.2, 13.3, 13.4, 13.5

  • 15. Implement Ollama integration service

    • Create OllamaService class in services/ollama_service.py
    • Implement connection to Ollama API using httpx
    • Configure OLLAMA_BASE_URL environment variable (default: http://localhost:11434)
    • Implement detect_speakers() method with structured prompt
    • Implement prompt builder for speaker detection
    • Implement JSON response parser
    • Implement is_available() health check method
    • Add timeout handling (30 seconds)
    • Requirements: 14.1, 14.2, 14.3, 14.6

  • * 15.1 Write property test for Ollama speaker detection

    • Property 28: Ollama speaker detection
    • Validates: Requirements 14.1

  • * 15.2 Write property test for JSON response parsing

    • Property 29: Ollama JSON response parsing
    • Validates: Requirements 14.3

  • * 15.3 Write unit tests for Ollama service

    • Test connection to Ollama
    • Test prompt construction
    • Test JSON parsing with various formats
    • Test timeout handling
    • Test error handling for unavailable service
    • Requirements: 14.1, 14.2, 14.3

  • 16. Integrate Ollama into role detection

    • Update RoleDetector to use OllamaService
    • Implement _detect_with_ollama() method
    • Add fallback logic: try Ollama first, then pattern matching
    • Implement role merging and deduplication across both methods
    • Handle Ollama unavailable gracefully
    • Log detection method used (AI vs pattern)
    • Requirements: 14.1, 14.4, 14.5

  • * 16.1 Write property test for Ollama fallback

    • Property 30: Ollama fallback behavior
    • Validates: Requirements 14.4

  • * 16.2 Write property test for role deduplication

    • Property 31: Role deduplication across detection methods
    • Validates: Requirements 14.5

  • * 16.3 Write unit tests for integrated role detection

    • Test with Ollama available
    • Test with Ollama unavailable
    • Test role merging from both sources
    • Test deduplication logic
    • Requirements: 14.4, 14.5

  • 17. Update documentation for Ollama integration

    • Document Ollama setup in README
    • Add instructions for installing and running Ollama
    • Document llama3:8B model requirement
    • Add OLLAMA_BASE_URL environment variable to .env.example
    • Update backend README with Ollama configuration
    • Add troubleshooting section for Ollama issues
    • Requirements: 14.6

  • 18. Final integration testing and polish

    • Test complete end-to-end workflow
    • Test with various text samples (short, long, multiple roles, no roles)
    • Test with unmarked dialogue text (relying on AI detection)
    • Test with marked text (pattern detection)
    • Test with Ollama available and unavailable
    • Test all language options (a, b, f, j)
    • Test error scenarios (network failures, invalid inputs)
    • Verify audio quality and playback
    • Test Docker deployment
    • Fix any bugs discovered during testing
    • Requirements: All
Reference in New Issue View Git Blame Copy Permalink