Compare commits
3 Commits
b5c53604a6
...
6f7330c491
| Author | SHA1 | Date | |
|---|---|---|---|
| 6f7330c491 | |||
| c310f11484 | |||
| bfdcc94b24 |
209
PIPER_TTS_INTEGRATION.md
Normal file
209
PIPER_TTS_INTEGRATION.md
Normal file
@@ -0,0 +1,209 @@
|
|||||||
|
# Piper TTS Integration Summary
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Successfully integrated Piper TTS as an alternative text-to-speech engine alongside Kokoro TTS, giving users the ability to choose between two engines with different strengths.
|
||||||
|
|
||||||
|
## What Was Added
|
||||||
|
|
||||||
|
### Backend Components
|
||||||
|
|
||||||
|
1. **PiperTTSService** (`backend/services/piper_tts_service.py`)
|
||||||
|
- Handles Piper TTS synthesis with pitch control
|
||||||
|
- Supports pitch shifting from -12 to +12 semitones per character
|
||||||
|
- Automatic voice model downloading
|
||||||
|
- Audio effects processing (pitch and speed)
|
||||||
|
|
||||||
|
2. **UnifiedTTSService** (`backend/services/unified_tts_service.py`)
|
||||||
|
- Manages multiple TTS engines (Kokoro and Piper)
|
||||||
|
- Runtime engine switching
|
||||||
|
- Unified API for both engines
|
||||||
|
- Handles engine-specific features (pitch control for Piper)
|
||||||
|
|
||||||
|
3. **New API Endpoints** (in `backend/main.py`)
|
||||||
|
- `GET /api/tts/engines` - List available engines and current selection
|
||||||
|
- `POST /api/tts/engines` - Switch active TTS engine
|
||||||
|
- `GET /api/tts/voices` - Get available voices for all engines
|
||||||
|
- Updated `POST /api/tts` to support `pitch_adjustments` parameter
|
||||||
|
|
||||||
|
4. **Installation Script** (`backend/install_piper.sh`)
|
||||||
|
- Automated Piper TTS installation for macOS/Linux
|
||||||
|
- Detects OS and architecture
|
||||||
|
- Installs to `/usr/local/bin`
|
||||||
|
|
||||||
|
### Frontend Components
|
||||||
|
|
||||||
|
1. **API Client Functions** (`frontend/src/api/tts.ts`)
|
||||||
|
- `getEngines()` - Fetch available engines
|
||||||
|
- `setEngine(engine)` - Switch active engine
|
||||||
|
- `getVoices()` - Get voices for all engines
|
||||||
|
- TypeScript interfaces for engine data
|
||||||
|
|
||||||
|
2. **UI Components** (`frontend/src/components/VoiceAssignmentSection.tsx`)
|
||||||
|
- Engine selector buttons (Kokoro vs Piper)
|
||||||
|
- Dynamic voice loading based on selected engine
|
||||||
|
- Visual feedback for active engine
|
||||||
|
- Pitch control capability notice for Piper
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
|
||||||
|
1. **Backend README** (`backend/README.md`)
|
||||||
|
- Multi-engine setup instructions
|
||||||
|
- Engine switching examples
|
||||||
|
- Pitch control documentation
|
||||||
|
|
||||||
|
2. **Piper TTS Guide** (`backend/PIPER_TTS_GUIDE.md`)
|
||||||
|
- Comprehensive installation guide
|
||||||
|
- Available voices documentation
|
||||||
|
- Pitch adjustment guidelines
|
||||||
|
- Usage examples and best practices
|
||||||
|
- Troubleshooting section
|
||||||
|
|
||||||
|
3. **Integration Summary** (this file)
|
||||||
|
|
||||||
|
## Engine Comparison
|
||||||
|
|
||||||
|
| Feature | Kokoro TTS | Piper TTS |
|
||||||
|
|---------|------------|-----------|
|
||||||
|
| **Quality** | High | Medium-High |
|
||||||
|
| **Speed** | Medium | Fast |
|
||||||
|
| **Voices** | 54 voices, 9 languages | 5+ English voices |
|
||||||
|
| **Pitch Control** | ❌ No | ✅ Yes (-12 to +12 semitones) |
|
||||||
|
| **Model Size** | ~500MB | ~10-50MB per voice |
|
||||||
|
| **Memory Usage** | Higher | Lower |
|
||||||
|
| **Languages** | 9 languages | English (more available) |
|
||||||
|
|
||||||
|
## How to Use
|
||||||
|
|
||||||
|
### Backend Setup
|
||||||
|
|
||||||
|
1. **Install Piper TTS** (optional):
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
./install_piper.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Set default engine** (optional):
|
||||||
|
```bash
|
||||||
|
# In backend/.env
|
||||||
|
TTS_ENGINE=piper # or kokoro (default)
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Install Python dependencies**:
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### Frontend Usage
|
||||||
|
|
||||||
|
1. **Select Engine**: In the Voice Assignment screen, click either "Kokoro TTS" or "Piper TTS" button
|
||||||
|
2. **Choose Voices**: Voice dropdown updates automatically with engine-specific voices
|
||||||
|
3. **Generate**: Audio is generated using the selected engine
|
||||||
|
|
||||||
|
### API Usage
|
||||||
|
|
||||||
|
**Switch Engine:**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts/engines \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"engine": "piper"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Generate with Pitch Control (Piper only):**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"text": "Alice: Hello!\nBob: Hi there!",
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "en_US-amy-medium"},
|
||||||
|
{"id": "Bob", "voice": "en_US-lessac-medium"}
|
||||||
|
],
|
||||||
|
"pitch_adjustments": [
|
||||||
|
{"id": "Alice", "pitch": 3},
|
||||||
|
{"id": "Bob", "pitch": -2}
|
||||||
|
]
|
||||||
|
}' \
|
||||||
|
--output audiobook.wav
|
||||||
|
```
|
||||||
|
|
||||||
|
## Pitch Control Guidelines
|
||||||
|
|
||||||
|
When using Piper TTS, you can adjust pitch per character:
|
||||||
|
|
||||||
|
- **+12 semitones**: One octave higher (very high-pitched)
|
||||||
|
- **+6 semitones**: Noticeably higher (child-like)
|
||||||
|
- **+3 semitones**: Slightly higher (feminine adjustment)
|
||||||
|
- **0 semitones**: Original voice (no change)
|
||||||
|
- **-3 semitones**: Slightly lower (masculine adjustment)
|
||||||
|
- **-6 semitones**: Noticeably lower (deep voice)
|
||||||
|
- **-12 semitones**: One octave lower (very deep)
|
||||||
|
|
||||||
|
**Recommended ranges:**
|
||||||
|
- Female characters: +2 to +5
|
||||||
|
- Male characters: -2 to -5
|
||||||
|
- Children: +5 to +8
|
||||||
|
- Elderly/deep voices: -4 to -8
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Run the engine selection test:
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
python test_engine_selection.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Files Modified/Created
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
- ✅ `backend/services/piper_tts_service.py` (new)
|
||||||
|
- ✅ `backend/services/unified_tts_service.py` (new)
|
||||||
|
- ✅ `backend/main.py` (modified - added endpoints)
|
||||||
|
- ✅ `backend/requirements.txt` (modified - added piper-tts, pydub)
|
||||||
|
- ✅ `backend/.env.example` (modified - added TTS_ENGINE)
|
||||||
|
- ✅ `backend/README.md` (modified - added multi-engine docs)
|
||||||
|
- ✅ `backend/PIPER_TTS_GUIDE.md` (new)
|
||||||
|
- ✅ `backend/install_piper.sh` (new)
|
||||||
|
- ✅ `backend/test_engine_selection.py` (new)
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
- ✅ `frontend/src/api/tts.ts` (modified - added engine functions)
|
||||||
|
- ✅ `frontend/src/components/VoiceAssignmentSection.tsx` (modified - added engine selector)
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
- ✅ `.kiro/specs/text-to-audiobook/tasks.md` (modified - added task 26)
|
||||||
|
- ✅ `PIPER_TTS_INTEGRATION.md` (new - this file)
|
||||||
|
|
||||||
|
## Git Commits
|
||||||
|
|
||||||
|
1. **feat: Add Piper TTS integration with pitch control support** (bfdcc94)
|
||||||
|
- Backend services and API endpoints
|
||||||
|
- Installation script and documentation
|
||||||
|
|
||||||
|
2. **feat: Add TTS engine selector UI in frontend** (c310f11)
|
||||||
|
- Frontend UI components
|
||||||
|
- API client functions
|
||||||
|
|
||||||
|
## Next Steps (Optional Enhancements)
|
||||||
|
|
||||||
|
1. **Pitch Control UI**: Add pitch sliders in the frontend for Piper engine
|
||||||
|
2. **Voice Previews**: Update preview functionality to work with Piper voices
|
||||||
|
3. **More Piper Voices**: Add support for additional Piper voice models
|
||||||
|
4. **Coqui TTS**: Integrate Coqui XTTS as a third engine option
|
||||||
|
5. **Voice Cloning**: Add voice cloning capabilities with Piper/Coqui
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Piper TTS requires separate installation (not included in Python packages)
|
||||||
|
- Voice models are downloaded automatically on first use
|
||||||
|
- Kokoro remains the default engine for backward compatibility
|
||||||
|
- Both engines can coexist and be switched at runtime
|
||||||
|
- Pitch control is only available with Piper TTS
|
||||||
|
|
||||||
|
## Support
|
||||||
|
|
||||||
|
For issues or questions:
|
||||||
|
- See `backend/PIPER_TTS_GUIDE.md` for detailed Piper documentation
|
||||||
|
- See `backend/README.md` for general backend setup
|
||||||
|
- Check troubleshooting sections in both guides
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
PORT=8000
|
PORT=8000
|
||||||
LOG_LEVEL=info
|
LOG_LEVEL=info
|
||||||
OLLAMA_BASE_URL=http://localhost:11434
|
OLLAMA_BASE_URL=http://localhost:11434
|
||||||
|
TTS_ENGINE=kokoro
|
||||||
|
|||||||
292
backend/PIPER_TTS_GUIDE.md
Normal file
292
backend/PIPER_TTS_GUIDE.md
Normal file
@@ -0,0 +1,292 @@
|
|||||||
|
# Piper TTS Integration Guide
|
||||||
|
|
||||||
|
This guide covers the Piper TTS engine integration, including installation, configuration, and usage.
|
||||||
|
|
||||||
|
## What is Piper TTS?
|
||||||
|
|
||||||
|
Piper is a fast, local neural text-to-speech system that runs entirely on your machine. It offers:
|
||||||
|
|
||||||
|
- **Fast synthesis**: Faster than Kokoro for most use cases
|
||||||
|
- **Pitch control**: Adjust voice pitch per character (-12 to +12 semitones)
|
||||||
|
- **Multiple voices**: Various English voices with different characteristics
|
||||||
|
- **Low resource usage**: Smaller models and lower memory footprint
|
||||||
|
- **Privacy**: All processing happens locally
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
### Automated Installation (Recommended)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
./install_piper.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
This script will:
|
||||||
|
1. Detect your OS and architecture
|
||||||
|
2. Download the appropriate Piper binary
|
||||||
|
3. Install it to `/usr/local/bin`
|
||||||
|
4. Verify the installation
|
||||||
|
|
||||||
|
### Manual Installation
|
||||||
|
|
||||||
|
1. Visit [Piper Releases](https://github.com/rhasspy/piper/releases/tag/v1.2.0)
|
||||||
|
2. Download the appropriate archive for your platform:
|
||||||
|
- **Linux x86_64**: `piper_amd64.tar.gz`
|
||||||
|
- **Linux ARM64**: `piper_arm64.tar.gz`
|
||||||
|
- **macOS**: `piper_macos_x64.tar.gz`
|
||||||
|
3. Extract the archive
|
||||||
|
4. Copy the `piper` binary to a directory in your PATH
|
||||||
|
|
||||||
|
### Verify Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
piper --version
|
||||||
|
```
|
||||||
|
|
||||||
|
You should see output like: `1.2.0`
|
||||||
|
|
||||||
|
## Available Voices
|
||||||
|
|
||||||
|
Piper supports multiple English voices:
|
||||||
|
|
||||||
|
| Voice ID | Language | Gender | Quality | Description |
|
||||||
|
|----------|----------|--------|---------|-------------|
|
||||||
|
| `en_US-lessac-medium` | US English | Male | Medium | Clear, neutral male voice |
|
||||||
|
| `en_US-libritts-high` | US English | Neutral | High | High-quality neutral voice |
|
||||||
|
| `en_US-amy-medium` | US English | Female | Medium | Clear female voice |
|
||||||
|
| `en_GB-alan-medium` | UK English | Male | Medium | British male voice |
|
||||||
|
| `en_GB-alba-medium` | UK English | Female | Medium | British female voice |
|
||||||
|
|
||||||
|
Voice models are automatically downloaded on first use (~10-50MB each).
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Set Piper as Default Engine
|
||||||
|
|
||||||
|
**Option 1: Environment Variable**
|
||||||
|
```bash
|
||||||
|
export TTS_ENGINE=piper
|
||||||
|
```
|
||||||
|
|
||||||
|
Add to your `.env` file:
|
||||||
|
```bash
|
||||||
|
TTS_ENGINE=piper
|
||||||
|
```
|
||||||
|
|
||||||
|
**Option 2: Runtime API Call**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts/engines \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"engine": "piper"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Voice Model Storage
|
||||||
|
|
||||||
|
Voice models are stored in:
|
||||||
|
```
|
||||||
|
backend/piper_models/
|
||||||
|
```
|
||||||
|
|
||||||
|
Models are downloaded automatically on first use. You can pre-download models manually if needed.
|
||||||
|
|
||||||
|
## Usage Examples
|
||||||
|
|
||||||
|
### Basic Text-to-Speech
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"text": "Hello, this is a test of Piper TTS.",
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Narrator", "voice": "en_US-lessac-medium"}
|
||||||
|
],
|
||||||
|
"speed": 1.0
|
||||||
|
}' \
|
||||||
|
--output test.wav
|
||||||
|
```
|
||||||
|
|
||||||
|
### Multi-Voice with Pitch Control
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"text": "Alice: Hello Bob!\nBob: Hi Alice, how are you?\nNarrator: They greeted each other warmly.",
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "en_US-amy-medium"},
|
||||||
|
{"id": "Bob", "voice": "en_US-lessac-medium"},
|
||||||
|
{"id": "Narrator", "voice": "en_GB-alan-medium"}
|
||||||
|
],
|
||||||
|
"speed": 1.0,
|
||||||
|
"pitch_adjustments": [
|
||||||
|
{"id": "Alice", "pitch": 4},
|
||||||
|
{"id": "Bob", "pitch": -3},
|
||||||
|
{"id": "Narrator", "pitch": 0}
|
||||||
|
]
|
||||||
|
}' \
|
||||||
|
--output audiobook.wav
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pitch Adjustment Guidelines
|
||||||
|
|
||||||
|
Pitch is measured in semitones (half-steps on a musical scale):
|
||||||
|
|
||||||
|
- **+12 semitones**: One octave higher (very high-pitched)
|
||||||
|
- **+6 semitones**: Noticeably higher (child-like)
|
||||||
|
- **+3 semitones**: Slightly higher (feminine adjustment)
|
||||||
|
- **0 semitones**: Original voice (no change)
|
||||||
|
- **-3 semitones**: Slightly lower (masculine adjustment)
|
||||||
|
- **-6 semitones**: Noticeably lower (deep voice)
|
||||||
|
- **-12 semitones**: One octave lower (very deep)
|
||||||
|
|
||||||
|
**Recommended ranges:**
|
||||||
|
- Female characters: +2 to +5
|
||||||
|
- Male characters: -2 to -5
|
||||||
|
- Children: +5 to +8
|
||||||
|
- Elderly/deep voices: -4 to -8
|
||||||
|
|
||||||
|
## Python API Usage
|
||||||
|
|
||||||
|
```python
|
||||||
|
from services.unified_tts_service import UnifiedTTSService
|
||||||
|
|
||||||
|
# Initialize with Piper engine
|
||||||
|
tts = UnifiedTTSService(engine="piper")
|
||||||
|
|
||||||
|
# Generate audiobook with pitch control
|
||||||
|
audio_bytes = await tts.generate_audiobook(
|
||||||
|
text="Alice: Hello!\nBob: Hi there!",
|
||||||
|
lang="a",
|
||||||
|
role_voice_map={
|
||||||
|
"Alice": "en_US-amy-medium",
|
||||||
|
"Bob": "en_US-lessac-medium"
|
||||||
|
},
|
||||||
|
speed=1.0,
|
||||||
|
pitch_adjustments={
|
||||||
|
"Alice": 3,
|
||||||
|
"Bob": -2
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Save to file
|
||||||
|
with open("output.wav", "wb") as f:
|
||||||
|
f.write(audio_bytes)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance Comparison
|
||||||
|
|
||||||
|
| Feature | Kokoro TTS | Piper TTS |
|
||||||
|
|---------|------------|-----------|
|
||||||
|
| Speed | Medium | Fast |
|
||||||
|
| Quality | High | Medium-High |
|
||||||
|
| Pitch Control | ❌ No | ✅ Yes |
|
||||||
|
| Model Size | ~500MB | ~10-50MB per voice |
|
||||||
|
| Memory Usage | Higher | Lower |
|
||||||
|
| Voice Variety | 10+ voices | 5+ voices (expandable) |
|
||||||
|
| Languages | 4 languages | English (more available) |
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Piper Not Found
|
||||||
|
|
||||||
|
**Error:** `piper: command not found`
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```bash
|
||||||
|
# Check if piper is in PATH
|
||||||
|
which piper
|
||||||
|
|
||||||
|
# If not found, reinstall
|
||||||
|
./install_piper.sh
|
||||||
|
|
||||||
|
# Or add to PATH manually
|
||||||
|
export PATH="/usr/local/bin:$PATH"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Voice Model Download Fails
|
||||||
|
|
||||||
|
**Error:** Model download timeout or failure
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
1. Check internet connection
|
||||||
|
2. Manually download models from [Piper releases](https://github.com/rhasspy/piper/releases)
|
||||||
|
3. Place `.onnx` and `.onnx.json` files in `backend/piper_models/`
|
||||||
|
|
||||||
|
### Audio Quality Issues
|
||||||
|
|
||||||
|
**Problem:** Distorted or robotic audio
|
||||||
|
|
||||||
|
**Solutions:**
|
||||||
|
- Reduce pitch shift amount (stay within ±6 semitones)
|
||||||
|
- Try different voice models
|
||||||
|
- Adjust speed (try 0.9 or 1.1)
|
||||||
|
- Check for text formatting issues
|
||||||
|
|
||||||
|
### Synthesis Timeout
|
||||||
|
|
||||||
|
**Error:** Piper synthesis times out
|
||||||
|
|
||||||
|
**Solutions:**
|
||||||
|
- Break text into smaller chunks
|
||||||
|
- Increase timeout in `piper_tts_service.py`
|
||||||
|
- Check system resources (CPU, memory)
|
||||||
|
|
||||||
|
## Advanced Configuration
|
||||||
|
|
||||||
|
### Custom Voice Models
|
||||||
|
|
||||||
|
To add custom Piper voice models:
|
||||||
|
|
||||||
|
1. Download `.onnx` and `.onnx.json` files
|
||||||
|
2. Place in `backend/piper_models/`
|
||||||
|
3. Update `AVAILABLE_VOICES` in `piper_tts_service.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
AVAILABLE_VOICES = {
|
||||||
|
"en_US-custom-voice": {"lang": "en", "gender": "female", "quality": "high"},
|
||||||
|
# ... existing voices
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adjusting Audio Effects
|
||||||
|
|
||||||
|
Modify `_apply_audio_effects()` in `piper_tts_service.py` to customize:
|
||||||
|
- Pitch shifting algorithm
|
||||||
|
- Speed adjustment method
|
||||||
|
- Audio processing pipeline
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Voice Selection**
|
||||||
|
- Use distinct voices for different characters
|
||||||
|
- Match voice gender to character when possible
|
||||||
|
- Use British voices for variety
|
||||||
|
|
||||||
|
2. **Pitch Adjustments**
|
||||||
|
- Start with small adjustments (±2-3 semitones)
|
||||||
|
- Test and iterate to find natural-sounding values
|
||||||
|
- Avoid extreme values (±10+) unless intentional
|
||||||
|
|
||||||
|
3. **Performance**
|
||||||
|
- Pre-download voice models for faster first use
|
||||||
|
- Cache generated audio when possible
|
||||||
|
- Use appropriate chunk sizes for long texts
|
||||||
|
|
||||||
|
4. **Quality**
|
||||||
|
- Use high-quality voice models when available
|
||||||
|
- Adjust speed slightly (0.95-1.05) for more natural pacing
|
||||||
|
- Combine with AI segmentation for best results
|
||||||
|
|
||||||
|
## Resources
|
||||||
|
|
||||||
|
- [Piper GitHub Repository](https://github.com/rhasspy/piper)
|
||||||
|
- [Piper Voice Samples](https://rhasspy.github.io/piper-samples/)
|
||||||
|
- [Voice Model Downloads](https://github.com/rhasspy/piper/releases/tag/v1.2.0)
|
||||||
|
- [Piper Documentation](https://github.com/rhasspy/piper/blob/master/README.md)
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Piper TTS is licensed under the MIT License.
|
||||||
@@ -13,6 +13,7 @@ The backend provides REST API endpoints for:
|
|||||||
|
|
||||||
- Python 3.10 or higher
|
- Python 3.10 or higher
|
||||||
- espeak-ng (system dependency for grapheme-to-phoneme conversion)
|
- espeak-ng (system dependency for grapheme-to-phoneme conversion)
|
||||||
|
- Piper TTS (optional, for alternative TTS engine with pitch control)
|
||||||
- Docker (optional, for containerized deployment)
|
- Docker (optional, for containerized deployment)
|
||||||
|
|
||||||
## Local Development Setup
|
## Local Development Setup
|
||||||
@@ -62,7 +63,33 @@ pip install --upgrade pip
|
|||||||
pip install -r requirements.txt
|
pip install -r requirements.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Ollama Setup (Optional but Recommended)
|
### 3. Piper TTS Setup (Optional)
|
||||||
|
|
||||||
|
For an alternative TTS engine with pitch control support:
|
||||||
|
|
||||||
|
**Installation:**
|
||||||
|
```bash
|
||||||
|
# Run the installation script
|
||||||
|
./install_piper.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Or install manually:
|
||||||
|
- Visit [Piper releases](https://github.com/rhasspy/piper/releases)
|
||||||
|
- Download the appropriate binary for your platform
|
||||||
|
- Extract and place in your PATH
|
||||||
|
|
||||||
|
**Verify installation:**
|
||||||
|
```bash
|
||||||
|
piper --version
|
||||||
|
```
|
||||||
|
|
||||||
|
**Benefits of Piper TTS:**
|
||||||
|
- ✨ Pitch shifting support (-12 to +12 semitones per role)
|
||||||
|
- ✨ Fast, lightweight neural TTS
|
||||||
|
- ✨ Multiple voice models available
|
||||||
|
- ✨ Lower resource usage than Kokoro
|
||||||
|
|
||||||
|
### 4. Ollama Setup (Optional but Recommended)
|
||||||
|
|
||||||
For AI-powered speaker detection and text segmentation, install Ollama:
|
For AI-powered speaker detection and text segmentation, install Ollama:
|
||||||
|
|
||||||
@@ -98,7 +125,7 @@ curl http://localhost:11434/api/tags
|
|||||||
- ✨ Handles narrative text without explicit role markers
|
- ✨ Handles narrative text without explicit role markers
|
||||||
- ✨ Falls back to pattern matching if unavailable
|
- ✨ Falls back to pattern matching if unavailable
|
||||||
|
|
||||||
### 4. Environment Configuration
|
### 5. Environment Configuration
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cp .env.example .env
|
cp .env.example .env
|
||||||
@@ -109,9 +136,10 @@ Edit `.env` if needed:
|
|||||||
PORT=8000
|
PORT=8000
|
||||||
LOG_LEVEL=info
|
LOG_LEVEL=info
|
||||||
OLLAMA_BASE_URL=http://localhost:11434 # Optional: Ollama API URL
|
OLLAMA_BASE_URL=http://localhost:11434 # Optional: Ollama API URL
|
||||||
|
TTS_ENGINE=kokoro # Options: kokoro, piper (default: kokoro)
|
||||||
```
|
```
|
||||||
|
|
||||||
### 5. Run the Server
|
### 6. Run the Server
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uvicorn main:app --reload --host 0.0.0.0 --port 8000
|
uvicorn main:app --reload --host 0.0.0.0 --port 8000
|
||||||
@@ -179,6 +207,43 @@ The Dockerfile includes:
|
|||||||
- Storage: ~500MB for Kokoro models
|
- Storage: ~500MB for Kokoro models
|
||||||
- CPU: Multi-core recommended
|
- CPU: Multi-core recommended
|
||||||
|
|
||||||
|
## TTS Engine Selection
|
||||||
|
|
||||||
|
The backend now supports multiple TTS engines that can be selected at runtime:
|
||||||
|
|
||||||
|
### Available Engines
|
||||||
|
|
||||||
|
1. **Kokoro TTS** (default)
|
||||||
|
- High-quality neural TTS
|
||||||
|
- Multiple voices per language
|
||||||
|
- No pitch control
|
||||||
|
- ~500MB model size
|
||||||
|
|
||||||
|
2. **Piper TTS**
|
||||||
|
- Fast, lightweight neural TTS
|
||||||
|
- Pitch shifting support (-12 to +12 semitones)
|
||||||
|
- Multiple voice models
|
||||||
|
- Lower resource usage
|
||||||
|
|
||||||
|
### Switching Engines
|
||||||
|
|
||||||
|
**Via Environment Variable:**
|
||||||
|
```bash
|
||||||
|
export TTS_ENGINE=piper # or kokoro
|
||||||
|
uvicorn main:app --reload
|
||||||
|
```
|
||||||
|
|
||||||
|
**Via API:**
|
||||||
|
```bash
|
||||||
|
# Get available engines
|
||||||
|
curl http://localhost:8000/api/tts/engines
|
||||||
|
|
||||||
|
# Set active engine
|
||||||
|
curl -X POST http://localhost:8000/api/tts/engines \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"engine": "piper"}'
|
||||||
|
```
|
||||||
|
|
||||||
## API Endpoints
|
## API Endpoints
|
||||||
|
|
||||||
### GET /api/health
|
### GET /api/health
|
||||||
@@ -243,6 +308,71 @@ curl -X POST http://localhost:8000/api/tts/roles \
|
|||||||
-d '{"text":"Alice: Hello!\nBob: Hi there!"}'
|
-d '{"text":"Alice: Hello!\nBob: Hi there!"}'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### GET /api/tts/engines
|
||||||
|
|
||||||
|
Get list of available TTS engines and current selection.
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"engines": [
|
||||||
|
{"name": "kokoro", "active": true},
|
||||||
|
{"name": "piper", "active": false}
|
||||||
|
],
|
||||||
|
"current": "kokoro"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8000/api/tts/engines
|
||||||
|
```
|
||||||
|
|
||||||
|
### POST /api/tts/engines
|
||||||
|
|
||||||
|
Set the active TTS engine.
|
||||||
|
|
||||||
|
**Request Body:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"engine": "piper"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "ok",
|
||||||
|
"engine": "piper"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts/engines \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"engine": "piper"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### GET /api/tts/voices
|
||||||
|
|
||||||
|
Get available voices for all TTS engines.
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"voices": {
|
||||||
|
"kokoro": ["af_bella", "af_sarah", "am_adam", "am_michael", ...],
|
||||||
|
"piper": ["en_US-lessac-medium", "en_US-amy-medium", ...]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8000/api/tts/voices
|
||||||
|
```
|
||||||
|
|
||||||
### POST /api/tts
|
### POST /api/tts
|
||||||
|
|
||||||
Generate multi-voice audiobook with AI-powered text segmentation.
|
Generate multi-voice audiobook with AI-powered text segmentation.
|
||||||
@@ -271,6 +401,7 @@ When Ollama is available, the system uses AI to intelligently split text and ass
|
|||||||
- `lang` (required): Language code (a=American English, b=British English, f=French, j=Japanese)
|
- `lang` (required): Language code (a=American English, b=British English, f=French, j=Japanese)
|
||||||
- `roles` (required): Array of role-to-voice mappings
|
- `roles` (required): Array of role-to-voice mappings
|
||||||
- `speed` (optional): Playback speed (0.5-1.5, default: 1.0)
|
- `speed` (optional): Playback speed (0.5-1.5, default: 1.0)
|
||||||
|
- `pitch_adjustments` (optional): Array of role-to-pitch mappings (Piper only, -12 to +12 semitones)
|
||||||
|
|
||||||
**Response:**
|
**Response:**
|
||||||
- Content-Type: `audio/wav`
|
- Content-Type: `audio/wav`
|
||||||
@@ -285,7 +416,7 @@ When Ollama is available, the system uses AI to intelligently split text and ass
|
|||||||
- Content-Disposition: `attachment; filename="audiobook.wav"`
|
- Content-Disposition: `attachment; filename="audiobook.wav"`
|
||||||
- Body: Audio file bytes
|
- Body: Audio file bytes
|
||||||
|
|
||||||
**Example:**
|
**Example (Kokoro):**
|
||||||
```bash
|
```bash
|
||||||
curl -X POST http://localhost:8000/api/tts \
|
curl -X POST http://localhost:8000/api/tts \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
@@ -301,6 +432,26 @@ curl -X POST http://localhost:8000/api/tts \
|
|||||||
--output audiobook.wav
|
--output audiobook.wav
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Example (Piper with pitch control):**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/tts \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"text": "Alice: Hello!\nBob: Hi!",
|
||||||
|
"lang": "a",
|
||||||
|
"roles": [
|
||||||
|
{"id": "Alice", "voice": "en_US-amy-medium"},
|
||||||
|
{"id": "Bob", "voice": "en_US-lessac-medium"}
|
||||||
|
],
|
||||||
|
"speed": 1.0,
|
||||||
|
"pitch_adjustments": [
|
||||||
|
{"id": "Alice", "pitch": 3},
|
||||||
|
{"id": "Bob", "pitch": -2}
|
||||||
|
]
|
||||||
|
}' \
|
||||||
|
--output audiobook.wav
|
||||||
|
```
|
||||||
|
|
||||||
## Kokoro TTS Configuration
|
## Kokoro TTS Configuration
|
||||||
|
|
||||||
### Supported Languages
|
### Supported Languages
|
||||||
|
|||||||
67
backend/install_piper.sh
Executable file
67
backend/install_piper.sh
Executable file
@@ -0,0 +1,67 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Script to install Piper TTS
|
||||||
|
|
||||||
|
echo "Installing Piper TTS..."
|
||||||
|
|
||||||
|
# Detect OS
|
||||||
|
OS="$(uname -s)"
|
||||||
|
ARCH="$(uname -m)"
|
||||||
|
|
||||||
|
case "${OS}" in
|
||||||
|
Linux*)
|
||||||
|
if [ "${ARCH}" = "x86_64" ]; then
|
||||||
|
PIPER_URL="https://github.com/rhasspy/piper/releases/download/v1.2.0/piper_amd64.tar.gz"
|
||||||
|
elif [ "${ARCH}" = "aarch64" ]; then
|
||||||
|
PIPER_URL="https://github.com/rhasspy/piper/releases/download/v1.2.0/piper_arm64.tar.gz"
|
||||||
|
else
|
||||||
|
echo "Unsupported architecture: ${ARCH}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
Darwin*)
|
||||||
|
if [ "${ARCH}" = "x86_64" ] || [ "${ARCH}" = "arm64" ]; then
|
||||||
|
PIPER_URL="https://github.com/rhasspy/piper/releases/download/v1.2.0/piper_macos_x64.tar.gz"
|
||||||
|
else
|
||||||
|
echo "Unsupported architecture: ${ARCH}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Unsupported OS: ${OS}"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# Create temp directory
|
||||||
|
TEMP_DIR=$(mktemp -d)
|
||||||
|
cd "${TEMP_DIR}"
|
||||||
|
|
||||||
|
# Download Piper
|
||||||
|
echo "Downloading Piper from ${PIPER_URL}..."
|
||||||
|
curl -L -o piper.tar.gz "${PIPER_URL}"
|
||||||
|
|
||||||
|
# Extract
|
||||||
|
echo "Extracting..."
|
||||||
|
tar -xzf piper.tar.gz
|
||||||
|
|
||||||
|
# Install to /usr/local/bin (requires sudo)
|
||||||
|
echo "Installing to /usr/local/bin (may require password)..."
|
||||||
|
sudo cp piper/piper /usr/local/bin/
|
||||||
|
sudo chmod +x /usr/local/bin/piper
|
||||||
|
|
||||||
|
# Cleanup
|
||||||
|
cd -
|
||||||
|
rm -rf "${TEMP_DIR}"
|
||||||
|
|
||||||
|
# Verify installation
|
||||||
|
if command -v piper &> /dev/null; then
|
||||||
|
echo "✓ Piper TTS installed successfully!"
|
||||||
|
piper --version
|
||||||
|
else
|
||||||
|
echo "✗ Installation failed"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "To download voice models, they will be automatically downloaded on first use."
|
||||||
|
echo "Or you can manually download from: https://github.com/rhasspy/piper/releases/tag/v1.2.0"
|
||||||
113
backend/main.py
113
backend/main.py
@@ -11,7 +11,7 @@ from pydantic import BaseModel, Field
|
|||||||
|
|
||||||
# Import services
|
# Import services
|
||||||
from services.role_detector import RoleDetector
|
from services.role_detector import RoleDetector
|
||||||
from services.tts_service import TTSService
|
from services.unified_tts_service import UnifiedTTSService
|
||||||
|
|
||||||
# Configure logging
|
# Configure logging
|
||||||
logging.basicConfig(
|
logging.basicConfig(
|
||||||
@@ -54,6 +54,11 @@ class RoleVoiceMapping(BaseModel):
|
|||||||
id: str = Field(..., description="Role identifier")
|
id: str = Field(..., description="Role identifier")
|
||||||
voice: str = Field(..., description="Kokoro voice ID")
|
voice: str = Field(..., description="Kokoro voice ID")
|
||||||
|
|
||||||
|
class RolePitchMapping(BaseModel):
|
||||||
|
"""Model for role-to-pitch mapping."""
|
||||||
|
id: str = Field(..., description="Role identifier")
|
||||||
|
pitch: int = Field(0, description="Pitch shift in semitones (-12 to +12)")
|
||||||
|
|
||||||
class TTSRequest(BaseModel):
|
class TTSRequest(BaseModel):
|
||||||
"""Request model for TTS generation endpoint."""
|
"""Request model for TTS generation endpoint."""
|
||||||
segments: Optional[List[ScriptSegment]] = Field(None, description="Optional edited script segments")
|
segments: Optional[List[ScriptSegment]] = Field(None, description="Optional edited script segments")
|
||||||
@@ -61,6 +66,7 @@ class TTSRequest(BaseModel):
|
|||||||
lang: str = Field(..., description="Language code (a, b, f, j)")
|
lang: str = Field(..., description="Language code (a, b, f, j)")
|
||||||
roles: List[RoleVoiceMapping] = Field(..., description="Role to voice mappings")
|
roles: List[RoleVoiceMapping] = Field(..., description="Role to voice mappings")
|
||||||
speed: Optional[float] = Field(1.0, description="Playback speed (0.5-1.5)")
|
speed: Optional[float] = Field(1.0, description="Playback speed (0.5-1.5)")
|
||||||
|
pitch_adjustments: Optional[List[RolePitchMapping]] = Field(None, description="Optional pitch adjustments per role (Piper only)")
|
||||||
|
|
||||||
class VoicePreviewRequest(BaseModel):
|
class VoicePreviewRequest(BaseModel):
|
||||||
"""Request model for voice preview endpoint."""
|
"""Request model for voice preview endpoint."""
|
||||||
@@ -75,6 +81,24 @@ class ErrorResponse(BaseModel):
|
|||||||
message: str = Field(..., description="Human-readable error message")
|
message: str = Field(..., description="Human-readable error message")
|
||||||
details: Optional[dict] = Field(None, description="Additional error details")
|
details: Optional[dict] = Field(None, description="Additional error details")
|
||||||
|
|
||||||
|
class EngineInfo(BaseModel):
|
||||||
|
"""Model for TTS engine information."""
|
||||||
|
name: str = Field(..., description="Engine identifier")
|
||||||
|
active: bool = Field(..., description="Whether this engine is currently active")
|
||||||
|
|
||||||
|
class EnginesResponse(BaseModel):
|
||||||
|
"""Response model for available engines."""
|
||||||
|
engines: List[EngineInfo] = Field(..., description="List of available TTS engines")
|
||||||
|
current: str = Field(..., description="Currently active engine")
|
||||||
|
|
||||||
|
class SetEngineRequest(BaseModel):
|
||||||
|
"""Request model for setting TTS engine."""
|
||||||
|
engine: str = Field(..., description="Engine to activate (kokoro or piper)")
|
||||||
|
|
||||||
|
class VoicesResponse(BaseModel):
|
||||||
|
"""Response model for available voices."""
|
||||||
|
voices: dict = Field(..., description="Dict mapping engine names to voice lists")
|
||||||
|
|
||||||
# Initialize FastAPI app
|
# Initialize FastAPI app
|
||||||
app = FastAPI(
|
app = FastAPI(
|
||||||
title="Text to Audiobook API",
|
title="Text to Audiobook API",
|
||||||
@@ -120,7 +144,7 @@ async def general_exception_handler(request: Request, exc: Exception):
|
|||||||
|
|
||||||
# Initialize services
|
# Initialize services
|
||||||
role_detector = RoleDetector()
|
role_detector = RoleDetector()
|
||||||
tts_service = TTSService()
|
tts_service = UnifiedTTSService()
|
||||||
|
|
||||||
# API Endpoints
|
# API Endpoints
|
||||||
@app.get("/api/health")
|
@app.get("/api/health")
|
||||||
@@ -129,6 +153,80 @@ async def health_check():
|
|||||||
logger.info("Health check requested")
|
logger.info("Health check requested")
|
||||||
return {"status": "ok"}
|
return {"status": "ok"}
|
||||||
|
|
||||||
|
@app.get("/api/tts/engines", response_model=EnginesResponse)
|
||||||
|
async def get_engines():
|
||||||
|
"""
|
||||||
|
Get list of available TTS engines and current selection.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
EnginesResponse with list of engines and current active engine
|
||||||
|
"""
|
||||||
|
logger.info("Engines list requested")
|
||||||
|
|
||||||
|
try:
|
||||||
|
available = tts_service.get_available_engines()
|
||||||
|
current = tts_service.engine_name
|
||||||
|
|
||||||
|
engines = [
|
||||||
|
EngineInfo(name=engine, active=(engine == current))
|
||||||
|
for engine in available
|
||||||
|
]
|
||||||
|
|
||||||
|
return EnginesResponse(engines=engines, current=current)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error getting engines: {str(e)}", exc_info=True)
|
||||||
|
raise
|
||||||
|
|
||||||
|
@app.post("/api/tts/engines")
|
||||||
|
async def set_engine(request: SetEngineRequest):
|
||||||
|
"""
|
||||||
|
Set the active TTS engine.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
request: SetEngineRequest with engine name
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Success message with new engine
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
400: If engine name is invalid
|
||||||
|
"""
|
||||||
|
logger.info(f"Engine change requested: {request.engine}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
tts_service.set_engine(request.engine)
|
||||||
|
return {"status": "ok", "engine": request.engine}
|
||||||
|
except ValueError as e:
|
||||||
|
logger.error(f"Invalid engine: {str(e)}")
|
||||||
|
return JSONResponse(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
content={
|
||||||
|
"error": "ValidationError",
|
||||||
|
"message": str(e),
|
||||||
|
"details": {"field": "engine"}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error setting engine: {str(e)}", exc_info=True)
|
||||||
|
raise
|
||||||
|
|
||||||
|
@app.get("/api/tts/voices", response_model=VoicesResponse)
|
||||||
|
async def get_voices():
|
||||||
|
"""
|
||||||
|
Get available voices for all TTS engines.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
VoicesResponse with dict mapping engine names to voice lists
|
||||||
|
"""
|
||||||
|
logger.info("Voices list requested")
|
||||||
|
|
||||||
|
try:
|
||||||
|
voices = tts_service.get_available_voices()
|
||||||
|
return VoicesResponse(voices=voices)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error getting voices: {str(e)}", exc_info=True)
|
||||||
|
raise
|
||||||
|
|
||||||
@app.post("/api/tts/roles", response_model=RolesResponse)
|
@app.post("/api/tts/roles", response_model=RolesResponse)
|
||||||
async def detect_roles(request: RoleDetectionRequest):
|
async def detect_roles(request: RoleDetectionRequest):
|
||||||
"""
|
"""
|
||||||
@@ -420,6 +518,11 @@ async def generate_audiobook(request: TTSRequest):
|
|||||||
# Convert roles list to dictionary for TTS service
|
# Convert roles list to dictionary for TTS service
|
||||||
role_voice_map = {role.id: role.voice for role in request.roles}
|
role_voice_map = {role.id: role.voice for role in request.roles}
|
||||||
|
|
||||||
|
# Convert pitch adjustments to dictionary if provided
|
||||||
|
pitch_map = None
|
||||||
|
if request.pitch_adjustments:
|
||||||
|
pitch_map = {p.id: p.pitch for p in request.pitch_adjustments}
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Generate audiobook using segments if provided, otherwise use text
|
# Generate audiobook using segments if provided, otherwise use text
|
||||||
if request.segments:
|
if request.segments:
|
||||||
@@ -428,7 +531,8 @@ async def generate_audiobook(request: TTSRequest):
|
|||||||
segments=[(seg.speaker, seg.text) for seg in request.segments],
|
segments=[(seg.speaker, seg.text) for seg in request.segments],
|
||||||
lang=request.lang,
|
lang=request.lang,
|
||||||
role_voice_map=role_voice_map,
|
role_voice_map=role_voice_map,
|
||||||
speed=request.speed
|
speed=request.speed,
|
||||||
|
pitch_adjustments=pitch_map
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
logger.info("Generating audiobook from text with automatic segmentation")
|
logger.info("Generating audiobook from text with automatic segmentation")
|
||||||
@@ -436,7 +540,8 @@ async def generate_audiobook(request: TTSRequest):
|
|||||||
text=request.text,
|
text=request.text,
|
||||||
lang=request.lang,
|
lang=request.lang,
|
||||||
role_voice_map=role_voice_map,
|
role_voice_map=role_voice_map,
|
||||||
speed=request.speed
|
speed=request.speed,
|
||||||
|
pitch_adjustments=pitch_map
|
||||||
)
|
)
|
||||||
|
|
||||||
# Create streaming response with proper headers
|
# Create streaming response with proper headers
|
||||||
|
|||||||
@@ -6,3 +6,5 @@ python-multipart==0.0.6
|
|||||||
pydantic==2.5.3
|
pydantic==2.5.3
|
||||||
numpy==1.26.3
|
numpy==1.26.3
|
||||||
httpx==0.26.0
|
httpx==0.26.0
|
||||||
|
piper-tts==1.2.0
|
||||||
|
pydub==0.25.1
|
||||||
|
|||||||
219
backend/services/piper_tts_service.py
Normal file
219
backend/services/piper_tts_service.py
Normal file
@@ -0,0 +1,219 @@
|
|||||||
|
"""
|
||||||
|
Piper TTS service for generating multi-voice audiobooks.
|
||||||
|
"""
|
||||||
|
import logging
|
||||||
|
import io
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
from typing import Dict, List, Tuple, Optional
|
||||||
|
import numpy as np
|
||||||
|
import soundfile as sf
|
||||||
|
from pydub import AudioSegment
|
||||||
|
from pydub.effects import speedup
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class PiperTTSService:
|
||||||
|
"""
|
||||||
|
Service for text-to-speech synthesis using Piper TTS.
|
||||||
|
|
||||||
|
Piper is a fast, local neural TTS system that supports multiple voices
|
||||||
|
and allows pitch modification through audio processing.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Default voice for unmapped roles
|
||||||
|
DEFAULT_VOICE = "en_US-lessac-medium"
|
||||||
|
|
||||||
|
# Sample rate for Piper TTS (typically 22050)
|
||||||
|
SAMPLE_RATE = 22050
|
||||||
|
|
||||||
|
# Available Piper voices with their characteristics
|
||||||
|
AVAILABLE_VOICES = {
|
||||||
|
# English voices
|
||||||
|
"en_US-lessac-medium": {"lang": "en", "gender": "male", "quality": "medium"},
|
||||||
|
"en_US-libritts-high": {"lang": "en", "gender": "neutral", "quality": "high"},
|
||||||
|
"en_US-amy-medium": {"lang": "en", "gender": "female", "quality": "medium"},
|
||||||
|
"en_GB-alan-medium": {"lang": "en", "gender": "male", "quality": "medium"},
|
||||||
|
"en_GB-alba-medium": {"lang": "en", "gender": "female", "quality": "medium"},
|
||||||
|
}
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
"""Initializes Piper TTS service."""
|
||||||
|
self.models_dir = os.path.join(os.path.dirname(__file__), "..", "piper_models")
|
||||||
|
os.makedirs(self.models_dir, exist_ok=True)
|
||||||
|
logger.info("PiperTTSService initialized")
|
||||||
|
|
||||||
|
def _check_piper_installed(self) -> bool:
|
||||||
|
"""Check if Piper is installed and accessible."""
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
["piper", "--version"],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=5
|
||||||
|
)
|
||||||
|
return result.returncode == 0
|
||||||
|
except (subprocess.SubprocessError, FileNotFoundError):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _download_voice_model(self, voice: str) -> bool:
|
||||||
|
"""
|
||||||
|
Download a Piper voice model if not already present.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
voice: Voice ID to download
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True if model is available, False otherwise
|
||||||
|
"""
|
||||||
|
model_path = os.path.join(self.models_dir, f"{voice}.onnx")
|
||||||
|
config_path = os.path.join(self.models_dir, f"{voice}.onnx.json")
|
||||||
|
|
||||||
|
# Check if model already exists
|
||||||
|
if os.path.exists(model_path) and os.path.exists(config_path):
|
||||||
|
return True
|
||||||
|
|
||||||
|
logger.info(f"Downloading Piper voice model: {voice}")
|
||||||
|
|
||||||
|
# Download using piper's built-in download
|
||||||
|
try:
|
||||||
|
subprocess.run(
|
||||||
|
["piper", "--model", voice, "--download-dir", self.models_dir],
|
||||||
|
check=True,
|
||||||
|
timeout=60
|
||||||
|
)
|
||||||
|
return os.path.exists(model_path)
|
||||||
|
except subprocess.SubprocessError as e:
|
||||||
|
logger.error(f"Failed to download voice model {voice}: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def synthesize_with_pitch(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
voice: str,
|
||||||
|
speed: float = 1.0,
|
||||||
|
pitch_shift: int = 0
|
||||||
|
) -> np.ndarray:
|
||||||
|
"""
|
||||||
|
Synthesizes text with Piper and applies pitch shifting.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Text to synthesize
|
||||||
|
voice: Piper voice ID
|
||||||
|
speed: Playback speed (0.5-2.0)
|
||||||
|
pitch_shift: Semitones to shift pitch (-12 to +12)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Audio samples as numpy array
|
||||||
|
"""
|
||||||
|
# Ensure voice model is available
|
||||||
|
if not self._download_voice_model(voice):
|
||||||
|
logger.warning(f"Voice {voice} not available, using default")
|
||||||
|
voice = self.DEFAULT_VOICE
|
||||||
|
self._download_voice_model(voice)
|
||||||
|
|
||||||
|
model_path = os.path.join(self.models_dir, f"{voice}.onnx")
|
||||||
|
|
||||||
|
# Generate audio with Piper
|
||||||
|
try:
|
||||||
|
# Run piper command
|
||||||
|
process = subprocess.Popen(
|
||||||
|
["piper", "--model", model_path, "--output-raw"],
|
||||||
|
stdin=subprocess.PIPE,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE
|
||||||
|
)
|
||||||
|
|
||||||
|
# Send text and get audio
|
||||||
|
stdout, stderr = process.communicate(input=text.encode('utf-8'), timeout=30)
|
||||||
|
|
||||||
|
if process.returncode != 0:
|
||||||
|
logger.error(f"Piper synthesis failed: {stderr.decode()}")
|
||||||
|
return np.array([], dtype=np.float32)
|
||||||
|
|
||||||
|
# Convert raw audio to numpy array
|
||||||
|
audio_data = np.frombuffer(stdout, dtype=np.int16)
|
||||||
|
audio_float = audio_data.astype(np.float32) / 32768.0
|
||||||
|
|
||||||
|
# Apply pitch shift and speed change using pydub
|
||||||
|
if pitch_shift != 0 or speed != 1.0:
|
||||||
|
audio_float = self._apply_audio_effects(
|
||||||
|
audio_float,
|
||||||
|
self.SAMPLE_RATE,
|
||||||
|
speed,
|
||||||
|
pitch_shift
|
||||||
|
)
|
||||||
|
|
||||||
|
return audio_float
|
||||||
|
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
logger.error("Piper synthesis timed out")
|
||||||
|
return np.array([], dtype=np.float32)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error in Piper synthesis: {e}")
|
||||||
|
return np.array([], dtype=np.float32)
|
||||||
|
|
||||||
|
def _apply_audio_effects(
|
||||||
|
self,
|
||||||
|
audio: np.ndarray,
|
||||||
|
sample_rate: int,
|
||||||
|
speed: float,
|
||||||
|
pitch_shift: int
|
||||||
|
) -> np.ndarray:
|
||||||
|
"""
|
||||||
|
Apply speed and pitch effects to audio using pydub.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
audio: Audio samples
|
||||||
|
sample_rate: Sample rate
|
||||||
|
speed: Speed multiplier
|
||||||
|
pitch_shift: Semitones to shift
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Processed audio samples
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Convert numpy to pydub AudioSegment
|
||||||
|
audio_int16 = (audio * 32768.0).astype(np.int16)
|
||||||
|
audio_bytes = audio_int16.tobytes()
|
||||||
|
|
||||||
|
audio_segment = AudioSegment(
|
||||||
|
audio_bytes,
|
||||||
|
frame_rate=sample_rate,
|
||||||
|
sample_width=2,
|
||||||
|
channels=1
|
||||||
|
)
|
||||||
|
|
||||||
|
# Apply pitch shift
|
||||||
|
if pitch_shift != 0:
|
||||||
|
# Pitch shift by changing frame rate then resampling
|
||||||
|
new_sample_rate = int(sample_rate * (2.0 ** (pitch_shift / 12.0)))
|
||||||
|
audio_segment = audio_segment._spawn(
|
||||||
|
audio_segment.raw_data,
|
||||||
|
overrides={'frame_rate': new_sample_rate}
|
||||||
|
)
|
||||||
|
audio_segment = audio_segment.set_frame_rate(sample_rate)
|
||||||
|
|
||||||
|
# Apply speed change
|
||||||
|
if speed != 1.0:
|
||||||
|
audio_segment = speedup(audio_segment, playback_speed=speed)
|
||||||
|
|
||||||
|
# Convert back to numpy
|
||||||
|
samples = np.array(audio_segment.get_array_of_samples())
|
||||||
|
audio_float = samples.astype(np.float32) / 32768.0
|
||||||
|
|
||||||
|
return audio_float
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error applying audio effects: {e}")
|
||||||
|
return audio
|
||||||
|
|
||||||
|
def get_available_voices(self) -> Dict[str, Dict[str, str]]:
|
||||||
|
"""
|
||||||
|
Returns dictionary of available Piper voices.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict mapping voice IDs to their characteristics
|
||||||
|
"""
|
||||||
|
return self.AVAILABLE_VOICES.copy()
|
||||||
283
backend/services/unified_tts_service.py
Normal file
283
backend/services/unified_tts_service.py
Normal file
@@ -0,0 +1,283 @@
|
|||||||
|
"""
|
||||||
|
Unified TTS service that supports multiple TTS engines (Kokoro, Piper).
|
||||||
|
"""
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
from typing import Dict, List, Tuple, Optional
|
||||||
|
from enum import Enum
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class TTSEngine(str, Enum):
|
||||||
|
"""Available TTS engines."""
|
||||||
|
KOKORO = "kokoro"
|
||||||
|
PIPER = "piper"
|
||||||
|
|
||||||
|
|
||||||
|
class UnifiedTTSService:
|
||||||
|
"""
|
||||||
|
Unified service that delegates to specific TTS engines.
|
||||||
|
Allows runtime selection between Kokoro and Piper TTS.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, engine: Optional[str] = None, ollama_service=None):
|
||||||
|
"""
|
||||||
|
Initialize unified TTS service.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
engine: TTS engine to use ('kokoro' or 'piper').
|
||||||
|
Defaults to TTS_ENGINE env var or 'kokoro'
|
||||||
|
ollama_service: Optional OllamaService for AI segmentation
|
||||||
|
"""
|
||||||
|
self.engine_name = engine or os.getenv("TTS_ENGINE", "kokoro")
|
||||||
|
self.ollama_service = ollama_service
|
||||||
|
|
||||||
|
# Lazy load engines
|
||||||
|
self._kokoro_service = None
|
||||||
|
self._piper_service = None
|
||||||
|
|
||||||
|
logger.info(f"UnifiedTTSService initialized with engine: {self.engine_name}")
|
||||||
|
|
||||||
|
@property
|
||||||
|
def kokoro_service(self):
|
||||||
|
"""Lazy load Kokoro service."""
|
||||||
|
if self._kokoro_service is None:
|
||||||
|
from .tts_service import TTSService
|
||||||
|
self._kokoro_service = TTSService(self.ollama_service)
|
||||||
|
return self._kokoro_service
|
||||||
|
|
||||||
|
@property
|
||||||
|
def piper_service(self):
|
||||||
|
"""Lazy load Piper service."""
|
||||||
|
if self._piper_service is None:
|
||||||
|
from .piper_tts_service import PiperTTSService
|
||||||
|
self._piper_service = PiperTTSService()
|
||||||
|
return self._piper_service
|
||||||
|
|
||||||
|
def set_engine(self, engine: str):
|
||||||
|
"""
|
||||||
|
Change the active TTS engine.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
engine: Engine name ('kokoro' or 'piper')
|
||||||
|
"""
|
||||||
|
if engine not in [e.value for e in TTSEngine]:
|
||||||
|
raise ValueError(f"Unknown engine: {engine}. Use 'kokoro' or 'piper'")
|
||||||
|
|
||||||
|
self.engine_name = engine
|
||||||
|
logger.info(f"Switched TTS engine to: {engine}")
|
||||||
|
|
||||||
|
def get_available_engines(self) -> List[str]:
|
||||||
|
"""
|
||||||
|
Get list of available TTS engines.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of engine names
|
||||||
|
"""
|
||||||
|
return [e.value for e in TTSEngine]
|
||||||
|
|
||||||
|
def get_available_voices(self) -> Dict[str, List[str]]:
|
||||||
|
"""
|
||||||
|
Get available voices for all engines.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict mapping engine names to lists of voice IDs
|
||||||
|
"""
|
||||||
|
voices = {}
|
||||||
|
|
||||||
|
# Kokoro voices (from documentation)
|
||||||
|
voices["kokoro"] = [
|
||||||
|
"af_bella", "af_nicole", "af_sarah", "af_sky",
|
||||||
|
"am_adam", "am_michael", "bf_emma", "bf_isabella",
|
||||||
|
"bm_george", "bm_lewis"
|
||||||
|
]
|
||||||
|
|
||||||
|
# Piper voices
|
||||||
|
try:
|
||||||
|
piper_voices = self.piper_service.get_available_voices()
|
||||||
|
voices["piper"] = list(piper_voices.keys())
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Could not load Piper voices: {e}")
|
||||||
|
voices["piper"] = []
|
||||||
|
|
||||||
|
return voices
|
||||||
|
|
||||||
|
async def generate_script_segments(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
roles: List[str]
|
||||||
|
) -> List[Tuple[str, str]]:
|
||||||
|
"""
|
||||||
|
Generate script segments with speaker assignments.
|
||||||
|
Uses AI when available, falls back to pattern matching.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Input text to segment
|
||||||
|
roles: List of detected role IDs/names
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of tuples: (speaker_name, text_content)
|
||||||
|
"""
|
||||||
|
# This is engine-agnostic, use Kokoro's implementation
|
||||||
|
return await self.kokoro_service.generate_script_segments(text, roles)
|
||||||
|
|
||||||
|
async def generate_audiobook(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
lang: str,
|
||||||
|
role_voice_map: Dict[str, str],
|
||||||
|
speed: float = 1.0,
|
||||||
|
pitch_adjustments: Optional[Dict[str, int]] = None
|
||||||
|
) -> bytes:
|
||||||
|
"""
|
||||||
|
Generate audiobook using the selected TTS engine.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Input text
|
||||||
|
lang: Language code
|
||||||
|
role_voice_map: Role to voice ID mapping
|
||||||
|
speed: Playback speed
|
||||||
|
pitch_adjustments: Optional dict of role to pitch shift (semitones)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Audio data as bytes (WAV format)
|
||||||
|
"""
|
||||||
|
if self.engine_name == TTSEngine.KOKORO:
|
||||||
|
# Kokoro doesn't support pitch adjustment
|
||||||
|
return await self.kokoro_service.generate_audiobook(
|
||||||
|
text, lang, role_voice_map, speed
|
||||||
|
)
|
||||||
|
|
||||||
|
elif self.engine_name == TTSEngine.PIPER:
|
||||||
|
# Use Piper with pitch support
|
||||||
|
return await self._generate_with_piper(
|
||||||
|
text, lang, role_voice_map, speed, pitch_adjustments or {}
|
||||||
|
)
|
||||||
|
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown engine: {self.engine_name}")
|
||||||
|
|
||||||
|
async def generate_audiobook_from_segments(
|
||||||
|
self,
|
||||||
|
segments: List[Tuple[str, str]],
|
||||||
|
lang: str,
|
||||||
|
role_voice_map: Dict[str, str],
|
||||||
|
speed: float = 1.0,
|
||||||
|
pitch_adjustments: Optional[Dict[str, int]] = None
|
||||||
|
) -> bytes:
|
||||||
|
"""
|
||||||
|
Generate audiobook from pre-segmented script.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
segments: List of (speaker_name, text_content) tuples
|
||||||
|
lang: Language code
|
||||||
|
role_voice_map: Role to voice ID mapping
|
||||||
|
speed: Playback speed
|
||||||
|
pitch_adjustments: Optional dict of role to pitch shift (semitones)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Audio data as bytes (WAV format)
|
||||||
|
"""
|
||||||
|
if self.engine_name == TTSEngine.KOKORO:
|
||||||
|
return await self.kokoro_service.generate_audiobook_from_segments(
|
||||||
|
segments, lang, role_voice_map, speed
|
||||||
|
)
|
||||||
|
|
||||||
|
elif self.engine_name == TTSEngine.PIPER:
|
||||||
|
return await self._generate_from_segments_with_piper(
|
||||||
|
segments, lang, role_voice_map, speed, pitch_adjustments or {}
|
||||||
|
)
|
||||||
|
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown engine: {self.engine_name}")
|
||||||
|
|
||||||
|
async def _generate_with_piper(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
lang: str,
|
||||||
|
role_voice_map: Dict[str, str],
|
||||||
|
speed: float,
|
||||||
|
pitch_adjustments: Dict[str, int]
|
||||||
|
) -> bytes:
|
||||||
|
"""Generate audiobook using Piper TTS with pitch control."""
|
||||||
|
import io
|
||||||
|
import numpy as np
|
||||||
|
import soundfile as sf
|
||||||
|
|
||||||
|
# Segment text using Kokoro's logic
|
||||||
|
segments = await self.kokoro_service.segment_text_with_ai(text, role_voice_map)
|
||||||
|
|
||||||
|
if segments is None:
|
||||||
|
segments = self.kokoro_service.segment_text(text, role_voice_map)
|
||||||
|
|
||||||
|
if not segments:
|
||||||
|
raise ValueError("No segments to synthesize")
|
||||||
|
|
||||||
|
# Synthesize each segment with Piper
|
||||||
|
audio_segments = []
|
||||||
|
for role_id, voice, content in segments:
|
||||||
|
pitch_shift = pitch_adjustments.get(role_id, 0)
|
||||||
|
logger.debug(
|
||||||
|
f"Synthesizing with Piper: role='{role_id}', voice='{voice}', "
|
||||||
|
f"pitch={pitch_shift}"
|
||||||
|
)
|
||||||
|
|
||||||
|
audio = self.piper_service.synthesize_with_pitch(
|
||||||
|
content, voice, speed, pitch_shift
|
||||||
|
)
|
||||||
|
audio_segments.append(audio)
|
||||||
|
|
||||||
|
# Concatenate
|
||||||
|
final_audio = np.concatenate([s for s in audio_segments if len(s) > 0])
|
||||||
|
|
||||||
|
# Convert to WAV bytes
|
||||||
|
buffer = io.BytesIO()
|
||||||
|
sf.write(buffer, final_audio, self.piper_service.SAMPLE_RATE, format='WAV')
|
||||||
|
buffer.seek(0)
|
||||||
|
|
||||||
|
logger.info(f"Generated Piper audiobook: {len(buffer.getvalue())} bytes")
|
||||||
|
return buffer.read()
|
||||||
|
|
||||||
|
async def _generate_from_segments_with_piper(
|
||||||
|
self,
|
||||||
|
segments: List[Tuple[str, str]],
|
||||||
|
lang: str,
|
||||||
|
role_voice_map: Dict[str, str],
|
||||||
|
speed: float,
|
||||||
|
pitch_adjustments: Dict[str, int]
|
||||||
|
) -> bytes:
|
||||||
|
"""Generate audiobook from segments using Piper TTS."""
|
||||||
|
import io
|
||||||
|
import numpy as np
|
||||||
|
import soundfile as sf
|
||||||
|
|
||||||
|
if not segments:
|
||||||
|
raise ValueError("No segments provided")
|
||||||
|
|
||||||
|
# Synthesize each segment
|
||||||
|
audio_segments = []
|
||||||
|
for speaker, text in segments:
|
||||||
|
voice = role_voice_map.get(speaker, self.piper_service.DEFAULT_VOICE)
|
||||||
|
pitch_shift = pitch_adjustments.get(speaker, 0)
|
||||||
|
|
||||||
|
logger.debug(
|
||||||
|
f"Synthesizing segment: speaker='{speaker}', voice='{voice}', "
|
||||||
|
f"pitch={pitch_shift}"
|
||||||
|
)
|
||||||
|
|
||||||
|
audio = self.piper_service.synthesize_with_pitch(
|
||||||
|
text, voice, speed, pitch_shift
|
||||||
|
)
|
||||||
|
audio_segments.append(audio)
|
||||||
|
|
||||||
|
# Concatenate
|
||||||
|
final_audio = np.concatenate([s for s in audio_segments if len(s) > 0])
|
||||||
|
|
||||||
|
# Convert to WAV bytes
|
||||||
|
buffer = io.BytesIO()
|
||||||
|
sf.write(buffer, final_audio, self.piper_service.SAMPLE_RATE, format='WAV')
|
||||||
|
buffer.seek(0)
|
||||||
|
|
||||||
|
logger.info(f"Generated Piper audiobook from segments: {len(buffer.getvalue())} bytes")
|
||||||
|
return buffer.read()
|
||||||
58
backend/test_engine_selection.py
Normal file
58
backend/test_engine_selection.py
Normal file
@@ -0,0 +1,58 @@
|
|||||||
|
"""
|
||||||
|
Test script for TTS engine selection functionality.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
import sys
|
||||||
|
from services.unified_tts_service import UnifiedTTSService
|
||||||
|
|
||||||
|
|
||||||
|
async def test_engine_selection():
|
||||||
|
"""Test switching between TTS engines."""
|
||||||
|
print("Testing TTS Engine Selection\n" + "="*50)
|
||||||
|
|
||||||
|
# Initialize service
|
||||||
|
print("\n1. Initializing UnifiedTTSService...")
|
||||||
|
tts = UnifiedTTSService()
|
||||||
|
print(f" Default engine: {tts.engine_name}")
|
||||||
|
|
||||||
|
# Get available engines
|
||||||
|
print("\n2. Getting available engines...")
|
||||||
|
engines = tts.get_available_engines()
|
||||||
|
print(f" Available: {engines}")
|
||||||
|
|
||||||
|
# Get available voices
|
||||||
|
print("\n3. Getting available voices...")
|
||||||
|
voices = tts.get_available_voices()
|
||||||
|
for engine, voice_list in voices.items():
|
||||||
|
print(f" {engine}: {len(voice_list)} voices")
|
||||||
|
if voice_list:
|
||||||
|
print(f" Examples: {voice_list[:3]}")
|
||||||
|
|
||||||
|
# Test switching to Piper
|
||||||
|
print("\n4. Switching to Piper engine...")
|
||||||
|
try:
|
||||||
|
tts.set_engine('piper')
|
||||||
|
print(f" Current engine: {tts.engine_name}")
|
||||||
|
except Exception as e:
|
||||||
|
print(f" ⚠️ Error: {e}")
|
||||||
|
print(" Note: Piper may not be installed. Run ./install_piper.sh")
|
||||||
|
|
||||||
|
# Switch back to Kokoro
|
||||||
|
print("\n5. Switching back to Kokoro engine...")
|
||||||
|
tts.set_engine('kokoro')
|
||||||
|
print(f" Current engine: {tts.engine_name}")
|
||||||
|
|
||||||
|
# Test invalid engine
|
||||||
|
print("\n6. Testing invalid engine...")
|
||||||
|
try:
|
||||||
|
tts.set_engine('invalid')
|
||||||
|
print(" ❌ Should have raised error!")
|
||||||
|
except ValueError as e:
|
||||||
|
print(f" ✓ Correctly rejected: {e}")
|
||||||
|
|
||||||
|
print("\n" + "="*50)
|
||||||
|
print("✓ Engine selection tests completed!")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(test_engine_selection())
|
||||||
@@ -256,6 +256,31 @@ export async function generateAudiobook(payload: GenerateRequest): Promise<Blob>
|
|||||||
return await response.blob();
|
return await response.blob();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* TTS Engine information
|
||||||
|
*/
|
||||||
|
export interface EngineInfo {
|
||||||
|
name: string;
|
||||||
|
active: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Response from engines endpoint
|
||||||
|
*/
|
||||||
|
export interface EnginesResponse {
|
||||||
|
engines: EngineInfo[];
|
||||||
|
current: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Response from voices endpoint
|
||||||
|
*/
|
||||||
|
export interface VoicesResponse {
|
||||||
|
voices: {
|
||||||
|
[engine: string]: string[];
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Request payload for voice preview
|
* Request payload for voice preview
|
||||||
*/
|
*/
|
||||||
@@ -266,6 +291,65 @@ export interface VoicePreviewRequest {
|
|||||||
text?: string;
|
text?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get available TTS engines
|
||||||
|
* @returns Promise resolving to engines list
|
||||||
|
* @throws ApiError if the request fails
|
||||||
|
*/
|
||||||
|
export async function getEngines(): Promise<EnginesResponse> {
|
||||||
|
const baseUrl = getApiBaseUrl();
|
||||||
|
const url = `${baseUrl}/api/tts/engines`;
|
||||||
|
|
||||||
|
const response = await fetchWithErrorHandling(url, {
|
||||||
|
method: 'GET',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
return await response.json();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Set the active TTS engine
|
||||||
|
* @param engine - Engine name to activate (kokoro or piper)
|
||||||
|
* @returns Promise resolving to success response
|
||||||
|
* @throws ApiError if the request fails
|
||||||
|
*/
|
||||||
|
export async function setEngine(engine: string): Promise<{ status: string; engine: string }> {
|
||||||
|
const baseUrl = getApiBaseUrl();
|
||||||
|
const url = `${baseUrl}/api/tts/engines`;
|
||||||
|
|
||||||
|
const response = await fetchWithErrorHandling(url, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
body: JSON.stringify({ engine }),
|
||||||
|
});
|
||||||
|
|
||||||
|
return await response.json();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get available voices for all engines
|
||||||
|
* @returns Promise resolving to voices by engine
|
||||||
|
* @throws ApiError if the request fails
|
||||||
|
*/
|
||||||
|
export async function getVoices(): Promise<VoicesResponse> {
|
||||||
|
const baseUrl = getApiBaseUrl();
|
||||||
|
const url = `${baseUrl}/api/tts/voices`;
|
||||||
|
|
||||||
|
const response = await fetchWithErrorHandling(url, {
|
||||||
|
method: 'GET',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
return await response.json();
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Generate a preview of a voice
|
* Generate a preview of a voice
|
||||||
* @param payload - Preview request with voice, language, and optional speed/text
|
* @param payload - Preview request with voice, language, and optional speed/text
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
import { useState } from 'react';
|
import { useState, useEffect } from 'react';
|
||||||
import type { Role } from '../api/tts';
|
import type { Role } from '../api/tts';
|
||||||
import { previewVoice } from '../api/tts';
|
import { previewVoice, getEngines, setEngine, getVoices } from '../api/tts';
|
||||||
|
|
||||||
type AgeCategory = 'young' | 'teen' | 'adult' | 'old';
|
type AgeCategory = 'young' | 'teen' | 'adult' | 'old';
|
||||||
|
|
||||||
@@ -161,6 +161,40 @@ export default function VoiceAssignmentSection({
|
|||||||
}: VoiceAssignmentSectionProps) {
|
}: VoiceAssignmentSectionProps) {
|
||||||
const [previewingRole, setPreviewingRole] = useState<string | null>(null);
|
const [previewingRole, setPreviewingRole] = useState<string | null>(null);
|
||||||
const [audioCache, setAudioCache] = useState<Map<string, string>>(new Map());
|
const [audioCache, setAudioCache] = useState<Map<string, string>>(new Map());
|
||||||
|
const [currentEngine, setCurrentEngine] = useState<string>('kokoro');
|
||||||
|
const [availableVoices, setAvailableVoices] = useState<VoiceOption[]>(KOKORO_VOICES);
|
||||||
|
const [isLoadingEngine, setIsLoadingEngine] = useState(false);
|
||||||
|
|
||||||
|
// Load current engine and voices on mount
|
||||||
|
useEffect(() => {
|
||||||
|
const loadEngineInfo = async () => {
|
||||||
|
try {
|
||||||
|
const [enginesData, voicesData] = await Promise.all([
|
||||||
|
getEngines(),
|
||||||
|
getVoices()
|
||||||
|
]);
|
||||||
|
|
||||||
|
setCurrentEngine(enginesData.current);
|
||||||
|
|
||||||
|
// Update available voices based on current engine
|
||||||
|
if (enginesData.current === 'piper' && voicesData.voices.piper) {
|
||||||
|
const piperVoices: VoiceOption[] = voicesData.voices.piper.map(voiceId => ({
|
||||||
|
id: voiceId,
|
||||||
|
label: voiceId.replace(/_/g, ' ').replace(/-/g, ' - '),
|
||||||
|
gender: voiceId.includes('amy') || voiceId.includes('alba') ? 'female' : 'male',
|
||||||
|
accent: voiceId.includes('GB') ? 'British' : 'American'
|
||||||
|
}));
|
||||||
|
setAvailableVoices(piperVoices);
|
||||||
|
} else {
|
||||||
|
setAvailableVoices(KOKORO_VOICES);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
console.error('Failed to load engine info:', error);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
loadEngineInfo();
|
||||||
|
}, []);
|
||||||
|
|
||||||
const handleVoiceSelect = (roleId: string, voiceId: string) => {
|
const handleVoiceSelect = (roleId: string, voiceId: string) => {
|
||||||
onVoiceChange(roleId, voiceId);
|
onVoiceChange(roleId, voiceId);
|
||||||
@@ -231,6 +265,36 @@ export default function VoiceAssignmentSection({
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const handleEngineChange = async (engine: string) => {
|
||||||
|
setIsLoadingEngine(true);
|
||||||
|
try {
|
||||||
|
await setEngine(engine);
|
||||||
|
setCurrentEngine(engine);
|
||||||
|
|
||||||
|
// Load voices for new engine
|
||||||
|
const voicesData = await getVoices();
|
||||||
|
|
||||||
|
if (engine === 'piper' && voicesData.voices.piper) {
|
||||||
|
const piperVoices: VoiceOption[] = voicesData.voices.piper.map(voiceId => ({
|
||||||
|
id: voiceId,
|
||||||
|
label: voiceId.replace(/_/g, ' ').replace(/-/g, ' - '),
|
||||||
|
gender: voiceId.includes('amy') || voiceId.includes('alba') ? 'female' : 'male',
|
||||||
|
accent: voiceId.includes('GB') ? 'British' : 'American'
|
||||||
|
}));
|
||||||
|
setAvailableVoices(piperVoices);
|
||||||
|
} else {
|
||||||
|
setAvailableVoices(KOKORO_VOICES);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Clear audio cache when switching engines
|
||||||
|
setAudioCache(new Map());
|
||||||
|
} catch (error) {
|
||||||
|
console.error('Failed to switch engine:', error);
|
||||||
|
} finally {
|
||||||
|
setIsLoadingEngine(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
const handleLanguageSelect = (e: React.ChangeEvent<HTMLSelectElement>) => {
|
const handleLanguageSelect = (e: React.ChangeEvent<HTMLSelectElement>) => {
|
||||||
onLanguageChange(e.target.value);
|
onLanguageChange(e.target.value);
|
||||||
};
|
};
|
||||||
@@ -250,6 +314,50 @@ export default function VoiceAssignmentSection({
|
|||||||
<p className="text-lg text-gray-600">Choose a voice for each character in your story.</p>
|
<p className="text-lg text-gray-600">Choose a voice for each character in your story.</p>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{/* TTS Engine Selector */}
|
||||||
|
<div className="p-6 bg-gradient-to-br from-indigo-50 to-purple-50 rounded-2xl border-2 border-indigo-200">
|
||||||
|
<label className="block text-base font-bold text-gray-900 mb-3">
|
||||||
|
TTS Engine
|
||||||
|
</label>
|
||||||
|
<div className="flex gap-4">
|
||||||
|
<button
|
||||||
|
onClick={() => handleEngineChange('kokoro')}
|
||||||
|
disabled={isLoadingEngine}
|
||||||
|
className={`flex-1 px-6 py-4 rounded-xl font-semibold transition-all duration-200 ${
|
||||||
|
currentEngine === 'kokoro'
|
||||||
|
? 'bg-indigo-600 text-white shadow-lg shadow-indigo-500/30'
|
||||||
|
: 'bg-white text-gray-700 border-2 border-gray-200 hover:border-indigo-300'
|
||||||
|
}`}
|
||||||
|
>
|
||||||
|
<div className="text-left">
|
||||||
|
<div className="font-bold">Kokoro TTS</div>
|
||||||
|
<div className="text-sm opacity-90">High quality, 54 voices</div>
|
||||||
|
</div>
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
onClick={() => handleEngineChange('piper')}
|
||||||
|
disabled={isLoadingEngine}
|
||||||
|
className={`flex-1 px-6 py-4 rounded-xl font-semibold transition-all duration-200 ${
|
||||||
|
currentEngine === 'piper'
|
||||||
|
? 'bg-indigo-600 text-white shadow-lg shadow-indigo-500/30'
|
||||||
|
: 'bg-white text-gray-700 border-2 border-gray-200 hover:border-indigo-300'
|
||||||
|
}`}
|
||||||
|
>
|
||||||
|
<div className="text-left">
|
||||||
|
<div className="font-bold">Piper TTS</div>
|
||||||
|
<div className="text-sm opacity-90">Fast, pitch control</div>
|
||||||
|
</div>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
{currentEngine === 'piper' && (
|
||||||
|
<div className="mt-3 p-3 bg-blue-50 border border-blue-200 rounded-lg">
|
||||||
|
<p className="text-sm text-blue-900">
|
||||||
|
<strong>✨ Pitch Control:</strong> Piper allows you to adjust voice pitch per character (coming soon in UI)
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
|
||||||
{/* Voice Assignments */}
|
{/* Voice Assignments */}
|
||||||
<div className="space-y-5">
|
<div className="space-y-5">
|
||||||
{roles.map((role, index) => {
|
{roles.map((role, index) => {
|
||||||
@@ -275,7 +383,7 @@ export default function VoiceAssignmentSection({
|
|||||||
className="flex-1 p-4 border-2 border-gray-200 rounded-xl focus:border-indigo-500 focus:ring-4 focus:ring-indigo-100 focus:outline-none bg-white text-base font-medium transition-all duration-200"
|
className="flex-1 p-4 border-2 border-gray-200 rounded-xl focus:border-indigo-500 focus:ring-4 focus:ring-indigo-100 focus:outline-none bg-white text-base font-medium transition-all duration-200"
|
||||||
>
|
>
|
||||||
<option value="">Select a voice...</option>
|
<option value="">Select a voice...</option>
|
||||||
{KOKORO_VOICES.map((voice) => (
|
{availableVoices.map((voice) => (
|
||||||
<option key={voice.id} value={voice.id}>
|
<option key={voice.id} value={voice.id}>
|
||||||
{voice.label}
|
{voice.label}
|
||||||
</option>
|
</option>
|
||||||
|
|||||||
Reference in New Issue
Block a user