Train Your Own Voice

Requirements

• Windows 10/11 with WSL2
• NVIDIA RTX 3060 (12GB VRAM) or comparable
• At least 50GB free disk space
• Internet connection
• Audio data: At least 30 minutes (recommended: 1-2 hours)

1. WSL2 Setup

1.1 Activate WSL (First Installation)

If you have never installed WSL before:

1. Open CMD (Command Prompt) in Windows
2. Type wsl and press Enter
3. Windows will report that WSL is not installed and ask if it should be installed
4. Press Enter to confirm
5. Windows will install WSL automatically
6. Restart your PC

1.2 Install Ubuntu

In CMD (Command Prompt) as Administrator:

wsl --install -d Ubuntu-22.04

After installation, you will be prompted to create a username and password.

IMPORTANT: Remember your username and password - you will need them later!

After setup, you will automatically be in the Ubuntu WSL environment.

1.3 Exit WSL Session

You now need to exit the current WSL session. Simply close the Ubuntu window (X button at top right) or type:

exit

💡 Note: An error message like "Error in installation process" may appear - this is normal and can be ignored. The installation is still successfully completed.

1.4 Set WSL2 as Default

Open a new CMD window as Administrator and execute:

wsl --set-default-version 2

wsl --set-default Ubuntu-22.04

1.5 Restart Ubuntu

wsl -d Ubuntu-22.04

Now you are back in the Ubuntu environment and can continue with step 2.

1.6 Test GPU Access (Optional)

To check if your NVIDIA GPU is recognized in WSL:

nvidia-smi

Expected result: You should see your GPU with VRAM information. CUDA is automatically included with WSL2!

2. Install System Packages

2.1 Update System

IMPORTANT: Run this first before installing packages!

sudo apt update && sudo apt upgrade -y

This may take a few minutes.

2.2 Base Packages

Build Tools:

sudo apt install -y build-essential ninja-build

Python and Development Tools:

sudo apt install -y python3.10 python3.10-venv python3.10-dev python3-pip

Audio Libraries:

sudo apt install -y libsndfile1 ffmpeg

espeak-ng:

sudo apt install -y espeak-ng

Git:

sudo apt install -y git

Additional Dependencies:

sudo apt install -y libportaudio2 portaudio19-dev

2.3 Update CMake (Version 3.26+ required)

Remove old CMake version:

sudo apt remove --purge cmake -y

sudo apt autoremove -y

Install CMake from Kitware:

wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | gpg --dearmor - | sudo tee /usr/share/keyrings/kitware-archive-keyring.gpg >/dev/null

echo 'deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ jammy main' | sudo tee /etc/apt/sources.list.d/kitware.list >/dev/null

sudo apt update

sudo apt install -y cmake

Check version (should be 3.26 or higher):

cmake --version

3. Setup Piper

3.1 Create Working Directory

cd ~

mkdir piper-training

cd piper-training

3.2 Clone Repository

git clone https://github.com/OHF-voice/piper1-gpl.git

cd piper1-gpl

3.3 Create Python Virtual Environment

python3 -m venv .venv

source .venv/bin/activate

💡 Successfully activated? You should now see (.venv) before your username in the command line.

pip install --upgrade pip setuptools wheel

3.4 Install PyTorch with CUDA Support

IMPORTANT: Install PyTorch first, BEFORE installing Piper!

PyTorch 2.0.1 with CUDA 11.8 (installation may take a while):

pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118

PyTorch Lightning:

pip install pytorch-lightning==2.0.9 lightning==2.0.9

Set NumPy to compatible version:

pip install "numpy<2.0"

3.5 Test PyTorch CUDA Access

python3 -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}'); print(f'CUDA device: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"None\"}')"

Expected result: CUDA available: True and your GPU name

3.6 Install Piper Training Dependencies

pip install -e .[train]

3.7 Build Cython Extension

chmod +x build_monotonic_align.sh

./build_monotonic_align.sh

💡 Performance warnings are normal and can be ignored!

3.8 scikit-build and Dev Build

pip install scikit-build

IMPORTANT: pip sometimes installs an old CMake version in venv - this must be removed:

pip uninstall cmake -y

Check CMake version (should be 3.26+):

cmake --version

Dev Build:

python3 setup.py build_ext --inplace

3.9 Download Checkpoint

mkdir -p ~/piper-training/checkpoints

cd ~/piper-training/checkpoints

Choose the checkpoint for your language (download only ONE!):

# German
wget https://huggingface.co/datasets/rhasspy/piper-checkpoints/resolve/main/en/en_US/lessac/medium/epoch%3D2164-step%3D1355540.ckpt

# English
wget https://huggingface.co/datasets/rhasspy/piper-checkpoints/resolve/main/en/en_US/lessac/medium/epoch%3D2164-step%3D1355540.ckpt

# Russian
wget https://huggingface.co/datasets/rhasspy/piper-checkpoints/resolve/main/ru/ru_RU/ruslan/medium/epoch%3D2436-step%3D1724372.ckpt

# Polish
wget https://huggingface.co/datasets/rhasspy/piper-checkpoints/resolve/main/pl/pl_PL/gosia/medium/epoch%3D5001-step%3D1457672.ckpt

# Romanian
wget https://huggingface.co/datasets/rhasspy/piper-checkpoints/resolve/main/ro/ro_RO/mihai/medium/epoch%3D7809-step%3D1558760.ckpt

# Spanish (Spain)
wget https://huggingface.co/datasets/rhasspy/piper-checkpoints/resolve/main/es/es_ES/davefx/medium/epoch%3D5629-step%3D1605020.ckpt

# Spanish (Mexico)
wget https://huggingface.co/datasets/rhasspy/piper-checkpoints/resolve/main/es/es_MX/ald/medium/epoch%3D9999-step%3D1753600.ckpt

# Czech
wget https://huggingface.co/datasets/rhasspy/piper-checkpoints/resolve/main/cs/cs_CZ/jirka/medium/epoch%3D8819-step%3D1435400.ckpt

💡 Note for Italian: There is currently no official Italian checkpoint. Use the German or English checkpoint as a base.

3.10 Prepare Directory Structure

cd ~/piper-training

mkdir -p voice-data/raw-audio

mkdir -p voice-data/audio

mkdir -p voice-data/cache

mkdir -p voice-data/output

4. Prepare Audio Data

4.1 Copy Audio Files to WSL

From Windows PowerShell:

copy C:\Pfad\zu\deinen\*.wav \\wsl$\Ubuntu-22.04\home\DEIN_USERNAME\piper-training\voice-data\raw-audio\

Or Windows Explorer:

Address: \\wsl$\Ubuntu-22.04\home\YOUR_USERNAME\piper-training\voice-data\raw-audio\

Copy files via drag & drop

💡 Note: The audio format doesn't matter - the automation script automatically converts all files to 22050 Hz Mono!

5. Automatic Processing with Whisper

5.1 Install Whisper

cd ~/piper-training/piper1-gpl

Activate Virtual Environment (if not already active):

source .venv/bin/activate

Install Whisper:

pip install openai-whisper pydub

5.2 Create Automation Script

Create the file with:

cd ~/piper-training

nano process_audio_auto.py

#!/usr/bin/env python3
"""
Automatische Audio-Verarbeitung für Piper Voice Training
- Konvertiert Audio zu 22050 Hz Mono
- Splittet in kurze Segmente
- Transkribiert mit Whisper
- Erstellt metadata.csv
"""

import os
import sys
import subprocess
from pathlib import Path
from pydub import AudioSegment
from pydub.silence import split_on_silence
import whisper
from tqdm import tqdm

# Konfiguration
RAW_AUDIO_DIR = "voice-data/raw-audio"
OUTPUT_AUDIO_DIR = "voice-data/audio"
METADATA_FILE = "voice-data/metadata.csv"
WHISPER_MODEL = "large"  # tiny, base, small, medium, large

# Segment-Einstellungen
MIN_SILENCE_LEN = 500  # ms
SILENCE_THRESH = -40   # dB
KEEP_SILENCE = 200     # ms
MIN_SEGMENT_LEN = 1000 # ms
MAX_SEGMENT_LEN = 15000 # ms

def convert_to_mono_22050(input_file, output_file):
    """Konvertiert Audio zu 22050 Hz Mono"""
    print(f"  Konvertiere: {input_file.name}")
    
    # Erst mit ffmpeg zu Standard-WAV konvertieren (behebt ADPCM-Problem)
    temp_standard_wav = f"/tmp/temp_standard_{input_file.stem}.wav"
    result = subprocess.run([
        'ffmpeg', '-i', str(input_file),
        '-acodec', 'pcm_s16le',  # Standard PCM 16-bit
        '-ar', '44100',           # Erstmal auf 44100 Hz
        '-ac', '1',               # Mono
        '-y',                     # Überschreiben
        temp_standard_wav
    ], capture_output=True, text=True)
    
    if result.returncode != 0:
        print(f"  FEHLER bei ffmpeg-Konvertierung: {result.stderr}")
        raise Exception(f"ffmpeg failed for {input_file}")
    
    # Jetzt mit pydub weiterverarbeiten
    audio = AudioSegment.from_file(temp_standard_wav)
    
    # Sample Rate auf 22050 Hz setzen
    if audio.frame_rate != 22050:
        audio = audio.set_frame_rate(22050)
    
    # Als WAV speichern
    audio.export(output_file, format="wav")
    
    # Temp-Datei löschen
    os.remove(temp_standard_wav)
    
    return audio

def split_audio(audio, base_name):
    """Splittet Audio in kurze Segmente"""
    print(f"  Splitte Audio...")
    
    chunks = split_on_silence(
        audio,
        min_silence_len=MIN_SILENCE_LEN,
        silence_thresh=SILENCE_THRESH,
        keep_silence=KEEP_SILENCE
    )
    
    segments = []
    for i, chunk in enumerate(chunks):
        chunk_len = len(chunk)
        
        # Nur Chunks zwischen MIN und MAX Länge
        if MIN_SEGMENT_LEN <= chunk_len <= MAX_SEGMENT_LEN:
            segments.append(chunk)
        
        # Zu lange Chunks weiter splitten
        elif chunk_len > MAX_SEGMENT_LEN:
            # In kleinere Teile aufteilen
            num_parts = int(chunk_len / MAX_SEGMENT_LEN) + 1
            part_len = chunk_len // num_parts
            
            for j in range(num_parts):
                start = j * part_len
                end = start + part_len if j < num_parts - 1 else chunk_len
                part = chunk[start:end]
                
                if len(part) >= MIN_SEGMENT_LEN:
                    segments.append(part)
    
    print(f"  {len(segments)} Segmente erstellt")
    return segments

def transcribe_segments(segments, segment_files):
    """Transkribiert alle Segmente mit Whisper"""
    print(f"\nLade Whisper-Modell '{WHISPER_MODEL}'...")
    model = whisper.load_model(WHISPER_MODEL)
    
    print(f"Transkribiere {len(segments)} Segmente...")
    transcriptions = []
    
    for i, (segment, file_path) in enumerate(tqdm(zip(segments, segment_files), total=len(segments))):
        # Segment temporär speichern für Whisper
        temp_file = f"/tmp/temp_segment_{i}.wav"
        segment.export(temp_file, format="wav")
        
        # Transkription
        result = model.transcribe(
            temp_file,
            language="de",  # Deutsch
            task="transcribe",
            fp16=True  # Nutzt GPU wenn verfügbar
        )
        
        text = result["text"].strip()
        transcriptions.append(text)
        
        # Temp-Datei löschen
        os.remove(temp_file)
    
    return transcriptions

def main():
    # Verzeichnisse erstellen
    os.makedirs(OUTPUT_AUDIO_DIR, exist_ok=True)
    
    # Alle Audio-Dateien finden
    raw_audio_path = Path(RAW_AUDIO_DIR)
    audio_files = list(raw_audio_path.glob("*.wav")) + \
                  list(raw_audio_path.glob("*.WAV")) + \
                  list(raw_audio_path.glob("*.mp3")) + \
                  list(raw_audio_path.glob("*.MP3")) + \
                  list(raw_audio_path.glob("*.m4a")) + \
                  list(raw_audio_path.glob("*.M4A")) + \
                  list(raw_audio_path.glob("*.flac")) + \
                  list(raw_audio_path.glob("*.FLAC"))
    
    if not audio_files:
        print(f"Keine Audio-Dateien in {RAW_AUDIO_DIR} gefunden!")
        sys.exit(1)
    
    print(f"Gefunden: {len(audio_files)} Audio-Datei(en)")
    print(f"Whisper-Modell: {WHISPER_MODEL}")
    print()
    
    all_segments = []
    all_segment_files = []
    segment_counter = 0
    
    # Jede Audio-Datei verarbeiten
    for audio_file in audio_files:
        print(f"Verarbeite: {audio_file.name}")
        
        # Konvertieren
        converted_file = f"/tmp/converted_{audio_file.stem}.wav"
        audio = convert_to_mono_22050(audio_file, converted_file)
        
        # Splitten
        segments = split_audio(audio, audio_file.stem)
        
        # Segmente speichern
        for segment in segments:
            filename = f"utt_{segment_counter:04d}.wav"
            filepath = os.path.join(OUTPUT_AUDIO_DIR, filename)
            segment.export(filepath, format="wav")
            
            all_segments.append(segment)
            all_segment_files.append(filename)
            segment_counter += 1
        
        # Temp-Datei löschen
        os.remove(converted_file)
        print()
    
    print(f"\nGesamt: {len(all_segments)} Segmente erstellt")
    
    # Transkription
    transcriptions = transcribe_segments(all_segments, all_segment_files)
    
    # metadata.csv erstellen
    print(f"\nErstelle {METADATA_FILE}...")
    with open(METADATA_FILE, "w", encoding="utf-8") as f:
        for filename, text in zip(all_segment_files, transcriptions):
            # Bereinige Text (entferne Zeilenumbrüche, etc.)
            text = text.replace("\n", " ").replace("\r", " ").strip()
            # Schreibe Zeile
            f.write(f"{filename}|{text}\n")
    
    print(f"✅ Fertig!")
    print(f"   Segmente: {len(all_segments)}")
    print(f"   Metadata: {METADATA_FILE}")
    print(f"\nÜberprüfe die metadata.csv und korrigiere bei Bedarf Fehler!")
    print(f"Dann starte das Training mit den Anweisungen in der Anleitung")

if __name__ == "__main__":
    main()

Save: Ctrl+O, Enter, Ctrl+X

5.3 Run Script

chmod +x process_audio_auto.py

cd ~/piper-training

source ~/piper-training/piper1-gpl/.venv/bin/activate

python3 process_audio_auto.py

Expected duration (RTX 3060, large model): 2 hours audio = approx. 30-60 minutes processing

5.4 Check Results

Count segments:

ls ~/piper-training/voice-data/audio/*.wav | wc -l

Check metadata.csv:

head -10 ~/piper-training/voice-data/metadata.csv

IMPORTANT: Check transcriptions and correct errors:

• Spell out numbers: "three" instead of "3"
• Correct names
• Add punctuation

nano ~/piper-training/voice-data/metadata.csv

6. Start Training

6.1 Activate Virtual Environment

cd ~/piper-training/piper1-gpl

Activate Virtual Environment (if not already active):

source .venv/bin/activate

6.2 Training Command

Adjust espeak_voice and max_epochs for your language:

python3 -m piper.train fit \
  --data.voice_name "meine_stimme" \
  --data.csv_path ~/piper-training/voice-data/metadata.csv \
  --data.audio_dir ~/piper-training/voice-data/audio/ \
  --model.sample_rate 22050 \
  --data.espeak_voice "de" \
  --data.cache_dir ~/piper-training/voice-data/cache/ \
  --data.config_path ~/piper-training/voice-data/output/config.json \
  --data.batch_size 16 \
  --ckpt_path ~/piper-training/checkpoints/*.ckpt \
  --trainer.max_epochs 4000 \
  --trainer.check_val_every_n_epoch 10

Language-specific Settings

Additional Parameters:

• voice_name: Name of your voice
• batch_size: 16 for RTX 3060 (reduce to 8 for OOM errors)
• ckpt_path: *.ckpt automatically finds your downloaded checkpoint

6.3 Monitor Training

Second terminal for GPU monitoring:

wsl
watch -n 1 nvidia-smi

Expected training time (RTX 3060):

• With checkpoint + 30 min audio: 2-4 hours
• With checkpoint + 1-2 hrs audio: 4-8 hours

6.4 Interrupt and Resume Training

Interrupt: Ctrl+C

Resume:

Find last checkpoint:

ls -lht ~/piper-training/piper1-gpl/lightning_logs/version_*/checkpoints/

Resume training (adjust espeak_voice and max_epochs for your language):

python3 -m piper.train fit \
  --ckpt_path ~/piper-training/piper1-gpl/lightning_logs/version_0/checkpoints/last.ckpt \
  --data.voice_name "meine_stimme" \
  --data.csv_path ~/piper-training/voice-data/metadata.csv \
  --data.audio_dir ~/piper-training/voice-data/audio/ \
  --model.sample_rate 22050 \
  --data.espeak_voice "de" \
  --data.cache_dir ~/piper-training/voice-data/cache/ \
  --data.config_path ~/piper-training/voice-data/output/config.json \
  --data.batch_size 16 \
  --trainer.max_epochs 4000

max_epochs per language: DE: 4000 | EN: 3000 | RU: 3300 | PL: 5900 | RO: 8700 | ES(ES): 6500 | ES(MX): 10800 | CS: 9700

7. Export and Test Model

7.1 Export to ONNX

cd ~/piper-training/piper1-gpl

Activate Virtual Environment (if not already active):

source .venv/bin/activate

Find best checkpoint:

ls -lht lightning_logs/version_*/checkpoints/

Export:

python3 -m piper.train.export_onnx \
  --checkpoint lightning_logs/version_0/checkpoints/epoch=999-step=10000.ckpt \
  --output-file ~/piper-training/voice-data/output/de_DE-meine_stimme-medium.onnx

7.2 Rename Config File

cd ~/piper-training/voice-data/output

cp config.json de_DE-meine_stimme-medium.onnx.json

ls -lh

7.3 Download and Test Piper CLI

cd ~/piper-training

wget https://github.com/rhasspy/piper/releases/download/2023.11.14-2/piper_linux_x86_64.tar.gz

tar -xzf piper_linux_x86_64.tar.gz

Test synthesis:

echo "This is a test of my new voice. I hope it sounds good!" | \
  ./piper/piper \
  --model voice-data/output/en_US-my_voice-medium.onnx \
  --output_file test_output.wav

Play audio:

sudo apt install -y sox

play test_output.wav

7.4 Copy to Windows

cp ~/piper-training/voice-data/output/en_US-my_voice-medium.onnx* /mnt/c/Users/YOUR_USERNAME/Desktop/

8. Improve Quality

8.1 Record More Audio (MOST IMPORTANT MEASURE!)

Recommended recording guidelines:

• At least 30 minutes (better: 1-2+ hours)
• Various sentences and sentence lengths
• Various emotions/emphasis
• Clear, distinct pronunciation
• Quiet environment (no background noise)
• Constant distance to microphone
• No clipping (level not too high)

Recording settings:

• Sample Rate: 22050 Hz
• Bit Depth: 16-bit
• Channels: Mono
• Format: WAV (uncompressed)

8.2 Adjust Training Parameters

For longer training:

--trainer.max_epochs 5000

For better quality (slower):

--data.batch_size 8

8.3 Retrain with More Data

1. Copy new audio files to voice-data/raw-audio/
2. Run process_audio_auto.py again
3. Resume training with old checkpoint

9. Troubleshooting

Problem: "CUDA out of memory"

Solution:

--data.batch_size 8  # or 4

Problem: "Weights only load failed"

Solution: Downgrade PyTorch

pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118

pip install pytorch-lightning==2.0.9 lightning==2.0.9

Problem: Training doesn't converge (Loss doesn't decrease)

Solutions:

• Check metadata.csv for typos
• Make sure audio files are correct
• Reduce learning rate: --optimizer.lr 0.00005

Problem: Voice sounds robotic

Causes:

• Too little training data
• Training too short
• Poor audio quality

Solutions:

• Record more audio (at least 30 min)
• Train longer (2000+ epochs)
• Improve audio quality

Problem: WSL is slow

IMPORTANT: ALWAYS work in the Linux filesystem (/home/...), NOT in the Windows filesystem (/mnt/c/...)!

Problem: Whisper is very slow

Solutions:

• Test GPU access: python3 -c "import torch; print(torch.cuda.is_available())"
• Smaller model: WHISPER_MODEL = "small"

Useful Commands

Shutdown WSL (from Windows):

wsl --shutdown

Start WSL:

wsl

Activate Virtual Environment:

source ~/piper-training/piper1-gpl/.venv/bin/activate

GPU Status:

nvidia-smi

Disk Space:

df -h

WSL Backup:

wsl --export Ubuntu-22.04 D:\wsl-backup.tar

WSL Restore:

wsl --import Ubuntu-22.04 D:\WSL\Ubuntu D:\wsl-backup.tar

Resources

• Piper GitHub
• Piper Checkpoints
• Whisper
• espeak-ng
• Complete Markdown Guide