// audio · enhuiz/vall-e

vall-e

vall-e é um projeto de áudio de código aberto de enhuiz (enhuiz/vall-e no GitHub). Tem 2.970 estrelas no GitHub e 399 forks, é publicado sob a licença MIT e foi atualizado pela última vez no GitHub em 10 de maio de 2023.

Unofficial PyTorch implementation of VALL-E, the neural codec language model for zero-shot TTS

★ 3,0 mil estrelas⑂ 399 forksMITatualizado mai 2023

Ver no GitHub ↗

▦ visual: o README deste projeto não inclui nenhuma captura incorporável,
então aqui vai um marcador. O guia abaixo está completo mesmo assim.

// dados do projeto

Dados do projeto

Repositório: enhuiz/vall-e
Estrelas no GitHub: 2.970
Forks: 399
Licença: MIT
Última atualização no GitHub: 10 de maio de 2023
Categoria: Áudio
Tipo de ficha: Guia de instalação completo

// como funciona

Como funciona

Clarify your goal
Understand this is a research implementation to train yourself: you will need your own audio/text data and a GPU setup DeepSpeed supports.
Clone the repo
Clone with submodules, or pip install directly from GitHub as the README shows.
Install and prepare data
Install the package (tested under Python 3.10.7), put .wav + .normalized.txt files in a data folder, quantize them and generate phonemes.
Train and synthesize
Create AR/NAR configs, train both models, export checkpoints, then run the synthesis CLI with text and a reference audio path.

// recursos

Por que as pessoas gostam tanto

AR + NAR models

Implements the autoregressive model for the first quantizer and the non-autoregressive model for the remaining quantizers.

EnCodec tokenizer based

Built on Facebook's EnCodec for audio tokenization and decoding from tokens.

Full training pipeline

Data quantization (vall_e.emb.qnt), phoneme generation (vall_e.emb.g2p) and DeepSpeed-based trainers for both models.

Synthesis CLI

After exporting AR and NAR checkpoints, synthesize speech from text plus a reference recording from the command line.

Configurable presets

YAML configs per dataset with different model presets defined in the code.

Colab toy example

A linked Google Colab notebook demonstrates the flow on a single utterance (the README notes it overfits and is not usable as a model).

// execute você mesmo

Execute você mesmo

Clone the repo
Clone VALL-E with submodules as the README instructs.
```
git clone https://github.com/enhuiz/vall-e.git
cd vall-e
```
Install the package
Alternatively install directly from GitHub with pip (Python 3.10.7 tested; a DeepSpeed-supported GPU is needed for training).
```
pip install git+https://github.com/enhuiz/vall-e
```
Synthesize after training
Export your trained AR/NAR checkpoints, then run the synthesis CLI with text, reference and output paths.
```
python -m vall_e <text> <ref_path> <out_path> --ar-ckpt zoo/ar.pt --nar-ckpt zoo/nar.pt
```

// casos de uso

O que você pode criar

Researching neural codec language models for TTS
Training a VALL-E-style model on a custom dataset
Studying AR/NAR two-stage speech generation
Prototyping zero-shot voice cloning in a research setting

// perguntas frequentes

Perguntas frequentes

Who maintains this VALL-E repo?

enhuiz maintains this unofficial implementation; VALL-E itself is a Microsoft Research model described in the cited arXiv paper.

Is it free?

The code is MIT-licensed. Note the README's notice that EnCodec is CC-BY-NC 4.0, so non-commercial terms apply to the EnCodec parts.

Does it need a paid API key?

No API key — it runs locally with PyTorch. But there is no pretrained checkpoint provided, so you must train the models yourself on GPU hardware.

How does it compare to ready-made TTS tools like gTTS or pyttsx3?

Those synthesize speech immediately. This repo is a research training implementation of a specific paper, with no shipped pretrained model.

How do I get started?

Os outros projetos de Áudio com mais estrelas no catálogo do LibreAI — a lista realista de alternativas se vall-e não for o ideal para você.

MoneyPrinterTurbo★ 128,3 mil
利用 AI 大模型和自动化工作流，根据主题或关键词一键生成高清短视频。Generate HD short videos from a topic or keyword with an automated AI workflow.
Whisper★ 109,9 mil
Robust Speech Recognition via Large-Scale Weak Supervision
GPT-SoVITS★ 62,3 mil
1 min voice data can also be used to train a good TTS model! (few shot voice cloning)
Real-Time-Voice-Cloning★ 60,2 mil
Clone a voice from 5 seconds of audio and generate arbitrary speech in real time
whisper.cpp★ 54,1 mil
Port of OpenAI's Whisper model in C/C++

// mais em audio

Mais em Áudio

Projetos de Áudio próximos no catálogo, os mais próximos em estrelas de vall-e: continue explorando dos dois lados dele.

GPA★ 3,1 mil
GPA — one tiny model that does ASR, TTS and voice conversion (General Purpose Audio).
MARS5-TTS★ 2,8 mil
MARS5 speech model from CAMB.AI — clone a voice from 5 seconds of audio with striking prosody
TTS-WebUI★ 3,3 mil
One Gradio + React WebUI for dozens of TTS, voice-cloning and music models — Bark to Kokoro.
VieNeu-TTS★ 2,8 mil
Vietnamese TTS with instant voice cloning — on-device, real-time CPU inference, 48 kHz audio
audio.cpp★ 3,4 mil
A pure C++ ggml inference engine for audio models — TTS, STT, VAD, voice conversion and music.
gTTS★ 2,6 mil
gTTS — Python library and CLI that turns text into spoken mp3 via Google Translate's speech API

Baixe o vall-e

Grátis e aberto: pegue o código-fonte, dê uma estrela no repositório e comece a criar.

Ver no GitHub ↗← Mais ferramentas de Áudio