Experience generating audio from text with chatterbox
Introduction
Text-to-Speech (TTS) technology is rapidly transforming how we interact with digital content, turning mere words on a screen into natural, engaging audio. Its utility extends far beyond simple convenience; TTS plays a crucial role in accessibility, allowing visually impaired individuals to consume information effortlessly by listening to articles, books, and web pages. In the fast-paced world of industry, TTS powers everything from virtual assistants like Siri and Alexa, making our devices conversational, to automated customer service systems that provide immediate spoken responses. Daily life is also rich with TTS applications, from GPS navigation systems guiding our commutes to audiobooks that bring stories to life during our workouts. The possibilities are truly expansive, constantly evolving to make information more accessible and interactive. It was this fascination with the power of voice technology that led me down a rabbit hole of exploration, eventually bringing me to the exciting open-source chatterbox-tts project on GitHub, a discovery that promised high-quality speech synthesis right at my fingertips. This post shares my initial tests and impressions with this remarkable tool.
Test
To truly put the text-to-speech capabilities of thisopen-source solution to the test, I meticulously replicated almost the entire sample provided directly on the Chatterbox GitHub site. In the following sections, I’ll walk you through the precise implementation steps I took and, most importantly, reveal the output results, offering a firsthand look at its performance.
- Environment preparation (on a MacBook Pro M4 Max) 🈁
python3 -m venv venv source venv/bin/activate pip install --upgrade pip # this installs all the required and necessary packkages pip install chatterbox-tts - The code (with some comments to make it more readable) 🧑💻
# --- Section 1: Setting Up Our Tools --- # First, we need to import the libraries that do the heavy lifting. # `torch` is the foundation, a powerful framework for building and running AI models. # `torchaudio` helps us work with audio data in Python, perfect for saving our generated speech. # `ChatterboxTTS` is the star of the show import torchaudio as ta from chatterbox.tts import ChatterboxTTS import os # --- Section 2: Smart Device Detection (Making it Fast!) --- # If you're on a newer Mac (with an M1, M2, M3, or M4 chip), you have an incredible # built-in GPU called 'MPS' (Metal Performance Shaders) that can speed things up! # This line intelligently detects if MPS is available; otherwise, it falls back to your CPU. device = "mps" if torch.backends.mps.is_available() else "cpu" # Define a `map_location` which tells PyTorch where to load our model data – either your fast GPU or CPU. map_location = torch.device(device) # When PyTorch loads pre-trained models from disk, it needs to know where to put them. # Sometimes, this 'location' isn't explicitly set, especially if the model was saved on a different type of device. # This little 'patch' ensures that no matter where the model came from, it always gets loaded onto # *your* chosen device (`mps` or `cpu`) for optimal performance. torch_load_original = torch.load # We save the original `torch.load` function first. def patched_torch_load(*args, **kwargs): # If the `map_location` isn't specified when loading, we add it! if 'map_location' not in kwargs: kwargs['map_location'] = map_location # Then, we call the original loading function with our (potentially updated) parameters. return torch_load_original(*args, **kwargs) # Finally, we replace PyTorch's default `load` function with our smarter, patched version. torch.load = patched_torch_load # A quick message to confirm which processing unit is being used! print(f"Using device: {device}") # --- Section 3: Loading Our Text-to-Speech Model --- # Getting our AI model ready! `ChatterboxTTS.from_pretrained(device=device)` # does two amazing things: # 1. It automatically downloads the pre-trained model weights (the 'brain' of the AI) # if you don't already have them, saving you a lot of setup hassle. # 2. It loads these weights onto your chosen `device` (MPS or CPU). # Wrapping this in a `try-except` block, just in case there's an issue with downloading or loading. try: model = ChatterboxTTS.from_pretrained(device=device) print("ChatterboxTTS model loaded successfully.") except Exception as e: print(f"Error loading ChatterboxTTS model: {e}") print("Please ensure you have an active internet connection and that the 'chatterbox-tts' library is correctly installed.") exit() # If the model can't load, we can't do anything, so we exit. # --- Section 4: The Text We Want to Hear --- # default as provided in the original sample text = "Today is the day. I want to move like a titan at dawn, sweat like a god forging lightning. No more excuses. From now on, my mornings will be temples of discipline. I am going to work out like the gods… every damn day." print(f"Text to synthesize: \"{text}\"") # --- Section 5: Customizing the Voice (Optional) --- # Try to generate the new text in that person's voice! # If you don't have a custom audio prompt, don't worry – the model will use its default voice. # Replace "YOUR_FILE.wav" with your actual audio file's path. AUDIO_PROMPT_PATH = "YOUR_FILE.wav" # <-- CHANGE THIS to your WAV file path, or set to None! # Check if the audio prompt file actually exists. # If it's the placeholder or not found, we'll just use the default voice. if not os.path.exists(AUDIO_PROMPT_PATH) or AUDIO_PROMPT_PATH == "YOUR_FILE.wav": print(f"Warning: Audio prompt file '{AUDIO_PROMPT_PATH}' not found or is default. Generating with default voice.") AUDIO_PROMPT_PATH = None # Setting to None tells the model to use its default voice. else: print(f"Using audio prompt from: {AUDIO_PROMPT_PATH}") # --- Section 6: Generating the Audio --- # This is the core command! `model.generate()` takes your text and turns it into sound. # - `audio_prompt_path`: (Optional) The path to your `.wav` file for voice customization. # - `exaggeration`: Controls how expressive the generated speech is. Higher values mean more emotion! # - `cfg_weight`: Stands for 'Classifier-Free Guidance weight'. It influences how much the model # adheres to the input text vs. generating more diverse speech. Adjust to fine-tune. try: wav = model.generate( text, audio_prompt_path=AUDIO_PROMPT_PATH, # Pass our audio prompt (or None) exaggeration=2.0, # Feel free to experiment with this value! cfg_weight=0.5 # And this one too! ) print("Audio waveform generated successfully.") except Exception as e: print(f"Error during audio generation: {e}") exit() # If audio generation fails, we stop here. # --- Section 7: Saving Your Masterpiece --- # Saving the generated audio. # `torchaudio.save()` writes our `wav` (the audio data) to a file. # `model.sr` gives us the 'sample rate' (samples per second) which is crucial for audio quality. output_filename = "test-2.wav" try: ta.save(output_filename, wav, model.sr) print(f"Audio saved to {output_filename}") except Exception as e: print(f"Error saving audio file: {e}") # Play your `test-2.wav` file and hear your words come to life! - Console logs during the application run 🏃
python generate_audio.py /Users/xxx/Devs/chatterbox-aam/venv/lib/python3.12/site-packages/perth/perth_net/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. from pkg_resources import resource_filename The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`. 0it [00:00, ?it/s] Using device: mps ve.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.70M/5.70M [00:00<00:00, 16.6MB/s] t3_cfg.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.13G/2.13G [01:27<00:00, 24.4MB/s] s3gen.safetensors: 54%|████████████████████████████████████████████████████████████████▊ | 566M/1.06G [00:23s3gen.safetensors: 55%|██████████████████████████████████████████████████████████████████ | 577M/1.06G [00:23s3gen.safetensors: 56%|███████████████████████████████████████████████████████████████████▎ | 587M/1.06G [00:24s3gen.safetensors: 57%|████████████████████████████████████████████████████████████████████▍ | 598M/1.06G [00:24s3gen.safetensors: 58%|█████████████████████████████████████████████████████████████████████▋ | 608M/1.06G [00:24s3gen.safetensors: 59%|██████████████████████████████████████████████████████████████████████▊ | 619M/1.06G [00:25s3gen.safetensors: 60%|████████████████████████████████████████████████████████████████████████ | 629M/1.06G [00:25s3gen.safetensors: 61%|█████████████████████████████████████████████████████████████████████████▎ | 640M/1.06G [00:26s3gen.safetensors: 62%|██████████████████████████████████████████████████████████████████████████▍ | 650M/1.06G [00:26s3gen.safetensors: 63%|███████████████████████████████████████████████████████████████████████████▋ | 661M/1.06G [00:27s3gen.safetensors: 64%|████████████████████████████████████████████████████████████████████████████▊ | 671M/1.06G [00:27s3gen.safetensors: 65%|██████████████████████████████████████████████████████████████████████████████ | 682M/1.06G [00:27s3gen.safetensors: 66%|███████████████████████████████████████████████████████████████████████████████▎ | 692M/1.06G [00:28s3gen.safetensors: 66%|████████████████████████████████████████████████████████████████████████████████▍ | 703M/1.06G [00:28s3gen.safetensors: 67%|█████████████████████████████████████████████████████████████████████████████████▋ | 713M/1.06G [00:29s3gen.safetensors: 68%|██████████████████████████████████████████████████████████████████████████████████▊ | 724M/1.06G [00:29s3gen.safetensors: 69%|████████████████████████████████████████████████████████████████████████████████████ | 734M/1.06G [00:29s3gen.safetensors: 70%|█████████████████████████████████████████████████████████████████████████████████████▎ | 744M/1.06G [00:30s3gen.safetensors: 71%|██████████████████████████████████████████████████████████████████████████████████████▍ | 755M/1.06G [00:30s3gen.safetensors: 72%|███████████████████████████████████████████████████████████████████████████████████████▋ | 765M/1.06G [00:31s3gen.safetensors: 73%|████████████████████████████████████████████████████████████████████████████████████████▊ | 776M/1.06G [00:31s3gen.safetensors: 74%|██████████████████████████████████████████████████████████████████████████████████████████ | 786M/1.06G [00:32s3gen.safetensors: 75%|███████████████████████████████████████████████████████████████████████████████████████████▎ | 797M/1.06G [00:32s3gen.safetensors: 76%|████████████████████████████████████████████████████████████████████████████████████████████▍ | 807M/1.06G [00:32s3gen.safetensors: 77%|█████████████████████████████████████████████████████████████████████████████████████████████▋ | 818M/1.06G [00:33s3gen.safetensors: 78%|██████████████████████████████████████████████████████████████████████████████████████████████▊ | 828M/1.06G [00:33s3gen.safetensors: 79%|████████████████████████████████████████████████████████████████████████████████████████████████ | 839M/1.06G [00:34s3gen.safetensors: 80%|█████████████████████████████████████████████████████████████████████████████████████████████████▎ | 849M/1.06G [00:34s3gen.safetensors: 81%|██████████████████████████████████████████████████████████████████████████████████████████████████▍ | 860M/1.06G [00:35s3gen.safetensors: 82%|███████████████████████████████████████████████████████████████████████████████████████████████████▋ | 870M/1.06G [00:35s3gen.safetensors: 83%|████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 881M/1.06G [00:36s3gen.safetensors: 84%|██████████████████████████████████████████████████████████████████████████████████████████████████████ | 891M/1.06G [00:36s3gen.safetensors: 85%|███████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 902M/1.06G [00:36s3gen.safetensors: 86%|████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 912M/1.06G [00:37s3gen.safetensors: 87%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 923M/1.06G [00:37s3gen.safetensors: 88%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 933M/1.06G [00:38s3gen.safetensors: 89%|████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 944M/1.06G [00:38s3gen.safetensors: 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 954M/1.06G [00:39s3gen.safetensors: 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 965M/1.06G [00:39s3gen.safetensors: 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 975M/1.06G [00:39s3gen.safetensors: 93%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 986M/1.06G [00:40s3gen.safetensors: 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 996M/1.06G [00:40s3gen.safetensors: 95%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 1.01G/1.06G [00:41s3gen.safetensors: 96%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 1.02G/1.06G [00:41s3gen.safetensors: 97%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 1.03G/1.06G [00:42s3gen.safetensors: 98%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 1.04G/1.06G [00:42s3gen.safetensors: 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 1.05G/1.06G [00:42s3gen.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.06G/1.06G [00:43s3gen.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.06G/1.06G [00:43<00:00, 24.4MB/s] tokenizer.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 25.5k/25.5k [00:00<00:00, 10.4MB/s] conds.pt: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 107k/107k [00:00<00:00, 26.5MB/s] /Users/alainairom/Devs/chatterbox-aam/venv/lib/python3.12/site-packages/diffusers/models/lora.py:393: FutureWarning: `LoRACompatibleLinear` is deprecated and will be removed in version 1.0.0. Use of `LoRACompatibleLinear` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`. deprecate("LoRACompatibleLinear", "1.0.0", deprecation_message) loaded PerthNet (Implicit) at step 250,000 ChatterboxTTS model loaded successfully. Text to synthesize: "Today is the day. I want to move like a titan at dawn, sweat like a god forging lightning. No more excuses. From now on, my mornings will be temples of discipline. I am going to work out like the gods… every damn day." Warning: Audio prompt file 'YOUR_FILE.wav' not found or is default. Generating with default voice. /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py:105: FutureWarning: `torch.backends.cuda.sdp_kernel()` is deprecated. In the future, this context manager will be removed. Please see `torch.nn.attention.sdpa_kernel()` for the new context manager, with updated signature. self.gen = func(*args, **kwds) LlamaModel is using LlamaSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model. Sampling: 0%| | 0/1000 [00:00<?, ?it/s]We detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format) Sampling: 29%|█████████████████████████████████▍ | 293/1000 [00:33<01:20, 8.76it/s] Audio waveform generated successfully. Audio saved to test-2.wav All the downloaded files could be found in “/Users/xxx/.cache/huggingface/hub”.
pwd /Users/xxx/.cache/huggingface/hub > ls models--ResembleAI--chatterbox version_diffusers_cache.txt version.txt - Upon execution, we obtained a perfectly clear WAV file with a precise length of 11 seconds. This output demonstrates the tool’s immediate readiness for use. What’s more, the prompt in the code is fully customizable, allowing you to replace it with any English text you desire. The resulting audio quality is quite impressive, making Chatterbox a compelling option for various TTS applications.
Conclusion
In conclusion, the versatility and increasing quality of Text-to-Speech technology continue to open new avenues for innovation, making digital interactions more dynamic and inclusive. As demonstrated with Chatterbox, the power of open-source tools allows developers and enthusiasts alike to harness cutting-edge AI for speech synthesis, achieving truly outstanding and customizable audio results without proprietary barriers. The future of voice-enabled applications looks incredibly bright, and readily available tools like Chatterbox are at the forefront of this exciting evolution.
Link
- Chatterbox: https://github.com/resemble-ai/chatterbox/tree/master
- Resemble AI: https://github.com/resemble-ai
Top comments (0)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.