RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR

[Envyana]

Fresh installation of Ubuntu 22.04, with cuda 11.7 I encountered this error when generating any model in webui.how to fix it?

"Traceback (most recent call last):
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/queueing.py", line 407, in call_prediction
output = await route_utils.call_process_api(
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/route_utils.py", line 226, in call_process_api
output = await app.get_blocks().process_api(
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/blocks.py", line 1550, in process_api
result = await self.call_function(
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/blocks.py", line 1185, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 851, in run
result = context.run(func, *args)
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/utils.py", line 661, in wrapper
response = f(*args, **kwargs)
File "/home/ptp-ai-001/voiceai/tts-generation-webui/src/musicgen/musicgen_tab.py", line 215, in generate
outputs_diffusion = mbd.tokens_to_wav(tokens)
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/models/multibanddiffusion.py", line 190, in tokens_to_wav
wav_diffusion = self.generate(emb=condition, size=wav_encodec.size())
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/models/multibanddiffusion.py", line 148, in generate
out += DP.generate(condition=emb, step_list=step_list, initial_noise=torch.randn_like(out))
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/models/multibanddiffusion.py", line 44, in generate
return self.schedule.generate_subsampled(model=self.model, initial=initial_noise, step_list=step_list,
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/modules/diffusion_schedule.py", line 272, in generate_subsampled
return self.sample_processor.return_sample(previous)
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/modules/diffusion_schedule.py", line 106, in return_sample
bands = self.split_bands(x)
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/bands.py", line 84, in forward
lows = self.lowpass(input)
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/lowpass.py", line 107, in forward
out = fft_conv1d(input, self.filters, stride=self.stride)
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/fftconv.py", line 119, in fft_conv1d
weight_z = _rfft(weight)
File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/fftconv.py", line 26, in _new_rfft
z = new_fft.rfft(x, dim=-1)
RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR
{
type: 'status',
endpoint: '/musicgen',
fn_index: 155,
time: 2024-04-30T03:10:49.133Z,
queue: true,
message: null,
stage: 'error',
code: undefined,
success: false
}"

[rsxdalv]

Hi, thanks for the report! According to my search, CUDA 11.8 should be your savior. This is the link to what I assume is related: pytorch/pytorch#88038

…

On Tue, Apr 30, 2024, 6:13 AM Envy42 ***@***.***> wrote: Fresh installation of Ubuntu 22.04, with cuda 11.7 I encountered this error when generating any model in webui. "Traceback (most recent call last): File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/queueing.py", line 407, in call_prediction output = await route_utils.call_process_api( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/route_utils.py", line 226, in call_process_api output = await app.get_blocks().process_api( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/blocks.py", line 1550, in process_api result = await self.call_function( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/blocks.py", line 1185, in call_function prediction = await anyio.to_thread.run_sync( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread return await future File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, *args) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/utils.py", line 661, in wrapper response = f(*args, **kwargs) File "/home/ptp-ai-001/voiceai/tts-generation-webui/src/musicgen/musicgen_tab.py", line 215, in generate outputs_diffusion = mbd.tokens_to_wav(tokens) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/models/multibanddiffusion.py", line 190, in tokens_to_wav wav_diffusion = self.generate(emb=condition, size=wav_encodec.size()) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/models/multibanddiffusion.py", line 148, in generate out += DP.generate(condition=emb, step_list=step_list, initial_noise=torch.randn_like(out)) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/models/multibanddiffusion.py", line 44, in generate return self.schedule.generate_subsampled(model=self.model, initial=initial_noise, step_list=step_list, File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/modules/diffusion_schedule.py", line 272, in generate_subsampled return self.sample_processor.return_sample(previous) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/modules/diffusion_schedule.py", line 106, in return_sample bands = self.split_bands(x) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/bands.py", line 84, in forward lows = self.lowpass(input) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/lowpass.py", line 107, in forward out = fft_conv1d(input, self.filters, stride=self.stride) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/fftconv.py", line 119, in fft_conv1d weight_z = _rfft(weight) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/fftconv.py", line 26, in _new_rfft z = new_fft.rfft(x, dim=-1) RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR { type: 'status', endpoint: '/musicgen', fn_index: 155, time: 2024-04-30T03:10:49.133Z, queue: true, message: null, stage: 'error', code: undefined, success: false }" — Reply to this email directly, view it on GitHub <#306>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTRXI66SNTUW3RGUV57ZZDY74D7JAVCNFSM6AAAAABG7N2WAWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3TANBRGQ2DCMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>