AI Pool
Log in
Discussions>PyTorch RuntimeError: CUDA error: an illegal memory access was encountered>

PyTorch RuntimeError: CUDA error: an illegal memory access was encountered

I've designed a network, which gives a weird error. It occurs randomly and can throw an exception in different epochs.

Traceback (most recent call last):
  File "/user/path/script.py", line 112, in <module>
    input0 = torch.randn(16, 3, 256).cuda()
RuntimeError: CUDA error: an illegal memory access was encountered

I'm using
PyTorch 1.6
Python 3.6
Ubuntu 18.04
CUDA 10.1
cuDNN 7.5
GPU Quadro RTX 8000

Deep LearningComputer VisionAI Development/Engineering
cudapytorchpytorch-memory
3 votesLP262.00
2 Answers
JW326.00
3

I had the same issue before when my code tried to multiply tensors on different device. torch.mul tried to multiple tensors on CPU and on GPU. Normally, PyTorch throws an error when you deal with tensors in different devices, but sometimes it does not. Make sure your tensors (even created on the fly) are on the same device.

Reply
JO295.00
2

Try to decrease the batch size. We do not have access to your code and the most probable reason is high batch size and it tries to allocate additional memory, which is not available.

Reply
Couldn't find what you were looking for?and we will find an expert to answer.
How helpful was this page?