nn.parallel.DistributedDataParallel报错，一块GPU时不报错
Warning: Traceback of forward call that caused the error:
File "train_YnetV2_dist.py", line 273, in <module>
pred_waypoint_map,pred_traj_map, pred_traj, pred_goal = model(feature_input,V_obs,A_obs,gt_waypoint_map)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 307, in forward
inter_feat, adjA = self.inter_features(V_obs, A_obs)
File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 342, in inter_features
inter_represent = self.stGcn(x, A)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 170, in forward
z = self.tcn(y).permute(0, 2, 1, 3)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
input = module(input)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 106, in forward
exponential_average_factor, self.eps)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/functional.py", line 1685, in batch_norm
training, momentum, eps, torch.backends.cudnn.enabled
(print_stack at /opt/conda/conda-bld/pytorch_1582013237873/work/torch/csrc/autograd/python_anomaly_mode.cpp:58)
Traceback (most recent call last):
File "train_YnetV2_dist.py", line 289, in <module>
loss.backward(retain_graph=True)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [5]] is at version 10; expected version 9 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Warning: Traceback of forward call that caused the error:
File "train_YnetV2_dist.py", line 273, in <module>
pred_waypoint_map,pred_traj_map, pred_traj, pred_goal = model(feature_input,V_obs,A_obs,gt_waypoint_map)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 307, in forward
inter_feat, adjA = self.inter_features(V_obs, A_obs)
File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 342, in inter_features
inter_represent = self.stGcn(x, A)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 170, in forward
z = self.tcn(y).permute(0, 2, 1, 3)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
input = module(input)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 106, in forward
exponential_average_factor, self.eps)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/functional.py", line 1685, in batch_norm
training, momentum, eps, torch.backends.cudnn.enabled
(print_stack at /opt/conda/conda-bld/pytorch_1582013237873/work/torch/csrc/autograd/python_anomaly_mode.cpp:58)
Traceback (most recent call last):
File "train_YnetV2_dist.py", line 289, in <module>
loss.backward(retain_graph=True)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [5]] is at version 10; expected version 9 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Warning: Traceback of forward call that caused the error:
File "train_YnetV2_dist.py", line 273, in <module>
pred_waypoint_map,pred_traj_map, pred_traj, pred_goal = model(feature_input,V_obs,A_obs,gt_waypoint_map)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 307, in forward
inter_feat, adjA = self.inter_features(V_obs, A_obs)
File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 342, in inter_features
inter_represent = self.stGcn(x, A)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 170, in forward
z = self.tcn(y).permute(0, 2, 1, 3)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
input = module(input)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 106, in forward
exponential_average_factor, self.eps)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/functional.py", line 1685, in batch_norm
training, momentum, eps, torch.backends.cudnn.enabled
(print_stack at /opt/conda/conda-bld/pytorch_1582013237873/work/torch/csrc/autograd/python_anomaly_mode.cpp:58)
Traceback (most recent call last):
File "train_YnetV2_dist.py", line 289, in <module>
loss.backward(retain_graph=True)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [5]] is at version 10; expected version 9 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Warning: Traceback of forward call that caused the error:
File "train_YnetV2_dist.py", line 273, in <module>
pred_waypoint_map,pred_traj_map, pred_traj, pred_goal = model(feature_input,V_obs,A_obs,gt_waypoint_map)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 307, in forward
inter_feat, adjA = self.inter_features(V_obs, A_obs)
File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 342, in inter_features
inter_represent = self.stGcn(x, A)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 170, in forward
z = self.tcn(y).permute(0, 2, 1, 3)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
input = module(input)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 106, in forward
exponential_average_factor, self.eps)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/functional.py", line 1685, in batch_norm
training, momentum, eps, torch.backends.cudnn.enabled
(print_stack at /opt/conda/conda-bld/pytorch_1582013237873/work/torch/csrc/autograd/python_anomaly_mode.cpp:58)
Traceback (most recent call last):
File "train_YnetV2_dist.py", line 289, in <module>
loss.backward(retain_graph=True)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [5]] is at version 10; expected version 9 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Traceback (most recent call last):
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in <module>
main()
File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/gs/home/wangrp/anaconda/envs/w01/bin/python', '-u', 'train_YnetV2_dist.py', '--local_rank=3']' returned non-zero exit status 1.
解决方案：
设置broadcast_buffers。原因不明
nn.parallel.DistributedDataParallel(model,device_ids=[device_id],output_device=device_id,\
broadcast_buffers=False,find_unused_parameters=True)