WIP: Workaround issues with cuFFT in W-Tiling (!131) · Merge requests · ResearchAndDevelopment / idg

Bram Veenboer requested to merge cuda-wtiling into master Jul 29, 2021
Trying to fix the following crash:
WSClean version 2.10.2 (2020-07-20)
This software package is released under the GPL version 3.
Author: André Offringa (offringa@gmail.com).

=== IMAGING TABLE ===
# Pol Ch JG ²G FG FI In Freq(MHz)
| Independent group:
+-+-J- 0 I 0 0 0 0 0 0 120-128 (666)
+-J- 1 I 1 0 1 1 0 0 128-136 (667)
+-J- 2 I 2 0 2 2 0 0 136-145 (667)
+-J- 3 I 3 0 3 3 0 0 145-153 (666)
+-J- 4 I 4 0 4 4 0 0 153-161 (667)
+-J- 5 I 5 0 5 5 0 0 161-169 (667)
Reordering...
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_127MHz.ms.sub.shift.avg.apply_infield.avg.ms [3]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_121MHz.ms.sub.shift.avg.apply_infield.avg.ms [0]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_123MHz.ms.sub.shift.avg.apply_infield.avg.ms [1]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_125MHz.ms.sub.shift.avg.apply_infield.avg.ms [2]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_130MHz.ms.sub.shift.avg.apply_infield.avg.ms [5]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_134MHz.ms.sub.shift.avg.apply_infield.avg.ms [7]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_129MHz.ms.sub.shift.avg.apply_infield.avg.ms [4]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_132MHz.ms.sub.shift.avg.apply_infield.avg.ms [6]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_138MHz.ms.sub.shift.avg.apply_infield.avg.ms [9]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_140MHz.ms.sub.shift.avg.apply_infield.avg.ms [10]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_136MHz.ms.sub.shift.avg.apply_infield.avg.ms [8]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_142MHz.ms.sub.shift.avg.apply_infield.avg.ms [11]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_146MHz.ms.sub.shift.avg.apply_infield.avg.ms [13]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_148MHz.ms.sub.shift.avg.apply_infield.avg.ms [14]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_150MHz.ms.sub.shift.avg.apply_infield.avg.ms [15]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_144MHz.ms.sub.shift.avg.apply_infield.avg.ms [12]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_152MHz.ms.sub.shift.avg.apply_infield.avg.ms [16]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_154MHz.ms.sub.shift.avg.apply_infield.avg.ms [17]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_156MHz.ms.sub.shift.avg.apply_infield.avg.ms [18]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_158MHz.ms.sub.shift.avg.apply_infield.avg.ms [19]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_160MHz.ms.sub.shift.avg.apply_infield.avg.ms [20]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_164MHz.ms.sub.shift.avg.apply_infield.avg.ms [22]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_162MHz.ms.sub.shift.avg.apply_infield.avg.ms [21]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_166MHz.ms.sub.shift.avg.apply_infield.avg.ms [23]
Finished reordering ./sub6asec_L686962_SB001_uv_12CFFDDA8t_168MHz.ms.sub.shift.avg.apply_infield.avg.ms [24]
Precalculating MF weights for Briggs'(-1.5) weighting...
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_121MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_123MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_125MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_127MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_129MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 1 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_129MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_130MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_132MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_134MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_136MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 1 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_136MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_138MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_140MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_142MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_144MHz.ms.sub.shift.avg.apply_infield.avg.ms
=Opening reordered part 1 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_144MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_146MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_148MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_150MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_152MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 1 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_152MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_154MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_156MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_158MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_160MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 1 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_160MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_162MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_164MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_166MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_168MHz.ms.sub.shift.avg.apply_infield.avg.ms
== Constructing PSF ==
IDG version 0.8:master:03b30672
Devices:
Quadro RTX 5000
Device memory : 15991 Mb / 16125 Mb (free / total)
Shared memory : 48.00 Kb
Clk frequency : 1815 Ghz
Mem frequency : 7001 Ghz
Number of SM : 48
Mem bus width : 256 bit
Mem bandwidth : 448 GB/s
Number of threads : 1024
Capability : 75
Unified memory : 1


Compiler flags:
-use_fast_math -lineinfo -src-in-ptx -arch=sm_75 -DNR_POLARIZATIONS=4 -I/opt/lofar/idg/include

Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_121MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_123MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_125MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_127MHz.ms.sub.shift.avg.apply_infield.avg.ms
Opening reordered part 0 spw 0 for ./sub6asec_L686962_SB001_uv_12CFFDDA8t_129MHz.ms.sub.shift.avg.apply_infield.avg.ms
Determining min and max w & theoretical beam size... DONE (w=[2.76188e-05:240992] lambdas, maxuvw=409387 lambda)
Determining min and max w & theoretical beam size... DONE (w=[2.80675e-05:242215] lambdas, maxuvw=415932 lambda)
Determining min and max w & theoretical beam size... DONE (w=[2.85162e-05:243430] lambdas, maxuvw=422477 lambda)
Determining min and max w & theoretical beam size... DONE (w=[2.8965e-05:246386] lambdas, maxuvw=429023 lambda)
Determining min and max w & theoretical beam size... DONE (w=[2.94137e-05:246997] lambdas, maxuvw=429554 lambda)
Theoretic beam = 480.18 masec
Detected 503.3 GB of system memory, usage not limited.
CUFFT Error at /opt/lofar/idg/src/idg-lib/src/CUDA/common/CUFFT.cpp:113 in function cufftPlanMany(&plan, 2, n, n, stride, dist, n, stride, dist, CUFFT_C2C, count): internal error
terminate called after throwing an instance of 'cufft::Error'
what(): internal error
./image.sh: line 6: 17475 Aborted wsclean -no-update-model-required -minuv-l 80.0 -size 23000 23000 -reorder -weight briggs -1.5 -parallel-reordering 4 -mgain 0.8 -data-column DATA -join-channels -channels-out 6 -auto-mask 3 -auto-threshold 1.0 -fit-spectral-pol 3 -pol i -name image_en1_yesfield_1asec -scale 0.4arcsec -minuv-l 80 -taper-gaussian 1.2asec -niter 1000000 -parallel-deconvolution 2048 -use-idg -idg-mode hybrid -aterm-kernel-size 32 -grid-with-beam -multiscale -temp-dir $PWD ./sub6asec*.ms
WIP: Workaround issues with cuFFT in W-Tiling

Merge request reports
