Segfault in DDECal gain solver
DP3
segfaults when trying to perform DDE calibration. To reproduce on DAS-6, you can use the following command:
DP3 msin.datacolumn=DATA msout= steps=[solve] solve.type=ddecal solve.mode=scalarphase solve.usebeammodel=True solve.beammode=array_factor solve.antennaconstraint=[] solve.h5parm=fast_phase_0.h5parm solve.llssolver=qr solve.maxiter=150 msin=/var/scratch/drafferty/run_elais_1/pipelines/predict_1/midbands.ms.mjd4941453483_field solve.nchan=10 msin.ntimes=75 numthreads=40 solve.onebeamperpatch=False solve.parallelbaselines=False solve.propagatesolutions=True solve.sagecalpredict=False solve.smoothnessconstraint=3000000.0 solve.smoothnessrefdistance=0.0 solve.smoothnessreffrequency=144000000.0 solve.solint=1 solve.solveralgorithm=hybrid solve.solverlbfgs.dof=200.0 solve.solverlbfgs.iter=4 solve.solverlbfgs.minibatches=1 solve.sourcedb=/var/scratch/drafferty/run_elais_1/skymodels/calibrate_2/calibration_skymodel.txt msin.starttime=19Jun2015/17:58:03.906 solve.stepsize=0.02 solve.tolerance=0.005 solve.uvlambdamin=150
And here is the backtrace:
Thread 1 "DP3" received signal SIGSEGV, Segmentation fault.
xsimd::kernel::detail::load_complex<xsimd::fma3<xsimd::avx2> > (lo=..., hi=...) at /var/scratch/loose/Software/rapthor/include/xsimd/types/../arch/./xsimd_avx.hpp:864
864 tmp1 = _mm256_extractf128_pd(lo, 1);
(gdb) bt
#0 xsimd::kernel::detail::load_complex<xsimd::fma3<xsimd::avx2> > (lo=..., hi=...) at /var/scratch/loose/Software/rapthor/include/xsimd/types/../arch/./xsimd_avx.hpp:864
#1 xsimd::kernel::load_complex_aligned<xsimd::fma3<xsimd::avx2>, double, double> (mem=<optimized out>)
at /var/scratch/loose/Software/rapthor/include/xsimd/types/../arch/././generic/xsimd_generic_memory.hpp:477
#2 xsimd::batch<std::complex<double>, xsimd::fma3<xsimd::avx2> >::load_aligned (src=<optimized out>) at /var/scratch/loose/Software/rapthor/include/xsimd/types/xsimd_batch.hpp:1171
#3 dp3::ddecal::SolverBase::Step (this=this@entry=0x964580, solutions=..., next_solutions=...) at /tmp/loose/rapthor-build/dp3/ddecal/gain_solvers/SolverBase.cc:135
#4 0x00007ffff7872954 in dp3::ddecal::ScalarSolver::Solve (this=<optimized out>, data=..., solutions=..., time=4941453484.0055609, stat_stream=0x0)
at /tmp/loose/rapthor-build/dp3/ddecal/gain_solvers/ScalarSolver.cc:78
#5 0x00007ffff7866d5f in dp3::ddecal::HybridSolver::RunSolver (this=<optimized out>, solver=..., available_iterations=@0x7fffffff6078: 150, result=..., solve_data=..., solutions=...,
time=4941453484.0055609, stat_stream=0x0)
at /var/software/spack/opt/spack/linux-rocky8-zen2/gcc-9.4.0/gcc-9.4.0-454bc6ws37ruxouizhw7fovuysktzozi/lib/gcc/x86_64-pc-linux-gnu/9.4.0/../../../../include/c++/9.4.0/bits/stl_construct.h:107
#6 0x00007ffff7866e3c in dp3::ddecal::HybridSolver::Solve (this=0xa234e0, solve_data=..., solutions=..., time=4941453484.0055609, stat_stream=0x0)
at /tmp/loose/rapthor-build/dp3/ddecal/gain_solvers/HybridSolver.cc:34
#7 0x00007ffff781935b in dp3::steps::DDECal::doSolve (this=0x9e5b80)
at /var/software/spack/opt/spack/linux-rocky8-zen2/gcc-9.4.0/gcc-9.4.0-454bc6ws37ruxouizhw7fovuysktzozi/lib/gcc/x86_64-pc-linux-gnu/9.4.0/../../../../include/c++/9.4.0/bits/stl_vector.h:1040
#8 0x00007ffff781ecd9 in dp3::steps::DDECal::process (this=<optimized out>, bufin=...) at /tmp/loose/rapthor-build/dp3/steps/DDECal.cc:717
#9 0x00007ffff79c0845 in dp3::steps::MSReader::process (this=this@entry=0x925d40, buffer=...)
at /var/software/spack/opt/spack/linux-rocky8-zen2/gcc-9.4.0/gcc-9.4.0-454bc6ws37ruxouizhw7fovuysktzozi/lib/gcc/x86_64-pc-linux-gnu/9.4.0/../../../../include/c++/9.4.0/bits/move.h:74
#10 0x00007ffff78c9c25 in dp3::base::Execute (parsetName=..., argc=argc@entry=33, argv=argv@entry=0x7fffffff8258)
at /var/software/spack/opt/spack/linux-rocky8-zen2/gcc-9.4.0/gcc-9.4.0-454bc6ws37ruxouizhw7fovuysktzozi/lib/gcc/x86_64-pc-linux-gnu/9.4.0/../../../../include/c++/9.4.0/bits/unique_ptr.h:153
#11 0x000000000040275b in main (argc=33, argv=0x7fffffff8258) at /tmp/loose/rapthor-build/dp3/base/Main.cc:74
I think the offending commit is:
commit 7cafdcb1b5db013a0e009a7be3c7032611624f87
Author: Maik Nijhuis <maik.nijhuis@triopsys.nl>
Date: Mon Oct 16 06:58:40 2023 +0000
AST-1410 Use XSimd in SolverBase::Step