Crash using TECConstraint in DDECal

Created by: tammojan

As reported by @revoltek. Using the latest master, the following parset causes a crash. Reproduced on a clean ubuntu 16.04 docker image with minimal requirements installed.

msin = forTJ1sec.MS
msin.datacolumn = DATA
msout = .

numthreads=1
steps=[sol]

sol.type = ddecal
sol.h5parm = test.h5
sol.maxiter = 300
sol.tolerance = 5e-4
sol.solint = 1
sol.nchan = 16
sol.stepsize = 0.2

sol.mode = tec # phaseonly, complexgain, tec, tecandphase
sol.propagatesolutions = True
sol.approximatetec = True
sol.maxapproxiter = 250
sol.approxtolerance = 1e-3
sol.usemodelcolumn = True

Below is the output of valgrind.

root@b2e81ea7da64:/data/fra# valgrind DPPP
==39361== Memcheck, a memory error detector
==39361== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==39361== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==39361== Command: DPPP
==39361==
MSReader
  input MS:       /data/fra/forTJ1sec.MS
  band            0
  startchan:      0  (0)
  nchan:          976  (0)
  ncorrelations:  4
  nbaselines:     666
  first time:     2015/10/25/23:00:04
  last time:      2015/10/25/23:00:04
  ntimes:         1
  time interval:  4.00556
  DATA column:    DATA
  WEIGHT column:  WEIGHT_SPECTRUM
  autoweight:     false
DDECal sol.
  H5Parm:              test.h5
  solint:              1
  nchan:               16
  directions:          [[]]
  use model column:    true
  tolerance:           0.0005
  max iter:            300
  flag unconverged:    false
     diverged only:    false
  propagate solutions: true
       converged only: false
  detect stalling:     true
  step size:           0.2
  mode (constraints):  tec
  approximate fitter:  true
  only predict:        false
  subtract model:      false
MSUpdater msout.
  MS:             /data/fra/forTJ1sec.MS
  datacolumn:     DATA
  weightcolumn    WEIGHT_SPECTRUM
  Compressed:     no

  flush:          0

Processing 1 time slots ...

0%==39361== Invalid read of size 8
==39361==    at 0x699E6B: operator/=<double> (complex:1334)
==39361==    by 0x699E6B: operator/<double> (complex:419)
==39361==    by 0x699E6B: TECConstraintBase::applyReferenceAntenna(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&) const (TECConstraint.cc:86)
==39361==    by 0x69CC4A: ApproximateTECConstraint::Apply(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (TECConstraint.cc:179)
==39361==    by 0x6922A0: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:304)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==  Address 0x12f62918 is 8 bytes after a block of size 592 alloc'd
==39361==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39361==    by 0x5453CE: allocate (new_allocator.h:104)
==39361==    by 0x5453CE: allocate (alloc_traits.h:491)
==39361==    by 0x5453CE: _M_allocate (stl_vector.h:170)
==39361==    by 0x5453CE: std::vector<std::complex<double>, std::allocator<std::complex<double> > >::_M_default_append(unsigned long) (vector.tcc:557)
==39361==    by 0x69282E: resize (stl_vector.h:676)
==39361==    by 0x69282E: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:245)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==
==39361== Invalid read of size 8
==39361==    at 0x699E70: operator/=<double> (complex:1334)
==39361==    by 0x699E70: operator/<double> (complex:419)
==39361==    by 0x699E70: TECConstraintBase::applyReferenceAntenna(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&) const (TECConstraint.cc:86)
==39361==    by 0x69CC4A: ApproximateTECConstraint::Apply(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (TECConstraint.cc:179)
==39361==    by 0x6922A0: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:304)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==  Address 0x12f62910 is 0 bytes after a block of size 592 alloc'd
==39361==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39361==    by 0x5453CE: allocate (new_allocator.h:104)
==39361==    by 0x5453CE: allocate (alloc_traits.h:491)
==39361==    by 0x5453CE: _M_allocate (stl_vector.h:170)
==39361==    by 0x5453CE: std::vector<std::complex<double>, std::allocator<std::complex<double> > >::_M_default_append(unsigned long) (vector.tcc:557)
==39361==    by 0x69282E: resize (stl_vector.h:676)
==39361==    by 0x69282E: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:245)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==
==39361== Invalid write of size 8
==39361==    at 0x699F33: operator= (complex:1254)
==39361==    by 0x699F33: TECConstraintBase::applyReferenceAntenna(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&) const (TECConstraint.cc:91)
==39361==    by 0x69CC4A: ApproximateTECConstraint::Apply(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (TECConstraint.cc:179)
==39361==    by 0x6922A0: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:304)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==  Address 0x12f62910 is 0 bytes after a block of size 592 alloc'd
==39361==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39361==    by 0x5453CE: allocate (new_allocator.h:104)
==39361==    by 0x5453CE: allocate (alloc_traits.h:491)
==39361==    by 0x5453CE: _M_allocate (stl_vector.h:170)
==39361==    by 0x5453CE: std::vector<std::complex<double>, std::allocator<std::complex<double> > >::_M_default_append(unsigned long) (vector.tcc:557)
==39361==    by 0x69282E: resize (stl_vector.h:676)
==39361==    by 0x69282E: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:245)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==
==39361== Invalid write of size 8
==39361==    at 0x699F37: operator= (complex:1254)
==39361==    by 0x699F37: TECConstraintBase::applyReferenceAntenna(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&) const (TECConstraint.cc:91)
==39361==    by 0x69CC4A: ApproximateTECConstraint::Apply(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (TECConstraint.cc:179)
==39361==    by 0x6922A0: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:304)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==  Address 0x12f62918 is 8 bytes after a block of size 592 alloc'd
==39361==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39361==    by 0x5453CE: allocate (new_allocator.h:104)
==39361==    by 0x5453CE: allocate (alloc_traits.h:491)
==39361==    by 0x5453CE: _M_allocate (stl_vector.h:170)
==39361==    by 0x5453CE: std::vector<std::complex<double>, std::allocator<std::complex<double> > >::_M_default_append(unsigned long) (vector.tcc:557)
==39361==    by 0x69282E: resize (stl_vector.h:676)
==39361==    by 0x69282E: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:245)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==
==39361== Invalid read of size 8
==39361==    at 0x699E6B: operator/=<double> (complex:1334)
==39361==    by 0x699E6B: operator/<double> (complex:419)
==39361==    by 0x699E6B: TECConstraintBase::applyReferenceAntenna(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&) const (TECConstraint.cc:86)
==39361==    by 0x69B48A: TECConstraint::Apply(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (TECConstraint.cc:125)
==39361==    by 0x69CC94: ApproximateTECConstraint::Apply(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (TECConstraint.cc:176)
==39361==    by 0x6922A0: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:304)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==  Address 0x12f62918 is 8 bytes after a block of size 592 alloc'd
==39361==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39361==    by 0x5453CE: allocate (new_allocator.h:104)
==39361==    by 0x5453CE: allocate (alloc_traits.h:491)
==39361==    by 0x5453CE: _M_allocate (stl_vector.h:170)
==39361==    by 0x5453CE: std::vector<std::complex<double>, std::allocator<std::complex<double> > >::_M_default_append(unsigned long) (vector.tcc:557)
==39361==    by 0x69282E: resize (stl_vector.h:676)
==39361==    by 0x69282E: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:245)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==
==39361== Invalid read of size 8
==39361==    at 0x699E70: operator/=<double> (complex:1334)
==39361==    by 0x699E70: operator/<double> (complex:419)
==39361==    by 0x699E70: TECConstraintBase::applyReferenceAntenna(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&) const (TECConstraint.cc:86)
==39361==    by 0x69B48A: TECConstraint::Apply(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (TECConstraint.cc:125)
==39361==    by 0x69CC94: ApproximateTECConstraint::Apply(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (TECConstraint.cc:176)
==39361==    by 0x6922A0: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:304)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==  Address 0x12f62910 is 0 bytes after a block of size 592 alloc'd
==39361==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39361==    by 0x5453CE: allocate (new_allocator.h:104)
==39361==    by 0x5453CE: allocate (alloc_traits.h:491)
==39361==    by 0x5453CE: _M_allocate (stl_vector.h:170)
==39361==    by 0x5453CE: std::vector<std::complex<double>, std::allocator<std::complex<double> > >::_M_default_append(unsigned long) (vector.tcc:557)
==39361==    by 0x69282E: resize (stl_vector.h:676)
==39361==    by 0x69282E: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:245)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==
==39361== Invalid write of size 8
==39361==    at 0x699F33: operator= (complex:1254)
==39361==    by 0x699F33: TECConstraintBase::applyReferenceAntenna(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&) const (TECConstraint.cc:91)
==39361==    by 0x69B48A: TECConstraint::Apply(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (TECConstraint.cc:125)
==39361==    by 0x69CC94: ApproximateTECConstraint::Apply(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (TECConstraint.cc:176)
==39361==    by 0x6922A0: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:304)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==  Address 0x12f62910 is 0 bytes after a block of size 592 alloc'd
==39361==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39361==    by 0x5453CE: allocate (new_allocator.h:104)
==39361==    by 0x5453CE: allocate (alloc_traits.h:491)
==39361==    by 0x5453CE: _M_allocate (stl_vector.h:170)
==39361==    by 0x5453CE: std::vector<std::complex<double>, std::allocator<std::complex<double> > >::_M_default_append(unsigned long) (vector.tcc:557)
==39361==    by 0x69282E: resize (stl_vector.h:676)
==39361==    by 0x69282E: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:245)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==
==39361== Invalid write of size 8
==39361==    at 0x699F37: operator= (complex:1254)
==39361==    by 0x699F37: TECConstraintBase::applyReferenceAntenna(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&) const (TECConstraint.cc:91)
==39361==    by 0x69B48A: TECConstraint::Apply(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (TECConstraint.cc:125)
==39361==    by 0x69CC94: ApproximateTECConstraint::Apply(std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (TECConstraint.cc:176)
==39361==    by 0x6922A0: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:304)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==  Address 0x12f62918 is 8 bytes after a block of size 592 alloc'd
==39361==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39361==    by 0x5453CE: allocate (new_allocator.h:104)
==39361==    by 0x5453CE: allocate (alloc_traits.h:491)
==39361==    by 0x5453CE: _M_allocate (stl_vector.h:170)
==39361==    by 0x5453CE: std::vector<std::complex<double>, std::allocator<std::complex<double> > >::_M_default_append(unsigned long) (vector.tcc:557)
==39361==    by 0x69282E: resize (stl_vector.h:676)
==39361==    by 0x69282E: MultiDirSolver::processScalar(std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > const&, std::vector<float*, std::allocator<float*> > const&, std::vector<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> >, std::allocator<std::vector<std::complex<float>*, std::allocator<std::complex<float>*> > > > const&, std::vector<std::vector<std::complex<double>, std::allocator<std::complex<double> > >, std::allocator<std::vector<std::complex<double>, std::allocator<std::complex<double> > > > >&, double, std::ostream*) (MultiDirSolver.cc:245)
==39361==    by 0x665F76: DP3::DPPP::DDECal::doSolve() (DDECal.cc:781)
==39361==    by 0x66C4EA: DP3::DPPP::DDECal::process(DP3::DPPP::DPBuffer const&) (DDECal.cc:974)
==39361==    by 0x4A5256: DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) (MSReader.cc:408)
==39361==    by 0x46584C: DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) (DPRun.cc:193)
==39361==    by 0x457815: main (DPPP.cc:86)
==39361==
....10....20....30....40....50....60....70....80....90....100%
Finishing processing ...

NaN/infinite data flagged in reader
===================================

Percentage of flagged visibilities detected per correlation:
  [89025,0,0,0] out of 650016 visibilities   [14%, 0%, 0%, 0%]
0 missing time slots were inserted




Total NDPPP time      16.84 real       16.27 user        0.96 system
   17.7% MSReader
   79.8% DDECal sol.
            2.3% of it spent in predict
           94.9% of it spent in estimating gains and computing residuals
            0.8% of it spent in writing gain solutions to disk
Iterations taken: [3|2]
    0.0% MSUpdater msout.
==39361==
==39361== HEAP SUMMARY:
==39361==     in use at exit: 143,194 bytes in 493 blocks
==39361==   total heap usage: 70,725 allocs, 70,232 frees, 320,547,088 bytes allocated
==39361==
==39361== LEAK SUMMARY:
==39361==    definitely lost: 56 bytes in 1 blocks
==39361==    indirectly lost: 48,928 bytes in 455 blocks
==39361==      possibly lost: 960 bytes in 3 blocks
==39361==    still reachable: 93,250 bytes in 34 blocks
==39361==         suppressed: 0 bytes in 0 blocks
==39361== Rerun with --leak-check=full to see details of leaked memory
==39361==
==39361== For counts of detected and suppressed errors, rerun with: -v
==39361== ERROR SUMMARY: 13908 errors from 8 contexts (suppressed: 0 from 0)