LINC issueshttps://git.astron.nl/RD/LINC/-/issues2024-03-27T14:26:55Zhttps://git.astron.nl/RD/LINC/-/issues/62LINC target crashes during save_logfiles2024-03-27T14:26:55ZRoland TimmermanLINC target crashes during save_logfilesI'm running into the issue that my LINC target seems to crash during the save_logfiles step:
```
�[1;30mINFO�[0m [workflow ] starting step save_logfiles
�[1;30mINFO�[0m [step save_logfiles] start
�[1;30mINFO�[0m [workflow ] starting ste...I'm running into the issue that my LINC target seems to crash during the save_logfiles step:
```
�[1;30mINFO�[0m [workflow ] starting step save_logfiles
�[1;30mINFO�[0m [step save_logfiles] start
�[1;30mINFO�[0m [workflow ] starting step save_results
�[1;30mINFO�[0m [step save_results] start
�[1;30mINFO�[0m [workflow ] starting step save_inspection
�[1;30mINFO�[0m [step save_inspection] start
�[1;30mINFO�[0m [job save_results] /.../tmp.FctKGFGVmu/tmpdir_LINC_target/00k9bj6v$ bash \
collect_files.sh
results
cp: missing destination file operand after 'results'
Try 'cp --help' for more information.
�[1;30mWARNING�[0m �[33m[job save_results] exited with status: 1�[0m
�[1;30mWARNING�[0m �[33m[job save_results] completed permanentFail�[0m
```
(Data path abbreviated for ease of read)
Full log attached to this issue. I just pulled again LINC two days ago, so I should be on the most recent version.
Please let me know if you need any other information!
Cheers,
Roland
[job_output_linc-target.txt](/uploads/4c43c01e3dc3f7b7fb0bd0418ce6a558/job_output_linc-target.txt)alexalexhttps://git.astron.nl/RD/LINC/-/issues/59HBA self-calibration stability2024-03-07T15:33:13ZFrits SweijenHBA self-calibration stability@shimwell is processing some fields now and the self-calibration procedure is not entirely stable yet (on galactic fields in this case). I'm opening this ticket to keep track of what we try or add to refine it.@shimwell is processing some fields now and the self-calibration procedure is not entirely stable yet (on galactic fields in this case). I'm opening this ticket to keep track of what we try or add to refine it.Frits SweijenFrits Sweijenhttps://git.astron.nl/RD/LINC/-/issues/57DP3 array shape error2024-03-20T11:07:48ZTimothy ShimwellDP3 array shape errorHi,
For a number of LINC target runs I am seeing the error:
dp3_execute_151/DP3_err.log
::::::::::::::
std exception detected: ArrayBase::operator()(b,e,i) - incorrectly specified
begin: [0, 0]
end: [72, 72]
incr: [1, 1]
array sha...Hi,
For a number of LINC target runs I am seeing the error:
dp3_execute_151/DP3_err.log
::::::::::::::
std exception detected: ArrayBase::operator()(b,e,i) - incorrectly specified
begin: [0, 0]
end: [72, 72]
incr: [1, 1]
array shape: [62, 62]
required: b >= 0; b <= e; e < shape; i >= 0
I gather from talking to @alex that is related to demix and more specifically the demix_timeres parameter. Not sure if the issue should be mitigated in LINC or fixed in DP3?
Thanksalexalexhttps://git.astron.nl/RD/LINC/-/issues/55Prevent loading all MODEL_DATA into RAM2024-02-12T11:23:43ZReinout van WeerenPrevent loading all MODEL_DATA into RAMhttps://git.astron.nl/RD/LINC/-/blob/master/scripts/Ateamclipper.py?ref_type=heads#L19
In general, for large (RAW) MS this can result in running out of the RAM, better and safer would be to loop over rows, something like
stepsize = 100...https://git.astron.nl/RD/LINC/-/blob/master/scripts/Ateamclipper.py?ref_type=heads#L19
In general, for large (RAW) MS this can result in running out of the RAM, better and safer would be to loop over rows, something like
stepsize = 100000 # update depending on how many rows you want to load into at once, now it's very small
for row in range(0,t.nrows(),stepsize):
#print(f"Doing {row} out of {t.nrows()}, (step: {stepsize})")
data = t.getcol(datacolumn,startrow=row,nrow=stepsize,rowincr=1)
# and now do what you want .....alexalexhttps://git.astron.nl/RD/LINC/-/issues/51Demix distance criteria2023-12-12T09:33:52ZTimothy ShimwellDemix distance criteriaPresently A team (CygA, CasA, TauA, VirA) are by default demixed if within 30 deg.
Maybe the criteria would be better to use the script from @yatawatta (https://git.astron.nl/RD/LINC/-/blob/master/scripts/tune_demixing_parameters.py) t...Presently A team (CygA, CasA, TauA, VirA) are by default demixed if within 30 deg.
Maybe the criteria would be better to use the script from @yatawatta (https://git.astron.nl/RD/LINC/-/blob/master/scripts/tune_demixing_parameters.py) to determine whether or not the field needs demixing - this would hopefully result in less than 30 deg for VirA for example (ticket #42). Sarod suggested using a cut off value of 0.5 so that might be a sensible default but would also be good if users could adjust this in the json file.alexalexhttps://git.astron.nl/RD/LINC/-/issues/42Demix of Virgo2024-01-18T21:48:45ZTimothy ShimwellDemix of VirgoHi,
I've been rerunning fields that were previously processed with PreFactor3 without demix but just using ateam-clipping. For the reprocessing im using LINC and the standard demix settings.
Unfortunately for all fields that ive reproc...Hi,
I've been rerunning fields that were previously processed with PreFactor3 without demix but just using ateam-clipping. For the reprocessing im using LINC and the standard demix settings.
Unfortunately for all fields that ive reprocessed around Virgo the data have got a fair bit worse. Its only 3 fields but thought to raise an issue as perhaps the default behaviour around this source could change. The fields I demixed were separated by 17.5, 10.5 and 12.3 deg away from Virgo.
Anyway, I wonder if its the sky model for Virgo not being much good or something else. Its 1250Jy at 144MHz so id have thought that demix should be working well.
Attached is an example of what i mean by "worse". Here the colour scale is the same on the two panels. The panel on the left is prefactor3 with ateam clipping and the panel on the right is default LINC (so demixing).
Thanks,
Tim
![Screenshot_2023-08-28_at_11.51.57](/uploads/ddcd71434696803e91c5da6abc0eea59/Screenshot_2023-08-28_at_11.51.57.png)alexalexhttps://git.astron.nl/RD/LINC/-/issues/34Target field skymodels2023-06-26T15:36:05ZTimothy ShimwellTarget field skymodelsHey,
More a feature than an issue. I was wondering if it possible to add LoTSS as an option into the target field sky model. LoTSS-DR2 catalogues can be retrieved through the Astron vo (https://vo.astron.nl/lotss_dr2/q/src_cone/form). ...Hey,
More a feature than an issue. I was wondering if it possible to add LoTSS as an option into the target field sky model. LoTSS-DR2 catalogues can be retrieved through the Astron vo (https://vo.astron.nl/lotss_dr2/q/src_cone/form). Whilst they only cover a quarter of the sky it still might be useful for target fields in that area.
The catalogues are pybsdf catalogues but in .fits format so I guess they'd also need converting to BBS format unless LINC could also accept .fits format pybdsf catalogues.alexalexhttps://git.astron.nl/RD/LINC/-/issues/24automatically recognising 160MHz clock obs?2022-12-15T14:26:26ZTimothy Shimwellautomatically recognising 160MHz clock obs?Presently for 170-230MHz obs with the 160MHz clock setup linc crashes with default settings as its tries to average the data to the wrong freq resolution. Instead to the json the user can add:
"avg_freqresolution": "39.00kHz",
"demix_fr...Presently for 170-230MHz obs with the 160MHz clock setup linc crashes with default settings as its tries to average the data to the wrong freq resolution. Instead to the json the user can add:
"avg_freqresolution": "39.00kHz",
"demix_freqresolution":"39.00kHz",
Just wondering though if its something that might be a simple just to automatically check which channel width is appropriate (e.g 160MHz or 200MHz clock) when determining the averaging?
Thanksalexalexhttps://git.astron.nl/RD/LINC/-/issues/20High time-free resolution2022-10-19T08:50:18ZHarish VedanthamHigh time-free resolutionI want to retain the high time-frequency resolution for some data. I tried to enforce that in LINC using "avg_timeresolution":1, "avg_freqresolution": "3.0517578125kHz". However, that did not work as the resulting concatenated MS wer...I want to retain the high time-frequency resolution for some data. I tried to enforce that in LINC using "avg_timeresolution":1, "avg_freqresolution": "3.0517578125kHz". However, that did not work as the resulting concatenated MS were all averaged down.
I then tried in addition to the above, "avg_timeresolution_concat": 1,"avg_freqresolution_concat": "3.0517578125kHz", and this worked.
So even though I have a working solution I thought that the run-time can be improved by averaging for the sake of calibration but then applying the solutions to the raw data without averaging. Is there a way I can implement this in the input Jason files?alexalexhttps://git.astron.nl/RD/LINC/-/issues/2Script to generate MS lists2022-11-21T10:51:18ZAlex KurekScript to generate MS listsI was unable to fork the project, so I upload it here - please consider adding to https://git.astron.nl/eosc/prefactor3-cwl/-/tree/master/test_jobs
It will generate
```json
{
"msin": [
{
"class": "Directory",
...I was unable to fork the project, so I upload it here - please consider adding to https://git.astron.nl/eosc/prefactor3-cwl/-/tree/master/test_jobs
It will generate
```json
{
"msin": [
{
"class": "Directory",
"path": "../data/L667520_SB000_uv.MS"
},
```
[...]
list that can be pasted to .json files.
[msInListPrep.zip](/uploads/8c586c607770384a63b4c12f3fa1b9dc/msInListPrep.zip)