Enthusiastic flagging?

Hi,

I was wondering if LINC is possibly flagging excessive amounts of data. We've processed a few hundred observations between 2018 and 2023 with LINC and find the following flagging fractions for the LOFAR HBA calibrator and target runs.

Calibrator (red initial and yellow final):

average_flagging_cal_LINC

Target (red intial; green prep; blue concat; yellow final)

average_flagging_target_LINC

Previously for e.g. LoTSS-DR2 which consisted of data between 2014 and 2020 we saw typically around 9% of the data flagged (see blue line in the following plot which is the average (dark) and individual pointing (semi transparent) flagging fraction as a function of freq).

LoTSS-DR2-pref1-flag

It could also be that the amount of RFI or bad data impacting the observations has severely increased in recently. Or that previously we didnt flag enough data. However just thought to mention it because its a lot of data.

Not sure if to help investigate the json file that records the flagging fractions could be a little finer (presently e.g. for calibrator it is just initial and final). This might help see if some particular steps are responsible.

Thanks,

Tim

Edited by Timothy Shimwell