Frequently asked questions

Contents:

• Where should I start with reading documentation about the tools?
• Does rficonsole work on WSRT / VLA / GMRT / ... data?
• I get bad results, how to solve?
• I suspect a presence of RFI in my final reduced data
• How does rficonsole handle multiple bands in the same observation?
• How does rficonsole handle multiple fields in the same observation?
• RFI console uses too much memory!
• RFI console does not seem to use CPU / does not progress
• Crashes with a 'FLAG is readonly' message
• Plotting does not work in the GUI
• How do I use this new, faster reading mode?
• Where is the newest version of the RFI tools?
• How can I transfer old strategies to newer versions of the tool?
• Does rficonsole store e.g. the statistics in a log?
• How does RFI console recognise RFI?
• I still have questions / contact info

Where should I start with reading documentation about the tools?

If you are reducing LOFAR data, you should at least read the LOFAR imaging cookbook. The NDPPP chapter contains also information about the "quality" tools that are included in the AOFlagger package. A first tutorial on the GUI can be found here: RFI gui tutorial, and an example to optimize the default strategy here: Strategy optimization example. More documentation might follow on this website, and of course scanning through this FAQ might solve some of your questions. See the "How does RFI console recognise RFI" faq item for some links to scientific papers about the flagger.

Does rficonsole work on WSRT / VLA / GMRT / ... data?

Yes, it does. As far as I know, it has already been tested and shown good results on LOFAR, WSRT, VLA and GMRT data. The only requirement is that the data is stored in the Aips++ / casa Measurement Set format (in my personal opinion, this is one of the best formats around for radio observations). If you have results for other telescopes, I'd love to hear about it.

Note though, that the settings have been optimized for LOFAR, and some of the settings might need to be changed to get good results. See the "I get bad results, how te solve?" question for more info.

I get bad results, how to solve?

Strong sources and different telescope properties can cause the default settings to fail in certain cases. The gui can help you a lot when trying to solve problems. Be sure to read the strategy optimization example. If you think rficonsole/rfigui is doing a bad job, I would suggest to experiment with the GUI to optimize the following settings:

• Horizontal resolution decrease factor before the sliding window fit action
  LOFAR has a high time resolution (currently 1 sec by default). If your telescope / observation has a longer integration time or a very strong source far from the phase centre, this should be set to 1 instead of 3).
• Kernel width (=kernel time size) of the sliding window fitting action
  Once again, this depends on integration time and source field.
• Kernel height (=kernel frequency size) of the sliding window fitting action
  This depends on the channel resolution and frequency behaviour. Should be fine in most cases, but if your band pass is very steep (such as with WSRT), you might need to make it smaller. Another (actually better) solution is to correct for the band pass before running rficonsole.
• Change the sensitivity of the SumThreshold action.
• Enlarge the number of iterations (slower, but more stable)

I have never seen an observation in which optimizing the settings did not work in the end, so don't give up. Make sure the phase centre does not change during the observation (see the question on multiple fields). You can always mail me and hand over a small part of your observation to let me take a look at the problem.

I suspect a presence of RFI in my final reduced data

That is well possible, even after using rficonsole. RFI console will only flag RFI that is (1) detectable and (2) not both broadband and continuous. Especially broadband in situ RFI (RFI generated by hardware near the telescope) can not be removed by flagging and might cause calibration failures and/or sidelobes while imaging. Other techniques should be used in such cases.

How does rficonsole handle multiple bands in the same observation?

It will treat them as separate baselines, which will be fine. If there is some problem, the best way is to split them into separate measurement sets. LOFAR never stores multiple bands in one measurement set by default, but WSRT does.

How does rficonsole handle multiple fields in the same observation?

I have not tested this, but it will probably treat the entire observation as a whole, and look at the concatenated baselines (i.e., time sort the data and assume the data should look the same). Therefore, if you sweep between a strong calibrator source and another field, rficonsole might flag your calibrator completely! Therefore, always split the measurement set when the phase centre changes. LOFAR will never store multiple sources in one measurement set, so for LOFAR this is not an issue.

RFI console uses too much memory!

Here are some hints on memory usage of rficonsole:

• First of all, the memory reported by top and similar is not the actual memory usage. It includes a so called "memory map", which is not actually used memory, but is hard disk space that is addressable as if it was memory. In most cases, this is the largest chunk reported by top. Therefore, as long as you don't get memory crashes or a lot of swapping, you should not worry about this huge number reported by top.
• The casacore library uses internally a caching mechanism that should speed up reading of the data. I am not sure how the data is cached, but I noticed that the cache will slowly grow until all physical available memory is occupied. This looks like a memory leak, but memory usage should stop growing as soon as swapping is about to occur.
• The minimal required memory is currently depending on three things: (1) the reading mode (direct or indirect) ; (2) the number of threads being used (defaults to 4) ; and (3) the size of a single baseline. The system will approximately use <THREAD COUNT> x 4 x <BASELINE SIZE> bytes of memory in direct mode and <THREAD COUNT> x 2.5 x <BASELINE SIZE> bytes in indirect mode. The size of a single baseline can be calculated like 8 (bytes/complex float) x 4 (polarizations) x 256 (channels) x 6000 (time steps) (thus 50 MB per baseline in this case, 0.5 GB in total, a rather typical LOFAR example). Hence, if the system really runs out of memory / starts swapping, decrease the number of threads as explained in the command line help of rficonsole and make sure to use the indirect reader.
• The software has been carefully tested for memory leaks, and there should not be any serious ones in it. A typical desktop machine should be able to run the software without problems.
• Don't run multiple rficonsole's on the same machine. Run them sequential, not in parallel.

RFI console does not seem to use CPU / does not progress

This probably means that reading the data takes a lot of time and the computing threads are awaiting new data. Reasons for slow IO can be:

• You are trying to run multiple rficonsole's on the same machine. Run them sequential, not in parallel.
• The machine is low on memory, causing the cache to be too small or swapping. See the question about memory usage to improve memory.
• Too many users on the storage node (/hard disk users).

Generally, the indirect reading mode is much faster than the direct reading mode.

Crashes with a 'FLAG is readonly' message

The full message is:

terminate called after throwing an instance of 'casa::TableInvOper'
what(): Invalid Table operation: FLAG is readonly (use the ROxxxColumn class)
Abort

This means you are trying to flag raw LOFAR data in which the LofarStMan storage manager is used for the flag column. This storage manager is read only, and rficonsole can not update these flags. The LOFAR imaging cookbook will tell you on changes, but the current way to solve this on the LOFAR cluster is:

makeFLAGwritable <MyObservation.MS>

This will take a few seconds to rewrite the flag column, after which you can run rficonsole.

Plotting does not work in the GUI

[update 2012-03: the GUI now uses an internal plotter, and this answer is probably no longer relevant]. Plotting requires several programs:

• Gnuplot, version 4.2 or higher
• ps2pdf, which is part of the Ghostscript software
• A pdf viewer. RFI gui will try to execute the command "./kpdf". Therefore, you will need a symlink in your working directory to this pdf viewer, and the symlink should be called "kpdf".

If plotting is not working, check the terminal; RFI gui will output errors there.

How do I use this new, faster reading mode?

The mode is called the indirect reading mode, and is described on the following page: reading mode.

What is the newest version of the RFI tools?

The latest version is the one installed on the LOFAR cluster. This version is build daily from the Astron Subversion repository and should be the preferred version at all times, if you have access to it. If not, you can use the packaged version from http://www.astro.rug.nl/rfi-software. If the version there is older than two months, mail me so that I package the latest version -- it is in heavy development and many bugs and fixes are made within a month.

How can I transfer old strategies to newer versions of the tool?

Newer versions can not always read strategies which are created by older tools. Sometimes, the changes are small and can be easily solved, but sometimes it is better to recreate the strategies. You can try changing the version of the strategy file with a text editor and load it with rficonsole (rficonsole -strategy old_strategy.rfis). It will probably reply with an exception about which action is causing the incompatibility. By comparing that part of the strategy with a newer strategy, you might be able to solve the problem.

Does rficonsole store e.g. the statistics in a log?

Since October 2010, RFI console uses the LOFAR logger, which in turn uses LogCplus. Standard logging configuration can be set up in a configuration file. The parameters -nolog and -v can be used to alter logging strategy. The standard unix program tee can be used to save the output to file while at the same time monitoring the standard output.

How does RFI console recognise RFI?

It estimates the astronomical signal and searches for statistical abnormalities that look like lines in the time/frequency plots. After initial detection, it applies a morphological algorithm to find likely contaminated samples. It has several tricks to enhance the speed. The following refereed papers describe parts of the detection technique in detail:

• Post-correlation radio frequency interference classification methods, MNRAS, Vol. 405, 1, June 2010, pp. 155-167
• A LOFAR RFI detection pipeline and its first results, PoS, RFI2010, March 2010.
• A morphological algorithm for improving radio-frequency interference detection, A&A, Vol. 539, A95, March 2012.

I still have questions / contact info

I love to have feedback in order to update this FAQ and get a grib on common problems. You can reach me on my e-mail address, <my-last-name>@astro.rug.nl, and my last name is offringa.