Offringa->RFI software->Reading mode

Reading mode

The AOFlagger used to read baselines by scanning a file for the data relevant for the currently required baseline. The mode is called the direct baseline reader. Since AOFlagger used to be limited by IO seeking and not by cpu performance, a new approach was implemented in which a measurement set is written to a temporary location in a different order: the indirect baseline reader. The increase in speed on large sets is on the order of several factors, typically around 3 or 4 times.

Because this approach uses a very large amount of disk space, it is not the default. In most cases however, people will run it on the lce nodes and there should be enough disk space on them to use this approach. One can then run rficonsole using the indirect baseline reader with the "-indirect-read" parameter, and the measurement set will be rewritten. The syntax will be similar to:

offringa@lce032:/data/scratch/offringa/temp$ rficonsole -indirect-read -j 8 SB4.MS

Please note that the current working directory will be used as a temporary storage location! Thus by running rficonsole like above, temporary files will be created in /data/scratch/offringa/temp that will take up the amount of space equal to the size of the sub-band (i.e. measurement set). So, for LOFAR, do not run this in your home directory but always on the local hd's of the nodes.

The output of rficonsole near the beginning will now be:

[..]
Initializing ms cache data...
Unique time stamps: 379
Bands: 1
Number of parts: 1
0% : +-+-+-Strategy...
0% : +-+-+-+-strategy...
0% : +-+-+-+-+-For each baseline...
Estimate of memory each thread will use: 11 MB.
0% : +-+-+-+-+-+-Initializing...
Requesting 9 x 105 x 1109 x 4 x 256 bytes of data
RW

...and will keep reporting R's and W's as long as it is rewriting the set. Each R and W means 1 GB of data has been respectively read or written. Rewriting 40 GB MS's takes in the order of 10 to 20 minutes.

In the case the AOFlagger might crash, please be sure to remove the temporary files (and send me a bug report if it is a bug ;-) ).