Offringa->RFI software->RFI gui tutorial

RFI GUI tutorial

Introduction

The GUI of the AOTools is aimed at analysing data of a single baseline at a time. Especially for problems related to RFI, this should be a very handy tool. The tool can do quite advanced data manipulation, but many of the advanced tricks are undocumented and/or somewhat hidden in the tool. The tool (including its documentation) is very much a work in progress.

This tutorial was written for the SVN version of 2011-01-21.

Starting the tool

Once you have installed the GUI, you should be able to start the GUI with "rfigui". If you are running the GUI remotely on a different machine, make sure you have your x server forwarded (either with ssh -X or ssh -Y depending on Mac/Linux). On the LOFAR CEP cluster, you have to import the LofIm environment with "use LofIm".

Opening a measurement set

After starting the GUI, the main window will appear (displayed on the right), from which most actions will be performed. To start working, open a measurement set. RFI gui has been written around the AIPS++/CASA measurement set format (directories with extension .MS), and works best with those formats. There is some limited support for opening UV FITS files (extension .uvfits) and raw RTCP output files (extension .raw).

To open a measurement set, select "Open directory" from the "File" menu. For opening a uv fits or raw file, select "Open file" instead. In general, RFI gui will never write changes to the measurement set; all options in the menu will be executed in memory, with the exception of custom made strategies.

After selecting your measurement set (abbrev. MS), a pop-up window will appear (displayed on the right) with settings for opening the set. In most cases, you'll want to keep the default settings.

The settings are:

After you press the "Open" button, the set will be opened and the first baseline in the set will be read. This might take some time.


Selecting a baseline

The first baseline in a MS set is often an autocorrelation. These have often been flagged entirely by the observatory, and you might therefore see a large purple window appear after opening a MS.

There are two options for going to a different baselines:

  1. Select "Go to..." from the "Go" menu. A window appears to select a baseline. The image on the right shows how the "Go to" window might look like when working with WSRT data. By default, your current baseline will be selected (versions < 2011-01-21 won't do this). Select your desired baseline and press "Load" to load the baseline. Once again, that might take time.
  2. By pressing the toolbar "Next" and "Previous" you can iterate quickly over baselines in the MS. The order in which will be iterated depends on the order they are stored in the MS (to be more exactly; the order in which the first time step has been stored in the set). Directly after loading a new set, the status bar in the bottom will show the baseline (if you have not moved with the mouse over the time frequency canvas).

The main window

Once the loading of the data is finished, you should see your baseline in the main window, along with the axes: frequency will be displayed vertically, time horizontally. To the right is the window displayed with a WSRT measurement open.

If your set has already been flagged, you should see the flags marked in fuchsia. As you can see on the right, the WSRT set has been flagged already. The "original flags" and "alternative flag" buttons turn respectively the fuchsia and the yellow flag markings on and off. When a new baseline is loaded, the yellow and fuchsia flags are the same, however, once you start executing strategies, the yellow ("alternative") flags will show the flags produced by the flagger.

The "plot" menu houses a lot of plotting operations, such as plotting the distribution, the bandpass and time vs. amplitude. These options require gnuplot, a pdf viewer and Ghostscript. If they don't work, check the FAQ page on that issue. All options always operate on the currently visible data. The plotting operations will also take in account flags, and not use them to draw the graph if appropriate, if marking of the flags is enabled (if both original and alternate flags are displayed, they will be combined).

The plot menu also houses a quick imager, that will be described later.

Flagging & strategies

The purpose of the RFI gui is to quickly experiment with different parameters and flagging algorithms. In this section, a quick overview to test and alter the flagger will be given.

Assuming that a baseline has been loaded as described above, a first try at flagging can be performed by selecting "Execute strategy" from the "Actions" menu. A pop-up will appear and show the progress. Note that, unlike the "RFI console" program, execution of the default flagger will be performed in memory only: the MS will not be changed (although, needless to say, it is generally a good practice to have a backup of data which is being worked on). Flagging a single baseline with the default strategy should be quick; less than a second with a few hundred time steps and channels, or in the order of tens of seconds when e.g. flagging thousands of time steps x 256 channels (which is common in LOFAR observations).

Once the strategy is finished, the "Alternative flags" button controls the visibility of the flags produced by the strategy. You might want to turn the "Original flags" button off, as these will be drawn on top of the alternative flags, and might occlude them.

A flagging algorithm is composed of a collection of actions which together form what is called a "strategy", which is comparable to a script in a programming language. Actions can have parameters that tune the flagger. For example; one action tries to subtract the celestial signal and passband by smoothing the data. This action has parameters to alter the way and the strength of the smoothing action.

Edit strategy window

The default strategy is optimized to be quick and accurate, and has been toroughly tested on LOFAR and WSRT data as well as using simulations. However, certain observations might still produce less satisfying results, for example when a telescope with a steeper passband is used and the fitter is too constrained, or when a celestial source produces strong fringes due to antenna sidelobes, and are being considered as RFI. Below I will describe some things which you can try to optimize the strategy for your situation.

To see and edit the strategy, go to the Actions menu in the main window and select "Edit strategy". A window similar to the picture on the right will appear. The largest space will be taken by the various actions, displayed hierarchically. In the root is "Strategy", which represents the starting point of execution. Below the strategy window are three rows of buttons. The first rows are used for editting the strategy, the second row handles file actions (save and open of a strategy) and the third row loads a new or default strategies. By clicking on an action, the parameters of the action appear (i.e., if the action has any). The hierarchical displayment allows grouping; e.g., the "For each polarisation" action iterates over all polarisations and execute its children once for each polarisation. Then, it combines the results of all runs and changes the three copies accordingly.

Internally, the strategy will keep three copies of the data in memory. These are referred to as "Original", "Revised" and "Contaminated". Most actions are applied on the "Contaminated" data, while the "Revised" is being used for storing the background fit. Finally, the "Original" image is kept to be able to restore any changes, e.g. after an iteration. Currently, it is not well documented what actions exactly do, which hopefully will become more clear in the future.

Adding an action

To add a new action:

The default strategy

The image of the edit window on the right shows the current default strategy (which is subject to change...). The strategy is explained in the article A LOFAR RFI detection pipeline and its first results (Offringa et al., 2010), which I refer to for a general understanding. The "Default" button resets the strategy. The "1", "2" and "3" buttons load the Default strategy encapsulated in a "For each baseline" action, including a "Write flags" action. This corresponds with the "quick", "default" and "best" strategies creatable by the "rfistrategy" binary, which are equal at present (we found that the best and quick strategies were almost equal qua performance, while the accuracy was considerably better, hence took that as default for all). If you load one of the 1,2,3 strategies and execute it, you would flag (and write those flags to..!) your whole set.

Here is a summary of the individual actions:

Be sure to read the strategy optimization example. There is also an item in the FAQ on what can be tried to solve certain issues. Do not forget to press the "Apply" button after changing parameters. If you forget this, the changes are not saved, and RFI gui will not warn you about it.

Executing a strategy in RFI console

It is possible to export a strategy from the GUI and execute it in RFI console, in order to quickly flag multiple sets with the same strategy. This is also handy when you want to flag the data on a different (faster :)) machine, or when you do not want to use the GUI for the actual flagging.

In order to do so, create a strategy that satisfactory flags a single baseline. Then, click the "FOB" button, which will add a "for each baseline" at the root of the strategy. You might want to change some of its parameters. Then, add a "Write flags" action as the last action of the For Each Baseline action, so that the results are actually written. Now, your strategy is ready to be executed on the entire set. You can "Save" the strategy and use it in RFI console (but you can also run it in RFI gui).

Vice versa, you can also load a Strategy created by the RFI strategy program, possibly to understand how the default strategies operate -- but be aware that those strategies write their changes.

Imaging capabilities

Imaging of the currently selected data can be done with "Set and show image plan" in the Plot menu. A window appears where you can switch between the UV plane and the imaging plane. This action is actually a short cut for four actions: (1) pressing "Show image plane"; (2) pressing "Clear" in the image plane; (3) pressing "Add to image plane" in the Plot menu of the main window; and (4) pressing "Apply weights" in the image window. The first action is trivial, the others will be described below.

The image window has a lot of buttons with undocumented abbreviations on them. They are: