Offringa->RFI software

RFI software

This page describes the software to detect and mitigate radio frequency interference (RFI) in a radio measurement set. The main algorithms are described in the articles "Post-correlation radio frequency interference classification methods'" (Offringa et al. 2010, MNRAS, Volume 405, Issue 1, pp. 155-167) and "A morphological algorithm for improving radio-frequency interference detection" (Offringa et al. 2012, A&A, Volume 539, Issue A95). Please acknowledge these papers in publications in which the software is used. The first one introduces the SumThreshold method and tests signal estimation methods, the latter introduces the Scale-invariant rank operator.

I've made a "Frequently Asked Questions" page for the software, a short RFI GUI tutorial and an example of how one can optimize the strategy.

Last release: version 2.1.0, 12 August 2012.

Contents:

Introduction

The RFI software presented here can automatically flag data and can be used to analyze the data in a measurement. The purpose of flagging is to mark samples that are affected by interfering sources such as radio stations, airplanes, electrical fences or other transmitting interferers.

The tools in the package are meant for offline use. The software package contains a graphical interface ("rfigui") that can be used to visualize a measurement set and analyze mitigation techniques. It also contains a console flagger ("rficonsole") that can execute a script of mitigation functions without the overhead of a graphical environment. All tools were written in C++.

The software has been tested extensively on many radio telescopes, especially on but not limited to low radio frequencies (150 MHz or lower) produced by the WSRT and LOFAR telescopes. It also works on single dish radio telescopes. I believe that it is both the fastest and most accurate flagger available, which is supported by the scientific articles.

The software is published under the GNU General Public License version 3.

Download

The latest version of the software can be downloaded here:

See the compilation instruction below.

Installation

The packages contain a part of the LOFAR software tree that contain the AOFlagger tools. Most astronomical workstations will have most of the required libraries.

Requirements

Below is a list of libraries that are required.

All the libraries are available as free and open software. The gtkmm libraries and headers are required for the gui but not for the console binary. If they are not present, cmake will warn about it, but will compile the console programs nevertheless.

I disencourage you to compile gtkmm yourself; it has many dependencies. Rather use your package manager (or ask your system administrator). Debian ships this library as "gtkmm-dev".

Some other libraries have to be available on the system (but are generally available). The complete list:

Instructions for compilation

The following steps explain the standard compilation procedure. It is probably recommendable that you start on a system where you have full administration rights, so you can easily find out what exact requirements are required.

  1. If you have not done so, install the required libraries. Most of the requirements can be fetched from Debian(like) repositories, with the exception of Casacore.
  2. Download and extract the software.
  3. Enter the directory ./LOFAR and create and enter a sub-directory called "build/gnu_opt": mkdir -p build/gnu_opt; cd build/gnu_opt
  4. Configure the package with cmake. If you installed all the libraries in their default location, you can issue the following command: cmake ../..
    CMake will tell you when some of the files can not be found. If you need to specify the location of libraries, e.g. for casacore and log4cpp, you can issue something like: cmake ../.. -DCASACORE_ROOT_DIR=/home/offringa/casacore-install -DLOG4CPLUS_LIBRARY=/home/offringa/lib/liblog4cplus.a -DLOG4CPLUS_INCLUDE_DIR=/home/offringa/include/
  5. If CMake succeeds, you can start the compilation: make (or make -j 8 for 8 threaded compilation).

After the last step, the executables will be in the ./LOFAR/build/gnu_opt/CEP/DP3/AOFlagger/src directory, ready for action!

Compilation issues

Usage instructions

rficonsole

The rficonsole program is the simplest way of executing a strategy. To flag RFI in a measurement set, type the following command:

./rficonsole <path‑to‑ms> [<another‑path‑to‑ms> ...]

This will run the default flagging strategy, which has the best balance between flagging quality and speed. The specified filename should normally be a .MS directory containing an AIPS++/Casa measurement set. You can type ./rficonsole without parameters to get information about its command. Currently, the following page is returned:

./rficonsole
Usage: ./rficonsole [-v] [-j <threadcount>] [-strategy <file.rfis>] [-indirect-read] [-nolog] [-skip-flagged] <ms1> [<ms2> [..]]
  -v will produce verbose output
  -j overrides the number of threads specified in the strategy
  -strategy specifies a possible customized strategy
  -indirect-read will reorder the measurement set before starting, which is normally faster
  -nolog will not use the LOFAR logger to output logging messages
  -skip-flagged will skip an ms if it has already been processed by RFI console according
   to its HISTORY table.
  -uvw reads uvw values (some strategies require them)
  -column <NAME> specify column to flag
Execute 'rfistrategy' without parameters for help on creating RFIS strategies.
rfistrategy

To customize the flagging process, you can run the rficonsole with an alternate strategy with the -strategy flag. You can create an .rfis-strategy file with the gui or with the rfistrategy command. rfistrategy is a console program. Run the rfistrategy command without parameters to get help:

./rfistrategy
RFI strategy file writer
This program will write an RFI strategy to a file, to run it with the
rficonsole or the rfigui.

Author: André Offringa (e-mail)

Usage: ./rfistrategy <profile> <filename>

Profiles:
        fast    Fastest strategy that provides a moderate
                result in quality. Will flag the measurement set using
                Stokes-I values, and by using a relatively small
                sliding window.
        average Best trade-off between speed and quality. Will
                flag the measurement set using stokes I values using
                an average sliding window size.
        best    Highest quality detection. Will flag each
                polarization individually, using a relatively large
                sliding window.
<filename> is the filename to which the strategy is written. This
file should have the extension ".rfis".

All profiles implement the SumThreshold method. The details of this
method are described in the article named "Post-correlation radio
frequency interference classification methods", submitted to MNRAS.

The .rfis files are XML files and can be changed by hand or with a script in e.g. a full (offline) processing pipeline.

rfigui

The rfigui is a complex analyzation and experimentation tool which I use to create and test alternate flagging strategies, quickly visualize data sets and their flags, make graphs and perform all kind of mutations on the data set. Most features are self-explanatory and can be found in the main menu. Some features are still under heavy development and might not work (correctly). Some of its features are:

Problems and solutions

Contact information

In case of questions, suggestions or feedback, please mail me. My e-mail address is <mylastname>@astro.rug.nl, and my last name is offringa.

Acknowledgments

Thanks to all who provided data to me for testing, in special Ger de Bruyn and Gianni Bernardi. Also thanks to Andy Biggs and Ger van Diepen for several fixes and hints. The RFI tools heavily use Casacore, thus their authors have been instrumental in creating the flagger.