diff --git a/.gitattributes b/.gitattributes index 3c53b0debcfdcee82ba8e4d7f48eaddf08851a68..bac69065a191e9850adac04c07ccfc308c8aaa59 100644 --- a/.gitattributes +++ b/.gitattributes @@ -121,11 +121,21 @@ CEP/DP3/AOFlagger/doc/site/faq.html -text CEP/DP3/AOFlagger/doc/site/gui-tutorial.html -text CEP/DP3/AOFlagger/doc/site/img/rfi-example-A0.png -text CEP/DP3/AOFlagger/doc/site/img/rfi-example-A1.png -text +CEP/DP3/AOFlagger/doc/site/img/rfi-example-A10.png -text +CEP/DP3/AOFlagger/doc/site/img/rfi-example-A11.png -text +CEP/DP3/AOFlagger/doc/site/img/rfi-example-A12.png -text +CEP/DP3/AOFlagger/doc/site/img/rfi-example-A13.png -text +CEP/DP3/AOFlagger/doc/site/img/rfi-example-A14.png -text +CEP/DP3/AOFlagger/doc/site/img/rfi-example-A15.png -text +CEP/DP3/AOFlagger/doc/site/img/rfi-example-A16.png -text CEP/DP3/AOFlagger/doc/site/img/rfi-example-A2.png -text CEP/DP3/AOFlagger/doc/site/img/rfi-example-A3.png -text CEP/DP3/AOFlagger/doc/site/img/rfi-example-A4.png -text CEP/DP3/AOFlagger/doc/site/img/rfi-example-A5.png -text CEP/DP3/AOFlagger/doc/site/img/rfi-example-A6.png -text +CEP/DP3/AOFlagger/doc/site/img/rfi-example-A7.png -text +CEP/DP3/AOFlagger/doc/site/img/rfi-example-A8.png -text +CEP/DP3/AOFlagger/doc/site/img/rfi-example-A9.png -text CEP/DP3/AOFlagger/doc/site/img/rfigui-editstrategywindow.png -text CEP/DP3/AOFlagger/doc/site/img/rfigui-goto.png -text CEP/DP3/AOFlagger/doc/site/img/rfigui-gotowindow.png -text diff --git a/CEP/DP3/AOFlagger/doc/site/faq.html b/CEP/DP3/AOFlagger/doc/site/faq.html index 0e0c346cbab239efaddedb33601373028300f44e..2d804fc0a5cfb5b8bebef8d291d879b6161a0035 100644 --- a/CEP/DP3/AOFlagger/doc/site/faq.html +++ b/CEP/DP3/AOFlagger/doc/site/faq.html @@ -23,14 +23,22 @@ <li>I still have questions / contact info</li> </ul> <h3>Where should I start with reading documentation about the tools?</h3> -<p>At present, there is not so much information on the RFI tools. If you are reducing LOFAR data, you should at least read the <a href="http://www.astron.nl/radio-observatory/lofar/lofar-imaging-cookbook">LOFAR imaging cookbook</a>. A first tutorial on the GUI can be found here: <a href="gui-tutorial.html">RFI gui tutorial</a>. More documentation might follow in this website, and of course scanning through this FAQ might solve some of your questions. See the "How does RFI console recognise RFI" faq item for some links to scientific papers about the flagger.</p> +<p>At present, there is not so much information on the RFI tools. If you +are reducing LOFAR data, you should at least read the +<a href="http://www.astron.nl/radio-observatory/lofar/lofar-imaging-cookbook">LOFAR imaging cookbook</a>. +A first tutorial on the GUI can be found here: +<a href="gui-tutorial.html">RFI gui tutorial</a>, and an example to +optimize the default strategy here: +<a href="strategy-example-a.html">Strategy optimization example</a>. +More documentation might follow on this website, and of course scanning +through this FAQ might solve some of your questions. See the "How does RFI console recognise RFI" faq item for some links to scientific papers about the flagger.</p> <h3>I like to use the GUI on LOFAR data but it is not installed</h3> <p>At the moment it is not, because the GUI needs GTKMM, which is not available at the cluster nodes. If you like to use the GUI, you can compile it yourself. See <a href="http://www.astro.rug.nl/rfi-software">http://www.astro.rug.nl/rfi-software</a> for information on downloading the source code, compiling and the dependencies. Alternatively, if you have access to the Groningen Kapteyn Institute network, you can mail me for instructions on how to start the GUI.</p> <h3>Does rficonsole work on WSRT / VLA / GMRT / ... data?</h3> <p>Yes, it does. As far as I know, it has already been tested and shown good results on LOFAR, WSRT, VLA and GMRT data. The only requirement is that the data is stored in the Aips++ / casa Measurement Set format (in my personal opinion, this is one of the best formats around for radio observations). If you have results for other telescopes, I'd love to hear about it.</p> <p>Note though, that the settings have been optimized for LOFAR, and some of the settings might need to be changed to get good results. See the "I get bad results, how te solve?" question for more info.</p> <h3>I get bad results, how to solve?</h3> -<p>Strong sources and different telescope properties can cause the default settings to fail in certain cases. The gui can help you a lot when trying to solve problems. If you think rficonsole/rfigui is doing a bad job, I would suggest to experiment with the GUI to optimize the following settings:</p> +<p>Strong sources and different telescope properties can cause the default settings to fail in certain cases. The gui can help you a lot when trying to solve problems. Be sure to read the <a href="strategy-example-a.html">strategy optimization example</a>. If you think rficonsole/rfigui is doing a bad job, I would suggest to experiment with the GUI to optimize the following settings:</p> <ul> <li>Horizontal resolution decrease factor before the sliding window fit action<br/> LOFAR has a high time resolution (currently 1 sec by default). If your telescope / observation has a longer integration time or a very strong source far from the phase centre, this should be set to 1 instead of 3).</li> diff --git a/CEP/DP3/AOFlagger/doc/site/gui-tutorial.html b/CEP/DP3/AOFlagger/doc/site/gui-tutorial.html index 36cc302be6450acf2eb6cc95d68c191cde36d446..6a4c2ded98fa988a7a65fcc80f4a84ac2b300a61 100644 --- a/CEP/DP3/AOFlagger/doc/site/gui-tutorial.html +++ b/CEP/DP3/AOFlagger/doc/site/gui-tutorial.html @@ -272,7 +272,9 @@ remove this action. Note that if the set contains flags and this action exists, the yellow output flags will be a combination of the fuchsia input flags and the flagging result.</li> </ul> -<p>There is an item in the <a href="faq.html">FAQ</a> on what can be tried +<p>Be sure to read <a href="strategy-example-a.html">the strategy +optimization example</a>. There is also an item in the +<a href="faq.html">FAQ</a> on what can be tried to solve certain issues. Do not forget to press the "Apply" button after changing parameters. If you forget this, the changes are not saved, and RFI gui will not warn you about it.</p> diff --git a/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A10.png b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A10.png new file mode 100644 index 0000000000000000000000000000000000000000..446af0b4b662e514e7d6778d3d7642dc3812f34d Binary files /dev/null and b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A10.png differ diff --git a/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A11.png b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A11.png new file mode 100644 index 0000000000000000000000000000000000000000..b434a66c243e02b67ac6ed812cbd06356c11e9e3 Binary files /dev/null and b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A11.png differ diff --git a/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A12.png b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A12.png new file mode 100644 index 0000000000000000000000000000000000000000..c76d15ec6eb6bcd319a0792391debe5ce80e00b5 Binary files /dev/null and b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A12.png differ diff --git a/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A13.png b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A13.png new file mode 100644 index 0000000000000000000000000000000000000000..65cf5496bb9041616cb94983baa2644e53a627a8 Binary files /dev/null and b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A13.png differ diff --git a/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A14.png b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A14.png new file mode 100644 index 0000000000000000000000000000000000000000..0c559aaf2e8819e1d008b91e541ac0b80a7ff417 Binary files /dev/null and b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A14.png differ diff --git a/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A15.png b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A15.png new file mode 100644 index 0000000000000000000000000000000000000000..e08e4ff4b91af41fa7d8084f27791e92c4e59db9 Binary files /dev/null and b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A15.png differ diff --git a/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A16.png b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A16.png new file mode 100644 index 0000000000000000000000000000000000000000..e987bc66fb6da3ccf6b6f90d53d0066361b80580 Binary files /dev/null and b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A16.png differ diff --git a/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A7.png b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A7.png new file mode 100644 index 0000000000000000000000000000000000000000..8a7d58ed6eb16777eb326256fb40f806f9009633 Binary files /dev/null and b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A7.png differ diff --git a/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A8.png b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A8.png new file mode 100644 index 0000000000000000000000000000000000000000..3ba68a1130de61b99e816793de9012a0c23a3986 Binary files /dev/null and b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A8.png differ diff --git a/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A9.png b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A9.png new file mode 100644 index 0000000000000000000000000000000000000000..11be208ce200ab959d73f88b2c3bca9905cb2573 Binary files /dev/null and b/CEP/DP3/AOFlagger/doc/site/img/rfi-example-A9.png differ diff --git a/CEP/DP3/AOFlagger/doc/site/index.html b/CEP/DP3/AOFlagger/doc/site/index.html index 6919db7a74869f94825aae73fe884b74bc568319..475456867a867f9d7f6e7d5a4abd17f7a539e29c 100644 --- a/CEP/DP3/AOFlagger/doc/site/index.html +++ b/CEP/DP3/AOFlagger/doc/site/index.html @@ -14,6 +14,7 @@ <li>Usage instructions</li> <li>Problems and solutions</li> <li>Contact information</li> +<li>Acknowledgment</li> </ul> <h3>Introduction</h3> <p>The RFI software presented here can automatically flag data and can be used to analyze the data in a measurement. The purpose of flagging is to mark samples that are affected by interfering sources such as radio stations, airplanes, electrical fences or other transmitting interferers.</p> @@ -126,6 +127,13 @@ frequency interference classification methods", submitted to MNRAS.</pre> </ul> <h3>Contact information</h3> <p>In case of questions, suggestions or feedback, please mail me. My e-mail address is <tt><mylastname>@astro.rug.nl</tt>, and my last name is offringa.</p> +<h3>Acknowledgement</h3> +<p>Thanks to all who provided data to me for testing, in special Ger de Bruyn +and Gianni Bernardi. Also thanks to Andy Biggs who was the first unfortunate +one to try it on a Mac and face all incompatibities for the first time. +Thanks to Ger van Diepen for several fixes and hints. +The RFI tools heavily use Casacore, thus their authors have been instrumental +in creating the flagger.</p> </body> </html> diff --git a/CEP/DP3/AOFlagger/doc/site/strategy-example-a.html b/CEP/DP3/AOFlagger/doc/site/strategy-example-a.html index 95fc8a89166e61ccaa4f4a44a664e78360abb05e..26e2b605e652cefa5e9d87b5592a5e0cad33fae5 100644 --- a/CEP/DP3/AOFlagger/doc/site/strategy-example-a.html +++ b/CEP/DP3/AOFlagger/doc/site/strategy-example-a.html @@ -2,12 +2,14 @@ <head><title>Offringa's RFI software</title></head> <body> <p><a href="http://www.astro.rug.nl/~offringa/">Offringa</a>-><a href="index.html">RFI software</a>->Strategy example A</p> -<h2>Strategy example A: daylight WSRT obervation with strong fringes</h2> +<h2>Strategy example A: daylight WSRT observation with strong fringes</h2> +<p>Author: André Offringa, 2011-01-28</p> <p> In this example I will try to demonstrate tweaking of some of the parameters -of the default stratregy when by itself it does not work +of the default strategy when by itself it does not work well enough. I will try to flag an WSRT observation -of B1834. Below is baseline RT0 x RT1, a 140m baseline.</p> +of B1834 with 10 second integration and 512 channels of 5 kHz each. +Below is baseline RT0 x RT1, a 140m baseline.</p> <img src="img/rfi-example-A0.png" /> <p>Notice the very strong fringes. I will use this baseline to experiment on, since it is a good case of RFI, plus the @@ -17,7 +19,7 @@ be a problem either.</p> <p>The strong fringes are due to the sun (the whitish area before 18:50 after which it evidently sets), and Cas A and Cyg A. Since the default strategy is LOFAR optimised, -The default flagged does a very bad job:</p> +The default flagger does a very bad job by flagging 33% of the data:</p> <img src="img/rfi-example-A1.png"> <p>While obviously the sun can be seen as an interfering source, I don't want to flag it, but rather try to subtract it at a later time. @@ -50,8 +52,8 @@ of the fringes. There are two visible problems at this point with the flagging: the fringes have not been removed completely and the pass-band is too steep. Remember that you are watching at Stokes I, and that an individual -(cross) polarizations might be the actual cause for a certain flag. -</p><p>A common problem is that too few iteration have been performed, +(cross) polarization might be the actual cause for a certain flag. +</p><p>A common problem is that too few iterations have been performed, causing the threshold to drop quickly in each iteration. This might cause instabilities. To make sure this is not a problem, I enlarge the number of iterations to "5" and the sensitivity start number of the @@ -69,10 +71,8 @@ in the differential image, hence the SumThreshold method will flag them. Several approaches can be taken now:</p> <ul> <li>The kernel of the background fit can be made smaller.</li> -<li>The SumThreshold method can be changed to not look for lines in -frequency direction.</li> -<li>The sensitivity of the SumThreshold method can be lowered.</li> -<li>The algorithm should not flag RMS-outlying time steps.</li> +<li>The sensitivity for broadband-like RFI can be lowered.</li> +<li>The total sensitivity of the SumThreshold method can be lowered.</li> </ul> <p>I will show the result of each. Obviously, the best solution can sometimes be a combination of these options.</p> @@ -128,10 +128,145 @@ an acceptable result, but not perfect.</p> high values in frequency direction. In a normal situations, broadband RFI such as lightning or electrical fences can produce these. However, besides the "fringe" -problem we are facing, there might be other reasons not to flag in +problem that is faced here, there might be other reasons not to flag in frequency direction, for example because one is searching for transients. -This would make us slightly more subject to leaked broad-band RFI.</p> +This would make us slightly more subject to leaked broad-band RFI, a +trade off that should be considered.</p> +<p>We go back to our default strategy with 5 iterations. +Two actions influence broadband flagging, that is, flagging of vertical +lines in the image. The first action is the SumThreshold, which does the +"per pixel" (yet combinatorial) tresholding. The SumThreshold has an option +named "In frequency direction". By deselecting this option in both SumThreshold +methods (note there are two SumThreshold actions which need to be changed +both!), the result drastically improves: (13% was flagged)</p> +<img src="img/rfi-example-A7.png" /> +<p>Intuitively, this improvement is not so strange; even by eye, +broad-band RFI is almost +impossible to distinguish in this set, unless it is very strong.</p> +<p>At this +point, we have to be careful. Note that this observation toke eight hours, +and since we have a 10 second integration, this image represents over 4000 +samples. Looking at the full image on a screen is not a good way of +differentiating broadband RFI from false positives due to fringes. For example, +when we zoom in (press "Zoom" in the "View" menu) on the first +two hours of the observation, we see this: (flags have been turned off)</p> +<img src="img/rfi-example-A8.png" /> +<p>This is broad-band RFI, which was impossible to see in the full image. +For those that are curious about its origin: that often remains a mystery +to me. +Sometimes, lightning causes such effects, but a starting car or even a camera +near the antenna might similarly cause these effects -- but the latter are +unlikely considering their length. The signal is not +dispersed, +so it is unlikely a celestial transient. Meteors often only reflect +terestrial RFI, which is not broadband. The sun +might cause such artefacts, though these lines are also present at the +end of the observation when +the sun is down. In short; we want to flag these lines, because they will +decrease the SNR. Fortunately, even without frequency direction +SumThresholding, these lines are all flagged:</p> +<img src="img/rfi-example-A9.png" /> +<p>The broad-band RFI has been flagged by the two +"Time selection" actions. These actions flag time steps that have an RMS +which differs significantly from its neighbours. If I would remove both +Time selection actions, this is the result:</p> +<img src="img/rfi-example-A10.png" /> +<p>My conclusion would be that the Time selection action was actually doing +a good thing. It lowers the ratio of flags from about 13% to +12.3%, but a large part of this 0.7% is actual unwanted interference. +The conclusion on the frequency direction flagging would be that the +RMS check should remain, but that SumThresholding in frequency direction +is doing not much good in this set.</p> +<h3>Sensitivity</h3> +<p>Once again I go back to my original strategy with 5 iterations. A last +parameter which is often good for solving issues, is the sensitivity of the +SumThreshold action. I can fix my original very bad result at once by +changing the base threshold of both SumThreshold actions to "2" instead +of its default "1". The result is a good result with 8.5% flags:</p> +<img src="img/rfi-example-A11.png" /> +<p>Note that this both solves most of the flagging problems of sun's fringes +and also solves the bandpass problem. However, also note that the frequency +changing interfering source has a bit more gaps, especially near 17.30 hours +at 138.3 MHz. +While this flagging would be good enough for further reduction, I would prefer +the previous changes and put the sensitivity somewhere in the middle +(by tweaking the end result).</p> <h3>Summary of modifications</h3> +<p>I have tried to isolate the effect of each important parameter in this +example. Taking all these changes together will enhance the result further. +The summary of changes and conclusions was:</p> +<ul> +<li>To make sure that stability issues are not cause by too few iteration, +I changed the Iteration action to iterate 5 times starting at threshold +factor 8, instead of iterating 2 times starting at threshold factor 4 +(see notes below about performance).</li> +<li>Smoothing in time direction was too strong. A good fit in time direction +was achieved by setting the "Change resolution size" to "1".</li> +<li>The bandpass problem could be fixed by removing the "Frequency selection" +action (but removing the bandpass prior to flagging would be better!).</li> +<li>Turning the SumThresholding in the frequency direction off improved +the strategy, while (even after zooming in) I could not detect introduced +false negatives by this action.</li> +<li>Turning the Time selection off did not significantly lower the false +positives, while some true RFI was left. The time selection action was kept.</li> +<li>Decreasing the sensitivity by increasing the base threshold made the system +more robused, but also introduced a few false negatives. Therefore, I would +suggest a slightly decreased sensitivity, to make sure the strategy never +starts "eating" whole parts of the data (after all, +this is only one baseline, other +baselines might be different). I set the base threshold from its default of +1.0 to a slightly less sensitive 1.2. This decreased the number of flags from 9.6% to 8.4%.</li> +</ul> +<p>This is the end result:</p> +<img src="img/rfi-example-A12.png" /> +<p>The fitted background looks like this:</p> +<img src="img/rfi-example-A13.png" /> +<p>Which is a very fair representation of the signal, as can also be seen +from the difference:</p> +<img src="img/rfi-example-A14.png" /> +<p>This image contains almost no artifacts from the signal, except for +decreased SNR when the sun is up and during fringes.</p> +<p>A remaining question is how well it works on other baselines. When checking +the other baselines, the parameters work very well on all of them. Consider +RT0 x RT6, which shows other signs of RFI:</p> +<img src="img/rfi-example-A15.png" /> +<p>The 10.1% flags produced by the optimized strategy:</p> +<img src="img/rfi-example-A16.png" /> +<h3>Execution on all baselines</h3> +<p>This strategy is ready for execution on all 91 WSRT baselines. This +can be done by (A) pressing the "FOB" button in the strategy window to +encapsulate the strategy in a "For each baseline" action, (B) add a "Write +flags" task as last action inside the "For each baseline" action to actually +write the flags, (C) export the strategy by pressing "Save" and entering a +name and (D) running rficonsole with the saved strategy.</p> +<h3>Performance considerations</h3> +<p>Finally, I would like to note that I increased the number of iterations to +"5" because I wanted to be sure that the background fit would be stable. In +practice, this is not necessary once all parameters are fixed. If you want +to make sure your strategy is also as fast as possible, the number of +iterations can be lowered. In this example, changing the number of iterations +back to "2" and the start sensitivity to "4" does not significantly change +the flags, but is 2.5x faster.</p> +<p>The "change resolution" action also has a lot of effect on the performance. +Once all parameter optimizations were applied, changing the resolution change +action back did not severely change the accuracy, so if performance would +be an issue, I would revert the change resolution optimization.</p> +<p>Wall-clock time performance on my desktop machine of flagging a +single baseline:</p> +<ul> +<li>5 iterations: 50 seconds, 8.4% flagged</li> +<li>5 iterations, decreased resolution: +42 seconds, 8.5% flagged</li> +<li>2 iterations: 26 seconds, 8.4% flagged</li> +<li>2 iterations, decreased resolution: +22 seconds, 8.3% flagged</li> +</ul> +<p>In all cases, no visible change in accuracy is visible. Note that when +flagging multiple baselines, strategies can be executed multithreaded, so +by using a multi core machine, the total time can be considerably lowered. +Flagging this set on an 8 core machine toke in the order of a few minutes.</p> +<h3>Acknowledgement</h3> +<p>I would like to thank Ger de Bruyn from Astron for providing the data +that has been used in this example.</p> </body> </html> -