Bug 1491: finished tutorial

CEP/DP3/AOFlagger/doc/site/gui-tutorial.html

@@ -140,14 +140,157 @@ Actions can have parameters
 that tune the flagger. For example; one action tries to subtract the celestial
 signal and passband by smoothing the data. This action has parameters to
 alter the way and the strength of the smoothing action.</p>
+<img src="img/rfigui-editstrategywindow.png" alt="Edit strategy window" align="right" />
 <p>The default strategy is optimized to be quick and accurate, and has
 been toroughly tested on LOFAR and WSRT data as well as using
-simulations. However, certain simulations might
-still produce less satisfying results, for example when a the telescope
+simulations. However, certain observations might
+still produce less satisfying results, for example when a telescope
 with a steeper passband is used and the fitter is too constrained, or when
 a celestial source produces strong fringes due to antenna sidelobes,
 and are being considered as RFI.</p>
+<p>To see and edit the strategy, go to the Actions menu in the main window
+and select "Edit strategy". A window similar to the picture on the right
+will appear. The largest space will be taken by the various actions,
+displayed hierarchically. In the root is "Strategy", which represents the
+starting point of execution. Below the strategy window are three rows of
+buttons. The first rows are used for editting the strategy, the second row
+handles file actions (save and open of a strategy) and the third row loads
+a new or default strategies. By clicking on an action, the parameters of the
+action appear (i.e., if the action has any). The hierarchical displayment
+allows grouping; e.g., the "For each polarisation" action iterates over all
+polarisations and execute its children once for each polarisation. Then,
+it combines the results of all runs and changes the three copies accordingly.
 </p>
+<p>Internally, the strategy will keep three copies of the data in memory.
+These are referred to as "Original", "Revised" and "Contaminated". Most
+actions are applied on the "Contaminated" data, while the "Revised" is being
+used for storing the background fit. Finally, the "Original" image is kept
+to be able to restore any changes, e.g. after an iteration. Currently,
+it is not well documented what actions exactly do, which hopefully will
+become more clear in the future. </p>
+<h3>Adding an action</h3>
+<p>To add a new action:</p>
+<ul>
+<li>Select the node which is going to be its parent. This parent should be
+a "container" node, that is, be able to have children nodes. If this is
+not the case, the "Add" button will be greyed out.</li>
+<li>Press the Add button. A long menu appears on the right with all the actions
+available.</li>
+<li>Select the right action.</li>
+<li>The new action will be selected. You might want to change its position
+inside the parent by pressing the Up and Down buttons.</li>
+</ul>
+<h3>The default strategy</h3>
+<p>The image of the edit window on the right shows the current default strategy (which is subject
+to change...). The strategy is explained in the article
+<a href="http://arxiv.org/abs/1007.2089">A LOFAR RFI detection pipeline and
+its first results (Offringa et al., 2010)</a>, which I refer to for a general
+understanding. The "Default" button resets the strategy. The "1", "2" and "3"
+buttons load the Default strategy encapsulated in a "For each baseline"
+action, including a "Write flags" action. This corresponds with the "quick",
+"default" and "best" strategies creatable by the "rfistrategy" binary, which
+are equal at present (we found that the best and quick strategies were almost
+equal qua performance, while the accuracy was considerably better, hence
+toke that as default for all). If you load one of the 1,2,3 strategies and
+execute it, you would flag (and write those flags to..!) your whole set.</p>
+<p>
+Here is a summary of the individual actions:</p>
+<ul>
+<li><em>Set no flags</em>: removes the flags in the Contaminated data.</li>
+<li><em>For each polariation</em> iterates over selected polarisations
+and executes all children for the selected polarisations. The result of the
+children will be combined and returned, if possible.</li>
+<li><em>On amplitude:</em> is a "For each complex component" action. Executes
+all children by iterating over certain complex derivations and will
+combine the results if possible. Possibilities are execution on
+amplitude, phase, real and imaginary values. In the default pipeline, only the
+amplitude value is executed. Note that if subtasks alter the amplitude values,
+this task cannot change this back into real and imaginary values.
+Nevertheless, it is often useful to see what the background fit has done,
+which can be seen by selecting the "Restore from amplitude" checkbox (do not
+forget to "Apply" the changes). The "background" toolbar button on the
+mainwindow will then show you the fitted background after the strategy
+has been executed. Very useful for analysis.</li>
+<li><em>Iterate 2 times:</em> is an iteration action. Executes its children
+a number of times. Will also change the threshold sensitivity
+exponentially, which can be controlled by its parameters. When accuracy
+is not good enough, it might help to increase the number of iterations. While
+it takes more time, the output should be more stable.</li>
+<li><em>SumThreshold:</em> performs the SumThreshold algorithm to flag samples.
+The result is stored in the Contaminated data. The method has been introduced
+and tested in the article
+<a href="http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2966.2010.16471.x/full">Post-correlation radio frequency interference classification methods
+(Offringa et al., 2010)</a>, and has since proven to be very accurate. 
+It has an important parameter: its base sensitivity. If you see too few
+flags, increase its sensitivity (by lowering the number), if you see too many
+flags, lower its sensitivity (by increasing the number).</li>
+<li><em>Combine flag results:</em> runs its children one by one and "ors"
+together all flags (in this case, the frequency and time selection are
+executed one after one other and then combined.
+The difference with just running the selections after each other without
+combining the flags, is that they will both be presented the flags
+that are produced by the SumThreshold, and the Time selection will not
+be presented the flags of the Frequency selection.</li>
+<li><em>Time selection:</em> selects times which have an acceptable RMS,
+compared to its neighbours and flags the other.</li>
+<li><em>Frequency selection:</em> equal to time selection, but then in
+frequency.</li>
+<li><em>Set contaminated=original</em>: will revert any changes being done
+to the contaminated data (but not change its flags). In the default
+strategy, this is used to iteratively get a better fit to the data. First,
+a very rough threshold is done, then a first fit while ignoring flagged data,
+then a better threshold and a better fit, etc.</li>
+<li><em>Change resolution</em>: downscales the images, then executes its
+children and then upscales the result. Scaling is done by nearest
+neighbour interpolation; not so accurate, but very fast. The reason for it
+being in the default pipeline, is to increase performance. The sliding window
+fit operation is one of the slowest operations. By performing it on a
+low resolution, the speed is significantly increased. If your measurement
+set has strong, fast fringes, it might be helpful to set the resolution change
+factors to "1" (i.e., no change). The kernel in the sliding window fit
+action is currently set in number of samples, which implies that the total
+size of the kernel is multiplied by the scale factor.</li>
+<li><em>Sliding window fit</em>: smoothes the "Contaminated data", stores
+the result in the "Revised data" and the difference in "Contaminated data".
+Keep in mind that the "Change resolution" action also effect the size of the
+kernels (see above). You can analyse the fitted background in the GUI by
+making sure that the "On amplitude" action saves the background (select
+its "Restore amplitudes" checkbox). </li>
+<li><em>Apply flags to all polarizations:</em> if a sample is flagged
+in one polarization, will flag all polarizations.</li>
+<li><em>Statistical flagging:</em> somewhat of a bad name for what its
+purpose became, this action flags samples that are close to large areas of
+flags. It's described in <a href="http://arxiv.org/abs/1007.2089">A LOFAR
+RFI pipeline and its first results</a>, where it is called a
+"density dillution".</li>
+<li><em>Select baselines</em>: assembles statistics on baselines. Needs a
+second action to output the baselines, which by default is added by
+RFI console.</li>
+<li><em>Or flags with original</em>: If flags exist in the MS, this action will
+keep those flags. If you want to "start over" and remove existing flags,
+remove this action. Note that if the set contains flags and this action
+exists, the yellow output flags will be a combination of the fuchsia input
+flags and the flagging result.</li>
+</ul>
+<p>There is an item in the <a href="faq.html">FAQ</a> on what can be tried
+to solve certain issues. Do not forget to press the "Apply" button after
+changing parameters. If you forget this, the changes are not saved,
+and RFI gui will not warn you about it.</p>
+<h3>Executing a strategy in RFI console</h3>
+<p>It is possible to export a strategy from the GUI and execute it in RFI
+console, in order to quickly flag multiple sets with the same strategy. This
+is also handy when you want to flag the data on a different (faster :))
+machine, or when you do not want to use the GUI for the actual flagging.</p>
+<p>In order to do so, create a strategy that satisfactory flags a single
+baseline. Then, click the "FOB" button, which will add a "for each baseline"
+at the root of the strategy. You might want to change some of its parameters.
+Then, add a "Write flags" action as the last action of the For Each Baseline
+action, so that the results are actually written. Now, your strategy is
+ready to be executed on the entire set. You can "Save" the strategy and use it
+in RFI console (but you can also run it in RFI gui).</p>
+<p>Vice versa, you can also load a Strategy created by the RFI strategy
+program, possibly to understand how the default strategies operate -- but be
+aware that those strategies write their changes.</p>
 <h3>Imaging capabilities</h3>
 <p>Imaging of the currently selected
 data can be done with "Set and show image plan" in the Plot menu. A window
@@ -169,7 +312,40 @@ not add new data to the uv plane, as you would weight this new data
 different then the previously added data. </li>
 <li>The "Add to image plane" menu item on the main window will grid and add
 the currently visible data to the uv plane, and update the weights
-accordingly. If you would press  </li>
+accordingly. By adding several baselines, you would add more and more data to
+the UV plane, hence improve the image.
+If you would add all baselines in your MS one by one, you would
+have manually imaged your observation much like a normal imager would
+do. You can actually create a strategy that does that (a "For each baseline"
+action with an "Image" action).</li>
+</ul>
+<p>The image window has a lot of buttons with undocumented abbreviations on
+them. They are:</p>
+<ul>
+<li>x1/4, x1/2, x1, x2, ... x128: these buttons Clear the current data and 
+weights and change scaling of (new) data in the UV plane. It is not so much
+meant as having accurate units, therefore you have to experiment which
+scaling will give you a good sampling of the uv plane (the uv tracks should
+be visible and preferably not be too small). With WSRT LFFE data (150 MHz), the
+longest baseline of 2,7 km is quite nicely shown with a scaling of 4x.
+Remember that zooming in on the UV plane means zooming out on the image plane.
+</li>
+<li>R: redraw the current image in the imager</li>
+<li>MS: the MS button ("Memory store") stores the image currently being
+shown in memory</li>
+<li>MR: draw the image that has been stored in memory. By clicking "R"
+and "MR" after each other, you can compare the stored image with the current
+image quickly.</li>
+<li>Mx: multiply the current image with the image in memory</li>
+<li>M-: Show memory minus current</li>
+<li>sqrt: Display the square root of the visible data. The negative values
+(displayed in red) will be calculated as -sqrt(-data), hence remain negative.
+</li>
+<li>S: (toggle button) keep the scale constant. Will not change the color scale
+when new data is shown, hence can make comparing easier.</li>
+<li>H/V: horizontal/vertical graph, experimental options.</li>
+<li>AT: angular transformation, experimental option.</li>
+</ul>
 </p>
 </body>
 </html>