faq.rst

FAQ

Connecting to devices

My device is unreachable, but the device logs say it's running fine?

The $HOSTNAME may have been incorrectly guessed by docker-compose/Makefile, or you accidently set it to an incorrect value. If you have $HOSTNAME set in the shell running make, try:

unset HOSTNAME
make build
make stop
make start

If this does not work, you need to set $HOSTNAME to something that resolves to your machine, both for external parties and for docker containers. See :ref:`corba`.

I get "API_CorbaException: TRANSIENT CORBA system exception: TRANSIENT_NoUsableProfile" when trying to connect to a device?

See the previous answer.

Docker

How do I prevent my containers from starting when I boot my computer?

You have to explicitly stop a container to prevent it from restarting. Use:

cd docker-compose
make stop <container>

or plain make stop to stop all of them.

Windows

How do I develop from Windows?

Our setup is Linux-based, so the easiest way to develop is by using WSL2, which lets you run a Linux distro under Windows. You'll need to:

• Install WSL2. See f.e. https://www.omgubuntu.co.uk/how-to-install-wsl2-on-windows-10
• Install Docker Desktop
• Enable the WSL2 backend in Docker Desktop
• We also recommend to install Windows Terminal

How do I run X11 applications on Windows?

If you need an X11 server on Windows:

• Install VcXsrv
• Disable access control during its startup,
• Use export DISPLAY=host.docker.internal:0 in WSL.

You should now be able to run X11 applications from WSL and Docker. Try running xterm or xeyes to test.

SSTs/XSTs

Some SSTs/XSTs packets do arrive, but not all, and/or the matrices remain zero?

So sst.nof_packets_received / xst.nof_packets_received is increasing, telling you packets are arriving. But they're apparently dropped or contain zeroes.

The sdp.set_defaults() command, followed by sst.set_defaults() / xst.set_defaults(), should reset that device to its default settings, which should result in a working system again. If not, or if the default configuration is not correct, check the following settings:

• sdp.TR_fpga_mask_RW[x] == True, to make sure we're actually configuring the FPGAs,
• sdp.FPGA_wg_enable_RW[x] == False, or the Waveform Generator might be replacing our the antenna data with zeroes,
• sdp.FPGA_processing_enabled_R[x] == True, to verify that the FPGAs are processing, or the values and timestamps will be zero,
• For XSTs, xst.FPGA_xst_processing_enabled_R[x] == True, to verify that the FPGAs are computing XSTs, or the values will be zero.

Furthermore, the sst and xst devices expose several packet counters to indicate where incoming packets were dropped before or during processing:

• nof_invalid_packets_R increases if packets arrive with an invalid header, or of the wrong statistic for this device,
• nof_packets_dropped_R increases if packets could not be processed because the processing queue is full, so the CPU cannot keep up with the flow,
• nof_payload_errors_R increases if the packet was marked by the FPGA to have an invalid payload, which causes the device to discard the packet,

I am not receiving any XSTs and/or SSTs packets from SDP!

Are you sure? If sst.nof_packets_received / xst.nof_packets_received is actually increasing, the packets are arriving, but are not parsable by the SST/XST device. If so, see the previous question.

The sdp.set_defaults() command, followed by sst.set_defaults() / xst.set_defaults(), should reset that device to its default settings, which should result in a working system again. If not, or if the default configuration is not correct, check the following settings:

• sdp.TR_fpga_mask_RW[x] == True, to make sure we're actually configuring the FPGAs,
• sdp.FPGA_communication_error_R[x] == False, to verify the FPGAs can be reached by SDP,
• SSTs:
  • sst.FPGA_sst_offload_enable_RW[x] == True, to verify that the FPGAs are actually emitting the SSTs,
  • sst.FPGA_sst_offload_hdr_eth_destination_mac_R[x] == <MAC of your machine's mtu=9000 interface>, or the FPGAs will not send it to your machine. Use f.e. ip addr on the host to find the MAC address of your interface, and verify that its MTU is 9000,
  • sst.FPGA_sst_offload_hdr_ip_destination_address_R[x] == <IP of your machine's mtu=9000 interface>, or the packets will be dropped by the network or the kernel of your machine,
  • sst.FPGA_sst_offload_hdr_ip_destination_address_R[x] == 5001, or the packets will not be sent to a port that the SST device listens on.
• XSTs:
  • xst.FPGA_sst_offload_enable_RW[x] == True, to verify that the FPGAs are actually emitting the SSTs,
  • xst.FPGA_xst_offload_hdr_eth_destination_mac_R[x] == <MAC of your machine's mtu=9000 interface>, or the FPGAs will not send it to your machine. Use f.e. ip addr on the host to find the MAC address of your interface, and verify that its MTU is 9000,
  • xst.FPGA_xst_offload_hdr_ip_destination_address_R[x] == <IP of your machine's mtu=9000 interface>, or the packets will be dropped by the network or the kernel of your machine,
  • xst.FPGA_xst_offload_hdr_ip_destination_address_R[x] == 5002, or the packets will not be sent to a port that the XST device listens on.

If this fails, see the next question.

I am still not receiving XSTs and/or SSTs, even though the settings appear correct!

Let's see where the packets get stuck. Let us assume your MTU=9000 network interface is called em2 (see ip addr to check):

• Check whether the data arrives on em2. Run tcpdump -i em2 udp -nn -vvv -c 10 to capture the first 10 packets. Verify:
  • The destination MAC must match that of em2,
  • The destination IP must match that of em2,
  • The destination port is correct (5001 for SST, 5002 for XST),
  • The source IP falls within the netmask of em2 (unless net.ipv4.conf.em2.rp_filter=0 is configured),
  • TTL >= 2,
• If you see no data at all, the network will have swallowed it. Try to use a direct network connection, or a hub (which broadcasts all packets, unlike a switch), to see what is being emitted by the FPGAs.
• Check whether the data reaches user space on the host:
  • Turn off the sst or xst device. This will not stop the FPGAs from sending.
  • Run nc -u -l -p 5001 -vv (or port 5002 for XSTs). You should see raw packets being printed.
  • If not, the Linux kernel is swallowing the packets, even before it can be sent to our docker container.
• Check whether the data reaches kernel space in the container:
  • Enter the docker device by running docker exec -it device-sst bash.
  • Run sudo bash to become root,
  • Run apt-get install -y tcpdump to install tcpdump,
  • Check whether packets arrive using tcpdump -i eth0 udp -c 10 -nn,
  • If not, Linux is not routing the packets to the docker container.
• Check whether the data reaches user space in the container:
  • Turn off the sst or xst device. This will not stop the FPGAs from sending.
  • Enter the docker device by running docker exec -it device-sst bash.
  • Run sudo bash to become root,
  • Run apt-get install -y netcat to install netcat,
  • Check whether packets arrive using nc -u -l -p 5001 -vv (or port 5002 for XSTs),
  • If not, Linux is not routing the packets to the docker container correctly.
• If still on error was found, you've likely hit a bug in our software.

Inspecting SST/XST packets

The fields sst.last_packet_R and xst.last_packet_R contain a raw dump of the last received packet for that statistic. Parsing these packets is aided greatly by using our packet parser:

from tangostationcontrol.devices.sdp.statistics_packet import SSTPacket, XSTPacket

# print the headers of the last received packets
print(SSTPacket(bytes(sst.last_packet_R)).header())
print(XSTPacket(bytes(xst.last_packet_R)).header())

Other containers

The ELK container won't start, saying "max virtual memory areas vm.max_map_count [65530] is too low"?

The ELK stack needs the vm.max_map_count sysctl kernel parameter to be at least 262144 to run. See :ref:`elk-kernel-settings`.