Skip to content
Snippets Groups Projects
Select Git revision
  • 04f3b545bdec072f1f73c8cb8f261d52bbcc881b
  • master default protected
  • revert-cs032-ccd-ip
  • deploy-components-parallel
  • fix-chrony-exporter
  • L2SS-2407-swap-iers-caltable-monitoring-port
  • L2SS-2357-fix-ruff
  • sync-up-with-meta-pypcc
  • stabilise-landing-page
  • all-stations-lofar2
  • v0.39.7-backports
  • Move-sdptr-to-v1.5.0
  • fix-build-ubuntu
  • tokens-in-env-files
  • fix-build
  • L2SS-2214-deploy-cdb
  • fix-missing-init
  • add-power-hardware-apply
  • L2SS-2129-Add-Subrack-Routine
  • Also-listen-internal-to-rpc
  • fix-build-dind
  • v0.55.5-r2 protected
  • v0.52.8-rc1 protected
  • v0.55.5 protected
  • v0.55.4 protected
  • 0.55.2.dev0
  • 0.55.1.dev0
  • 0.55.0.dev0
  • v0.54.0 protected
  • 0.53.2.dev0
  • 0.53.1.dev0
  • v0.52.3-r2 protected
  • remove-snmp-client
  • v0.52.3 protected
  • v0.52.3dev0 protected
  • 0.53.1dev0
  • v0.52.2-rc3 protected
  • v0.52.2-rc2 protected
  • v0.52.2-rc1 protected
  • v0.52.1.1 protected
  • v0.52.1 protected
41 results

developer.rst

Blame
  • Jan David Mol's avatar
    L2SS-445: Added documentation for adding log sources
    Jan David Mol authored
    bf1f4f0f
    History
    Code owners
    Assign users and groups as approvers for specific file changes. Learn more.
    developer.rst 6.33 KiB

    Developer information

    This chapter describes key areas useful for developers.

    Docker compose

    The docker setup is managed using make in the docker-compose directory. Key commands are:

    • make status to check which containers are running,

    • make build <container> to rebuild the image for the container,

    • make build-nocache <container> to rebuild the image for the container from scratch,

    • make restart <container> to restart a specific container, for example to effectuate a code change.

    • make clean to remove all images and containers, and the tangodb volume. To do a deeper clean, we need to remove all volumes and rebuild all containers from scratch:

      make clean
      docker volume prune
      docker build-nocache

    Since the Python code is taken from the host when the container starts, restarting is enough to use the code you have in your local git repo. Rebuilding is unnecessary.

    Docker networking

    The Docker containers started use a virtual network to communicate among each other. This means that:

    • Containers address each other by a host name equal to the container name (f.e. elk for the elk stack, and databaseds for the TANGO_HOST),
    • localhost cannot be used within the containers to access ports of other containers.
    • host.docker.internal resolves to the actual host running the containers,
    • All ports used by external parties need to be exposed explicitly in the docker-compose files. The container must open the same port as is thus exposed, or the port will not be reachable.

    The networks are defined in docker-compose/networks.yml:

    The $NETWORK_MODE defaults to tangonet in the docker-compose/Makefile.

    CORBA

    Tango devices use CORBA, which require all servers to be able to reach each other directly. Each CORBA device opens a port and advertises its address to the CORBA broker. The broker then forwards this address to any interested clients. A device within a docker container cannot know under which name it can be reached, however, and any port opened needs to be exposed explicitly in the docker-compose file for the device. To solve all this, we assign a unique port to each device, and explictly tell CORBA to use that port, and what the hostname is under which others can reach it. Each device thus has these lines in their compose file:

    ports:
      - "5701:5701" # unique port for this DS
    entrypoint:
      # configure CORBA to _listen_ on 0:port, but tell others we're _reachable_ through ${HOSTNAME}:port, since CORBA
      # can't know about our Docker port forwarding
      - python3 -u /opt/lofar/tango/devices/devices/sdp/sdp.py LTS -v -ORBendPoint giop:tcp:0:5701 -ORBendPointPublish giop:tcp:${HOSTNAME}:5701

    Specifying the wrong $HOSTNAME or port can make your device unreachable, even if it is running. Note that $HOSTNAME is advertised as is, that is, it is resolved to an IP address by any client that wants to connect. This means the $HOSTNAME needs to be correct for both the other containers, and external clients.

    The docker-compose/Makefile tries to set a good default for $HOSTNAME, but you can override it by exporting the environment variable yourself (and run make restart <container> to effectuate the change).

    For more information, see:

    Logging

    The ELK stack collects the logs from the containers, as well as any external processes that send theirs. It is the Logstash part of ELK that is responsible for this. The following interfaces are available for this purpose:

    Interface Port Note
    Syslog 1514/udp Recommended over TCP, as the ELK stack might be down.
    Syslog 1514/tcp  
    JSON 5959/tcp From python, recommended is the LogStash Async module.
    Beats 5044/tcp Use FileBeat to watch logs locally, and forward them to ELK.

    We recommend making sure the contents of your log lines are parsed correctly, especially if logs are routed to the Syslog input. These configurations are stored in docker-compose/elk/logstash/conf.d. An example:

    Log from Python

    The common.lofar_logging module provides an easy way to log to the ELK stack from a Python Tango device.

    Log from Docker

    Not all Docker containers run our Python programs, and can forward the logs themselves. For those, we use the syslog log driver in Docker. Extend the docker compose files with:

    Logs forwarded in this way are provided with the container name, their timestamp, and a log level guessed by Docker. It is thus wise to parse the message content further in Logstash (see above).