Skip to content
Snippets Groups Projects
Commit 04f3b545 authored by Stefano Di Frischia's avatar Stefano Di Frischia
Browse files

Merge branch 'master' into L2SS-429-timescaledb-container

parents 36dba6c7 9a5cd4fb
Branches
Tags
1 merge request!177Resolve L2SS-429 "Timescaledb container"
Showing
with 201 additions and 26 deletions
asyncua
# Do not put tangostationcontrol dependencies here
astropy
python-logstash-async
gitpython
PyMySQL[rsa]
sqlalchemy
docker
# requirements to build tangocontrol
GitPython >= 3.1.24 # BSD
......@@ -16,4 +16,10 @@ services:
- control
ports:
- "9090:9090"
logging:
driver: syslog
options:
syslog-address: udp://${LOG_HOSTNAME}:1514
syslog-format: rfc3164
tag: "{{.Name}}"
restart: unless-stopped
......@@ -33,3 +33,10 @@ services:
- /usr/bin/supervisord
- --configuration
- /etc/supervisor/supervisord.conf
logging:
driver: syslog
options:
syslog-address: udp://${LOG_HOSTNAME}:1514
syslog-format: rfc3164
tag: "{{.Name}}"
restart: unless-stopped
......@@ -16,5 +16,7 @@ RUN cd /sdptr && \
./configure && \
bash -c "make -j `nproc` install"
COPY simulator.conf /sdptr/src/simulator.conf
WORKDIR /sdptr/src
CMD ["sdptr", "--type=LTS", "--configfile=uniboard.conf", "--nodaemon"]
CMD ["sdptr", "--type=simulator", "--configfile=simulator.conf", "--nodaemon"]
# sdptr.conf
# configuration file for the SDP Translator.
#
# this config file holds settings for all [type].
#
# # settings per type
# [LB_CORE] # [ant_band_station_type]
# n_fpgas = 16 # 8 or 16
# first_pfga_nr = 0 # 0 for LB or 16 for HB
# ip_prefix = 10.99. # first part of ip (last part is hardware dependent)
# n_beamsets = 1 # 1 for 'LB', 'HB Remote' and 'HB International' and 2 for 'HB Core'
[simulator]
n_fpgas = 16
first_fpga_nr = 0
ip_prefix = 127.0.
n_beamsets = 1
Subproject commit 774d39a40ca19c9d979ad22565e57b4af3e9a831
Subproject commit dddb23ff587f6e9c837cdb77e7955e94272eca6f
......@@ -28,6 +28,12 @@ services:
- tangodb:/var/lib/mysql
ports:
- "3306:3306"
logging:
driver: syslog
options:
syslog-address: udp://${LOG_HOSTNAME}:1514
syslog-format: rfc3164
tag: "{{.Name}}"
restart: unless-stopped
databaseds:
......@@ -55,4 +61,10 @@ services:
- "2"
- -ORBendPoint
- giop:tcp::10000
logging:
driver: syslog
options:
syslog-address: udp://${LOG_HOSTNAME}:1514
syslog-format: rfc3164
tag: "{{.Name}}"
restart: unless-stopped
......@@ -59,3 +59,40 @@ For more information, see:
- https://huihoo.org/ace_tao/ACE-5.2+TAO-1.2/TAO/docs/ORBEndpoint.html
- http://omniorb.sourceforge.net/omni42/omniNames.html
- https://sourceforge.net/p/omniorb/svn/HEAD/tree/trunk/omniORB/src/lib/omniORB/orbcore/tcp/tcpEndpoint.cc
Logging
-------------------------
The ELK stack collects the logs from the containers, as well as any external processes that send theirs. It is the *Logstash* part of ELK that is responsible for this. The following interfaces are available for this purpose:
+-------------+------------+-------------------------------------------------------------------------------------------------------------+
| Interface | Port | Note |
+=============+============+=============================================================================================================+
| Syslog | 1514/udp | Recommended over TCP, as the ELK stack might be down. |
+-------------+------------+-------------------------------------------------------------------------------------------------------------+
| Syslog | 1514/tcp | |
+-------------+------------+-------------------------------------------------------------------------------------------------------------+
| JSON | 5959/tcp | From python, recommended is the `LogStash Async <https://pypi.org/project/python-logstash-async/>`_ module. |
+-------------+------------+-------------------------------------------------------------------------------------------------------------+
| Beats | 5044/tcp | Use `FileBeat <https://www.elastic.co/beats/filebeat>`_ to watch logs locally, and forward them to ELK. |
+-------------+------------+-------------------------------------------------------------------------------------------------------------+
We recommend making sure the contents of your log lines are parsed correctly, especially if logs are routed to the *Syslog* input. These configurations are stored in ``docker-compose/elk/logstash/conf.d``. An example:
.. literalinclude:: ../../docker-compose/elk/logstash/conf.d/22-parse-tango-rest.conf
Log from Python
`````````````````
The ``common.lofar_logging`` module provides an easy way to log to the ELK stack from a Python Tango device.
Log from Docker
`````````````````
Not all Docker containers run our Python programs, and can forward the logs themselves. For those, we use the ``syslog`` log driver in Docker. Extend the ``docker compose`` files with:
.. literalinclude:: ../../docker-compose/rest.yml
:start-at: logging:
:end-before: restart:
Logs forwarded in this way are provided with the container name, their timestamp, and a log level guessed by Docker. It is thus wise to parse the message content further in Logstash (see above).
......@@ -66,7 +66,9 @@ SSTs/XSTs
Some SSTs/XSTs packets do arrive, but not all, and/or the matrices remain zero?
``````````````````````````````````````````````````````````````````````````````````````````````````````````````
So ``sst.nof_packets_received`` / ``xst.nof_packets_received`` is increasing, telling you packets are arriving. But they're apparently dropped or contain zeroes. First, check the following settings:
So ``sst.nof_packets_received`` / ``xst.nof_packets_received`` is increasing, telling you packets are arriving. But they're apparently dropped or contain zeroes.
The ``sdp.set_defaults()`` command, followed by ``sst.set_defaults()`` / ``xst.set_defaults()``, should reset that device to its default settings, which should result in a working system again. If not, or if the default configuration is not correct, check the following settings:
- ``sdp.TR_fpga_mask_RW[x] == True``, to make sure we're actually configuring the FPGAs,
- ``sdp.FPGA_wg_enable_RW[x] == False``, or the Waveform Generator might be replacing our the antenna data with zeroes,
......@@ -84,7 +86,7 @@ I am not receiving any XSTs and/or SSTs packets from SDP!
Are you sure? If ``sst.nof_packets_received`` / ``xst.nof_packets_received`` is actually increasing, the packets are arriving, but are not parsable by the SST/XST device. If so, see the previous question.
Many settings need to be correct for the statistics emitted by the SDP FPGAs to reach our devices correctly. Here is a brief overview:
The ``sdp.set_defaults()`` command, followed by ``sst.set_defaults()`` / ``xst.set_defaults()``, should reset that device to its default settings, which should result in a working system again. If not, or if the default configuration is not correct, check the following settings:
- ``sdp.TR_fpga_mask_RW[x] == True``, to make sure we're actually configuring the FPGAs,
- ``sdp.FPGA_communication_error_R[x] == False``, to verify the FPGAs can be reached by SDP,
......
......@@ -15,11 +15,11 @@ ELK
To monitor the logs remotely, or to browse older logs, use the *ELK stack* that is included on the station, and served on http://localhost:5601. ELK, or ElasticSearch + Logstash + Kibana, is a popular log collection and querying system. Currently, the following logs are collected in our ELK installation:
- Logs of all devices,
- Logs of the Jupyter notebook server.
- Logs of the Docker containers.
If you browse to the ELK stack (actually, it is Kibana providing the GUI), your go-to is the *Discover* view at http://localhost:5601/app/discover. There, you can construct (and save, load) a dashboard that provides a custom view of the logs, based on the *index pattern* ``logstash-*``. There is a lot to take in, and there are excellent Kibana tutorials on the web.
To get going, use for example `this dashboard <http://localhost:5601/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-60m,to:now))&_a=(columns:!(extra.lofar_id,level,message),filters:!(),index:'1e8ca200-1be0-11ec-a85f-b97e4206c18b',interval:auto,query:(language:kuery,query:''),sort:!())>`_, which shows the logs of the last hour, with some useful columns added to the default timestamp and message columns. Expand the time range if no logs appear, to look further back. You should see something like:
To get going, use for example `this dashboard <http://localhost:5601/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1h,to:now))&_a=(columns:!(extra.lofar_id,program,level,message),filters:!(),index:'1e8ca200-1be0-11ec-a85f-b97e4206c18b',interval:auto,query:(language:kuery,query:'extra.lofar_id.keyword%20:%20*'),sort:!())>`_, which shows the logs of the last hour, with some useful columns added to the default timestamp and message columns. Expand the time range if no logs appear, to look further back. You should see something like:
.. image:: elk_last_hour.png
......
......@@ -9,10 +9,10 @@ fi
# copy file into container to read it from container, as the file's location
# in the container won't be the same as on the host.
docker cp "${file}" dsconfig:/tmp/dsconfig-load-settings.json || exit 1
docker cp "${file}" "${CONTAINER_NAME_PREFIX}"dsconfig:/tmp/dsconfig-update-settings.json || exit 1
# write settings
docker exec -it dsconfig json2tango --write /tmp/dsconfig-load-settings.json
# update settings, Do not change -i into -it this will break integration tests in gitlab ci!
docker exec -i "${CONTAINER_NAME_PREFIX}"dsconfig json2tango --write /tmp/dsconfig-update-settings.json
# somehow json2tango does not return 0 on success
exit 0
......@@ -9,16 +9,22 @@ if [ -z "$LOFAR20_DIR" ]; then
LOFAR20_DIR=$(readlink -f "${LOFAR20_DIR_RELATIVE}")
fi
# Start and stop sequence
cd "$LOFAR20_DIR/docker-compose" || exit 1
# Make sure builds are recent, and use our building parameters.
make build
# Start and stop sequence
make stop device-sdp device-recv device-sst device-unb2 device-xst sdptr-sim recv-sim unb2-sim apsct-sim apspu-sim
make start databaseds dsconfig jupyter elk
make start databaseds dsconfig elk
# Give dsconfig and databaseds time to start
sleep 15
sleep 60
# Update the dsconfig
"${LOFAR20_DIR}"/sbin/update_ConfigDb.sh "${LOFAR20_DIR}"/CDB/integration_ConfigDb.json
# Do not remove `bash`, otherwise statement ignored by gitlab ci shell!
bash "${LOFAR20_DIR}"/sbin/update_ConfigDb.sh "${LOFAR20_DIR}"/CDB/LOFAR_ConfigDb.json
bash "${LOFAR20_DIR}"/sbin/update_ConfigDb.sh "${LOFAR20_DIR}"/CDB/stations/simulators_ConfigDb.json
cd "$LOFAR20_DIR/docker-compose" || exit 1
make start sdptr-sim recv-sim unb2-sim apsct-sim apspu-sim
......@@ -28,12 +34,16 @@ sleep 5
make start device-sdp device-recv device-sst device-unb2 device-xst
# Give the devices time to start
sleep 5
# Give devices time to restart
# TODO(Corne Lukken): Use a nicer more reliable mechanism
sleep 60
# Start the integration test
cd "$LOFAR20_DIR/docker-compose" || exit 1
make start integration-test
# Give devices time to restart
sleep 60
# Run the integration test with the output displayed on stdout
docker start -a integration-test
docker start -a "${CONTAINER_NAME_PREFIX}"integration-test
......@@ -9,10 +9,10 @@ fi
# copy file into container to read it from container, as the file's location
# in the container won't be the same as on the host.
docker cp "${file}" dsconfig:/tmp/dsconfig-update-settings.json || exit 1
docker cp "${file}" "${CONTAINER_NAME_PREFIX}"dsconfig:/tmp/dsconfig-update-settings.json || exit 1
# update settings
docker exec -it dsconfig json2tango --write --update /tmp/dsconfig-update-settings.json
# update settings, Do not change -i into -it this will break integration tests in gitlab ci!
docker exec -i "${CONTAINER_NAME_PREFIX}"dsconfig json2tango --write --update /tmp/dsconfig-update-settings.json
# somehow json2tango does not return 0 on success
exit 0
[DEFAULT]
test_path=${TESTS_DIR:-./tangostationcontrol/test}
top_dir=./
......@@ -3,7 +3,7 @@
This code provides an attribute_wrapper class in place of attributes for tango devices. the attribute wrappers contain additional code
that moves a lot of the complexity and redundant code to the background.
The tango Device class is also abstracted further to a "hardware_device" class. This class wraps
The tango Device class is also abstracted further to a "lofar_device" class. This class wraps
The only things required on the users part are to declare the attributes using the attribute_wrapper (see `example/example_device`),
declare what client the attribute has to use in the initialisation and provide support for the used clients.
......
......@@ -2,4 +2,12 @@
# order of appearance. Changing the order has an impact on the overall
# integration process, which may cause wedges in the gate later.
pbr>=2.0 # Apache-2.0
asyncua >= 0.9.90 # LGPLv3
PyMySQL[rsa] >= 1.0.2 # MIT
sqlalchemy >= 1.4.26 #MIT
GitPython >= 3.1.24 # BSD
snmp >= 0.1.7 # GPL3
h5py >= 3.5.0 # BSD
psutil >= 5.8.0 # BSD
docker >= 5.0.3 # Apache 2
python-logstash-async >= 2.3.0 # MIT
\ No newline at end of file
[metadata]
name = TangoStationControl
name = tangostationcontrol
version = attr: tangostationcontrol.__version__
summary = LOFAR 2.0 Station Control
description_file =
README.md
......@@ -20,11 +21,36 @@ classifier =
Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9
Programming Language :: Python :: 3.10
[files]
package_dir=./
[options]
package_dir=
=./
packages=find:
python_requires = >=3.6
[entry_points]
[options.packages.find]
where=./
[options.entry_points]
console_scripts =
SDP = SDP:main
RECV = RECV:main
l2ss-boot = tangostationcontrol.devices.boot:main
l2ss-docker-device = tangostationcontrol.devices.docker_device:main
l2ss-observation = tangostationcontrol.devices.observation:main
l2ss-observation-control = tangostationcontrol.devices.observation_control:main
l2ss-receiver = tangostationcontrol.devices.recv:main
l2ss-sdp = tangostationcontrol.devices.sdp.sdp:main
l2ss-sst = tangostationcontrol.devices.sdp.sst:main
l2ss-unb2 = tangostationcontrol.devices.unb2:main
l2ss-xst = tangostationcontrol.devices.sdp.xst:main
l2ss-statistics-reader = tangostationcontrol.statistics_writer.statistics_reader:main
l2ss-statistics-writer = tangostationcontrol.statistics_writer.statistics_writer:main
# The following entry points should eventually be removed / replaced
l2ss-cold-start = tangostationcontrol.toolkit.lts_cold_start:main
l2ss-hardware-device-template = tangostationcontrol.examples.HW_device_template:main
l2ss-ini-device = tangostationcontrol.examples.load_from_disk.ini_device:main
l2ss-parse-statistics-packet = tangostationcontrol.devices.sdp.statistics_packet:main
l2ss-random-data = tangostationcontrol.test.devices.random_data:main
l2ss-snmp = tangostationcontrol.examples.snmp.snmp:main
l2ss-version = tangostationcontrol.common.lofar_version:main
import setuptools
with open('requirements.txt') as f:
required = f.read().splitlines()
# Requires: setup.cfg
setuptools.setup(install_requires=required)
from util.lofar_git import get_version
from tangostationcontrol.common.lofar_version import get_version
__version__ = get_version()
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment