TMSS

The TMSS project is developing a new software application for the specification, scheduling, and administration of LOFAR observations.

See the TMSS Software Design for more details.

Getting Started

To make changes in the Lofar Repo you need a JIRA-ticket, which are for TMSS typical identified as TMSS-[ID]. For example TMSS-179. Get the code to your local Linux machine

git clone https://git.astron.nl/ro/lofar.git
cd lofar
git checkout -b TMSS-179 master

Now you can make changes in your local TMSS-[ID] branch. When committing changed don't forget to add TMSS-[ID] in your comment

git commit -m "TMSS-179: A pretty nice feature"

Prerequisites

Before you can build the TMSS component, you need to install some packages, like The Django and Postgress. Also you need to be sure that some environment variables are set correctly.
The next example shows install of packages in CentOs. You can do the same for Ubuntu with apt-install, some package names are different

yum install -y epel-release
yum install -y cmake gcc-c++ make log4cplus log4cplus-devel python3 python3-libs python3-devel boost readline-devel boost-devel binutils-devel boost-python36 boost-python36-devel gettext which openldap-devel npm nodejs git java-11-openjdk
yum erase -y postgresql postgresql-server postgresql-devel
yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
yum install -y postgresql96 postgresql96-server postgresql96-devel
RUN pip3 install cython kombu lxml requests pygcn xmljson mysql-connector-python python-dateutil django djangorestframework djangorestframework-xml ldap==1.0.2 flask fabric coverage python-qpid-proton PyGreSQL numpy h5py psycopg2 testing.postgresql Flask-Testing scipy Markdown django-filter python-ldap python-ldap-test ldap3 djangorestframework django-jsonforms django-json-widget django-jsoneditor drf-yasg flex swagger-spec-validator django-auth-ldap mozilla-django-oidc jsonschema

For install postgresql-server > 9.2 on Centos7 you need to check https://www.postgresql.org/download/linux/redhat/ how to to it.

Setting the path

cd $/HOME/lofar
mkdir -p build_tmss/gnucxx11_opt
cd $HOME/lofar/build/gnucxx_opt
export LD_LIBRARY_PATH=/opt/casacore/lib:`pwd`/lib:$LD_LIBRARY_PATH
export PATH=/opt/casacore/bin:`pwd`/bin:`pwd`/sbin:$PATH
export PYTHONPATH=/opt/casacore/lib:`pwd`/lib/python3.6/site-packages:$PYTHONPATH

Installing and running TMSS

Build and install TMSS

cd $/HOME/lofar/build_tmss/gnucxx11_opt
cmake -DBUILD_PACKAGES=TMSS ../..
make -j 5
make install

Running TMSS

Now run a TMSS instance on your local machine:

cd installed 
# That's $HOME/lofar/build/gnucxx11_opt/installed
source ./lofarinit.sh
tmss_test_environment -p 8000

Ok, TMSS should be running, check it at: http://127.0.0.1:8000/api

Running TMSS with MAC Scheduler

One of the clients of TMSS is the MAC Scheduler. The MAC Scheduler will poll TMSS for new observations.

The MAC Scheduler will be build in a docker container. The next steps gives the "MAC build" instructions on your local machine cd to you local source root (that's $HOME/lofar in examples used on this page)

docker build -t ci_base -f Docker/lofar-ci/Dockerfile_ci_base .
docker build -t ci_mac -f Docker/lofar-ci/Dockerfile_ci_mac .
docker run --rm -it --net=host -v $HOME/lofar:$HOME/lofar ci_mac

Now you're running the ci_mac image with your local sources mounted into it. That way you can use your favorite editor on your local machine, and commit changes to git from your local machine using your own credentials etc.Ok, let's build MACScheduler:

Run in the ci_mac docker container, make sure you use another build folder:

cd $HOME/lofar
mkdir -p build_mac/gnucxx11_opt
cd build_mac/gnucxx11_opt
cmake -DBUILD_PACKAGES=MainCU -DUSE_LOG4CPLUS=false -DWINCC_ROOT_DIR=/opt/WinCC_OA/3.14/ -DBLITZ_ROOT_DIR=/opt/blitz/ ../..
make -j 5
make install

Now it's build (in the container) Let's run it (in the container)

cd installed
source ./lofarinit.sh
MACScheduler

Continuous Integration

Every commit/push of code to the remote repo branch in gitlab will result in starting the Continuous Integration gitlab 'build' pipeline which is described in .gitlab-ci.yml This pipeline consist of 6 main stages:

1. Prepare
   • LTA docker image
   • SAS docker image
2. Build
   • LTA ingest
   • RA Services
   • TMSS
3. Unit test
   • LTA ingest
   • RA Services
   • TMSS
4. Integration Test
   • LTA ingest
   • RA Services
   • TMSS
5. Create Docker Container
   • TMSS and push to Nexus Repository
6. Deploy (manual action)
   • TMSS Test
   • TMSS Acceptance

Unit Test

Unit Test can be run with the command ctest that's how the 'build' pipeline will execute it. For development convenience, it is also possible to run the TMSS python unittest in PyCharm IDE.

See the TMSS landscape overview.

Integration Test

Integration tests are python unit test with @integration_test decorator. The session authentication test is a typical integration test for TMSS.

Deployment

The Deployment can only be executed manually by pushing the button after the gitlab pipeline has accomplished all stages of the pipeline successfully. Deployment of TMSS consist of pushing the TMSS docker container to Lofar Test environment scu199.control.lofar and to the Acceptance environment tmss-ua.control.lofar.