Merge branch 'L2SS-235-Archive-PCC-attribute' into 'master'

Resolve L2SS-235 "Archive pcc attribute"

Closes L2SS-235 and L2SS-236

See merge request !108

devices/toolkit/README.md

+# Tango Archiving Framework
+
+The Archiver class in archiver.py defines the methods to manage the device attributes archiving allowed by Tango.
+
+The main components (and the relative Docker containers) are:
+
+- Configuration Manager (container: hdbpp-cm): Device server that assists in adding, modifying, moving, deleting an Attribute to/from the archiving system
+- Event Subscriber (container: hdbpp-es): The EventSubscriber TANGO device server, is the archiving system engine. On typical usage, it will subscribe to archive events on request by the ConfigurationManager device. The EventSubscriber is designed to start archiving all the already configured Attributes, even if the ConfigurationManager is not running. Moreover, being a TANGO device, the EventSubscriber configuration can be managed with Jive.
+- Archiving DBMS (container: archiver-maria-db): Specific Database devoted to storing attribute values.
+- (Optional) HDB++ Viewer (container: hdbpp-viewer): Standalone JAVA application designed to monitor signals coming from database
+
+## Archiver creation
+When an Archiver object is created, we can define three of its properties:
+- the ConfigurationManager name (Tango namespace)
+- at least one EventSubscriber name (Tango namespace)
+- the default context archiving for the subscribers. This means that a default archiving strategy will be applied to
+all the attributes. Of course this strategy can be tuned individually for each attribute if needed.
+Archiving strategies are ['ALWAYS','RUN','SHUTDOWN','SERVICE']
+- ALWAYS:always stored
+- RUN:stored during run
+- SHUTDOWN:stored during shutdown
+- SERVICE:stored during maintenance activities
+
+## Add an attribute
+When adding an attribute to the archiving framework, we must define the following properties:
+- the EventSubscriber name that will take charge of the attribute
+- the archiving strategy (4 options defined above)
+- the attribute polling period (it should have been already defined in TangoDB)
+- the archive event period (MOST IMPORTANT, it defines the frequency rate at which an attribute is archived in the DBMS)
+
+It is important to understand that, when an attribute is successfully added to the EventSubscriber list, the archiving begins without an explicit 'Start' command, rather it follows the archiving strategy already defined.
+
+The 'Start' command is used instead during a session when an attribute has been paused/stopped for any reason, or it has raised some kind of issue.
+
+## Difference between Stop and Remove an attribute
+When stopping an attribute archiving, the framework does not remove it from the list.
+This means that archiving is stopped for the current session, but if the device is restarted,  the attribute archiving will be restarted as well.
+In order to definitely stop the archiving, the attribute must be removed from the attribute list.
+
+## Update an attribute
+If we want to update the archiving properties of an attribute (e.g. the archive event period), there is a relative method.
+It must be noted that the updating is not istantaneous because, following the framework architecture, an attribute must be first removed from the EventSubscriber list and then re-added with the new properties.

devices/toolkit/archiver.py

 #! /usr/bin/env python3
 
+from logging import raiseExceptions
+import traceback
 from clients.attribute_wrapper import attribute_wrapper
-from tango import DeviceProxy
+from tango import DeviceProxy, AttributeProxy
 from datetime import datetime, timedelta
 
+import time
 from sqlalchemy import create_engine, and_
 from sqlalchemy.orm import sessionmaker
 from .archiver_base import *
@@ -12,32 +15,216 @@ class Archiver():
     """
     The Archiver class implements the basic operations to perform attributes archiving
     """
-    def __init__(self, cm_name: str = 'archiving/hdbpp/confmanager01', es_name: str = 'archiving/hdbpp/eventsubscriber01'):
+    def __init__(self, cm_name: str = 'archiving/hdbpp/confmanager01', es_name: str = 'archiving/hdbpp/eventsubscriber01', context: str = 'RUN'):
         self.cm_name = cm_name
         self.cm = DeviceProxy(cm_name)
+        try: 
+            cm_state = self.cm.state() # ping the device server
+            if cm_state is 'FAULT':
+                print('Configuration Manager is in FAULT state')
+                print(self.cm.status())
+                return
+        except:
+            print(traceback.format_exc())
+            return
         self.es_name = es_name
         self.es = DeviceProxy(es_name)
+        self.cm.write_attribute('Context',context)    # Set default Context Archiving for all the subscribers
 
-    def add_attribute_to_archiver(self, attribute: str, polling_period: float = 1000, event_period: float = 1000, strategy: str = 'ALWAYS'):
+    def add_attribute_to_archiver(self, attribute_name: str, polling_period: int = 1000, event_period: int = 1000, strategy: str = 'RUN'):
         """
         Takes as input the attribute name, polling period (ms), event period (ms) and archiving strategy, 
         and adds the selected attribute to the subscriber's list of archiving attributes.
         The ConfigurationManager and EventSubscriber devices must be already up and running.
         The archiving-DBMS must be already properly configured.
         """
-        self.cm.write_attribute('SetAttributeName', attribute)
-        self.cm.write_attribute('SetArchiver', self.es_name)
-        self.cm.write_attribute('SetStrategy', strategy)
-        self.cm.write_attribute('SetPollingPeriod', int(polling_period))
-        self.cm.write_attribute('SetPeriodEvent', int(event_period))
-        self.cm.AttributeAdd()
+        if (len(attribute_name.split('/'))!=4): 
+            raise AttributeFormatException 
+        try:
+            self.cm.write_attribute('SetAttributeName', attribute_name)
+            self.cm.write_attribute('SetArchiver', self.es_name)
+            self.cm.write_attribute('SetStrategy', strategy)
+            self.cm.write_attribute('SetPollingPeriod', polling_period)
+            self.cm.write_attribute('SetPeriodEvent', event_period)
+            self.cm.AttributeAdd()
+            print('Attribute %s added to archiving list!' % attribute_name)
+        except Exception as e:
+            if 'already archived' not in str(e).lower():
+                traceback.format_exc()
+            else: 
+                print('Attribute %s already in archiving list!' % attribute_name)
 
-    def remove_attribute_from_archiver(self, attribute: str):
+    def add_attributes_to_archiver(self,device_name,global_archive_period:int = None, exclude:list = ['Status','State']):
+        """
+        Add sequentially all the attributes of the selected device in the event subscriber list, if not already present
+        """
+        d = DeviceProxy(device_name)
+        attrs_list = list(d.get_attribute_list()) # cast to list otherwise removal is not allowed
+        try: 
+            for a in exclude: attrs_list.remove(a)
+        except:
+            pass
+        for a in attrs_list:
+            attr_fullname = str(device_name+'/'+a).lower()
+            attr_proxy = AttributeProxy(attr_fullname)
+            if attr_proxy.is_polled() is True:   # if not polled attribute is also not archived
+                try:
+                    if self.es.AttributeList is None or not(self.cm.AttributeSearch(a)):
+                        polling_period = attr_proxy.get_poll_period()  
+                        archive_period = global_archive_period or int(attr_proxy.get_property('archive_period')['archive_period'][0])                 
+                        self.add_attribute_to_archiver(attr_fullname,polling_period=polling_period,
+                            event_period=archive_period)
+                        #time.sleep(0.5)
+                except:
+                    print(traceback.format_exc())
+    
+    def remove_attribute_from_archiver(self, attribute_name:str):
         """
         Stops the data archiving of the attribute passed as input, and remove it from the subscriber's list. 
         """
-        self.cm.AttributeStop(attribute)
-        self.cm.AttributeRemove(attribute)       
+        if (len(attribute_name.split('/'))!=4): 
+            raise AttributeFormatException 
+        try:
+            self.cm.AttributeStop(attribute_name)
+            self.cm.AttributeRemove(attribute_name)
+            print('Attribute %s removed!' % attribute_name)
+        except Exception as e:
+            if 'attribute not found' not in str(e).lower():
+                traceback.format_exc()
+            else: 
+                print('Attribute %s not found!' % attribute_name)      
+    
+    def remove_attributes_by_device(self,device_name:str):
+        """
+        Stops the data archiving of all the attributes of the selected device, and remove them from the
+        subscriber's list
+        """
+        d = DeviceProxy(device_name)
+        attrs_list = d.get_attribute_list()
+        for a in attrs_list:
+            try:
+                attr_fullname = str(device_name+'/'+a).lower()
+                self.remove_attribute_from_archiver(attr_fullname)
+            except:
+                print(traceback.format_exc())
+
+    def start_archiving_attribute(self, attribute_name:str):
+        """
+        Starts the archiving of the attribute passed as input.
+        The attribute must be already present in the subscriber's list
+        """
+        if (len(attribute_name.split('/'))!=4): 
+            raise AttributeFormatException 
+        try:
+            self.cm.AttributeStart(attribute_name)
+        except Exception as e:
+            if 'attribute not found' not in str(e).lower():
+                traceback.format_exc()
+            else: 
+                print('Attribute %s not found!' % attribute_name)
+    
+    def stop_archiving_attribute(self, attribute_name:str):
+        """
+        Stops the archiving of the attribute passed as input.
+        The attribute must be already present in the subscriber's list
+        """
+        if (len(attribute_name.split('/'))!=4): 
+            raise AttributeFormatException
+        try:
+            self.cm.AttributeStop(attribute_name)
+        except Exception as e:
+            if 'attribute not found' not in str(e).lower():
+                traceback.format_exc()
+            else: 
+                print('Attribute %s not found!' % attribute_name)
+    
+    def check_and_add_attribute_in_archiving_list(self, attribute_name:str):
+        """
+        Check if an attribute is in the archiving list
+        """
+        if (len(attribute_name.split('/'))!=4): 
+            raise AttributeFormatException
+        # Add attribute if not present in event subscriber list
+        try:
+            if self.es.AttributeList is None or not(self.cm.AttributeSearch(attribute_name)):
+                self.add_attribute_to_archiver(attribute_name)
+        except:
+            print(traceback.format_exc())
+        return attribute_name
+    
+    def update_archiving_attribute(self, attribute_name: str, polling_period: int = 1000, event_period: int = 1000, strategy: str = 'RUN'):
+        """
+        Update the archiving properties of an attribute already in a subscriber list
+        """
+        try:
+            self.remove_attribute_from_archiver(attribute_name)
+            time.sleep(1)
+            self.add_attribute_to_archiver(attribute_name,polling_period,event_period,strategy)
+            time.sleep(1)
+            self.start_archiving_attribute(attribute_name)
+        except:
+            print(traceback.format_exc())
+    
+    def get_subscriber_attributes(self,es_name:str = None):
+        """
+        Return the list of attributes managed by the event subscriber
+        """
+        if es_name is not None:
+            es = DeviceProxy(es_name)
+        else: 
+            es = self.es
+        attrs = es.AttributeList or []
+        return attrs
+    
+    def get_subscriber_errors(self,es_name:str = None):
+        """
+        Return a dictionary of the attributes currently in error, defined as AttributeName -> AttributeError
+        """
+        if es_name is not None:
+            es = DeviceProxy(es_name)
+        else: 
+            es = self.es
+        try:
+            attrs = es.AttributeList or []
+            errs = es.AttributeErrorList or []
+            return dict((a,e) for a,e in zip(attrs,errs) if e)
+        except:
+            print('No attribute errors in the subscriber')
+            return {}
+    
+    def get_attribute_errors(self,attribute_name:str):
+        """
+        Return the error related to the attribute
+        """
+        if (len(attribute_name.split('/'))!=4): 
+            raise AttributeFormatException
+        errs_dict = self.get_subscriber_errors()
+        for e in errs_dict:
+            if attribute_name in e:
+                return errs_dict.get(e)
+        return None
+    
+    def get_subscriber_load(self,use_freq:bool=True,es_name:str = None):
+        """
+        Return the estimated load of an archiver, in frequency of records or number
+        of attributes
+        """
+        if es_name is not None:
+            es = DeviceProxy(es_name)
+        else: 
+            es = self.es
+        if use_freq:
+            return str(es.AttributeRecordFreq)+(' events/period' )
+        else:
+            return len(es.AttributeList or []) 
+    
+class AttributeFormatException(Exception):
+    """
+    Exception that handles wrong attribute naming
+    """
+    def __init__(self, message="Wrong Tango attribute format! Try: domain/family/member/attribute (e.g. LTS/PCC/1/temperature)"):
+        self.message = message
+        super().__init__(self.message)
 
 class Retriever():
     """

docker-compose/lofar-device-base/lofar-requirements.txt

@@ -2,3 +2,5 @@ opcua >= 0.98.9
 astropy
 python-logstash-async
 gitpython
+PyMySQL[rsa]
+sqlalchemy

jupyter-notebooks/PCC_archive_all_attributes.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "3191bdf1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sys, time\n",
+    "import numpy as np\n",
+    "sys.path.append('/hosthome/tango/devices')\n",
+    "from toolkit.archiver import Archiver,Retriever\n",
+    "from toolkit.archiver_base import *\n",
+    "from matplotlib import pyplot as plt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "e2d12232",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "True\n"
+     ]
+    }
+   ],
+   "source": [
+    "from common.lofar_environment import isProduction\n",
+    "print(isProduction())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "81e08b9f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[]"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "archiver = Archiver()\n",
+    "archiver.get_subscriber_attributes()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "884ff1ff",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "OFF\n"
+     ]
+    }
+   ],
+   "source": [
+    "device_name = 'LTS/PCC/1'\n",
+    "d=DeviceProxy(device_name) \n",
+    "state = str(d.state())\n",
+    "print(state)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "0f6e65b0",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Attribute lts/pcc/1/ant_mask_rw added to archiving list!\n",
+      "Attribute lts/pcc/1/clk_enable_pwr_r added to archiving list!\n",
+      "Attribute lts/pcc/1/clk_i2c_status_r added to archiving list!\n",
+      "Attribute lts/pcc/1/clk_pll_error_r added to archiving list!\n",
+      "Attribute lts/pcc/1/clk_pll_locked_r added to archiving list!\n",
+      "Attribute lts/pcc/1/clk_monitor_rate_rw added to archiving list!\n",
+      "Attribute lts/pcc/1/clk_translator_busy_r added to archiving list!\n",
+      "Attribute lts/pcc/1/hba_element_beamformer_delays_r added to archiving list!\n",
+      "Attribute lts/pcc/1/hba_element_beamformer_delays_rw added to archiving list!\n",
+      "Attribute lts/pcc/1/hba_element_led_r added to archiving list!\n",
+      "Attribute lts/pcc/1/hba_element_led_rw added to archiving list!\n",
+      "Attribute lts/pcc/1/hba_element_lna_pwr_r added to archiving list!\n",
+      "Attribute lts/pcc/1/hba_element_lna_pwr_rw added to archiving list!\n",
+      "Attribute lts/pcc/1/hba_element_pwr_r added to archiving list!\n",
+      "Attribute lts/pcc/1/hba_element_pwr_rw added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_adc_lock_r added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_attenuator_r added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_attenuator_rw added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_band_r added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_band_rw added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_i2c_status_r added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_id_r added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_led0_r added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_led0_rw added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_led1_r added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_led1_rw added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_mask_rw added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_monitor_rate_rw added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_pwr_dig_r added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_temperature_r added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_translator_busy_r added to archiving list!\n",
+      "Attribute lts/pcc/1/rcu_version_r added to archiving list!\n",
+      "Device is now in ON state\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Start the device\n",
+    "if state == \"OFF\":\n",
+    "    if isProduction():\n",
+    "        archiver.add_attributes_to_archiver(device_name,global_archive_period=1000)\n",
+    "    else:\n",
+    "        archiver.remove_attributes_by_device(device_name)\n",
+    "    time.sleep(1)\n",
+    "    d.initialise()\n",
+    "    time.sleep(1)\n",
+    "state = str(d.state())\n",
+    "if state == \"STANDBY\":\n",
+    "    d.on()\n",
+    "state = str(d.state())\n",
+    "if state == \"ON\":\n",
+    "    print(\"Device is now in ON state\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "8efd3dc1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "('tango://databaseds:10000/lts/pcc/1/ant_mask_rw',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/clk_enable_pwr_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/clk_i2c_status_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/clk_pll_error_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/clk_pll_locked_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/clk_monitor_rate_rw',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/clk_translator_busy_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/hba_element_beamformer_delays_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/hba_element_beamformer_delays_rw',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/hba_element_led_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/hba_element_led_rw',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/hba_element_lna_pwr_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/hba_element_lna_pwr_rw',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/hba_element_pwr_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/hba_element_pwr_rw',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_adc_lock_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_attenuator_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_attenuator_rw',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_band_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_band_rw',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_i2c_status_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_id_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_led0_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_led0_rw',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_led1_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_led1_rw',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_mask_rw',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_monitor_rate_rw',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_pwr_dig_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_temperature_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_translator_busy_r',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_version_r')"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "archiver.get_subscriber_attributes()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "a1222d19",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'tango://databaseds:10000/lts/pcc/1/clk_enable_pwr_r': 'Read value for attribute CLK_Enable_PWR_R has not been updated',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/clk_i2c_status_r': 'Read value for attribute CLK_I2C_STATUS_R has not been updated',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/clk_pll_error_r': 'Read value for attribute CLK_PLL_error_R has not been updated',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/clk_pll_locked_r': 'Read value for attribute CLK_PLL_locked_R has not been updated',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/clk_translator_busy_r': 'Read value for attribute CLK_translator_busy_R has not been updated',\n",
+       " 'tango://databaseds:10000/lts/pcc/1/rcu_version_r': 'Storing Error: mysql_stmt_bind_param() failed, err=Buffer type is not supported'}"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Archiver managing methods\n",
+    "archiver.get_subscriber_errors()\n",
+    "\n",
+    "#e = archiver.get_attribute_errors('lts/pcc/1/rcu_temperature_r')\n",
+    "#print(e)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "174bbcdb",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "1586.0 events/period  -> Number of archiving events per minute\n"
+     ]
+    }
+   ],
+   "source": [
+    "l = archiver.get_subscriber_load()\n",
+    "print(l,\" -> Number of archiving events per minute\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f060b0b6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#archiver.update_archiving_attribute('lts/pcc/1/rcu_pwr_dig_r')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f626d029",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Turn off the device\n",
+    "d.off()\n",
+    "\n",
+    "# Leave commented by default\n",
+    "archiver.remove_attributes_by_device(device_name)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "13c3b97d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initialise the retriever object and print the archived attributes in the database\n",
+    "retriever = Retriever()\n",
+    "#retriever.get_all_archived_attributes()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f176c20e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Retrieve records in the last n hours (works even with decimals)\n",
+    "\n",
+    "# Use alternatively one of the following two methods to retrieve data (last n hours or interval)\n",
+    "records= retriever.get_attribute_value_by_hours(attr_fq_name,hours=0.1)\n",
+    "#records = retriever.get_attribute_value_by_interval(attr_fq_name,'2021-09-01 16:00:00', '2021-09-01 16:03:00')\n",
+    "\n",
+    "if not records:\n",
+    "    print('Empty result!')\n",
+    "else:\n",
+    "    # Convert DB Array records into Python lists\n",
+    "    data = build_array_from_record(records,records[0].dim_x_r)\n",
+    "    # Extract only the value from the array \n",
+    "    array_values = get_values_from_record(data)\n",
+    "\n",
+    "#records\n",
+    "#data\n",
+    "#array_values"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "StationControl",
+   "language": "python",
+   "name": "stationcontrol"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter-notebooks/PCC_archive_attribute.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "42e7f25a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sys, time\n",
+    "import numpy as np\n",
+    "sys.path.append('/hosthome/tango/devices')\n",
+    "from toolkit.archiver import Archiver,Retriever\n",
+    "from toolkit.archiver_base import *\n",
+    "from matplotlib import pyplot as plt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1f025912",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from common.lofar_environment import isProduction\n",
+    "print(isProduction())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e0656e2d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Define an attribute for archiving\n",
+    "device_name = 'LTS/PCC/1'\n",
+    "d=DeviceProxy(device_name) \n",
+    "state = str(d.state())\n",
+    "print(device_name,'is',state)\n",
+    "\n",
+    "archiver = Archiver()\n",
+    "\n",
+    "# Attribute chosen to be archived\n",
+    "attr_name = 'rcu_temperature_r'\n",
+    "attr_fq_name = str(device_name+'/'+attr_name).lower()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "153d9420",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Print the list of the attributes in the event subscriber\n",
+    "# If any attribute is present, its archiving will begin when device will reach ON state,\n",
+    "# Otherwise, attribute will be added to the list at the device initializing phase only in PRODUCTION mode\n",
+    "archiver.get_subscriber_attributes()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2ebb00f8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Start the device\n",
+    "if state == \"OFF\":\n",
+    "    if isProduction():\n",
+    "        archiver.check_and_add_attribute_in_archiving_list(attr_fq_name)\n",
+    "    else:\n",
+    "        archiver.remove_attribute_from_archiver(attr_fq_name)\n",
+    "    time.sleep(1)\n",
+    "    d.initialise()\n",
+    "    time.sleep(1)\n",
+    "state = str(d.state())\n",
+    "if state == \"STANDBY\":\n",
+    "    d.on()\n",
+    "state = str(d.state())\n",
+    "if state == \"ON\":\n",
+    "    print(\"Device is now in ON state\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "75163627",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Modify attribute archiving features\n",
+    "archiver.update_archiving_attribute(attr_fq_name,polling_period=1000,event_period=5000,strategy='RUN')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7814715e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Add attribute to the archiving list (starts the archiving if device is running)\n",
+    "\n",
+    "# Archiving strategies are ['ALWAYS','RUN','SHUTDOWN','SERVICE']\n",
+    "#Read [0]\tALWAYS:always stored\n",
+    "#Read [1]\tRUN:stored during run\n",
+    "#Read [2]\tSHUTDOWN:stored during shutdown\n",
+    "#Read [3]\tSERVICE:stored during maintenance activities\n",
+    "\n",
+    "archiver.add_attribute_to_archiver(attr_fq_name, polling_period=1000, event_period=1000, strategy='RUN')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "52a27abb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Stop the attribute archiving but do not remove it from the list\n",
+    "# This means that archiving is stopped for the current session, but if the device is restarted, \n",
+    "# the attribute archiving will be restarted as well\n",
+    "# In order to definitely stop the archiving, the attribute must be removed from the attribute list (go to last cell)\n",
+    "archiver.stop_archiving_attribute(attr_fq_name)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c064e337",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Starts the attribute archiving if it was stopped\n",
+    "archiver.start_archiving_attribute(attr_fq_name)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d199916c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initialise the retriever object and print the archived attributes in the database\n",
+    "retriever = Retriever()\n",
+    "retriever.get_all_archived_attributes()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "80e2a560",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Retrieve records in the last n hours (works even with decimals)\n",
+    "\n",
+    "# Use alternatively one of the following two methods to retrieve data (last n hours or interval)\n",
+    "records= retriever.get_attribute_value_by_hours(attr_fq_name,hours=0.1)\n",
+    "#records = retriever.get_attribute_value_by_interval(attr_fq_name,'2021-09-01 16:00:00', '2021-09-01 16:03:00')\n",
+    "\n",
+    "if not records:\n",
+    "    print('Empty result!')\n",
+    "else:\n",
+    "    # Convert DB Array records into Python lists\n",
+    "    data = build_array_from_record(records,records[0].dim_x_r)\n",
+    "    # Extract only the value from the array \n",
+    "    array_values = get_values_from_record(data)\n",
+    "\n",
+    "#records\n",
+    "#data\n",
+    "#array_values"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "64c8e060",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Extract and process timestamps for plotting purposes\n",
+    "def get_timestamps(data,strformat):\n",
+    "    timestamps = []\n",
+    "    for i in range(len(data)):\n",
+    "        timestamps.append(data[i][0].recv_time.strftime(strformat))\n",
+    "    return timestamps\n",
+    "timestamps = get_timestamps(data,\"%Y-%m-%d %X\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "59a0c05c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Plot of array values\n",
+    "\n",
+    "heatmap = np.array(array_values,dtype=np.float)\n",
+    "fig = plt.figure()\n",
+    "plt.rcParams['figure.figsize'] = [128, 64]\n",
+    "#plt.rcParams['figure.dpi'] = 128\n",
+    "ax = fig.add_subplot(111)\n",
+    "im = ax.imshow(heatmap, interpolation='nearest',cmap='coolwarm')\n",
+    "ax.set_xlabel('Array index')\n",
+    "ax.set_ylabel('Timestamp')\n",
+    "ax.set_xlim([0,(records[0].dim_x_r)-1])\n",
+    "ax.set_xticks(np.arange(0,records[0].dim_x_r))\n",
+    "\n",
+    "ax.set_yticks(range(0,len(timestamps)))\n",
+    "ax.set_yticklabels(timestamps,fontsize=4)\n",
+    "\n",
+    "# Comment the previous two lines and uncomment the following line if there are too many timestamp labels\n",
+    "#ax.set_yticks(range(0,len(timestamps),10))\n",
+    "\n",
+    "ax.set_title('Archived data for '+ attr_fq_name)\n",
+    "ax.grid()\n",
+    "cbar = fig.colorbar(ax=ax, mappable=im, orientation='horizontal')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1c753ed9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Count number of archive events per minute\n",
+    "archiver.get_subscriber_load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a0e8dcab",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Turn off the device\n",
+    "d.off()\n",
+    "# Remove attribute from archiving list\n",
+    "#archiver.remove_attribute_from_archiver(attr_fq_name)\n",
+    "#archiver.remove_attributes_by_device(device_name)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "StationControl",
+   "language": "python",
+   "name": "stationcontrol"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}