## Updates to incorporate Hdbpp-Python as our Retriever
The library in [libhdbpp-python](https://gitlab.com/tango-controls/hdbpp/libhdbpp-python) implements an AbstractReader and multiple Readers (i.e. what is called Retriever in our repo) following the relative hdb++ engine. Currently (March 2022), only MariaDB implementation is on the master branch, while Timescaledb is on development in [libhdbpp-python-timescaledb-branch](https://gitlab.com/tango-controls/hdbpp/libhdbpp-python/-/tree/package_and_timescaledb_support).
### Approach
The Reader relies upon hard-coded SQL scripts inside the Python methods, managed as strings.
Even our first version of the Retriever used this approach, but then we decided to overcome the hardcoded SQL scripts using the SQLAlchemy Python library, which has led to a more stable, reliable and customizable code.
### Compatibility
The libhdbpp-reader is compatible with our code and our archiver setup, as demonstrated in [demonstrator](../../../jupyter-notebooks/HdbppReader_demonstrator.ipynb).
### Functionalities in libhdbpp-python-reader
These are the functionalities implemented in the libhdbpp-reader:
- get_connection() : Return the connection object to avoid a client to open one for custom queries.
- get_attributes(active=False, pattern='') : Queries the database for the current list of archived attributes
- is_attribute_archived(attribute, active=False): Returns if an attribute has values in DB
- get_last_attribute_value(attribute) : Returns last value inserted in DB for an attribute
- get_last_attributes_values(attributes, columns = 'time, r_value'): Returns last values inserted in DB for a list of attributes
- get_attribute_values(attribute,start_date, stop_date=None,decimate=None,**params): Returns attribute values between start and stop dates
- get_attributes_values(attributes,start_date, stop_date=None,decimate=None,correlate = False,columns = 'time, r_value',**params): Returns attributes values between start and stop dates, using decimation or not, correlating the values or not.
### TODO List for our repository
The Reader in libhdpp-python has roughly the same functionalities of ours, but if we need to align our methods to the AbstractReader, we must:
- replace the methods/parameter names
- introduce the concept of active Attribute (boolean flag that indicates if the attribute is being currently archived)
- add the decimate parameter (avg, count, etc..)
- add the correlation parameter ('if True, data is generated so that there is available data for each timestamp of each attribute')
- add a more general pattern parameter to retrieve methods
### TODO List for libhdbpp-python in case of our contribuition
Since we experimented that SQLAlchemy Python library adds many more benefits compared to the use of bare SQL-strings, in case of our contribuition to the libhdbpp-python, these functionalities must be added to the tango-repository:
- install and import SQLAlchemy library
- development of an ArchiverBase class that maps the DBMS-schema (we've had alredy both the TimescaleDB and MariaDB version)
- replacement of the SQL-strings in Reader methods with classes/methods representing relative DB tables/scripts
- Add some small extra functionalities to match our developed methods (get_attribute_format(), get_attribute_tablename(), etc...)
### Linting and other general issues to be fixed in libhdbpp-python code
If we want that pipeline doesn't raise errors we need to fix the following errors:
- imported but unused packages (F401)
- use of bare exceptions (B001)
- use of mutable data structures for argument defaults (B006)
- Xenon complexity of some methods (acceptable for now)
