diff --git a/tangostationcontrol/tangostationcontrol/toolkit/TODO_HdbppPython.md b/tangostationcontrol/tangostationcontrol/toolkit/TODO_HdbppPython.md new file mode 100644 index 0000000000000000000000000000000000000000..a72618b3fffac8b4c94f6b46d4bb1353db637c67 --- /dev/null +++ b/tangostationcontrol/tangostationcontrol/toolkit/TODO_HdbppPython.md @@ -0,0 +1,47 @@ +# To Do List + +## Updates to incorporate Hdbpp-Python as our Retriever + +The library in [libhdbpp-python](https://gitlab.com/tango-controls/hdbpp/libhdbpp-python) implements an AbstractReader and multiple Readers (i.e. what is called Retriever in our repo) following the relative hdb++ engine. Currently (March 2022), only MariaDB implementation is on the master branch, while Timescaledb is on development in [libhdbpp-python-timescaledb-branch](https://gitlab.com/tango-controls/hdbpp/libhdbpp-python/-/tree/package_and_timescaledb_support). + +### Approach +The Reader relies upon hard-coded SQL scripts inside the Python methods, managed as strings. +Even our first version of the Retriever used this approach, but then we decided to overcome the hardcoded SQL scripts using the SQLAlchemy Python library, which has led to a more stable, reliable and customizable code. + +### Compatibility +The libhdbpp-reader is compatible with our code and our archiver setup, as demonstrated in [demonstrator](../../../jupyter-notebooks/HdbppReader_demonstrator.ipynb). + +### Functionalities in libhdbpp-python-reader +These are the functionalities implemented in the libhdbpp-reader: +- get_connection() : Return the connection object to avoid a client to open one for custom queries. +- get_attributes(active=False, pattern='') : Queries the database for the current list of archived attributes +- is_attribute_archived(attribute, active=False): Returns if an attribute has values in DB +- get_last_attribute_value(attribute) : Returns last value inserted in DB for an attribute +- get_last_attributes_values(attributes, columns = 'time, r_value'): Returns last values inserted in DB for a list of attributes +- get_attribute_values(attribute,start_date, stop_date=None,decimate=None,**params): Returns attribute values between start and stop dates +- get_attributes_values(attributes,start_date, stop_date=None,decimate=None,correlate = False,columns = 'time, r_value',**params): Returns attributes values between start and stop dates, using decimation or not, correlating the values or not. + +### TODO List for our repository +The Reader in libhdpp-python has roughly the same functionalities of ours, but if we need to align our methods to the AbstractReader, we must: +- replace the methods/parameter names +- introduce the concept of active Attribute (boolean flag that indicates if the attribute is being currently archived) +- add the decimate parameter (avg, count, etc..) +- add the correlation parameter ('if True, data is generated so that there is available data for each timestamp of each attribute') +- add a more general pattern parameter to retrieve methods + +### TODO List for libhdbpp-python in case of our contribuition +Since we experimented that SQLAlchemy Python library adds many more benefits compared to the use of bare SQL-strings, in case of our contribuition to the libhdbpp-python, these functionalities must be added to the tango-repository: +- install and import SQLAlchemy library +- development of an ArchiverBase class that maps the DBMS-schema (we've had alredy both the TimescaleDB and MariaDB version) +- replacement of the SQL-strings in Reader methods with classes/methods representing relative DB tables/scripts +- Add some small extra functionalities to match our developed methods (get_attribute_format(), get_attribute_tablename(), etc...) + +### Linting and other general issues to be fixed in libhdbpp-python code +If we want that pipeline doesn't raise errors we need to fix the following errors: +- imported but unused packages (F401) +- use of bare exceptions (B001) +- use of mutable data structures for argument defaults (B006) +- Xenon complexity of some methods (acceptable for now) + + +