Skip to content
Snippets Groups Projects
Commit 8367a3ac authored by Jan David Mol's avatar Jan David Mol
Browse files

Prepopulate some notebooks

parent 93a832a8
No related branches found
No related tags found
No related merge requests found
......@@ -36,6 +36,8 @@ RUN fix-permissions /home/${NB_USER}/.jupyter/lab
USER ${NB_UID}
COPY default-notebooks/* /home/${NB_USER}/
WORKDIR "${HOME}"
# Configure using git from Jupyter Lab. Since the web service is open, we can't know who is
......
%% Cell type:markdown id:1a806252-c92a-4d4f-b74d-b79b0b782cf8 tags:
# Station State Transitions
The station can be in the following states:
* `OFF`: Everything is powered off. This is also the state the software assumes when powering up (after roll out or LCU reboot),
* `HIBERNATE`: Connections to translators are on line. System is monitored.
* `STANDBY`: Critical systems are powered and initialised.
* `ON`: Everything is powered and initialised. Station is ready for observations.
Valid transitions are: `OFF` -> `HIBERNATE` <-> `STANDBY` <-> `ON`. If the LCU is booted and nothing is known, the following transitions fully reset a station: `OFF` -> `HIBERNATE` -> `STANDBY` -> `HIBERNATE` -> `STANDBY` -> `ON` as that will force powering key hardware off before turning everything on. This ensures that the final `HIBERNATE` -> `STANDBY` -> `ON` starts with a consistent state.
The `stationmanager` device is used to query about the station state and to perform transitions. It exposes the following:
* `stationmanager.station_hibernate()`: Transition to `HIBERNATE`,
* `stationmanager.station_standby()`: Transition to `STANDBY`,
* `stationmanager.station_off()`: Transition to `OFF`,
* `stationmanager.station_state_R.name`: Current/last station state,
* `stationmanager.station_state_transitioning_R`: Station is currently transitioning between states,
* `stationmanager.requested_station_state_R`: Requested station state (different from current when transitioning),
* `stationmanager.last_requested_transition_exceptions_R`: Exceptions thrown by last transition. This is filled when the transition has finished/stopped.
%% Cell type:code id:7e3f8189-862f-40b8-b8ed-80605200880c tags:
``` python
# Enable write access for the whole notebook
write_access.enable()
```
%% Cell type:code id:dc277fdc-0e1f-4e51-92a3-383d1f781be6 tags:
``` python
# Helper functions to track state transitions
import time
def timeit():
"""Decorator that prints how long the decorated function took."""
def wrapper(func):
def inner(*args, **kwargs):
start = time.monotonic()
try:
return func(*args, **kwargs)
finally:
end = time.monotonic()
print(f"Time elapsed in {func.__name__}: {end-start:.2f} seconds")
return inner
return wrapper
def report_transition_until_done():
if stationmanager.station_state_transitioning_R:
print(f"Transitioning {stationmanager.station_state_r.name} -> {stationmanager.requested_station_state_r.name}", end="", flush=True)
while stationmanager.station_state_transitioning_R:
print(".", end="",flush=True)
time.sleep(1)
print()
error_str = ''.join(['\n * %s' % ex for ex in stationmanager.last_requested_transition_exceptions_R])
if error_str:
print(f"In state {stationmanager.station_state_r.name}. Last transition errors: {error_str}")
else:
print(f"In state {stationmanager.station_state_r.name}. No errors occurred.")
@timeit()
def transition(func):
try:
# call transition function
func()
except DevFailed as ex:
if ex.args[-1].reason != "API_DeviceTimedOut":
raise
# report until done
report_transition_until_done()
```
%% Cell type:markdown id:cab12787-53e2-4118-bbbf-de5806b3cd92 tags:
We need to go OFF->HIBERNATE->STANDBY->HIBERNATE->STANDBY->ON:
* Most of the time, the station is actually in "ON" but SC rebooted.
* STANDBY->HIBERNATE is needed to explicitly turn the hardware off, which in turn is required to turn it on consistently.
%% Cell type:code id:e157803f-ab26-4a3e-b402-8f5cd251e769 tags:
``` python
# Transition to HIBERNATE
transition(stationmanager.station_hibernate)
```
%% Output
In state HIBERNATE. Last transition errors:
* UNB2(stat/unb2/h0): DevFailed: Failed to execute command_inout on device stat/unb2/h0, command power_hardware_off
Time elapsed in transition: 0.01 seconds
%% Cell type:code id:ef3c86a6-da73-40e1-bdf1-04c0dc2080ad tags:
``` python
# Transition to STANDBY
transition(stationmanager.station_standby)
```
%% Output
Transitioning ON -> STANDBY.........................
In state STANDBY. No errors occurred.
Time elapsed in transition: 85.21 seconds
%% Cell type:code id:31448896-16cc-44b2-84aa-f8a4faee75e2 tags:
``` python
# Transition to HIBERNATE
transition(stationmanager.station_hibernate)
```
%% Output
In state HIBERNATE. No errors occurred.
Time elapsed in transition: 16.19 seconds
%% Cell type:code id:672cba78-159b-4d74-8917-ae8ad2ad04a1 tags:
``` python
# Transition to STANDBY
transition(stationmanager.station_standby)
```
%% Output
Transitioning ON -> STANDBY......................................................
In state STANDBY. No errors occurred.
Time elapsed in transition: 114.23 seconds
%% Cell type:code id:7fd714c3-c0c2-4452-a298-8068538c2ad2 tags:
``` python
# Transition to ON
transition(stationmanager.station_on)
```
%% Output
Transitioning STANDBY -> ON..................................................................................................
In state ON. Last transition errors:
* EC(stat/ec/1): DevFailed: Failed to execute command_inout on device stat/ec/1, command initialise
Time elapsed in transition: 158.39 seconds
%% Cell type:markdown id:745c3ec8-17fa-46e6-95aa-b62d2b7a629c tags:
# Station Verification
The cells below verify various aspects of a working station.
%% Cell type:code id:3c3b9418-4230-4433-9e8a-d40168d0ecc9 tags:
``` python
# Read the current station state, should be ON
stationmanager.station_state_r
```
%% Output
<station_state_R.ON: 3>
%% Cell type:code id:6c736f5c-f7d7-416d-b39e-8a5c6d394ce8 tags:
``` python
# Check for errors. Acceptable are:
# EC(stat/ec/1): DevFailed: Failed to execute command_inout on device stat/ec/1, command initialise
stationmanager.last_requested_transition_exceptions_R
```
%% Output
('EC(stat/ec/1): DevFailed: Failed to execute command_inout on device stat/ec/1, command initialise',)
%% Cell type:code id:418f187d-3cdf-4363-8c43-46cce110e8d0 tags:
``` python
# Check if statistics arrive. All these counters should be non-zero
for b in [bst_l, bst_h0, bst_h1, bst_h, sst_l, sst_h0, sst_h1, sst_h, xst_l, xst_h0, xst_h1, xst_h]:
if b is not None:
print(f"{b}: {b.nof_packets_received_r}")
```
%% Output
BST(stat/bst/lba): 1933
BST(stat/bst/hba0): 1929
BST(stat/bst/hba1): 1925
SST(stat/sst/lba): 371136
SST(stat/sst/hba0): 92592
SST(stat/sst/hba1): 92352
XST(stat/xst/lba): 277776
XST(stat/xst/hba0): 23100
XST(stat/xst/hba1): 23040
%% Cell type:code id:79554a1e-9770-4d26-804e-9a53d2554631 tags:
``` python
# Check if tracking works. All these should report True
for d in [digitalbeam_h0, digitalbeam_h1, digitalbeam_h, digitalbeam_l, tilebeam_h0, tilebeam_h1, tilebeam_h]:
if d is not None:
print(f"{d}: {d.tracking_enabled_r}")
```
%% Output
DigitalBeam(stat/digitalbeam/hba0): True
DigitalBeam(stat/digitalbeam/hba1): True
DigitalBeam(stat/digitalbeam/lba): True
TileBeam(stat/tilebeam/hba0): True
TileBeam(stat/tilebeam/hba1): True
%% Cell type:code id:69078849-7ecf-4f4f-83d4-0c052e3052dc tags:
``` python
# Verify that Prometheus is working, and everything is scraped. Result should all be >0
from http import HTTPStatus
import urllib
import requests
import json
import datetime
# construct query
end = datetime.datetime.now()
start = end - datetime.timedelta(minutes=5)
query=urllib.parse.quote("scrape_samples_scraped")
url=f"http://prometheus.service.consul:9090/api/v1/query_range?query={query}&start={start.strftime('%Y-%m-%dT%H:%M:%SZ')}&end={end.strftime('%Y-%m-%dT%H:%M:%SZ')}&step=60"
http_result=requests.get(url)
# report what we fetched
print(url,"=>",http_result)
# parse results
result = json.loads(http_result.text)
scrape_samples_scraped = {row['metric']['instance']: row['values'][0][1] for row in result['data']['result']}
from pprint import pprint
pprint(scrape_samples_scraped)
```
%% Output
http://prometheus.service.consul:9090/api/v1/query_range?query=scrape_samples_scraped&start=2024-07-12T08:13:56Z&end=2024-07-12T08:18:56Z&step=60 => <Response [200]>
{'chrony-exporter': '99',
'consul.service.consul:8500': '930',
'device-afh': '6314',
'device-afl': '2791',
'device-aps': '177',
'device-apsct': '264',
'device-apspu': '294',
'device-beamlet': '2160',
'device-bst': '567',
'device-calibration': '89',
'device-ccd': '102',
'device-configuration': '71',
'device-digitalbeam': '5031',
'device-ec': '36',
'device-metadata': '82',
'device-observationcontrol': '3262',
'device-pcon': '68',
'device-psoc': '68',
'device-recvh': '5215',
'device-recvl': '3784',
'device-sdp': '3696',
'device-sdpfirmware': '1287',
'device-sst': '615',
'device-stationmanager': '85',
'device-temperaturemanager': '72',
'device-tilebeam': '380',
'device-unb2': '1335',
'device-xst': '1260',
'grafana': '2746',
'jupyter': '955',
'lcu-ipmi': '121',
'loki': '1576',
'node-exporter': '4775',
'nomad': '2008',
'pcon0-snmp': '95',
'pcon1-snmp': '95',
'prometheus': '936',
'rpc-metrics': '20',
's3-metrics': '785',
'snmp-exporter': '544',
'sync-iers': '48',
'white-rabbit-snmp': '2948'}
%% Cell type:code id:e9dbafe8-8ccd-412a-a260-daab86f7746c tags:
``` python
# Start an observation on LBA (which is available on all stations)
start = datetime.datetime.now()
stop = start + datetime.timedelta(minutes=5)
field_spec = {
"observation_id": 1234567,
"start_time": start.isoformat(),
"stop_time": stop.isoformat(),
"antenna_set": "ALL",
"station": stationmanager.station_name_r,
"antenna_field": "LBA",
"filter": "LBA_30_90",
"SAPs": [{
"subbands": [10, 20, 30],
"pointing": { "angle1": 1.5, "angle2": 0, "direction_type": "J2000" }
},{
"subbands": [10, 20, 30],
"pointing": { "angle1": 0, "angle2": 1.0, "direction_type": "J2000" }
}]
}
observation_spec = {
"station": stationmanager.station_name_r,
"antenna_fields": [field_spec]
}
import json
with write_access:
observationcontrol.add_observation(json.dumps(observation_spec))
```
%% Cell type:code id:5a569240-bae7-4eda-9b20-a60daa85a719 tags:
``` python
# Verify that the observation is running. This should now return True
1234567 in observationcontrol.running_observations_r
```
%% Cell type:code id:4946f754-0023-4007-9a65-2a1771a66324 tags:
``` python
# Verify whether the station was actually configured for the observation
# Should all be ('J2000', '1.5rad', '0rad')
print(digitalbeam_l.pointing_direction_r[0:3])
# Should all be ('J2000', '0rad', '1.0rad')
print(digitalbeam_l.pointing_direction_r[3:6])
```
%% Cell type:code id:a69e1500-5b51-4c0a-8fca-b63289f3e2eb tags:
``` python
# Stop observation
with write_access:
observationcontrol.stop_all_observations_now()
```
%% Cell type:code id:13ccea03-9d70-414f-aaac-a5223788d95c tags:
``` python
# Are they all stopped? This should return []
observationcontrol.running_observations_r
```
%% Cell type:code id:ec67aa9d-af41-45f3-a44f-dd4f1a4e24b5 tags:
``` python
unb2_l0.unb2tr_monitor_rate_rw
```
%% Output
---------------------------------------------------------------------------
DevFailed Traceback (most recent call last)
Cell In[32], line 1
----> 1 unb2_l0.unb2tr_monitor_rate_rw
File /opt/conda/lib/python3.11/site-packages/tango/device_proxy.py:462, in __DeviceProxy__getattr(self, name)
460 attr_info = self.__get_attr_cache().get(name_l)
461 if attr_info:
--> 462 return __get_attribute_value(self, attr_info, name)
464 if name_l in self.__get_pipe_cache():
465 return self.read_pipe(name)
File /opt/conda/lib/python3.11/site-packages/tango/device_proxy.py:402, in __get_attribute_value(self, attr_info, name)
400 return __async_get_attribute_value(self, attr_info, name)
401 else:
--> 402 return __sync_get_attribute_value(self, attr_info, name)
File /opt/conda/lib/python3.11/site-packages/tango/device_proxy.py:394, in __sync_get_attribute_value(self, attr_info, name)
393 def __sync_get_attribute_value(self, attr_info, name):
--> 394 attr_value = self.read_attribute(name).value
395 return __update_enum_values(attr_info, attr_value)
File /opt/conda/lib/python3.11/site-packages/tango/green.py:234, in green.<locals>.decorator.<locals>.greener(obj, *args, **kwargs)
232 green_mode = access("green_mode", None)
233 executor = get_object_executor(obj, green_mode)
--> 234 return executor.run(fn, args, kwargs, wait=wait, timeout=timeout)
File /opt/conda/lib/python3.11/site-packages/tango/green.py:124, in AbstractExecutor.run(self, fn, args, kwargs, wait, timeout)
122 # Synchronous (no delegation)
123 if not self.asynchronous or not self.in_executor_context():
--> 124 return fn(*args, **kwargs)
125 # Asynchronous delegation
126 accessor = self.delegate(fn, *args, **kwargs)
File /opt/conda/lib/python3.11/site-packages/tango/device_proxy.py:594, in __DeviceProxy__read_attribute(self, value, extract_as)
593 def __DeviceProxy__read_attribute(self, value, extract_as=ExtractAs.Numpy):
--> 594 return __check_read_attribute(self._read_attribute(value, extract_as))
File /opt/conda/lib/python3.11/site-packages/tango/device_proxy.py:162, in __check_read_attribute(dev_attr)
160 def __check_read_attribute(dev_attr):
161 if dev_attr.has_failed:
--> 162 raise DevFailed(*dev_attr.get_err_stack())
163 return dev_attr
DevFailed: DevFailed[
DevError[
desc = It is currently not allowed to read attribute UNB2TR_monitor_rate_RW
origin = void Tango::Device_3Impl::read_attributes_no_except(const Tango::DevVarStringArray&, Tango::AttributeIdlData&, bool, std::vector<long int>&) at (/src/cppTango/src/server/device_3.cpp:522)
reason = API_AttrNotAllowed
severity = ERR]
DevError[
desc = Failed to read_attribute on device stat/unb2/l0, attribute unb2tr_monitor_rate_rw
origin = virtual Tango::DeviceAttribute Tango::DeviceProxy::read_attribute(const string&) at (/src/cppTango/src/client/devapi_base.cpp:5581)
reason = API_AttributeFailed
severity = ERR]
]
%% Cell type:code id:8649c3f9-2a46-45ae-827b-2c7998a9a090 tags:
``` python
```
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment