Skip to content
Snippets Groups Projects
Unverified Commit 81ddb58a authored by Drew Devereux's avatar Drew Devereux
Browse files

[MCCS-163] Update adminMode state machine and write docs for it

parent 5a2a62b6
Branches
No related tags found
No related merge requests found
Showing
with 285 additions and 48 deletions
...@@ -2,35 +2,130 @@ ...@@ -2,35 +2,130 @@
State Machine State Machine
============= =============
The state machine modules implements SKA's two fundamental state machines: the The state machine module implements three fundamental SKA state
base device state machine, and the observation state machine. machines:
Base device state machine * the admin mode state machine
------------------------- * the operational state (opState, represented in TANGO devices by TANGO
The base device state machine provides basic state needed for all devices, state) state machine
covering initialisation, off and on states, and a fault state. This state * the observation state machine.
machine is implemented by all SKA Tango devices that inherit from these LMC
base classes, though some devices with standby power modes may need to Admin mode state machine
implement further states. ------------------------
The admin mode state machine allows for transitions between the five
administrative modes:
.. figure:: images/device_state_diagram.png
:width: 80% * NOT_FITTED: this is the lowest state of readiness, representing
:alt: Diagram of the device state machine, taken from SKA design devices that cannot be deployed without some external action, such as
documentation, showing the state machine as designed plugging hardware in or updating network settings.)
* RESERVED: the device is fitted but redundant to other devices. It is
Diagram of the device state machine, taken from SKA design ready to take over should other devices fail.
documentation, showing the state machine as designed * OFFLINE: the device has been declared by SKA operations not currently
to be used for operations (or whatever other function it provides)
* MAINTENANCE: the device cannot be used for science purposes but can be
.. figure:: images/DeviceStateMachine.png operationed for engineering / maintenance purposes, such as testing,
debugging, etc
* ONLINE: the device can be used for science purposes.
The admin mode state machine allows for
* any transition between the modes NOT_FITTED, RESERVED and OFFLINE
(e.g. an unfitted device being fitted as a redundant or non-redundant
device, a redundant device taking over when another device fails, etc)
* any transition between the modes OFFLINE, MAINTENANCE and ONLINE (e.g.
an online device being taken offline or put into maintenance mode to
diagnose a fault, a faulty device moving between maintenance and
offline mode as it undergoes sporadic periods of diagnosis.
Diagrams of the admin mode state machine are shown below.
.. figure:: images/AdminModeStateMachine.png
:alt: Diagram of the admin mode state machine, as designed
Diagram of the admin mode state machine, as designed
..
COMMENTED OUT FOR NOW
.. figure:: images/AdminModeStateMachine_autogenerated.png
:alt: Diagram of the admin mode state machine, as implemented
Diagram of the admin mode state machine, automatically generated from
the implementation. The equivalence of this diagram to the diagram
above demonstrates that the machine has been implemented as designed.
Operational state state machine
-------------------------------
The operational state (opState) machine represents the operational state
of a SKA device. It is represented in TANGO devices using the TANGO
"state", so the states used are a subset of the TANGO states: INIT,
FAULT, DISABLE, STANDBY, OFF and ON.
* INIT: the device is currently initialising
* FAULT: the device has experienced an error from which it could not
recover.
* DISABLE: the device is in its lowest state of readiness, from which
it may take some time to become fully operational. For example, if the
device manages hardware, that hardware may be switched off.
* STANDBY: the device is unready, but can be made ready quickly. For
example, if the device manages hardware, that hardware may be in a
low-power standby mode.
* OFF: the device is fully operational but is not currently in use
* ON: the device is in use
The operational state state machine allows for:
* transition from INIT or FAULT into any of the three "readiness states"
DISABLE, STANDBY and OFF.
* all transitions between these three "readiness states" DISABLE,
STANDBY and OFF.
* transition between OFF and ON.
.. figure:: images/OperationalStateStateMachine_decoupled.png
:alt: Diagram of the operational state state machine, as designed,
ignoring coupling with admin mode
Diagram of the operational state (opState) state machine, as
designed, ignoring coupling with admin mode
Unfortunately, operational state is inextricably coupled with admin
mode: there are admin modes that imply disablement, and operational
states such as ON should not be possible in such admin modes.
To facilitate this, the entire operational state state machine is
accessible only when the admin mode is ONLINE or MAINTENANCE. When in
any other admin mode, the only permitted operational states are INIT,
FAULT and DISABLE. This constraint is implemented into the operational
state state machine by
* three extra states: INIT_ADMIN, FAULT_ADMIN and DISABLED_ADMIN
* two extra transition triggers: "admin_on" and "admin_off", which allow
for transition between INIT and INIT_ADMIN; FAULT and FAULT_ADMIN; and
DISABLE and DISABLE_ADMIN.
This implementation minimises the coupling between admin mode and
operational state, allowing the two machines to be conceptualised almost
separately.
Diagrams of the operational state state machine are shown below.
.. figure:: images/OperationalStateStateMachine_coupled.png
:alt: Diagram of the operational state state machine, as designed,
showing coupling with admin mode
Diagram of the operational state (opState) state machine, as
designed, showing coupling with admin mode
..
COMMENTED OUT FOR NOW
.. figure:: images/OperationalStateStateMachine_coupled_autogenerated.png
:width: 80% :width: 80%
:alt: Diagram of the device state machine, automatically generated :alt: Diagram of the admin operational state state machine, as implemented
from the state machine as specified in code.
Diagram of the device state machine, automatically generated from the Diagram of the operational state state machine, automatically
state machine as specified in code. The equivalence of this diagram to generated from the implementation. The equivalence of this diagram
the diagram previous demonstrates that the machine has been to the diagram above demonstrates that the machine has been
implemented as designed. implemented as designed.
...@@ -39,22 +134,24 @@ Observation state machine ...@@ -39,22 +134,24 @@ Observation state machine
The observation state machine is implemented by devices that manage The observation state machine is implemented by devices that manage
observations (currently only subarray devices). observations (currently only subarray devices).
.. figure:: images/ADR-8.png .. figure:: images/ObservationStateMachine_adr8.png
:width: 80% :width: 80%
:alt: Diagram of the observation state machine, as decided and published in ADR-8. :alt: Diagram of the observation state machine, as decided and
published in ADR-8.
Diagram of the observation state machine, as decided and published in ADR-8. Diagram of the observation state machine, as decided and published in
ADR-8.
.. figure:: images/ObservationStateMachine.png .. figure:: images/ObservationStateMachine_autogenerated.png
:width: 80% :width: 80%
:alt: Diagram of the observation state machine, automatically generated from :alt: Diagram of the observation state machine, automatically
the state machine as specified in code. generated from the implementation
Diagram of the observation state machine, automatically generated from Diagram of the observation state machine, automatically generated from
the state machine as specified in code. The equivalance of this the implementation. The equivalance of this diagram to the diagram
diagram to the diagram previous demonstrates that the machine has been previous demonstrates that the machine has been implemented in
implemented in conformance with ADR-8. conformance with ADR-8.
API API
--- ---
......
docs/source/images/AdminModeStateMachine.png

8.66 KiB

docs/source/images/DeviceStateMachine.png

119 KiB

docs/source/images/OperationalStateStateMachine_coupled.png

38.2 KiB

docs/source/images/OperationalStateStateMachine_decoupled.png

21.9 KiB

docs/source/images/device_state_diagram.png

564 KiB

...@@ -421,7 +421,7 @@ class DeviceStateModel: ...@@ -421,7 +421,7 @@ class DeviceStateModel:
"to_notfitted": ("admin_on", "to_notfitted"), "to_notfitted": ("admin_on", "to_notfitted"),
"to_offline": ("admin_on", "to_offline"), "to_offline": ("admin_on", "to_offline"),
"to_maintenance": ("admin_off", "to_maintenance"), "to_maintenance": ("admin_off", "to_maintenance"),
"to_online": ("admin_off", "to_maintenance"), "to_online": ("admin_off", "to_online"),
"init_started": ("init_started", None), "init_started": ("init_started", None),
"init_succeeded_disable": ("init_succeeded_disable", None), "init_succeeded_disable": ("init_succeeded_disable", None),
"init_succeeded_standby": ("init_succeeded_standby", None), "init_succeeded_standby": ("init_succeeded_standby", None),
......
...@@ -222,7 +222,7 @@ class AdminModeStateMachine(Machine): ...@@ -222,7 +222,7 @@ class AdminModeStateMachine(Machine):
states = ["RESERVED", "NOT_FITTED", "OFFLINE", "MAINTENANCE", "ONLINE"] states = ["RESERVED", "NOT_FITTED", "OFFLINE", "MAINTENANCE", "ONLINE"]
transitions = [ transitions = [
{ {
"source": "NOT_FITTED", "source": ["NOT_FITTED", "OFFLINE"],
"trigger": "to_reserved", "trigger": "to_reserved",
"dest": "RESERVED", "dest": "RESERVED",
}, },
...@@ -232,17 +232,17 @@ class AdminModeStateMachine(Machine): ...@@ -232,17 +232,17 @@ class AdminModeStateMachine(Machine):
"dest": "NOT_FITTED", "dest": "NOT_FITTED",
}, },
{ {
"source": ["NOT_FITTED", "MAINTENANCE", "ONLINE"], "source": ["RESERVED", "NOT_FITTED", "MAINTENANCE", "ONLINE"],
"trigger": "to_offline", "trigger": "to_offline",
"dest": "OFFLINE", "dest": "OFFLINE",
}, },
{ {
"source": "OFFLINE", "source": ["OFFLINE", "ONLINE"],
"trigger": "to_maintenance", "trigger": "to_maintenance",
"dest": "MAINTENANCE", "dest": "MAINTENANCE",
}, },
{ {
"source": "OFFLINE", "source": ["OFFLINE", "MAINTENANCE"],
"trigger": "to_online", "trigger": "to_online",
"dest": "ONLINE", "dest": "ONLINE",
}, },
......
...@@ -7,11 +7,6 @@ ...@@ -7,11 +7,6 @@
"ONLINE": {} "ONLINE": {}
}, },
"transitions": [ "transitions": [
{
"from": "RESERVED",
"to": "NOT_FITTED",
"trigger": "to_notfitted"
},
{ {
"from": "NOT_FITTED", "from": "NOT_FITTED",
"to": "RESERVED", "to": "RESERVED",
...@@ -22,6 +17,21 @@ ...@@ -22,6 +17,21 @@
"to": "OFFLINE", "to": "OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
}, },
{
"from": "RESERVED",
"to": "NOT_FITTED",
"trigger": "to_notfitted"
},
{
"from": "RESERVED",
"to": "OFFLINE",
"trigger": "to_offline"
},
{
"from": "OFFLINE",
"to": "RESERVED",
"trigger": "to_reserved"
},
{ {
"from": "OFFLINE", "from": "OFFLINE",
"to": "NOT_FITTED", "to": "NOT_FITTED",
...@@ -42,10 +52,20 @@ ...@@ -42,10 +52,20 @@
"to": "OFFLINE", "to": "OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
}, },
{
"from": "MAINTENANCE",
"to": "ONLINE",
"trigger": "to_online"
},
{ {
"from": "ONLINE", "from": "ONLINE",
"to": "OFFLINE", "to": "OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
},
{
"from": "ONLINE",
"to": "MAINTENANCE",
"trigger": "to_maintenance"
} }
] ]
} }
\ No newline at end of file
...@@ -91,6 +91,11 @@ ...@@ -91,6 +91,11 @@
"to": "INIT_OFFLINE", "to": "INIT_OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
}, },
{
"from": "INIT_MAINTENANCE",
"to": "INIT_ONLINE",
"trigger": "to_online"
},
{ {
"from": "INIT_MAINTENANCE", "from": "INIT_MAINTENANCE",
"to": "DISABLE_MAINTENANCE", "to": "DISABLE_MAINTENANCE",
...@@ -116,6 +121,11 @@ ...@@ -116,6 +121,11 @@
"to": "FAULT_MAINTENANCE", "to": "FAULT_MAINTENANCE",
"trigger": "fatal_error" "trigger": "fatal_error"
}, },
{
"from": "INIT_ONLINE",
"to": "INIT_MAINTENANCE",
"trigger": "to_maintenance"
},
{ {
"from": "INIT_ONLINE", "from": "INIT_ONLINE",
"to": "INIT_OFFLINE", "to": "INIT_OFFLINE",
...@@ -156,6 +166,11 @@ ...@@ -156,6 +166,11 @@
"to": "INIT_MAINTENANCE", "to": "INIT_MAINTENANCE",
"trigger": "to_maintenance" "trigger": "to_maintenance"
}, },
{
"from": "INIT_OFFLINE",
"to": "INIT_RESERVED",
"trigger": "to_reserved"
},
{ {
"from": "INIT_OFFLINE", "from": "INIT_OFFLINE",
"to": "INIT_NOTFITTED", "to": "INIT_NOTFITTED",
...@@ -206,6 +221,11 @@ ...@@ -206,6 +221,11 @@
"to": "INIT_NOTFITTED", "to": "INIT_NOTFITTED",
"trigger": "to_notfitted" "trigger": "to_notfitted"
}, },
{
"from": "INIT_RESERVED",
"to": "INIT_OFFLINE",
"trigger": "to_offline"
},
{ {
"from": "INIT_RESERVED", "from": "INIT_RESERVED",
"to": "DISABLE_RESERVED", "to": "DISABLE_RESERVED",
...@@ -226,6 +246,11 @@ ...@@ -226,6 +246,11 @@
"to": "FAULT_OFFLINE", "to": "FAULT_OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
}, },
{
"from": "FAULT_MAINTENANCE",
"to": "FAULT_ONLINE",
"trigger": "to_online"
},
{ {
"from": "FAULT_MAINTENANCE", "from": "FAULT_MAINTENANCE",
"to": "DISABLE_MAINTENANCE", "to": "DISABLE_MAINTENANCE",
...@@ -256,6 +281,11 @@ ...@@ -256,6 +281,11 @@
"to": "FAULT_OFFLINE", "to": "FAULT_OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
}, },
{
"from": "FAULT_ONLINE",
"to": "FAULT_MAINTENANCE",
"trigger": "to_maintenance"
},
{ {
"from": "FAULT_ONLINE", "from": "FAULT_ONLINE",
"to": "DISABLE_ONLINE", "to": "DISABLE_ONLINE",
...@@ -291,6 +321,11 @@ ...@@ -291,6 +321,11 @@
"to": "FAULT_MAINTENANCE", "to": "FAULT_MAINTENANCE",
"trigger": "to_maintenance" "trigger": "to_maintenance"
}, },
{
"from": "FAULT_OFFLINE",
"to": "FAULT_RESERVED",
"trigger": "to_reserved"
},
{ {
"from": "FAULT_OFFLINE", "from": "FAULT_OFFLINE",
"to": "FAULT_NOTFITTED", "to": "FAULT_NOTFITTED",
...@@ -341,6 +376,11 @@ ...@@ -341,6 +376,11 @@
"to": "FAULT_NOTFITTED", "to": "FAULT_NOTFITTED",
"trigger": "to_notfitted" "trigger": "to_notfitted"
}, },
{
"from": "FAULT_RESERVED",
"to": "FAULT_OFFLINE",
"trigger": "to_offline"
},
{ {
"from": "FAULT_RESERVED", "from": "FAULT_RESERVED",
"to": "DISABLE_RESERVED", "to": "DISABLE_RESERVED",
...@@ -361,6 +401,11 @@ ...@@ -361,6 +401,11 @@
"to": "DISABLE_OFFLINE", "to": "DISABLE_OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
}, },
{
"from": "DISABLE_MAINTENANCE",
"to": "DISABLE_ONLINE",
"trigger": "to_online"
},
{ {
"from": "DISABLE_MAINTENANCE", "from": "DISABLE_MAINTENANCE",
"to": "STANDBY_MAINTENANCE", "to": "STANDBY_MAINTENANCE",
...@@ -391,6 +436,11 @@ ...@@ -391,6 +436,11 @@
"to": "DISABLE_OFFLINE", "to": "DISABLE_OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
}, },
{
"from": "DISABLE_ONLINE",
"to": "DISABLE_MAINTENANCE",
"trigger": "to_maintenance"
},
{ {
"from": "DISABLE_ONLINE", "from": "DISABLE_ONLINE",
"to": "STANDBY_ONLINE", "to": "STANDBY_ONLINE",
...@@ -431,6 +481,11 @@ ...@@ -431,6 +481,11 @@
"to": "DISABLE_NOTFITTED", "to": "DISABLE_NOTFITTED",
"trigger": "to_notfitted" "trigger": "to_notfitted"
}, },
{
"from": "DISABLE_OFFLINE",
"to": "DISABLE_RESERVED",
"trigger": "to_reserved"
},
{ {
"from": "DISABLE_OFFLINE", "from": "DISABLE_OFFLINE",
"to": "FAULT_OFFLINE", "to": "FAULT_OFFLINE",
...@@ -456,6 +511,11 @@ ...@@ -456,6 +511,11 @@
"to": "DISABLE_NOTFITTED", "to": "DISABLE_NOTFITTED",
"trigger": "to_notfitted" "trigger": "to_notfitted"
}, },
{
"from": "DISABLE_RESERVED",
"to": "DISABLE_OFFLINE",
"trigger": "to_offline"
},
{ {
"from": "DISABLE_RESERVED", "from": "DISABLE_RESERVED",
"to": "FAULT_RESERVED", "to": "FAULT_RESERVED",
......
...@@ -212,6 +212,11 @@ ...@@ -212,6 +212,11 @@
"to": "INIT_OFFLINE", "to": "INIT_OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
}, },
{
"from": "INIT_MAINTENANCE",
"to": "INIT_ONLINE",
"trigger": "to_online"
},
{ {
"from": "INIT_MAINTENANCE", "from": "INIT_MAINTENANCE",
"to": "DISABLE_MAINTENANCE", "to": "DISABLE_MAINTENANCE",
...@@ -237,6 +242,11 @@ ...@@ -237,6 +242,11 @@
"to": "FAULT_MAINTENANCE", "to": "FAULT_MAINTENANCE",
"trigger": "fatal_error" "trigger": "fatal_error"
}, },
{
"from": "INIT_ONLINE",
"to": "INIT_MAINTENANCE",
"trigger": "to_maintenance"
},
{ {
"from": "INIT_ONLINE", "from": "INIT_ONLINE",
"to": "INIT_OFFLINE", "to": "INIT_OFFLINE",
...@@ -277,6 +287,11 @@ ...@@ -277,6 +287,11 @@
"to": "INIT_MAINTENANCE", "to": "INIT_MAINTENANCE",
"trigger": "to_maintenance" "trigger": "to_maintenance"
}, },
{
"from": "INIT_OFFLINE",
"to": "INIT_RESERVED",
"trigger": "to_reserved"
},
{ {
"from": "INIT_OFFLINE", "from": "INIT_OFFLINE",
"to": "INIT_NOTFITTED", "to": "INIT_NOTFITTED",
...@@ -327,6 +342,11 @@ ...@@ -327,6 +342,11 @@
"to": "INIT_NOTFITTED", "to": "INIT_NOTFITTED",
"trigger": "to_notfitted" "trigger": "to_notfitted"
}, },
{
"from": "INIT_RESERVED",
"to": "INIT_OFFLINE",
"trigger": "to_offline"
},
{ {
"from": "INIT_RESERVED", "from": "INIT_RESERVED",
"to": "DISABLE_RESERVED", "to": "DISABLE_RESERVED",
...@@ -347,6 +367,11 @@ ...@@ -347,6 +367,11 @@
"to": "FAULT_OFFLINE", "to": "FAULT_OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
}, },
{
"from": "FAULT_MAINTENANCE",
"to": "FAULT_ONLINE",
"trigger": "to_online"
},
{ {
"from": "FAULT_MAINTENANCE", "from": "FAULT_MAINTENANCE",
"to": "DISABLE_MAINTENANCE", "to": "DISABLE_MAINTENANCE",
...@@ -377,6 +402,11 @@ ...@@ -377,6 +402,11 @@
"to": "FAULT_OFFLINE", "to": "FAULT_OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
}, },
{
"from": "FAULT_ONLINE",
"to": "FAULT_MAINTENANCE",
"trigger": "to_maintenance"
},
{ {
"from": "FAULT_ONLINE", "from": "FAULT_ONLINE",
"to": "DISABLE_ONLINE", "to": "DISABLE_ONLINE",
...@@ -412,6 +442,11 @@ ...@@ -412,6 +442,11 @@
"to": "FAULT_MAINTENANCE", "to": "FAULT_MAINTENANCE",
"trigger": "to_maintenance" "trigger": "to_maintenance"
}, },
{
"from": "FAULT_OFFLINE",
"to": "FAULT_RESERVED",
"trigger": "to_reserved"
},
{ {
"from": "FAULT_OFFLINE", "from": "FAULT_OFFLINE",
"to": "FAULT_NOTFITTED", "to": "FAULT_NOTFITTED",
...@@ -462,6 +497,11 @@ ...@@ -462,6 +497,11 @@
"to": "FAULT_NOTFITTED", "to": "FAULT_NOTFITTED",
"trigger": "to_notfitted" "trigger": "to_notfitted"
}, },
{
"from": "FAULT_RESERVED",
"to": "FAULT_OFFLINE",
"trigger": "to_offline"
},
{ {
"from": "FAULT_RESERVED", "from": "FAULT_RESERVED",
"to": "DISABLE_RESERVED", "to": "DISABLE_RESERVED",
...@@ -482,6 +522,11 @@ ...@@ -482,6 +522,11 @@
"to": "DISABLE_OFFLINE", "to": "DISABLE_OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
}, },
{
"from": "DISABLE_MAINTENANCE",
"to": "DISABLE_ONLINE",
"trigger": "to_online"
},
{ {
"from": "DISABLE_MAINTENANCE", "from": "DISABLE_MAINTENANCE",
"to": "STANDBY_MAINTENANCE", "to": "STANDBY_MAINTENANCE",
...@@ -512,6 +557,11 @@ ...@@ -512,6 +557,11 @@
"to": "DISABLE_OFFLINE", "to": "DISABLE_OFFLINE",
"trigger": "to_offline" "trigger": "to_offline"
}, },
{
"from": "DISABLE_ONLINE",
"to": "DISABLE_MAINTENANCE",
"trigger": "to_maintenance"
},
{ {
"from": "DISABLE_ONLINE", "from": "DISABLE_ONLINE",
"to": "STANDBY_ONLINE", "to": "STANDBY_ONLINE",
...@@ -552,6 +602,11 @@ ...@@ -552,6 +602,11 @@
"to": "DISABLE_NOTFITTED", "to": "DISABLE_NOTFITTED",
"trigger": "to_notfitted" "trigger": "to_notfitted"
}, },
{
"from": "DISABLE_OFFLINE",
"to": "DISABLE_RESERVED",
"trigger": "to_reserved"
},
{ {
"from": "DISABLE_OFFLINE", "from": "DISABLE_OFFLINE",
"to": "FAULT_OFFLINE", "to": "FAULT_OFFLINE",
...@@ -577,6 +632,11 @@ ...@@ -577,6 +632,11 @@
"to": "DISABLE_NOTFITTED", "to": "DISABLE_NOTFITTED",
"trigger": "to_notfitted" "trigger": "to_notfitted"
}, },
{
"from": "DISABLE_RESERVED",
"to": "DISABLE_OFFLINE",
"trigger": "to_offline"
},
{ {
"from": "DISABLE_RESERVED", "from": "DISABLE_RESERVED",
"to": "FAULT_RESERVED", "to": "FAULT_RESERVED",
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment