Update all the scripts/configuration to latest 'best known'

All the files have been updated to the last known 'best' state (as tested by me).

The deployment of Spack is now handled by Ansible, this is the biggest change in this MR. Other parts of the scripts have merely been updated or cleaned-up wherever needed.