esap-general issueshttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues2023-07-12T12:31:21Zhttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/211Rucio URL needs to be configured in two places2023-07-12T12:31:21ZYan GrangeRucio URL needs to be configured in two placesWhen I set up my own ESAP that needs to talk to its own Rucio instance, I have to configure the URL both in the envrionment through the RUCIO_HOST and RUCIO_AUTH_HOST as well as in the `esap/esap/esap_config.yaml` file which contains a `...When I set up my own ESAP that needs to talk to its own Rucio instance, I have to configure the URL both in the envrionment through the RUCIO_HOST and RUCIO_AUTH_HOST as well as in the `esap/esap/esap_config.yaml` file which contains a `url` and a `user_url`. This is not really documented and may lead to some weird errors because the URL used to list the scopes is taken from the environment while the URL that is used to find the files within a scope is taken from the latter.
I am looking into this so I may propose a fix today.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/210Make OIDC audience configurable2023-07-12T12:31:21ZYan GrangeMake OIDC audience configurableWhen trying to connect ESAP to the SKAO Rucio, this failed because the `aud` of the OIDC request is expected to be slightly different in this case. This is an extension of the original aud (adding an extra `https://wlcg.cern.ch/jwt/v1/an...When trying to connect ESAP to the SKAO Rucio, this failed because the `aud` of the OIDC request is expected to be slightly different in this case. This is an extension of the original aud (adding an extra `https://wlcg.cern.ch/jwt/v1/any` to it, but I think it may be nicer to make this configurable using an environment variable.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/209Zooniverse connector returns invalid results2023-01-23T15:45:36ZKlaas Kliffenkliffen@astron.nlZooniverse connector returns invalid resultsWhen credentials are provided, the Zooniverse connector should only return your own datasets, instead it returns all datasets.When credentials are provided, the Zooniverse connector should only return your own datasets, instead it returns all datasets.Hugh DickinsonHugh Dickinsonhttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/208Apply the react rules of hooks in a correct way in ESAP GUI2023-02-27T15:16:06ZKlaas Kliffenkliffen@astron.nlApply the react rules of hooks in a correct way in ESAP GUICurrently some `useEffect` hooks don't have the correct dependencies (https://blog.logrocket.com/understanding-react-exhaustive-deps-linting-warning/) The linting has been disabled for them since they require a major refactoring (both st...Currently some `useEffect` hooks don't have the correct dependencies (https://blog.logrocket.com/understanding-react-exhaustive-deps-linting-warning/) The linting has been disabled for them since they require a major refactoring (both structural and application logic).
See the `FIXME` from this MR: https://git.astron.nl/astron-sdc/escape-wp5/esap-gui/-/merge_requests/109
The major failure mode is an infinite render loop, which is caused by circular updates, where a (re-)render triggers an update. This should not happen in React as state should always flow down and render should be idempotent.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/207There is no IDA Service Category documentation2023-01-16T19:12:14ZJohn SwinbankThere is no IDA Service Category documentationThere's a page for it at https://git.astron.nl/astron-sdc/escape-wp5/esap-api-gateway/-/wikis/Service-Categories/Interactive-Analysis-Service-Category, but it is disappointingly empty.There's a page for it at https://git.astron.nl/astron-sdc/escape-wp5/esap-api-gateway/-/wikis/Service-Categories/Interactive-Analysis-Service-Category, but it is disappointingly empty.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/206Remove unused Rucio App in ESAP API GW2023-01-16T15:06:54ZKlaas Kliffenkliffen@astron.nlRemove unused Rucio App in ESAP API GWThe django app is not used and can be removed to avoid confusion with the Rucio in the query app.The django app is not used and can be removed to avoid confusion with the Rucio in the query app.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/205GUI freezes on Interactive Processing tab when selecting DLaaS without any DL...2023-01-16T08:21:34ZKlaas Kliffenkliffen@astron.nlGUI freezes on Interactive Processing tab when selecting DLaaS without any DLaaS configured in ESAP APITrying to access `response.data.results[0]` on `Interactive.js:189` which is `undefined` in case no DLaaS is configured in ESAP API.Trying to access `response.data.results[0]` on `Interactive.js:189` which is `undefined` in case no DLaaS is configured in ESAP API.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/204Apertif Collections “Imaging” or “Timedomain” query fails2023-01-07T11:36:39ZJohn SwinbankApertif Collections “Imaging” or “Timedomain” query failsIf I go to https://sdc-dev.astron.nl/esap-gui/archives/apertif/query and set “Apertif Collections” equal to “Imaging” before hitting submit, I get the message:
```
138, ["Error: ''collection''. Could not translate key: 'collection' for ...If I go to https://sdc-dev.astron.nl/esap-gui/archives/apertif/query and set “Apertif Collections” equal to “Imaging” before hitting submit, I get the message:
```
138, ["Error: ''collection''. Could not translate key: 'collection' for value: 'imaging'. Available translation parameters: {'target': 'target__icontains', 'title': 'target__icontains', 'ra': 'view_ra', 'dec': 'view_dec', 'fov': 'view_fov', 'dataproduct_level': 'calibrationLevel', 'dataproduct_type': 'dataProductType', 'dataproduct_subtype': 'dataProductSubT_and_dataProductSubType__in=uncalibratedVisibility_and_page_size=50"]
```
(and similarly for collection “Timedomain”).
If I set collections to “All” the query works as expected.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/202Update react-router-dom for ESAP GUI to version 62022-11-09T08:58:48ZKlaas Kliffenkliffen@astron.nlUpdate react-router-dom for ESAP GUI to version 6Currently we are using version 5. It requires some rework in the code: https://dev.to/arunavamodak/react-router-v5-vs-v6-dp0Currently we are using version 5. It requires some rework in the code: https://dev.to/arunavamodak/react-router-v5-vs-v6-dp0https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/201Make esap-gui fully compatible with modern Node2023-01-09T15:22:56ZJohn SwinbankMake esap-gui fully compatible with modern NodeThe `node:lts` Docker image recently switched from providing Node v16.18 to v18.12. Our [build broke](https://git.astron.nl/astron-sdc/esap-gui/-/jobs/413799) as a result.
One https://git.astron.nl/astron-sdc/esap-gui/-/merge_requests/1...The `node:lts` Docker image recently switched from providing Node v16.18 to v18.12. Our [build broke](https://git.astron.nl/astron-sdc/esap-gui/-/jobs/413799) as a result.
One https://git.astron.nl/astron-sdc/esap-gui/-/merge_requests/105 we've put in a temporary fix to unbreak the build, but in practice we ought to revert that hack and do the upgrade “properly”.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/200Why do we use inconsistent Python Docker images?2023-01-09T15:09:53ZJohn SwinbankWhy do we use inconsistent Python Docker images?- esap-api-gateway's `esap/Dockerfile` says `FROM python:3.10-slim`.
- esap-api-gateway's `.gitlab-ci.yml` uses `image: python:latest` for `run-unit-tests`.
- esap-worker's `Dockerfile` starts with `FROM python:3.10-alpine`.
[Somebody o...- esap-api-gateway's `esap/Dockerfile` says `FROM python:3.10-slim`.
- esap-api-gateway's `.gitlab-ci.yml` uses `image: python:latest` for `run-unit-tests`.
- esap-worker's `Dockerfile` starts with `FROM python:3.10-alpine`.
[Somebody on the internet](https://pythonspeed.com/articles/alpine-docker-python/) told me that using Alpine is a bad idea (but I didn't actually test for myself). Regardless of that, though, it seems like we should be consistent everywhere unless there's a reason not to be. And we definitely don't want to be testing with a version of Python (3.11) that's different from the version we're developing with (3.10), which means we should likely never using anything `:latest`.Klaas Kliffenkliffen@astron.nlJohn SwinbankKlaas Kliffenkliffen@astron.nlhttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/193Zooniverse tests load but do not run2022-10-17T14:21:26ZPierre ChanialZooniverse tests load but do not runLow-priority, since these tests require authentication, which the E2E testing does not provide.Low-priority, since these tests require authentication, which the E2E testing does not provide.Pierre ChanialJohn SwinbankPierre Chanialhttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/189Some MyBinder notebooks fail to launch automatically2023-01-09T15:14:38ZPierre ChanialSome MyBinder notebooks fail to launch automaticallyMyBinder is correctly initialized, but the notebook won't launch automatically
- CDS MOCPy
- Dark matter constraints from dwarf galaxies: a data-driven LAT analysis
- Escape Template ProjectMyBinder is correctly initialized, but the notebook won't launch automatically
- CDS MOCPy
- Dark matter constraints from dwarf galaxies: a data-driven LAT analysis
- Escape Template ProjectPierre ChanialPierre Chanialhttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/179Orchestration for container execution2023-01-24T17:22:47ZDave MOrchestration for container executionProof of concept orchestration service that uses the metadata from #131 to initialise the environment for a Docker or Singularity container.Proof of concept orchestration service that uses the metadata from #131 to initialise the environment for a Docker or Singularity container.ESAP version 2.0 releaseDave MDave Mhttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/175Review and repair the VO query pages2023-01-24T17:20:54ZDave MReview and repair the VO query pagesThe VO query pages need a detailed review of all the issues and a plan for fixing them.
----
Example issues:
Remove hard coded limits from VO queries
https://git.astron.nl/astron-sdc/esap-api-gateway/-/blob/master/esap/query/api/serv...The VO query pages need a detailed review of all the issues and a plan for fixing them.
----
Example issues:
Remove hard coded limits from VO queries
https://git.astron.nl/astron-sdc/esap-api-gateway/-/blob/master/esap/query/api/services/ivoa.py#L171-L177
The VO query page mixes up the general IVOA registry with four specific protocols (TAP, SIA, SSA, and SCS).
* "Query this dataset" button should be "Query the registry"
* "Query this dataset" button should be "Query [TAP|SIA|SSA|SCS] services"
The description for the 'VO registry' is wrong - it describes the TAP protocol.
The 'documentation' link for the 'VO registry' actually points ot a registry instance.
The title for SSA says 'Table Access Protocol (SSA)'
The text for SSA has way too much detail.
Selecting [Query this dataset] for the 'VO registry'. Using the default keyword 'apertif' we get 1 TAP service at Astron. If we change the keyword to '2mass' we get 6 TAP services, including GAVO.
Select [x] 'GAVO DC TAP' and [Query selected VO Resources]
The table metadata reveals details of the JSON structure which are not relevant to the science user.
The first element shows the service URL and the number of 'keys'.
```
http://dc.zah.uni-heidelberg.de/tap => 30 keys
```
* The URL is not useful to a science user, we should display the service name 'GAV TAP service'
* The text '30 keys' should be '30 catalogs'
* The number '30' is truncated, the GAV data center has 110 catalogs.
* The metadata tree reveals details of the raw JSON response
```
gaia: 3 keys
fields: 27 keys
source_id 4 keys
name
description
units
datatype
```
should be displayed as
```
gaia: 3 tables
27 columns
source_id
name
description
units
datatype
```
The default query is:
```
SELECT TOP 100 * from ivoa.obscore WHERE obs_collection='apertif-dr1' and dataproduct_subtype='continuum'
```
This instance happens to have 'ivoa.obscore', but the obs_collection and dataproduct_subtype are not relevant.
Changing to a simpler query
```
SELECT TOP 100 * from ivoa.obscore
```
Select first two items from first page
```
ppakm31/data/PPAK_M31_F5_Ha6563.fits
ppakm31/data/PPAK_M31_F5_SII6730.fits
```
Select second page
Get the same set of results as the first page
Check the contents of our shopping basket
Source (archive) listed as 'vo_reg' when it should be the GAVO data center.
The record is just an array of values, no metadata. Not even the original column names.
```
{
"archive":"vo_reg",
"record":[
"image",
"",
"2",
"PPAKM31",
"ppakm31/data/PPAK_M31_F5_Ha6563.fits",
....
....
]
}
```
Once we have looked at our shopping basket, we can't get back to the query results.
The [back] button goes back to the start of the VO reg query sequence, with default search terms.
Change the query to only select four columns
```
SELECT TOP 100 obs_id, s_ra, s_dec, facility_name, preview from ivoa.obscore
```
Select 3rd and 4th rows
Check the contents of our shopping basket
The record is just an array of values, no metadata. Not even the original column names.
```
{
"archive":"vo_reg",
"record":[
"ppakm31/data/PPAK_M31_F5_SII6716.fits",
"11.105143194205871",
"41.626154777484786",
"Calar Alto 3.5m telescope",
"http://dc.zah.uni-heidelberg.de/getproduct/ppakm31/data/PPAK_M31_F5_SII6716.fits?preview=True"
]
}
```
Change the query to select different columns
```
SELECT TOP 100 s_fov, s_region from ivoa.obscore
```
```
KeyError: "Got KeyError when attempting to get a value for field `thumbnail` on serializer
`CreateAndRunQuerySerializer`.\nThe serializer field might be named incorrectly and not match
any attribute or key on the `dict` instance.\nOriginal exception text was: 'thumbnail'."
```
Seems that a thumbnail (preview) column is required or the parser fails.
----
Back to the first page of VO query, and select 'Simple Image Access (SIA)'
[Query this dataset] jumps to IVOA query page, with ServiceType set to 'TAP: Tables' not 'SIA: Images'
Manually select 'SIA: Images'and change keyword to '2mass'. Results show 9 rows
Select the first row '2MASS All-Sky Atlas Image Service' and [Query selected service]
* Metadata query fails to complete
* Page fails to load ...ESAP version 2.0 releasehttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/168Handle async Job in queued state which failed2022-08-22T13:43:23ZKlaas Kliffenkliffen@astron.nlHandle async Job in queued state which failedSometimes an async job might get stuck in the queued state. The job might have been dispatched via Celery, but ( often due to mis-configuration) it can not be started by the worker.
However, this somehow does not trigger re-submission o...Sometimes an async job might get stuck in the queued state. The job might have been dispatched via Celery, but ( often due to mis-configuration) it can not be started by the worker.
However, this somehow does not trigger re-submission of a celery task. Check the celery configuration for possible solutions.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/167Refreshing of JWT for Job page2022-08-22T13:21:41ZKlaas Kliffenkliffen@astron.nlRefreshing of JWT for Job pageCurrently tokens are only available for 1 hour and should be refreshed. The current mechanisms in place don't seem to work.
It relates to #14 where the shopping basket can not be saved (although that one is about storing a dirty version...Currently tokens are only available for 1 hour and should be refreshed. The current mechanisms in place don't seem to work.
It relates to #14 where the shopping basket can not be saved (although that one is about storing a dirty version locally and is more of a workaround).
I think it should be possible to actually have the refresh tokens work, but this requires some investigation.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/164Alternative sources of “workflows”2022-08-05T11:51:33ZJohn SwinbankAlternative sources of “workflows”Currently, all “workflows” are fetched from a hard-coded Zenodo project corresponding to the OSSR.
In practice, this should be configurable (and perhaps we should also support more than one source at a time). Ideally, one could imagine ...Currently, all “workflows” are fetched from a hard-coded Zenodo project corresponding to the OSSR.
In practice, this should be configurable (and perhaps we should also support more than one source at a time). Ideally, one could imagine being able to specify a Zenodo project by name. Future extensions could include complete different (non-Zenodo) sources.
Credit to @volodymyr and @ghughes1066 for the suggestion.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/163Display more information about workflow origin.2022-08-05T11:49:13ZJohn SwinbankDisplay more information about workflow origin.@volodymyr and @ghughes1066 suggest then ESAP displays workflows it should also display information about whether they have been fetched from the OSSR, hard-coded locally into the database, or have some other origin. (I think this is a g...@volodymyr and @ghughes1066 suggest then ESAP displays workflows it should also display information about whether they have been fetched from the OSSR, hard-coded locally into the database, or have some other origin. (I think this is a good idea!)https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/162Display more information about workflow validation.2022-08-05T11:48:14ZJohn SwinbankDisplay more information about workflow validation.@volodymyr and @ghughes1066 suggest then ESAP displays workflows it could also display information about the last time the workflow was checked (e.g. through automated testing; see also the work being carried out by @pchanial (https://gi...@volodymyr and @ghughes1066 suggest then ESAP displays workflows it could also display information about the last time the workflow was checked (e.g. through automated testing; see also the work being carried out by @pchanial (https://git.astron.nl/groups/astron-sdc/escape-wp5/-/epics/12).