esap-general issueshttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues2022-08-22T13:21:42Zhttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/14Handle expired token and CORS errors (2 separate issues)2022-08-22T13:21:42ZNico VermaasHandle expired token and CORS errors (2 separate issues)These are 2 separate issues, but both related to token communication
- [ ] When the IAM id_token becomes invalid after 1 hour then ESAP-GUI gets in a unpredictable state where the user can not save the shopping basket anymore (https://s...These are 2 separate issues, but both related to token communication
- [ ] When the IAM id_token becomes invalid after 1 hour then ESAP-GUI gets in a unpredictable state where the user can not save the shopping basket anymore (https://support.astron.nl/jira/browse/SDC-271)
- [x] When the oidc refresh token tries to check if the id_token is still valid (default every 15 minutes) then it reports a CORS error (since moving the backend away from port 5555 to 80/443).
Currently the 'OIDC_RENEW_ID_TOKEN_EXPIRY_SECONDS' value in the backend has been set to 3600 to avoid the error before the token becomes invalid anyway. (https://support.astron.nl/jira/browse/SDC-272)MS31Nico VermaasNico Vermaashttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/47Use a conventional repository layout for esap-api-gateway2022-07-21T11:25:33ZJohn SwinbankUse a conventional repository layout for esap-api-gatewayesap-api-gateway has a slightly surprising layout for its repository: the root directory contains almost nothing but an `esap` directory, which is a Python package (ie, it contains an `__init__.py` file), but also contains a bunch of sta...esap-api-gateway has a slightly surprising layout for its repository: the root directory contains almost nothing but an `esap` directory, which is a Python package (ie, it contains an `__init__.py` file), but also contains a bunch of static assets (`esap/configuration_assets`), a Dockerfile (`esap/Dockerfile`), some logs (`esap/logs/esap.log` -- why are we versioning logs?!), a variety of different types of configuration (SQLite databases, Python code, etc, in `esap/esap`), information on dependencies (`esap/requirements`), and a bunch of additional Docker configuration (in `esap/docker`, including `esap/docker/ska` — despite the name! — is marked as being not specific to SKA).
It would be nice to rationalize this. In particular, we should:
- Not mix configuration and assets with code;
- Not version logs at all;
- Move the docker-compose stuff into a separate repository, since it isn't really part of esap-api-gateway;
- Probably some other stuff.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/57Bi-directional SAMP integration2022-04-25T07:12:40ZJohn SwinbankBi-directional SAMP integration@vermaas writes:
> SAMP integration is ... in one direction only. Data only flows from the samphub to ESAP.
> I am not aware if there is a requirement to also let meta data flow from ESAP to the samphub?
My response: there is no hard r...@vermaas writes:
> SAMP integration is ... in one direction only. Data only flows from the samphub to ESAP.
> I am not aware if there is a requirement to also let meta data flow from ESAP to the samphub?
My response: there is no hard requirement, but I think we should aspire to providing bi-directional SAMP. This issue exists to record that on the backlog, rather than as an immediate call-to-arms.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/61Document Git workflow2022-09-28T06:30:47ZJohn SwinbankDocument Git workflowWe should have a set of standard for interacting with WP5 git repositories. In particular, this should include:
- General git hygiene, well formed commits, etc;
- Preparing your work for merge and submitting a merge request;
- Expectati...We should have a set of standard for interacting with WP5 git repositories. In particular, this should include:
- General git hygiene, well formed commits, etc;
- Preparing your work for merge and submitting a merge request;
- Expectations around testing and review.John SwinbankJohn Swinbankhttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/63Document semantics of ESAP GitLab issues2022-09-28T06:30:44ZJohn SwinbankDocument semantics of ESAP GitLab issuesThere are a bunch of labels attached to GitLab issues. What do they mean?
For example, in #60, we added a “Done” label. Why? Why not just close it? Probably there's a good reason, but let's document it. :-)There are a bunch of labels attached to GitLab issues. What do they mean?
For example, in #60, we added a “Done” label. Why? Why not just close it? Probably there's a good reason, but let's document it. :-)https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/65Asynchronous queries2023-02-02T10:30:23ZJohn SwinbankAsynchronous queriesSome external services will return results asynchronously. A good historical example of this is [SDSS CasJobs](https://skyserver.sdss.org/CasJobs/); it's also supported (optionally?) by some(?) VO services, and it will certainly be requi...Some external services will return results asynchronously. A good historical example of this is [SDSS CasJobs](https://skyserver.sdss.org/CasJobs/); it's also supported (optionally?) by some(?) VO services, and it will certainly be required for many queries of the Rubin archive.
We need to support it within ESAP. This will include (presumably)
- Enabling the user to submit a query that doesn't return an immediate result;
- Presenting them with a list of queries that they have submitted which are in progress;
- Notifying them when results become available;
- Enabling them to retrieve and view the results at their leisure.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/66Mint DOIs2022-04-25T07:14:32ZJohn SwinbankMint DOIsThe ESAP requirements (R-8; see [deliverable D5.2](https://projectescape.eu/sites/default/files/D5.2%20Detailed%20project%20plan%20for%20WP5.pdf)) tell us that “Users should be able to assign PIDs to every digital object that is part of ...The ESAP requirements (R-8; see [deliverable D5.2](https://projectescape.eu/sites/default/files/D5.2%20Detailed%20project%20plan%20for%20WP5.pdf)) tell us that “Users should be able to assign PIDs to every digital object that is part of a Research Object”).
Currently, we've not sketched out what this means. As zeroth-order implementation might be some means of assigning a DOI to the contents of the shopping basket. Is that practical? Are there other possible interpretations of what's needed here?https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/67Develop a plan for provenance tracking2022-04-25T07:14:44ZJohn SwinbankDevelop a plan for provenance trackingThe [ESAP Vision](https://git.astron.nl/groups/astron-sdc/escape-wp5/-/wikis/ESAP/ESAP-Vision#provenance) tells us that ESAP should track provenance for its data products. It expands on that to say:
> In general, it seems impossible for...The [ESAP Vision](https://git.astron.nl/groups/astron-sdc/escape-wp5/-/wikis/ESAP/ESAP-Vision#provenance) tells us that ESAP should track provenance for its data products. It expands on that to say:
> In general, it seems impossible for ESAP to provide full provenance information given the nature of its workflow. In particular, while ESAP can track the queries used to select an input dataset, it cannot track whatever IDA is performed by the user (which may include filtering or otherwise modifying the input data). **Suggestion:** the shopping basket should track the source of each entry (that is, what query was run to generate it, etc), and record what other services the data has been round-tripped through.
It's also been suggested that we look at the IVOA Provenance Model.
We should do that, and come up with a concrete proposal for what's required here.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/68Add to shopping basket from IDA or other services2022-04-25T07:14:59ZJohn SwinbankAdd to shopping basket from IDA or other servicesWe currently support users reading their shopping basket from IDA or other services. However, they should also be able to write to it, so that they can add newly-generated data products to the basket.
Question for debate: should they be...We currently support users reading their shopping basket from IDA or other services. However, they should also be able to write to it, so that they can add newly-generated data products to the basket.
Question for debate: should they be able to modify existing entries, or just add new ones?Hugh DickinsonHugh Dickinsonhttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/71Change TAP Query to use async mode2023-02-02T10:30:29ZSteliosChange TAP Query to use async modeCurrently we use Pyvo to run a query and return the results, using the `search` method as seen here:
https://git.astron.nl/astron-sdc/esap-api-gateway/-/blob/master/esap/query/api/services/vo.py#L141
This runs a synchronous query, which...Currently we use Pyvo to run a query and return the results, using the `search` method as seen here:
https://git.astron.nl/astron-sdc/esap-api-gateway/-/blob/master/esap/query/api/services/vo.py#L141
This runs a synchronous query, which means that the client will wait until the results are received, after which the connection is lost, and the results can not be referenced anymore, which means we have to store them in the client.
In order to meet the requirement of being able to handle and pass along 1000+ rows to the shopping cart, this doesn't scale with the sychronous query. It does however with the "async" method of TAP, which is run asynchronously, and results are stored at a referencable URL.
Pyvo has a method that we can use in place of the sync query, allowing us to then work on storing / adding the reference to the results in the shopping client.
https://pyvo.readthedocs.io/en/latest/api/pyvo.dal.TAPService.html#pyvo.dal.TAPService.run_asynchttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/73Provide development workflow for Jupyter notebooks in IDA systems2022-04-25T07:24:06ZJohn SwinbankProvide development workflow for Jupyter notebooks in IDA systemsCurrently, the user can access a Jupyter notebook provided either through OSSR or hard-coded in the ESAP configuration and deploy it on an analysis service (e.g. JIVE or SKAO BinderHub).
It should be possible for the user to develop **a...Currently, the user can access a Jupyter notebook provided either through OSSR or hard-coded in the ESAP configuration and deploy it on an analysis service (e.g. JIVE or SKAO BinderHub).
It should be possible for the user to develop **and save** their own notebooks. In particular, they should be able to:
- Create a new notebook from scratch, and save it for their personal future use.
- Edit a notebook retrieved from the OSSR, and save it for their personal future use.
For bonus points it _may_ also be possible to publish the {new,modified} notebook through the OSSR, but:
- This is not required for successful completion of this ticket, and
- Users should not be required to publish to the OSSR in order to save their notebook.
Possible implementations might include:
- Providing an easy workflow for users to save their notebooks to Git{Hub,Lab}.
- Providing a way to store notebooks to the Data Lake (see also #72).https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/78Support associated data in ESAP queries2022-04-25T07:24:36ZJohn SwinbankSupport associated data in ESAP queriesOften, a user will first run a query to find the core science data they are interested in (e.g., an image of their favourite object), but then will want to download not just that data but related data. This would include, say, calibratio...Often, a user will first run a query to find the core science data they are interested in (e.g., an image of their favourite object), but then will want to download not just that data but related data. This would include, say, calibration products which can be applied to the science data.
For example, see https://archive.eso.org/downloadportal/a87339c7-1bcc-48d9-8cc2-ac8577d4f183 — for a given science product, the ESO archive here provides the option to download:
- Associated processed calibrations
- Raw calibrations
- Night logs
ESAP should similarly allow the user to “drill down” from their science results to access related products without needing to run an entirely new query.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/79Support authentication with external services2022-04-25T07:24:43ZJohn SwinbankSupport authentication with external servicesSome services have their own authentication system which is not related to ESCAPE IAM. It should be possible for the user to authenticate with and access these services from within ESAP. The user should remain authenticated with ESCAPE ...Some services have their own authentication system which is not related to ESCAPE IAM. It should be possible for the user to authenticate with and access these services from within ESAP. The user should remain authenticated with ESCAPE IAM *in addition* to their authentication with the external service.
For example, the user may need to log in to ESO or Rubin Observatory archives to access proprietary data products. After they have logged in, they should be able to add those data products to their shopping basket, and otherwise process them as usual using ESAP.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/80Persistent workspaces within ESAP2022-04-22T13:09:40ZJohn SwinbankPersistent workspaces within ESAPInteractive data analysis sessions should be persistent within ESAP. That is, when a user launches a Jupyter notebook (or similar), they should have access to a persistent filesystem on which they can store and share data, rather than th...Interactive data analysis sessions should be persistent within ESAP. That is, when a user launches a Jupyter notebook (or similar), they should have access to a persistent filesystem on which they can store and share data, rather than that filesystem simply disappearing when the notebook is closed.
This likely means that the filesystem is based on the data lake, using DLaaS (or similar).https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/81Download data from the shopping basket2022-04-22T13:09:48ZJohn SwinbankDownload data from the shopping basketWhen I run a query against e.g. Apertif in ESAP, I am able to download the resultant data, view thumbnails, etc.
When I add the data to the shopping basket, those options vanish: I can simply see some metadata relating to the data I sto...When I run a query against e.g. Apertif in ESAP, I am able to download the resultant data, view thumbnails, etc.
When I add the data to the shopping basket, those options vanish: I can simply see some metadata relating to the data I stored.
This should be changed so that I can download the data, view thumbnails, etc, directly within the basket interface.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/82Share the shopping basket with other users2022-04-22T13:10:02ZJohn SwinbankShare the shopping basket with other usersPer [this ESO use case](https://project.escape2020.de/issues/137), we should make it possible for users to share their shopping basket with other users.
This would likely take the form of multiple users having complete read/write access...Per [this ESO use case](https://project.escape2020.de/issues/137), we should make it possible for users to share their shopping basket with other users.
This would likely take the form of multiple users having complete read/write access to a single basket (with appropriate locking, etc). However, it may also be interesting for one user to be able to take a copy of another user's basket.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/85Header information with VO results2022-04-25T07:26:51ZYan GrangeHeader information with VO resultsWhen I get results in the shopping basket through SAMP (so I assume this will be true for the VO in general) the data looks something like this:
```
| samp | {"archive":"samp","record":["0692110101",47.188035,48.99925,"2012-08-05 23:00:4...When I get results in the shopping basket through SAMP (so I assume this will be true for the VO in general) the data looks something like this:
```
| samp | {"archive":"samp","record":["0692110101",47.188035,48.99925,"2012-08-05 23:00:46.0","2012-08-06 01:02:44.0",7318,2318,"2013-08-23 00:00:00.0","http://nxsa.n1data.lan/nxsa-sl/servlet/data-action-aio?obsno=0692110101&name=OIMAGE&level=PPS&extension=FTZ","XMM EPIC Image. Target: 1SWASPJ030845+485957",2,[650,650],[-0.00111111111111111,0.00111111111111111],"image/fits"]}
```
The last item (record) is basically a list of table contents, but there is no header information available to show what the dat in the column means. I think it would be very useful if that was added to the output :)Nico VermaasNico Vermaashttps://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/93Rucio seems not to find everything2022-07-18T13:26:21ZYan GrangeRucio seems not to find everythingI really don't know what is happening here but if I query the Datalake for the `LOFAR_ASTRON_PANDEY` scope for objects of type `file` that will result in zero files. It will find two files if I actually select DIDs as object type.
Maybe...I really don't know what is happening here but if I query the Datalake for the `LOFAR_ASTRON_PANDEY` scope for objects of type `file` that will result in zero files. It will find two files if I actually select DIDs as object type.
Maybe the issue could be that this view is showing all top-level objects (ie hides all datasets/files/container that themselves are part of a dataset/container since especially that scope has a pretty complete structure to it (by design) :)https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/99D8.2 — Propose an appropriate logical connection between shopping basket entr...2022-04-22T13:26:44ZJohn SwinbankD8.2 — Propose an appropriate logical connection between shopping basket entries and eossr entriesThe goal of this ticket is to identify the information that will help users locate relevant software given the contents of their shopping basket. For example, this might include finding image processing software if their basket includes ...The goal of this ticket is to identify the information that will help users locate relevant software given the contents of their shopping basket. For example, this might include finding image processing software if their basket includes images, or software specialised to work with a particular instrument if their basket includes data from that instrument.
The format and representation of that information is not in scope for now — focus on the logical process, not the technical representation.
Success criteria: a simple list of logical information to cross-match between shopping basket and OSSR. This list can be expanded in the future.https://git.astron.nl/astron-sdc/escape-wp5/esap-general/-/issues/100Register graphical application in eossr2022-04-22T13:23:23ZJohn SwinbankRegister graphical application in eossre.g. container with Topcat & Aladin.e.g. container with Topcat & Aladin.Dave MDave M