|
|
|
# Add an archive
|
|
|
|
|
|
|
|
In ESAP, data discovery model is composed by three types of complementary entities:
|
|
|
|
|
|
|
|
- **Archives:** Data sources associated with an observatory/facility research infrastructure or organization that provides access to the datasets, e.g., SKA, CTA, LHC…
|
|
|
|
- **Catalogues:** Specific data resources, e.g., data tables, imaging catalogues…
|
|
|
|
- **Datasets:** A queryable source, i.e., a list of images, data files or some sort of collection. Many datasets can reference the same catalogue, but may access different components, e.g., different columns of a table. Datasets belong uniquely to a single archive.
|
|
|
|
|
|
|
|
ESAP data discovery schematic:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# Prerequisites
|
|
|
|
|
|
|
|
ESAP flow: users send a query to the front-end, the front-end communicates it to the backend, that query its data resources. As a prerequisite, you must have a **REST API with access to the archives you want to upload into ESAP**.
|
|
|
|
|
|
|
|
# 1. Set up your local Development ESAP instance
|
|
|
|
|
|
|
|
Instructions on how to set your local development ESAP instance are [here](/Build-&-Deploy-ESAP/Development-mode-(localhost)).
|
|
|
|
|
|
|
|
Once installed, create a Django superuser by typing in a terminal:
|
|
|
|
|
|
|
|
```
|
|
|
|
python manage.py createsuperuser --settings=esap.settings.dev
|
|
|
|
```
|
|
|
|
|
|
|
|
To access the backend:
|
|
|
|
|
|
|
|
```
|
|
|
|
cd esap-api-gateway
|
|
|
|
python3 -m venv env
|
|
|
|
source env/bin/activate
|
|
|
|
cd esap
|
|
|
|
./run.sh
|
|
|
|
```
|
|
|
|
|
|
|
|
Then go to: [http://localhost:5555/esap-api/query](http://localhost:5555/esap-api/query)
|
|
|
|
|
|
|
|
To access the front-end:
|
|
|
|
|
|
|
|
```
|
|
|
|
cd esap-api-gateway/esap/esap-gui
|
|
|
|
npm run start
|
|
|
|
```
|
|
|
|
|
|
|
|
The GUI will opens¡ automatically in your browser
|
|
|
|
|
|
|
|
# 2. Backend
|
|
|
|
|
|
|
|
In general, you need to:
|
|
|
|
|
|
|
|
- **Update the database.**
|
|
|
|
- **Implement 2 python files:**
|
|
|
|
|
|
|
|
```
|
|
|
|
esap-api-gateway/esap/esap/configuration/{name_configuration}.py
|
|
|
|
esap-api-gateway/esap/query/api/services/{name_connector}.py
|
|
|
|
```
|
|
|
|
- **Update 1 python file:**
|
|
|
|
|
|
|
|
```
|
|
|
|
esap-api-gateway/esap/query/api/services/query_controler.py
|
|
|
|
```
|
|
|
|
|
|
|
|
## 2.1 Update the database
|
|
|
|
|
|
|
|
The ESAP API gateway has a database per app (query, rucio, ida…) and the database configuration is SQLite3. To access the database, in your development environment go to: [http://localhost:5555/esap-api/admin/query](http://localhost:5555/esap-api/admin/query). This opens the Django Admin application. You need the Django superuser account with admin privileges to edit the databases.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**(+)Add Archive:** (CTA will be used as an example from here on). Fill all the boxes for your Archive and make a note of the URI.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
By saving this and mark the box "Visible" and Order "5", I added CTA to the GUI in the fifth position.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**(+) Add Catalogue:** Fill all the boxes for your catalogue. Make a note of the URI and the parameters.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
By saving this, I added CTA to the GUI
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**(+) Add Dataset:** Fill all the boxes for your dataset. Select the your dataset catalogue and dataset archive. Make a note of the service connector: the service connector is composed any the name of a module and the name of a class separated by a dot: {module}.{class}
|
|
|
|
|
|
|
|
###### 
|
|
|
|
|
|
|
|
## 2.2 Configuration file
|
|
|
|
|
|
|
|
This file specifies the fields in the GUI to search for your dataset, so how the GUI is going to be populated and what fields are going to be shown. Create a python file:
|
|
|
|
|
|
|
|
```
|
|
|
|
esap-api-gateway/esap/esap/configuration/{name_configuration}.py
|
|
|
|
```
|
|
|
|
|
|
|
|
The structure form is ∫specified by 2 data structures:
|
|
|
|
|
|
|
|
- `query_schema`: Mandatory. Basic form of the query required.
|
|
|
|
- `uri_schema`: Optional. Extra attributes on individual fields (to get more information about this, see Hugh Dickinson’s talks, links provided at the end of this documents).
|
|
|
|
|
|
|
|
Here I show as an example `cta.py`: each box in the GUI is defined by one block:
|
|
|
|
|
|
|
|
```
|
|
|
|
title = "cta"
|
|
|
|
logo = "https://upload.wikimedia.org/wikipedia/commons/d/d1/Cherenkov_Telescope_Array_logo.jpg"
|
|
|
|
|
|
|
|
query_schema = {
|
|
|
|
"name": "cta",
|
|
|
|
"title": "CTA Query",
|
|
|
|
"type": "object",
|
|
|
|
"properties": {
|
|
|
|
"site": {
|
|
|
|
"type": "string",
|
|
|
|
"title": "site",
|
|
|
|
"default": "La Palma",
|
|
|
|
"enum": ["La Palma"],
|
|
|
|
"enumNames": ["La Palma"]
|
|
|
|
},
|
|
|
|
"azimut": {
|
|
|
|
"type": "string",
|
|
|
|
"title": "azimut",
|
|
|
|
"default": "20 deg",
|
|
|
|
"enum": ["20 deg"],
|
|
|
|
"enumNames": ["20 deg"]
|
|
|
|
},
|
|
|
|
"altitude": {
|
|
|
|
"type": "string",
|
|
|
|
"title": "altitude",
|
|
|
|
"default": "0 deg",
|
|
|
|
"enum": ["0 deg"],
|
|
|
|
"enumNames": ["0 deg"]
|
|
|
|
},
|
|
|
|
"particle": {
|
|
|
|
"type": "string",
|
|
|
|
"title": "particle",
|
|
|
|
"default": "proton",
|
|
|
|
"enum": ["proton","gamma"],
|
|
|
|
"enumNames": ["Proton","Gamma"]
|
|
|
|
},
|
|
|
|
"catalog": {
|
|
|
|
"type": "string",
|
|
|
|
"title": "Catalog",
|
|
|
|
"enum": ["cta"],
|
|
|
|
"enumNames": ["Cta"]
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
ui_schema = {#"keyword": {"ui:help": "e.g. CTA"},
|
|
|
|
#"particle": {"ui:placeholder": "Proton"},
|
|
|
|
"catalog": {"ui:widget": "hidden"}}
|
|
|
|
```
|
|
|
|
|
|
|
|
And this is how it looks in the GUI: 
|
|
|
|
|
|
|
|
## 2.3 Service connector The service connector is a python class.
|
|
|
|
|
|
|
|
This file communicates with your archive REST API to retrieve the data. I is necessary to create one service connector per dataset. Create a python file in:
|
|
|
|
|
|
|
|
```
|
|
|
|
esap-api-gateway/esap/query/api/services/{name_connector}.py
|
|
|
|
```
|
|
|
|
|
|
|
|
Two functions:
|
|
|
|
|
|
|
|
- `construct_query`: returns a list of query strings that can be interpreted by `run_query`.
|
|
|
|
- `run_query`: receives as input the string built by `construct_query`, retrieves the requested data and sends it to the front-end.
|
|
|
|
|
|
|
|
## 2.4 Connect service connector with the backend
|
|
|
|
|
|
|
|
Edit the Python file:
|
|
|
|
|
|
|
|
```
|
|
|
|
esap/query/api/services/query_controler.py
|
|
|
|
```
|
|
|
|
|
|
|
|
Import your service, as you defined in the + Add Dataset {module} step as your service connector. In the formula instantiate_connector, add your own service.
|
|
|
|
|
|
|
|
# 3. Front-end
|
|
|
|
|
|
|
|
The ESAP front-end is written using the JavaScript React framework.
|
|
|
|
|
|
|
|
In general, you need to:
|
|
|
|
|
|
|
|
Implement 2 Javascript files:
|
|
|
|
|
|
|
|
```
|
|
|
|
esap-api-gateway/esap/esap-gui/src/utils/form/parse{your_archive_Name}Form.js
|
|
|
|
esap-api-gateway/esap/esap-gui/src/components/services/query_results/{your_archive_Name}Results.js
|
|
|
|
```
|
|
|
|
|
|
|
|
Update 3 Javascript files:
|
|
|
|
|
|
|
|
```
|
|
|
|
esap-api-gateway/esap/esap-gui/src/utils/form/parseQueryForm.js
|
|
|
|
esap-api-gateway/esap/esap-gui/src/components/query/QueryCatalogs.js
|
|
|
|
esap-api-gateway/esap/esap-gui/src/components/services/query_results/QueryResults.js
|
|
|
|
```
|
|
|
|
|
|
|
|
1. Implement the Javascript file:
|
|
|
|
|
|
|
|
```
|
|
|
|
esap-api-gateway/esap/esap-gui/src/utils/form/parse{your_archive_Name}Form.js
|
|
|
|
```
|
|
|
|
|
|
|
|
For example, take one of the other files, such as `parseZenodoForm.js`, and change “Zenodo” for your archive name.
|
|
|
|
|
|
|
|
2. Update the Javascript file:
|
|
|
|
|
|
|
|
```
|
|
|
|
esap-api-gateway/esap/esap-gui/src/utils/form/parseQueryForm.js
|
|
|
|
```
|
|
|
|
|
|
|
|
Import the file `/parse{your_archive_Name}Form.js` you previously implemented, and add it to the function `parseQueryForm`. This file determines which catalogue was selected and prepares a query.
|
|
|
|
|
|
|
|
3. Implement the Javascript file:
|
|
|
|
|
|
|
|
```
|
|
|
|
esap-api-gateway/esap/esap-gui/src/components/services/query_results/{your_archive_Name}Results.js
|
|
|
|
```
|
|
|
|
|
|
|
|
4. Update the Javascript file:
|
|
|
|
|
|
|
|
```
|
|
|
|
esap-api-gateway/esap/esap-gui/src/components/services/query_results/QueryResults.js
|
|
|
|
```
|
|
|
|
|
|
|
|
Import the file `/parse{your_archive_Name}Form.js` you previously implemented, and add it to the function `parseQueryForm`.
|
|
|
|
|
|
|
|
5. Update the Javascript file:
|
|
|
|
|
|
|
|
```
|
|
|
|
esap-api-gateway/esap/esap-gui/src/components/query/QueryCatalogs.js
|
|
|
|
```
|
|
|
|
|
|
|
|
Add your catalogue to the function `QueryCatalogs`.
|
|
|
|
|
|
|
|
* **Useful material**: Talks by Hugh Dickinson: "Integrating ESAP with the Zooniverse" [recording](https://indico.in2p3.fr/event/22271/) and [slides](https://docs.google.com/presentation/d/1Y4i97njODImYyZx4C-OcElsxvgW9h99bdIzv1UQKBe0/edit?usp=sharing). "Adding new Data Services to ESAP" [recording](https://astron.zoom.us/rec/play/aLxaNZvjcRWuhBSOlfnJA-u2FYRECstQCnS-KPlnXlDJsIfP6JGbtipc4tvdkUk_VubZ2bWxnmU3TUMU.XA5AlrPPK7aKp5ds?continueMode=true&\_x_zm_rtaid=YN2iHbKzRtGd9poZLqRJdQ.1669712873928.f62a7a4144ee66016384be1ce4b895ed&\_x_zm_rhtaid=992) and [slides](https://docs.google.com/presentation/d/1a5Qh_dzlWw-YC2ibggKwKOSY8lgubsynRmeU6-M5vcI/edit#slide=id.p). |
