Skip to content
Snippets Groups Projects
Select Git revision
  • 701d2a59efbc13d2f634e8c8fc5e1fb825f977e8
  • main default protected
  • trunk
3 results

esap-db-python-client

  • Open with
  • Download source code
  • Your workspaces

      A workspace is a virtual sandbox environment for your code in GitLab.

      No agents available to create workspaces. Please consult Workspaces documentation for troubleshooting.

    • Name Last commit Last update
    esap_client
    tests
    .gitignore
    .pre-commit-config.yaml
    README.md
    poetry.lock
    pyproject.toml
    README.md

    ESAP-DB

    ESAP-DB is an ESAP component providing managed database services. This Python package provides an interface to ESAP-DB.

    Installation

    After cloning the esap-db-python-client repository:

    git clone git@git.astron.nl:astron-sdc/esap-db-python-client.git
cd esap-db-python-client
poetry install
poetry shell

    Creation of a project

    In order to use ESAP-DB in Python, a client needs to be instantiated:

    from esap_client import ESAPClient
client = ESAPClient()

    A project defines a scope, in which collections of tables can be created. Let's create our first project:

    project = client.create_project('my_project', 'My first project 😀')

    We can check that the project has been created:

    client.projects

    Creation of a dataset

    In this project, let's create a dataset, i.e., a group of tables:

    dataset = project.create_dataset('my_dataset', 'My first dataset 😃')

    We can check that the dataset has been created:

    project.datasets

    Creation of a table from a Pandas DataFrame

    Pandas DataFrames can be used as a data source.

    import numpy as np
import pandas as pd
df = pd.DataFrame({
    'a': np.random.randn(10),
    'b': np.random.randn(10),
    'c': np.random.randn(10),
})
table = dataset.create_table_from(df, "my_pandas_table")

    The returned table object does not hold any data. It is a proxy to an actual database table, managed by ESAP-DB. We can check that the table has been added to the dataset:

    dataset.tables

    Creation of a table from an external resource

    External CSV files can be imported into a table:

    table = dataset.create_table_from('https://www.data.gouv.fr/fr/datasets/r/c0f59f00-3ab2-4f31-8a05-d317b43e9055', sep=';')
df = table.aspandas()

    Creation of a table from a query

    The tables from different datasets can be combined in a query.

    import pandas as pd

dataset1 = project.create_dataset('dataset1')
df_fruits = pd.DataFrame({'x': 6 * ['fruit'], 'y': list('🍓🥝🍇🍐🍏🍍')})
fruits = dataset1.create_table_from(df_fruits, 'fruits')
fruits.aspandas()

dataset2 = project.create_dataset('dataset2')
df_vegetables = pd.DataFrame({'x': 5 * ['vegetable'], 'y': list('🥑🌽🥒🍆🥦')})
vegetables = dataset2.create_table_from(df_vegetables, 'vegetables')
vegetables.aspandas()

dataset3 = project.create_dataset('dataset3')
query = """
SELECT * FROM dataset1.fruits
UNION
SELECT * FROM dataset2.vegetables
ORDER BY x, y
"""
food = dataset3.create_table_as(query, 'food')
food.aspandas()

    Creation of a table from an ESAP ESFRI query

    We can store the result of an ESAP-API query into a table that belongs to the dataset that we have just created.

    query = {
    "level": "raw",
    "collection": "imaging",
    "ra": 342.16,
    "dec": 33.94,
    "fov": 10,
    "archive_uri": "apertif"
}
table = dataset.create_table_from_esap_gateway_query(query, 'apertif_table', 'Apertif cone search')

    The new table apertif_table can be imported by the Python client as Pandas dataframes:

    table.aspandas()