@@ -9,14 +9,14 @@ For filling the ldv specification database with data dumps from the LTA (tables
## Create combined table
First create a table in accordance with the `DataProduct` model:
> CREATE TABLE lofardata_dataproduct(
obs_id CHARACTER(15),
oid_source CHARACTER(15),
dataproduct_source CHARACTER(20),
dataproduct_type CHARACTER(50),
project CHARACTER(50),
location CHARACTER(50),
activity CHARACTER(50),
surl CHARACTER(200),
obs_id VARCHAR(15),
oid_source VARCHAR(15),
dataproduct_source VARCHAR(20),
dataproduct_type VARCHAR(50),
project VARCHAR(50),
location VARCHAR(50),
activity VARCHAR(50),
surl VARCHAR(200),
filesize BIGINT,
additional_meta JSON);
...
...
@@ -28,9 +28,9 @@ There are 3 tables in the ldvadmin database:
They all need to be queried to retrieve the wanted information via the commands below. Note that these queries transform information to the wanted model.
raw_dataproducts:
>`INSERT INTO lofardata_dataproduct
>INSERT INTO lofardata_dataproduct
SELECT obsid AS obs_id, obsid_source AS oid_source, 'LOFAR LTA', dp_type AS dataproduct_type, project, location, activity, uri AS surl, size AS filesize, json_build_object('dysco_compression', dysco, 'antenna_set', antenna_set, 'instrument_filter', instrument_filter) AS additional_meta
FROM astrowise.raw_dataproducts;`
FROM astrowise.raw_dataproducts;
pl_dataproducts:
>INSERT INTO lofardata_dataproduct
...
...
@@ -47,7 +47,7 @@ Since the surl index in our model is unique but the combined table, by above com
#### Create unique id:
> ALTER TABLE lofardata_dataproduct ADD COLUMN id bigserial PRIMARY KEY;
#### Delete duplicate surls:
> DELETE FROM lofardata_dataproduct x USING lofardata_dataproduct y WHERE x.id < y.id AND x.surl = y.surl;`
> DELETE FROM lofardata_dataproduct x USING lofardata_dataproduct y WHERE x.id < y.id AND x.surl = y.surl;
## Dump and restore
On the database server, dump the newly combined table with the correct format and the date in the filename for future reference as follows:
