Fix incorrect installation of CWL files

Many CWL files were being installed in the wrong directory. This
has been fixed. This commit fixes RAP-249.

setup.py

 from setuptools import setup
 import os
-import pathlib
 
 
 def read(rel_path):
@@ -12,6 +11,13 @@ def read(rel_path):
         return fp.read()
 
 
+data_files = []
+for top in ('lofar-cwl/steps', 'steps', 'subworkflow', 'workflows'):
+    for root, _, files in os.walk(top):
+        cwl_files = [os.path.join(root, f) for f in files if f.endswith('.cwl')]
+        if cwl_files:
+            data_files.append((os.path.join('share/prefactor', root), cwl_files))
+
 setup(
     name='prefactor3-cwl',
     version='0.3.0',
@@ -28,7 +34,5 @@ setup(
     install_requires=[
         'prefactor@git+https://github.com/lofar-astron/prefactor.git'
     ],
-    data_files=[(os.path.join('share', 'prefactor', dir),
-                 [str(p) for p in pathlib.Path(dir).glob('**/*.cwl')]) for dir in
-                ('lofar-cwl', 'steps', 'subworkflow', 'workflows')]
+    data_files=data_files
 )

[David Rafferty]

Does "root" appear twice here (once in cwl_files and once in the this line)?

[Marcel Loose]

Yes, but that's on purpose. data_files is a list of tuples of (dir, files) (see https://docs.python.org/3/distutils/setupscript.html#installing-additional-files), where dir must contain the destination directory, and files the list of files that must be installed in that directory.

[Marcel Loose]

BTW: I do have a different implementation for lines 14-19 that doesn't use os.walk, but glob.glob instead. It's a bit slower, but maybe you consider it more readable:

cwl_files = collections.defaultdict(list)
for dir in ('lofar-cwl', 'steps', 'subworkflow', 'workflows'):
    for file in glob.glob(os.path.join(dir, '**/*.cwl'), recursive=True):
        cwl_files[os.path.join('share', 'prefactor', os.path.dirname(file))].append(file)
data_files = cwl_files.items()

[David Rafferty]

The version with os.walk is fine with me -- though I did indeed misread it initially :)