Recipe rework to allow to redefine parameters and directives

[cimendes]

Currently, the recipe module allows a recipe to by defined as a string of compoents, such as:

available_recipes = {
    "dengue": "integrity_coverage fastqc_trimmomatic filter_poly remove_host bowtie retrieve_mapped megahit",
}

To increase the flexibility of this feature, I suggest reworking how the recipes are implemented to allow the (re)definition of parameters and directives, tailoring each recipe to the purpose it's intended for. For this, there's two foreseeable options:

1. Keep the recipe definition as a sting and implement a step in the engine to setup the directives and parameters. For example:

"test_recipe": "fastq trimmomatic spades={'params':{'A':2}}"

This would require the least changes to flowcraft's engine but it's a cumbersome solution for the implementation of complex recipes.

2. Change de data structure of the recipe to a dictionary. It allows the integration of fairly complex processes but increases the complexity.

[{"input": {"process": "init", "lane": 1},
"output": {"process": "integrity_coverage", "lane": 1}},
{"input": {"process": "integrity_coverage", "lane": 1},
"output": {"process": "fastqc", "lane": 1}}
]

[ODiogoSilva]

Just to provide real example of the same pipeline for both options:

recipe = """
integrity_coverage={'params':'{'minCoverage':'15'}'}
fastqc_trimmomatic
fastqc
check_coverage={'params':'{minCoverage:20}'}
spades
process_spades={'params':'{'spadesMinContigLen':'250','spadesMaxContigs':'10'}'}
pilon
"""

recipe = [{
	"input": {"process": "__init__", "lane": 1},
	"output": {"process": "integrity_coverage", "lane": 1, "params": {"minCoverage": 15}}
}, {
	"input": {"process": "integrity_coverage", "lane": 1},
	"output": {"process": "fastqc_trimmomatic", "lane": 1}
}, {
	"input": {"process": "fastqc_trimmomatic", "lane": 1},
	"output": {"process": "fastqc", "lane": 1}
}, {
	"input": {"process": "fastqc", "lane": 1},
	"output": {"process": "check_coverage", "lane": 1, "params": {"minCoverage": 15}}
}, {
	"input": {"process": "check_coverage", "lane": 1},
	"output": {"process": "spades", "lane": 1}
}, {
	"input": {"process": "spades", "lane": 1},
	"output": {"process": "process_spades", "lane": 1, "params": {'spadesMinContigLen':'250','spadesMaxContigs':'10'}}
}, {
	"input": {"process": "process_spades", "lane": 1},
	"output": {"process": "pilon", "lane": 1}
}]

The second option seems much more verbose, but this is something that only needs to be done once. The first has the advantage of the simplicity, but the second will be more easily extensible if we want to provide more information to processes (other than params)

[tiagofilipe12]

What if we have a function that accepts as arguments a pipeline string and a dictionary of the params to change? Something like:

def recipe_generator(p_string, params_dict):
    # take the p_string and the params_dict and make a recipe like Diogo's second option
    return recipe

Example params_dict

params_dict = {
    "proc1": {"param1": "lala", "param2": "lele"}, 
    "proc2": {"param1": "lolo", "param2": "lili"}
}

or if we want to scale this beyond params:

params_dict = {
    "proc1": {
        "params": {"param1": "lala", "param2": "lele"}, 
        "directives": {"cpus": 2, "memory": "2GB"} 
    }, 
    "proc2": { 
        "params": {"param1": "lolo", "param2": "lili"}
    }
}

The idea with the p_string argument would be to have a simple pipeline string without any complicated special dictionaries like @ODiogoSilva first option. E.g.:

p_string= "integrity_coverage fastqc_trimmomatic fastqc check_coverage spades process_spades pilon"

Then, in the end "available_recipes" would take something like this:

available_recipes = {
    "dengue": recipe_generator("integrity_coverage fastqc_trimmomatic filter_poly remove_host bowtie retrieve_mapped megahit", params_dict)
}

May be this later part can be improved but this is just an idea....

[ODiogoSilva]

I really like this hybrid idea of having a pipeline string and a params dictionary as the input of a fuction/class. However, some modifications are required to allow the same process to be placed multiple times in the same pipeline string. A possible solution would be:

Pipeline string:

p_string = "reads_download ( trimmomatic spades#1 | spades#2 )"

Params dictionary:

directives = {
    "trimmomatic": {
        "params": { ... }, 
        "directives": { ... } 
    }, 
    "spades#1": { 
        "params": { ... },
        "directives": { ... }
    },
    "spades#2": { 
        "params": { ... },
	"directives": { ... }
    }
}

Then, these objects could be provided to a Recipe class:

my_recipe = Recipe(p_string, directives)

Where my_recipe would be the processed pipeline string, ready to enter the pipeline parser and the engine.

This class could then contain methods that deal with the provided directives and produce the correct pipeline string automatically:

class Recipe:

    def __init__(self, p_string, directives):

         return self.process_directives(p_string, directives)

     def process_directives(p_string, directives):
          # code 

The reason for using a class instead of a function is that we may want to add additional methods to the preparation of the recipe.

[tiagofilipe12]

Even better, I think we can define classes that inherit the main Recipe to define in proper "code containers" (rather than spreading it through the main available_recipes dict) each recipe params_dict and p_string. What do you think?

[cimendes]

I like this solution of an hybrid approach to define the recipe and I agree with @tiagofilipe12 to move this to a subclass of Recipe. It cleans up the code nicely and the new approach allows for the necessary flexibility to tailor params and directories to whatever task is needed.