A pipeline framework for python
Documentation | ChangeLog | Examples | API
- Easy to use
- Nearly zero-configuration
- Nice logging
- Highly extendable
- Cloud support naively
pip install -U pipenexample.py
from pipen import Proc, Pipen, run class P1(Proc): """Sort input file""" input = "infile" input_data = ["/tmp/data.txt"] output = "outfile:file:intermediate.txt" script = "cat {{in.infile}} | sort > {{out.outfile}}" class P2(Proc): """Paste line number""" requires = P1 input = "infile:file" output = "outfile:file:result.txt" script = "paste <(seq 1 3) {{in.infile}} > {{out.outfile}}" # class MyPipeline(Pipen): # starts = P1 if __name__ == "__main__": # MyPipeline().run() run("MyPipeline", starts=P1)> echo -e "3\n2\n1" > /tmp/data.txt > python example.py04-17 16:19:35 I core _____________________________________ __ 04-17 16:19:35 I core ___ __ \___ _/__ __ \__ ____/__ | / / 04-17 16:19:35 I core __ /_/ /__ / __ /_/ /_ __/ __ |/ / 04-17 16:19:35 I core _ ____/__/ / _ ____/_ /___ _ /| / 04-17 16:19:35 I core /_/ /___/ /_/ /_____/ /_/ |_/ 04-17 16:19:35 I core 04-17 16:19:35 I core version: 0.17.3 04-17 16:19:35 I core 04-17 16:19:35 I core ╔═══════════════════════════ MYPIPELINE ════════════════════════════╗ 04-17 16:19:35 I core ║ My pipeline ║ 04-17 16:19:35 I core ╚═══════════════════════════════════════════════════════════════════╝ 04-17 16:19:35 I core plugins : verbose v0.14.1 04-17 16:19:35 I core # procs : 2 04-17 16:19:35 I core profile : default 04-17 16:19:35 I core outdir : /home/pwwang/github/pipen/examples/MyPipeline-output 04-17 16:19:35 I core cache : True 04-17 16:19:35 I core dirsig : 1 04-17 16:19:35 I core error_strategy : ignore 04-17 16:19:35 I core forks : 1 04-17 16:19:35 I core lang : bash 04-17 16:19:35 I core loglevel : info 04-17 16:19:35 I core num_retries : 3 04-17 16:19:35 I core scheduler : local 04-17 16:19:35 I core submission_batch: 8 04-17 16:19:35 I core template : liquid 04-17 16:19:35 I core workdir : /home/pwwang/github/pipen/examples/.pipen/MyPipeline 04-17 16:19:35 I core plugin_opts : 04-17 16:19:35 I core template_opts : filters={'realpath': <function realpath at 0x7fc3eba12... 04-17 16:19:35 I core : globals={'realpath': <function realpath at 0x7fc3eba12... 04-17 16:19:35 I core Initializing plugins ... 04-17 16:19:36 I core 04-17 16:19:36 I core ╭─────────────────────────────── P1 ────────────────────────────────╮ 04-17 16:19:36 I core │ Sort input file │ 04-17 16:19:36 I core ╰───────────────────────────────────────────────────────────────────╯ 04-17 16:19:36 I core P1: Workdir: '/home/pwwang/github/pipen/examples/.pipen/MyPipeline/P1' 04-17 16:19:36 I core P1: <<< [START] 04-17 16:19:36 I core P1: >>> ['P2'] 04-17 16:19:36 I verbose P1: in.infile: /tmp/data.txt 04-17 16:19:36 I verbose P1: out.outfile: /home/pwwang/github/pipen/examples/.pipen/MyPipeline/P1/0/output/intermediate .txt 04-17 16:19:38 I verbose P1: Time elapsed: 00:00:02.051s 04-17 16:19:38 I core 04-17 16:19:38 I core ╭═══════════════════════════════ P2 ════════════════════════════════╮ 04-17 16:19:38 I core ║ Paste line number ║ 04-17 16:19:38 I core ╰═══════════════════════════════════════════════════════════════════╯ 04-17 16:19:38 I core P2: Workdir: '/home/pwwang/github/pipen/examples/.pipen/MyPipeline/P2' 04-17 16:19:38 I core P2: <<< ['P1'] 04-17 16:19:38 I core P2: >>> [END] 04-17 16:19:38 I verbose P2: in.infile: /home/pwwang/github/pipen/examples/.pipen/MyPipeline/P1/0/output/intermediate .txt 04-17 16:19:38 I verbose P2: out.outfile: /home/pwwang/github/pipen/examples/MyPipeline-output/P2/result.txt 04-17 16:19:41 I verbose P2: Time elapsed: 00:00:02.051s 04-17 16:19:41 I core MYPIPELINE: 100%|██████████████████████████████| 2/2 [00:06<00:00, 0.35 procs/s] > cat ./MyPipeline-output/P2/result.txt 1 1 2 2 3 3See more examples at examples/ and a more realcase example at:
https://github.com/pwwang/pipen-report/tree/master/example
Plugins make pipen even better.
pipen-annotate: Use docstring to annotate pipen processespipen-args: Command line argument parser for pipenpipen-board: Visualize configuration and running of pipen pipelines on the webpipen-diagram: Draw pipeline diagrams for pipenpipen-dry: Dry runner for pipen pipelinespipen-filters: Add a set of useful filters for pipen templates.pipen-lock: Process lock for pipen to prevent multiple runs at the same time.pipen-log2file: Save running logs to file for pipenpipen-poplog: Populate logs from jobs to running log of the pipelinepipen-report: Generate report for pipenpipen-runinfo: Save running information to file for pipenpipen-verbose: Add verbosal information in logs for pipen.pipen-gcs: A plugin for pipen to handle files in Google Cloud Storage.pipen-deprecated: A pipen plugin to mark processes as deprecated.pipen-cli-init: A pipen CLI plugin to create a pipen project (pipeline)pipen-cli-ref: Make reference documentation for processespipen-cli-require: A pipen cli plugin check the requirements of a pipelinepipen-cli-run: A pipen cli plugin to run a process or a pipelinepipen-cli-gbatch: A pipen cli plugin to submit pipeline to Google Batch Jobs