Welcome to Compi’s documentation!

Basics

• Introduction
  • What is Compi?
  • Main features
  • The Compi ecosystem
  • Installation
• Writing pipelines
  • The XML file for pipelines
  • A pipeline example
  • Defining pipeline parameters
  • Defining tasks
  • Defining tasks metadata
  • Validating a pipeline
  • Viewing the pipeline as a graph
• Running pipelines
  • Sample pipeline
  • Executing the pipeline using a parameters file
  • Executing the pipeline using command-line pipeline parameters
  • Resuming a pipeline
  • Advanced execution control
• Application development
  • A practical example

Advanced

• Logging
  • Show task standard outputs
  • Remove Compi messages
  • Log task outputs into files
  • Log only specific tasks
  • Exclude tasks from logging
• Partial execution
  • Executing a single pipeline task
  • Run the pipeline until a specific task
  • Run all the dependencies of a specific task
  • Start the pipeline execution at a specific task
  • Start the pipeline execution after a specific task
  • Running a subpath of tasks
• Custom runners
  • What are Compi runners
  • Creating a custom runner
  • Examples of useful runners
• Custom interpreters
  • How to define an interpreter for a task
  • Examples
• Common patterns
  • Resume pipeline execution in Docker images of Compi pipelines
  • Run a task only if a directory exists and have files
  • Use Docker images from a Docker image of a Compi pipeline
• FAQ
  • Why does the log file of a Python task is not updated?
  • Does compi-dk require compi to be installed?
  • How to make a parameter optional?
  • Is it possible to control the maximum number of parallel tasks of a specific task?