Compmake: keep calm and carry on

Contents   ::   Features and limitations  »

Fork me on GitHub

Compmake: keep calm and carry on

compmake is a non-obtrusive module that provides Makefile–like facilities to your Python computations, including:

  • Familiar commands such as make, clean, etc.
  • Zero-effort parallelization.
  • Caching of temporary results: you can interrupt your program (CTRL-C), and restart it without losing what was already computed.
  • A console for inspecting failures and partial completion.

compmake has been designed primarily for handling long computational-intensive batch processes. It assumes that the computational layout is fixed and that all intermediate results can be cached to disk. If these two conditions are met, you can use compmake to gain considerable peace of mind. Read Why using compmake.

Installation

The simplest way to install compmake is:

$ easy_install compmake

or, alternatively, using pip install compmake. You can also fork the project on GitHub.

Basic usage

To use compmake, you have to minimally modify your Python program, such that it can understand the processing layout and the opportunities for parallelization.

images/initial.png

Download a demo example.py that you can try.

You would run the modified program using:

$ compmake example -c make            # runs serially

Parallel execution: To run jobs in parallel, use the parmake command:

$ compmake example -c "parmake n=6"   # runs at most 6 in parallel

There are all sorts of configuration options for being nice to other users of the machine; for example, Compmake can be instructed not to start other jobs if the CPU or memory usage is already above a certain percentage:

$ compmake --max_cpu_load=50 --max_mem_load=50 example -c "clean; parmake"

Console: Moreover, by running compmake example only, you have access to a console that allows you to inspect the status of the computations, cleaning/remaking jobs, etc.

Selective re-make: You can selectively remake part of the computations. For example, suppose that you modify the draw() function, and you want to rerun only the last step. You can achieve that by:

$ compmake example -c "remake draw*"

compmake will reuse part of the computations (func1 and func2) but it will redo the last step.

Failure tolerance: If some of the jobs fail (e.g., they throw an exception), compmake will go forward with the rest.

To try this behavior, download the file example_fail.py. If you run:

$ compmake example_fail -c "parmake n=4"

you will see how compmake completes all jobs that can be completed. If you run again:

$ compmake example_fail -c "make"

Compmake will try again to run the jobs that failed.

Feedback

Compmake is currently developed by Andrea Censi. Contributors are most welcome. Please use the issue tracker on github for bugs and feature requests.

Contents   ::   Features and limitations  »