Compmake: keep calm and carry on¶
compmake is a non-obtrusive module that provides Makefile–like facilities to your Python computations, including:
- Familiar commands such as make, clean, etc.
- Zero-effort parallelization.
- Caching of temporary results: you can interrupt your program (CTRL-C), and restart it without losing what was already computed.
- A console for inspecting failures and partial completion.
compmake has been designed primarily for handling long computational-intensive batch processes. It assumes that the computational layout is fixed and that all intermediate results can be cached to disk. If these two conditions are met, you can use compmake to gain considerable peace of mind. Read Why using compmake.
The simplest way to install compmake is:
$ easy_install compmake
or, alternatively, using pip install compmake. You can also fork the project on GitHub.
To use compmake, you have to minimally modify your Python program, such that it can understand the processing layout and the opportunities for parallelization.
Download a demo example.py that you can try.
You would run the modified program using:
$ compmake example -c make # runs serially
Parallel execution: To run jobs in parallel, use the parmake command:
$ compmake example -c "parmake n=6" # runs at most 6 in parallel
There are all sorts of configuration options for being nice to other users of the machine; for example, Compmake can be instructed not to start other jobs if the CPU or memory usage is already above a certain percentage:
$ compmake --max_cpu_load=50 --max_mem_load=50 example -c "clean; parmake"
Console: Moreover, by running compmake example only, you have access to a console that allows you to inspect the status of the computations, cleaning/remaking jobs, etc.
Selective re-make: You can selectively remake part of the computations. For example, suppose that you modify the draw() function, and you want to rerun only the last step. You can achieve that by:
$ compmake example -c "remake draw*"
compmake will reuse part of the computations (func1 and func2) but it will redo the last step.
Failure tolerance: If some of the jobs fail (e.g., they throw an exception), compmake will go forward with the rest.
To try this behavior, download the file example_fail.py. If you run:
$ compmake example_fail -c "parmake n=4"
you will see how compmake completes all jobs that can be completed. If you run again:
$ compmake example_fail -c "make"
Compmake will try again to run the jobs that failed.
Still interested? Start with the tutorial Basic compmake usage.