Contents
Single-host multiprocessing¶
To use single-host multiprocessing, instead of using make, simply use parmake:
@: parmake [jobs]
Optionally, you can specify the number of processes to spawn:
@: parmake n=11 [jobs]
If you don’t specify a number, Compmake will use the number of detected processors. If your jobs are IO-bound rather than CPU-bound, you should specify a larger number.
What’s happening under the hood is that Compmake spawns n workers thread using the multiprocessing module. So be aware that each job will run in a different process.
Troubleshooting problems¶
Compmake turns your original program into a parallel program. For you, it is almost as easy as it could ever get. However, there might be some issues to be aware about that are documented here.
Some libraries are picky¶
Some libraries don’t like to run in child processes. For example, matplotlib on OS X will complain and not work properly. The errors might mention exec(). For example, this is thrown by matplotlib:
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
The solution is to divide your tasks in processing tasks and visualization tasks. Use parmake to run the former group, and then run make for the latter.