Recently I’ve had users of CIRCE in need of being able to take a set of input values and iterate over them in parallel in Matlab to save time. Most people would consider utilizing the Parallel Computing Toolbox, but that would require modification of code, as well as a license for the toolbox itself. I have an alternative to allow users to run their Matlab code in parallel with minimal modification in conjunction with an arraytask in Sun Grid Engine. I’m sure other schedulers support arraytasks, so this can be adapted to whatever scheduler is in production.
To review- an arraytask is a type of HPC job that allows you to run a piece of software multiple times simultaneously, each time with a different set of inputs. This is usually done for compute tasks that are embarassingly parallel in nature. Here’s an example of a simple arraytask in SGE:
#!/bin/bash #$ -N my_app_array_run #$ -o output.$JOB_ID #$ -cwd #$ -pe smp 1 #$ -l h_rt=00:45:00 #$ -t 1-48 ./my_app inputfile.$SGE_TASK_ID
This creates 48 tasks to run the program ./my_app with the argument inputfile.1 to inputfile.48 with each task. Now, how do we use this tool with Matlab?
Environment variables. First, use this to export the appropriate value to the environment:
#!/bin/bash #$ -cwd #$ -l h_rt=1:00:00 #$ -j y #$ -N matlab_arraytask #$ -o output.$JOB_ID # HOW MANY TASKS? #$ -t 6 task=1 # SPECIFY A LIST OF INPUTS EQUAL TO NUMBER OF TASKS for i in 1 2 3 4 5 6; do if [[ "$task" -eq "$SGE_TASK_ID" ]]; then INPUTVALUE=$i fi let task=$task+1 done export INPUTVALUE matlab -nodisplay -r function
Next, add this to your Matlab code to convert the environment variable into a Matlab variable:
inputvalue = str2num(getenv('INPUTVALUE'));
With this, and sufficient computing resources, you can cut the time needed to run all of your inputs to the time needed to run just one.