[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

parallel tasks with rake ?

quixoticsycophant

11/1/2006 3:34:00 PM

I was excited to switch my makefile system to rake, but I just noticed
rake lacks a (GNU) make option I can't live without: -j4. It's like
"make & ; make & ; make & ; make &" but with the parallel jobs
properly synchronized. On a quad-core machine this makes a world of
difference, being around four times faster than a simple 'make'.

As I understand it, rake should be able to determine which targets can
be built in parallel just as (GNU) make does. rake's multitask
feature isn't quite right because it shouldn't be up to the user to
determine parallel vs non-parallel tasks. In fact, the user should
probably never decide parallelism by fiat; even if the user happens to
be correct, it creates a maintenance problem whereby additional tasks
will eventually render the parallel assertion false.

Perhaps there is something I'm missing about 'multitask' ?
6 Answers

Wilson Bilkovich

11/1/2006 3:49:00 PM

0

On 11/1/06, quixoticsycophant@yahoo.com <quixoticsycophant@yahoo.com> wrote:
> I was excited to switch my makefile system to rake, but I just noticed
> rake lacks a (GNU) make option I can't live without: -j4. It's like
> "make & ; make & ; make & ; make &" but with the parallel jobs
> properly synchronized. On a quad-core machine this makes a world of
> difference, being around four times faster than a simple 'make'.
>
> As I understand it, rake should be able to determine which targets can
> be built in parallel just as (GNU) make does. rake's multitask
> feature isn't quite right because it shouldn't be up to the user to
> determine parallel vs non-parallel tasks. In fact, the user should
> probably never decide parallelism by fiat; even if the user happens to
> be correct, it creates a maintenance problem whereby additional tasks
> will eventually render the parallel assertion false.
>
> Perhaps there is something I'm missing about 'multitask' ?
>
>

How does Make figure out which jobs can be run in parallel? In my
experience, it just runs everything in parallel, and things break when
the tasks aren't safely parallelizable. (e.g. doing make -j4 install
on a FreeBSD port)

quixoticsycophant

11/1/2006 4:19:00 PM

0


Wilson Bilkovich wrote:
> On 11/1/06, quixoticsycophant@yahoo.com <quixoticsycophant@yahoo.com> wrote:
> > I was excited to switch my makefile system to rake, but I just
> > noticed rake lacks a (GNU) make option I can't live without: -j4.
> > It's like "make & ; make & ; make & ; make &" but with the
> > parallel jobs properly synchronized. On a quad-core machine this
> > makes a world of difference, being around four times faster than a
> > simple 'make'.
> >
> > As I understand it, rake should be able to determine which targets
> > can be built in parallel just as (GNU) make does. rake's
> > multitask feature isn't quite right because it shouldn't be up to
> > the user to determine parallel vs non-parallel tasks. In fact,
> > the user should probably never decide parallelism by fiat; even if
> > the user happens to be correct, it creates a maintenance problem
> > whereby additional tasks will eventually render the parallel
> > assertion false.
> >
> > Perhaps there is something I'm missing about 'multitask' ?
>
> How does Make figure out which jobs can be run in parallel? In my
> experience, it just runs everything in parallel, and things break when
> the tasks aren't safely parallelizable. (e.g. doing make -j4 install
> on a FreeBSD port)

The Make program figures it out from the dependency graph.
Unfortunately, the 'install' target you mention is a fake target which
depends on nothing. Such uses of 'install' are common but incorrect.
As a workaround, 'install' (and any other wrongly written targets)
should be treated as a special case and should not be run with -j.

Hugh Sasse

11/1/2006 4:55:00 PM

0

quixoticsycophant

11/1/2006 7:11:00 PM

0


Hugh Sasse wrote:
> Would marking it as a .PHONY help? I see little advice in the GNU
> make manual about how to get this sort of thing right for parallel
> make. Where should one look for such info?

First of all, the workaround isn't so bad:

$ make -j4 && make install

In order for 'make -j4 install' to work, the install target must
simply depend on everything it needs. Needless to say, most Makefile
writers assume a 'make && make install' invocation and don't write
proper prerequisites for the install target. In other words, they
assume a single-process make.

Marking the target .PHONY just means it will run unconditionally (so
for example Make won't be confused by a file named 'install') and
won't affect the correctness or incorrectness of the dependencies.

To get back to the original point, the dependency graph is what
decides parallelism, not the other way around. Unless I misunderstand
the rake docs, the 'multitask' feature of rake is dangerous and should
be removed (see my original post).

Rake is a very nice tool, but it needs a -j equivalent in order to be
considered as an alternative to make (at least in my case). I put it
out to enterprising rubyists to give rake genuine parallel task
execution.

James Gray

11/1/2006 7:32:00 PM

0

On Nov 1, 2006, at 1:15 PM, quixoticsycophant@yahoo.com wrote:

> Rake is a very nice tool, but it needs a -j equivalent in order to be
> considered as an alternative to make (at least in my case). I put it
> out to enterprising rubyists to give rake genuine parallel task
> execution.

I bet the pieces of this could be written up as a fun Ruby Quiz or
two. Determining what can be done in parallel from the dependancy
graph in particular sounds like a great quiz topic. Then you could
Rakify the solutions and submit a patch. Just a thought...

James Edward Gray II

guillaume.marcais

11/1/2006 8:33:00 PM

0

quixoticsycophant@yahoo.com wrote:
> To get back to the original point, the dependency graph is what
> decides parallelism, not the other way around.

In the current implementation, is the dependency graph explicitly built
by rake? Or does it simply recursively run the dependencies as it goes
along?

Guillaume.