Asp Forum - rake task dependencies: via timestamps table in database

jandot

6/6/2008 8:43:00 AM

Hi all,

There is some interest in the bioinformatics community for using rake
as a workflow tool (see e.g. http://www.bioinformaticszen.com/2008/05/organised-bioinformatics-ex...).
Rake could be ideal for this type of work: a typical workflow will
take data and perform a first set of conversions on it (i.e. a task),
followed by a second set of conversions (that is dependent on the
first task), and so on.

However, bioinformaticians try to keep their data in databases rather
than files. And we found we need some workarounds to get dependencies
working. Does anyone know if it would be very difficult to add
functionality to rake to check a meta table in a database for
timestamps of tasks rather than looking at timestamps of files? I was
thinking of a table looking like the one below:

table: meta
task
modified_on
==============================================
001_load_data
20080602_0831
002_calculate_averages 20080602_0845
003_make_histogram_of_averages 20080602_0851

The rakefile would then contain:

task :001_load_data do
<do stuff>
<automatically update record in meta table>
end

task :002_calculate_averages => [:001_load_data] do
<do stuff>
<automatically update record in meta table>
end

task :003_make_histogram_of_averages => [:002_calculate_averages] do
<do stuff>
<automatically update record in meta table>
end

So if we had reloaded the data (001), then the timestamp for that task
in the meta table would be later than the one for task 002. As a
result, task 002 would automatically have to be rerun if we were to
run task 003.

I'd very much like to know if anyone has an idea how rake can be
extended this way. Basically, the dependency checker has to be
extended to look into a fixed table in a database...

Many thanks,
Jan Aerts

-
=================================
Dr Jan Aerts
Senior Bioinformatician
Genome Dynamics and Evolution Group
Wellcome Trust Sanger Institute
Hinxton
Cambridge CB10 1SA
UK

phone: +44 (0)1223 - 494732
web: http://www.sanger.ac.uk/Tea...

10 Answers

Pit Capitain

6/9/2008 8:17:00 AM

2008/6/6 jandot <jan.aerts@gmail.com>:
> There is some interest in the bioinformatics community for using rake
> as a workflow tool (...)
> However, bioinformaticians try to keep their data in databases rather
> than files. And we found we need some workarounds to get dependencies
> working. Does anyone know if it would be very difficult to add
> functionality to rake to check a meta table in a database for
> timestamps of tasks rather than looking at timestamps of files?

Hi Jan, if you look at the source code of rake's FileTask, you'll see
that this shouldn't be very difficult. The code consists of only four
methods and is easy to read. Feel free to ask again if you have more
questions.

Regards,
Pit

jandot

6/11/2008 2:15:00 PM

On Jun 9, 9:17 am, Pit Capitain <pit.capit...@gmail.com> wrote:
> 2008/6/6 jandot <jan.ae...@gmail.com>:
>
> > There is some interest in the bioinformatics community for using rake
> > as a workflow tool (...)
> > However, bioinformaticians try to keep their data in databases rather
> > than files. And we found we need some workarounds to get dependencies
> > working. Does anyone know if it would be very difficult to add
> > functionality to rake to check a meta table in a database for
> > timestamps of tasks rather than looking at timestamps of files?
>
> Hi Jan, if you look at the source code of rake's FileTask, you'll see
> that this shouldn't be very difficult. The code consists of only four
> methods and is easy to read. Feel free to ask again if you have more
> questions.
>
> Regards,
> Pit

Thanks for that pointer, Pit. I think I got quite far now based on
FileTask. But something is still wrong. The trouble is that I have no
idea where, so can't really ask specific questions...
It looks like the block passed to a task is not executed.

I've put what I already have on github: http://github.com/jandot/biorake/t...

There's a sample directory with an example Rakefile that should work
once the extension is fixed. In addition, there are two test suites
copied from the file tests. Unfortunately, many of the tests still
fail.

If anybody could have a look at the tests and help to get them
running, I would be very thankfull.

Cheers,
jan.

Fattuchus

8/16/2010 9:03:00 AM

On Aug 15, 11:05 pm, poisoned rose <pro...@realitychex.com> wrote:
> Whew. Just look at all the pitiful cowards letting off steam with
> shallow, knowingly unfair shots at me, sensing that "the coast is
> clear." And counting on cheap, validating backslaps from others (which
> they'll get, of course). So now TNMM can pretend he's equipped to debate
> music with me...Rich can pretend that I'm picking fights with him rather
> than the reverse...Bip Bop can pretend that I'm just a "troll," etc.
> etc. All without the pesky intrusion of counterarguments and/or reality.
> Well, don't let me rain on the parade. Though I guess this post may
> demand another 20 or so overcompensating replies, in order for all the
> petulant chicks to rebuild that desired illusion of henhouse triumph.
>
> Otherwise, I feel like dropping out for a spell. There was something
> just too world-turned-upside-down about that Joanna Newsom thread, as I
> watched Rich fumble for an angle to start another hypocritical fight and
> realized that he had actually sunk lower than Topaz (who was far more
> constructive in comparison). Yikes. How much more Bizarro-universe can
> this forum become? (BTW, the song lyric is actually "A.D. 1," not "81."
> Pun. The overemphasis on JN's birth year reminds me of the fudged "28IF"
> hoopla.)
>
> PS Most interesting to see our closet racist Bip Bop at last openly
> courting our open racist TNMM for camaraderie. But since he didn't poke
> any RMBers' own vulnerable spots, of course his latest "those cretinous
> Negros I see on the street" snarl was diplomatically ignored.

He's baaaaaaccckkkk.

topaz

8/16/2010 12:05:00 PM

Donna

8/16/2010 1:53:00 PM

topaz wrote:
>
> On Aug 15, 11:05 pm, poisoned rose <pro...@realitychex.com> wrote:
>
> > There was something
> > just too world-turned-upside-down about that Joanna Newsom thread, as I
> > watched Rich fumble for an angle to start another hypocritical fight and
> > realized that he had actually sunk lower than Topaz (who was far more
> > constructive in comparison). Yikes. How much more Bizarro-universe can
> > this forum become?
>
>
> Thank you for calling me "constructive".

Let me guess.... You're the type who likes to see the glass as half
full.

BlackMonk

8/16/2010 5:14:00 PM

"poisoned rose" <prose9@realitychex.com> wrote in message
news:prose9-7BBEAD.20055515082010@reserved-multicast-range-not-delegated.example.com...

> PS Most interesting to see our closet racist Bip Bop at last openly
> courting our open racist TNMM for camaraderie. But since he didn't poke
> any RMBers' own vulnerable spots, of course his latest "those cretinous
> Negros I see on the street" snarl was diplomatically ignored.

It was noticed. Not much of a surprise.

topaz

8/16/2010 8:24:00 PM

On Aug 16, 9:52 am, Donna <tom.r...@ix.netcom.com> wrote:
> topaz wrote:
>
> > On Aug 15, 11:05 pm, poisoned rose <pro...@realitychex.com> wrote:
>
> > > There was something
> > > just too world-turned-upside-down about that Joanna Newsom thread, as I
> > > watched Rich fumble for an angle to start another hypocritical fight and
> > > realized that he had actually sunk lower than Topaz (who was far more
> > > constructive in comparison). Yikes. How much more Bizarro-universe can
> > > this forum become?
>
> > Thank you for calling me "constructive".
>
> Let me guess.... You're the type who likes to see the glass as half
> full.

Well I am not perfect but I try to be positive and give praise when I
can. Considering what many people are dealing with I guess I should
be grateful that I can "see the glass". Having an attitude of
gratitude for all of us really is the way to go. After all I could be
blind God forbid!

The Walrus was Danny

8/17/2010 11:45:00 PM

> Otherwise, I feel like dropping out for a spell.

Oh Guru, don't be silly. I Like you...and I actually do..I think your
posts are OK...just a bit misinterperated that's all..I wouldn't have
started the thread if I hadn't have missed you..don't drop
out...you'll miss my NYC visit as it happens blow by blow!! Me posting
up in the NG from the Hotel on the West side! Surely that's not to be
missed!

Danny

Eric Ramon

8/18/2010 12:32:00 AM

On Aug 17, 4:44 pm, TheWalrusWasDanny <dannyisthewal...@tesco.net>
wrote:
> > Otherwise, I feel like dropping out for a spell.
>
> Oh Guru, don't be silly. I Like you...and I actually do..I think your
> posts are OK...just a bit misinterperated that's all..I wouldn't have
> started the thread if I hadn't have missed you..don't drop
> out...you'll miss my NYC visit as it happens blow by blow!! Me posting
> up in the NG from the Hotel on the West side! Surely that's not to be
> missed!
>
> Danny

be careful. As I remember from my New York days, what they call a
"hotel" might be disappointing. Check for bed bugs!

The Walrus was Danny

8/18/2010 1:22:00 AM

On 18 Aug, 01:31, Eric Ramon <ramon.e...@gmail.com> wrote:
> On Aug 17, 4:44 pm, TheWalrusWasDanny <dannyisthewal...@tesco.net>
> wrote:
>
> > > Otherwise, I feel like dropping out for a spell.
>
> > Oh Guru, don't be silly. I Like you...and I actually do..I think your
> > posts are OK...just a bit misinterperated that's all..I wouldn't have
> > started the thread if I hadn't have missed you..don't drop
> > out...you'll miss my NYC visit as it happens blow by blow!! Me posting
> > up in the NG from the Hotel on the West side! Surely that's not to be
> > missed!
>
> > Danny
>
> be careful. As I remember from my New York days, what they call a
> "hotel" might be disappointing. Check for bed bugs!

Oh God Eric...you're talking to a man who has slept in Aleppo,
Syria...damascus...amman...cairo...istanbul..vynlnius...beijing...I
have no problem with all that sort of stuff..anyway it's on the West
Side and looks pretty posh to me!!

Danny

comp.lang.ruby

rake task dependencies: via timestamps table in database

jandot

Pit Capitain

jandot

Fattuchus

topaz

Donna

BlackMonk

topaz

The Walrus was Danny

Eric Ramon

The Walrus was Danny

x Login to ForumsZone