Asp Forum - Strategies for autoloading ruby files

Luke A. Kanies

8/1/2006 9:54:00 PM

Hi all,

Following up on my post about complicated problems, here's a much
simpler problem that I need to tackle.

There are many parts of my framework that are extensible, and I want
people to be able to create extensions without modifying the core
framework, akin to how you don't have to specifically load each model
and controller in Rails.

Is there a relatively standard way of doing that with Ruby? This is
what I'm doing in one of my projects, Facter[1]:

# Now see if we can find any other facts
$:.each do |dir|
fdir = File.join(dir, "facter")
if FileTest.exists?(fdir) and FileTest.directory?(fdir)
Dir.glob("#{fdir}/*.rb").each do |file|
# Load here, rather than require, because otherwise
# the facts won't get reloaded if someone calls
# "loadfacts". Really only important in testing,
# but, well, it's important in testing.
begin
load file
rescue => detail
warn "Could not load %s: %s" %
[file, detail]
end
end
end
end

I've tried further copying Rails by checking that loaded files create
the objects it seems they should create (e.g., here they should create a
fact named after the file), but users constantly complained about those
warnings.

Is this the best way? Any other recommended mechanisms?

I'd actually like the autoloading to work either on-demand (e.g., look
for 'facter/myfact' if I ask for the 'myfact' fact), or load all of them
right now.

Hmm. In fact, it'd be ideal if you could write a module that allows you
to register autoload paths (e.g., I'd use 'facter', 'puppet/type',
'puppet/type/package', 'puppet/type/service', and lots more), and then
just have hooks into that module so that you could autoload all
instances, or just load a specific name.

1 - http://reducti.../proje...

--
The Number 1 Sign You Have Nothing to Do at Work...
The 4th Division of Paperclips has overrun the Pushpin Infantry
and General White-Out has called for a new skirmish.
---------------------------------------------------------------------
Luke Kanies | http://reducti... | http://m...

9 Answers

Ara.T.Howard

8/1/2006 10:02:00 PM

Brian Palmer

8/1/2006 10:04:00 PM

You can use `Module.const_missing` to autoload a file the first time
the class in that file is mentioned in the code. So you could write
an `Object.const_missing(name)` method that translates the constant
into a file name of an expected format, then load that file on-the-fly.

Amusingly, this is exactly the code example that's given in the
Pickaxe version 2 under Module.const_missing, along with a note that
the code is in very poor style and is a "perverse kind of autoload
functionality".

:)

-- Brian

On Aug 1, 2006, at 3:53 PM, Luke Kanies wrote:

> Hi all,
>
> Following up on my post about complicated problems, here's a much
> simpler problem that I need to tackle.
>
> There are many parts of my framework that are extensible, and I
> want people to be able to create extensions without modifying the
> core framework, akin to how you don't have to specifically load
> each model and controller in Rails.
>
> Is there a relatively standard way of doing that with Ruby? This
> is what I'm doing in one of my projects, Facter[1]:
>
> # Now see if we can find any other facts
> $:.each do |dir|
> fdir = File.join(dir, "facter")
> if FileTest.exists?(fdir) and FileTest.directory?(fdir)
> Dir.glob("#{fdir}/*.rb").each do |file|
> # Load here, rather than require, because
> otherwise
> # the facts won't get reloaded if someone calls
> # "loadfacts". Really only important in testing,
> # but, well, it's important in testing.
> begin
> load file
> rescue => detail
> warn "Could not load %s: %s" %
> [file, detail]
> end
> end
> end
> end
>
> I've tried further copying Rails by checking that loaded files
> create the objects it seems they should create (e.g., here they
> should create a fact named after the file), but users constantly
> complained about those warnings.
>
> Is this the best way? Any other recommended mechanisms?
>
> I'd actually like the autoloading to work either on-demand (e.g.,
> look for 'facter/myfact' if I ask for the 'myfact' fact), or load
> all of them right now.
>
> Hmm. In fact, it'd be ideal if you could write a module that
> allows you to register autoload paths (e.g., I'd use 'facter',
> 'puppet/type', 'puppet/type/package', 'puppet/type/service', and
> lots more), and then just have hooks into that module so that you
> could autoload all instances, or just load a specific name.
>
> 1 - http://reducti.../proje...
>
> --
> The Number 1 Sign You Have Nothing to Do at Work...
> The 4th Division of Paperclips has overrun the Pushpin Infantry
> and General White-Out has called for a new skirmish.
> ---------------------------------------------------------------------
> Luke Kanies | http://reducti... | http://m...
>
>

James Gray

8/1/2006 10:12:00 PM

On Aug 1, 2006, at 5:04 PM, Brian Palmer wrote:

> You can use `Module.const_missing` to autoload a file the first
> time the class in that file is mentioned in the code. So you could
> write an `Object.const_missing(name)` method that translates the
> constant into a file name of an expected format, then load that
> file on-the-fly.

Ruby has a function for this though:

$ ri -T Kernel#autoload
-------------------------------------------------------- Kernel#autoload
autoload(module, filename) => nil
------------------------------------------------------------------------
Registers filename to be loaded (using Kernel::require) the first
time that module (which may be a String or a symbol) is accessed.

autoload(:MyModule, "/usr/local/lib/modules/my_module.rb")

James Edward Gray II

Luke A. Kanies

8/1/2006 10:23:00 PM

Brian Palmer wrote:
> You can use `Module.const_missing` to autoload a file the first time the
> class in that file is mentioned in the code. So you could write an
> `Object.const_missing(name)` method that translates the constant into a
> file name of an expected format, then load that file on-the-fly.

Ironically, I use almost no constants in my code. Parent classes get
constants, but subclasses are almost always created dynamically[1], like so:

Puppet::Type.newtype(:user) do
...
end

Then I always refer to that class by the name 'user' and never by the
constant. This is partially because it's easier for me, but also
because I've written a custom language (no, not a DSL[2]) for Puppet,
and most of those names are exposed in that language. E.g., you create
a 'user' type like above, and users can create user elements in Puppet:

user { luke:
comment => "Luke Kanies",
uid => 100,
shell => bash,
ensure => exists
}

Puppet supports lots of package types (e.g., dpkg, apt, rpm,
darwinports, gems) and service types (init, smf, etc.); I specify the
name when I create the class, and users can select those types by name,
either in the language or directly in Ruby if they're directly accessing
the library.

In other words, my problem is not how to hook into the autoload, it's
the guts behind it.

> Amusingly, this is exactly the code example that's given in the Pickaxe
> version 2 under Module.const_missing, along with a note that the code is
> in very poor style and is a "perverse kind of autoload functionality".

1 - I do assign constants to them, but only because class inspection
during error printing did not work consistently unless there's a
constant assigned to the class. I just tried reproducing this but could
not; I don't remember the details well enough, I guess.

2 - See the FAQ, http://reducti.../projects/puppe..., the
fourth question.

--
An expert is a person who has made all the mistakes that can be made
in a very narrow field. - Niels Bohr
---------------------------------------------------------------------
Luke Kanies | http://reducti... | http://m...

Brian Palmer

8/1/2006 10:53:00 PM

On Aug 1, 2006, at 4:11 PM, James Edward Gray II wrote:

> Ruby has a function for this though:
>
> $ ri -T Kernel#autoload
> --------------------------------------------------------
> Kernel#autoload
> autoload(module, filename) => nil
> ----------------------------------------------------------------------
> --
> Registers filename to be loaded (using Kernel::require) the first
> time that module (which may be a String or a symbol) is accessed.
>
> autoload(:MyModule, "/usr/local/lib/modules/my_module.rb")
>
> James Edward Gray II
>

Cool, I didn't realize that was already baked in to Kernel.

On Aug 1, 2006, at 4:22 PM, Luke Kanies wrote:

> Puppet supports lots of package types (e.g., dpkg, apt, rpm,
> darwinports, gems) and service types (init, smf, etc.); I specify
> the name when I create the class, and users can select those types
> by name, either in the language or directly in Ruby if they're
> directly accessing the library.
>
> In other words, my problem is not how to hook into the autoload,
> it's the guts behind it.

I misunderstood your question, sorry about that.

-- Brian

Luke A. Kanies

8/1/2006 11:03:00 PM

ara.t.howard@noaa.gov wrote:
>
> this may be helpful (or not! ;-))
>
> http://codeforpeople.com/lib/ruby...

Clearly, I should always check your codebase before asking any questions. :)

> i use it for plugins in my satellite processing system like so
>
>
> class PlugIn
> def self.inherited other
> Dynaload::export self, 'plugin' => true
> end
> end
>
>
> then plugins are written something like
>
>
> require 'plugin'
>
> class MyPlugIn < Plugin
> #
> # stuff
> #
> end
>
>
> then, assume that is saved in myplugin.rb, then i can do
>
>
> require 'dynaload'
>
> loaded = Dynaload::dynaload 'myplugin.rb'
>
> plugins = loaded.select{|klass, attributes| attributes['plugin'] == true}
>
>
> make sense?

Yeah. Half of my question, though, was figuring out which files to
load, although that's easy enough to resolve, I guess. I'm assuming it
does get moderately expensive as I do more of these, but I normally load
by name (e.g., look for 'puppet/type/package/dpkg.rb' anywhere in the
search path), and only sometimes load all files in all search directories.

I started out using the self.inherited method for all of this kind of
work, but it just got too cumbersome eventually, at least partially
because the object passed to 'inherited' has not yet been initialized.
That is, with the following code:

class MyClass < ParentClass
@name = :myclass
end

When 'inherited' is called, '@name' is not set, so you have to stick it
in an array and then do shenanigans later to make it act like a hash.
This is clearly a trivial example, because 'name' can (usually) be
autodetermined from the class name, but you also can't test whether
methods are defined, for instance. With the 'new<thing>' methods, I can
do any initialization on the class that I want (e.g., setting @name
using accessor methods), class_eval the passed block, and then do any
analysis I need. See [1].

When I switched from using inherited to creating 'new<thing>' methods,
it unleashed a flood of productivity. I was astounded -- I barely slept
for two weeks because so much was suddenly so easy when it had been so
difficult for so long.

1 -
http://reducti.../downloads/puppet/apidocs/classes/Puppet/Type.ht...

Notice all of the other 'new*' methods around this one. I use it
everywhere and I love it.

--
Neonle will continue to be rude, and will nretend that you had a small
stroke which makes you unable to say or see the letter "n". Stunid
nractical joke, if you ask me. Bunch of noon-heads, huh?
-- Fred Barling, Humorscope
---------------------------------------------------------------------
Luke Kanies | http://reducti... | http://m...

Ara.T.Howard

8/1/2006 11:15:00 PM

Luke A. Kanies

8/1/2006 11:30:00 PM

ara.t.howard@noaa.gov wrote:
> module Plugable
> def self.included other
> ...
> end
> end
>
> class MyPlugin
> include Plugable
> end

At this point, I'm looking at creating an Autoload class, and classes
wanting to autoload will create an instance:

Puppet::Autoload.new(self, "puppet/type")

They'll then be able to store that instance, or retrieve it from the class:

Puppet::Autoload[self].load(:package)

That'll handle the mechanics of the loading. I'll have an additional
:loadall method that will search through $:, loading everything that
matches.

I want to keep the work around plugin loading -- that is, what to do
with the class and how to initialize it -- in the class supporting
plugins (i.e., the class with the 'new<thing>' method), because most
initialization is pretty custom. I suppose I could start with a default
method in a module, though; I hadn't thought much about that.

A significant part of this for me is that I have lots of classes that
are related to each other -- the Puppet::Type class is related to its
subclasses, the Puppet::Type::Package class is related to the
Puppet::Type::Package::PkgType classes (ugh, constants are hard), the
'service' class is related to each 'svctype' class, etc. Calling
'new<thing>' directly on the related class makes that relationship very
explicit and pleasantly clear.

> yes - i've got a fair number of class factories lying around myself.
> Class.new is powerfull since you can easily parameterize your classes.

I *love* Class.new and Module.new. Ruby was great before I discovered
them, but with them, I'm definitely hooked. Well, Class.new and
class_eval, anyway.

> on a related note i used 'traits' alot with this sort of thing since it
> avoids
> needing to initialize all those class vars directly down the inheritence
> chain.

Most of the variables involved are actually class instance variables
(which, I understand, others don't use that much, but which I use
pervasively and could not live without).

I need to just add traits into my codebase; I think it would simplify a
lot of what I'm doing.

/me adds one more item to the todo

> i'll read over some over of you code but gotta run to get the kid...

Comments, positive or negative, are always appreciated.

--
Today I dialed a wrong number...The other person said, "Hello?" and
I said, "Hello, could I speak to Joey?"...
They said, "Uh...I don't think so...he's only 2 months old."
I said, "I'll wait." -- Steven Wright
---------------------------------------------------------------------
Luke Kanies | http://reducti... | http://m...

F. Senault

8/2/2006 9:49:00 AM

Le 01 août à 23:53, Luke Kanies a écrit :

> Is this the best way? Any other recommended mechanisms?

I don't know about recommended, but I've made two or three programs
using plugins, and here's my method, in the main script :

Dir.glob($conf[:path]) do |f|
$mtime = File.mtime(f)
begin
require "#{f}"
rescue LoadError => err
puts err
end
end
Plugin.plugins.each |p| do
...
end

class Plugin
@@plugins = []
@@mtimes = {}
def Grapher.inherited(c)
@@plugins.push(c)
@@mtimes[c.to_s] = $mtime
end
def Plugin.plugins
@@plugins
end
def Plugin.mtimes
@@mtimes
end
end

And the plugins just have to :

class LittlePlugin < Plugin
...
end

Note : I needed here to know the modification time of the plugin to
eventually reinitialize some data. I think that using a global variable
is more than clunky. Any thoughts on a better way ?

Fred
--
The DBAs will be whining because when you kill -9 Oracle, you get bits
and pieces left all over the place, and when re-started Oracle gets all
squeamish and frightened about what happened to it's predecessor. (Its
very literal pre-decessor...) (Dan Holdsworth in the SDM)

comp.lang.ruby

Strategies for autoloading ruby files

Luke A. Kanies

Ara.T.Howard

Brian Palmer

James Gray

Luke A. Kanies

Brian Palmer

Luke A. Kanies

Ara.T.Howard

Luke A. Kanies

F. Senault

x Login to ForumsZone