Asp Forum - Memcached: Dealing with unknown keys

Tony

2/14/2008 4:46:00 AM

I was wondering if there is a way to handle reading keys when you don't
know what the keys are, using memcache-client...
--
Posted via http://www.ruby-....

13 Answers

Ben Bleything

2/14/2008 6:34:00 PM

On Thu, Feb 14, 2008, Tony Garcia wrote:
> I was wondering if there is a way to handle reading keys when you don't
> know what the keys are, using memcache-client...

Do you mean discovering what keys are stored in the cache or reading the
value of MYSTERY KEYS(tm)?

Ben

Marcin Raczkowski

2/14/2008 7:40:00 PM

Tony

2/14/2008 8:55:00 PM

Ben Bleything wrote:
> On Thu, Feb 14, 2008, Tony Garcia wrote:
>> I was wondering if there is a way to handle reading keys when you don't
>> know what the keys are, using memcache-client...
>
> Do you mean discovering what keys are stored in the cache or reading the
> value of MYSTERY KEYS(tm)?
>
> Ben

I suppose a little more clarification would help:

I need to pass information (for example, an email address) based on user
name from a rails application to a disconnected ruby process through the
cache. The external process has no way to know what users are being
passed to it. I had wanted to use the user name as the key, but it seems
that I can't find the values of current keys in the cache - at least not
with memcache-client.

So far, about the best I've found is to set the cache as a hash:
Cache["users"] = { "user1"=>"user1@example.com",
"user2"=>"user2@example.com" }

Is there any better way?
--
Posted via http://www.ruby-....

Ben Bleything

2/14/2008 10:11:00 PM

On Fri, Feb 15, 2008, Tony Garcia wrote:
> I need to pass information (for example, an email address) based on user
> name from a rails application to a disconnected ruby process through the
> cache. The external process has no way to know what users are being
> passed to it. I had wanted to use the user name as the key, but it seems
> that I can't find the values of current keys in the cache - at least not
> with memcache-client.

It sounds like you're trying to use the cache as a message queue, when
you should probably be using both in tandem... that is, store user data
in the cache, but send messages along the queue with the user name, so
the disconnected process can look it up in the cache.

Caching in general (and memcache in particular) are intended to speed up
slow data access, not to act as data repositories.

> So far, about the best I've found is to set the cache as a hash:
> Cache["users"] = { "user1"=>"user1@example.com",
> "user2"=>"user2@example.com" }

If you need to iterate over the usernames and the iterator has no way of
knowing what they are, that's your best bet. Not a great idea, though.

Ben

Tony

2/14/2008 10:34:00 PM

Ben Bleything wrote:
> On Fri, Feb 15, 2008, Tony Garcia wrote:
>> I need to pass information (for example, an email address) based on user
>> name from a rails application to a disconnected ruby process through the
>> cache. The external process has no way to know what users are being
>> passed to it. I had wanted to use the user name as the key, but it seems
>> that I can't find the values of current keys in the cache - at least not
>> with memcache-client.
>
> It sounds like you're trying to use the cache as a message queue, when
> you should probably be using both in tandem... that is, store user data
> in the cache, but send messages along the queue with the user name, so
> the disconnected process can look it up in the cache.

Only the username, email address, and a code are being stored - there
are no messages or other items that need to be passed along.

> Caching in general (and memcache in particular) are intended to speed up
> slow data access, not to act as data repositories.

Granted - and I am using memcache as a queue in this case. I've done a
lot of looking and can't find much else that will do what I need - and
Joyent has memcache already installed.

If anyone has a better suggestion, I'd love to hear it. This is the best
I've been able to figure out on my own.

>> So far, about the best I've found is to set the cache as a hash:
>> Cache["users"] = { "user1"=>"user1@example.com",
>> "user2"=>"user2@example.com" }
>
> If you need to iterate over the usernames and the iterator has no way of
> knowing what they are, that's your best bet. Not a great idea, though.

I'm not too thrilled with using that method either, but as I said, it's
the best I can come up with on my own.

Any suggestions?
--
Posted via http://www.ruby-....

Marcin Raczkowski

2/14/2008 10:41:00 PM

WEll.

There are libraries ready to do exactly what you want.
I dunno why you try to reinvite the well that so many people did before you.

anyway check out ruby-talk archive some days ago someone created library
to crate message queue using memcache - not that i think it's good idea
- there are much better solutions - or even tuplespace which is in
stdlib doing just that.

anyway have phun

Ben Bleything

2/14/2008 10:43:00 PM

On Fri, Feb 15, 2008, Tony Garcia wrote:
> Only the username, email address, and a code are being stored - there
> are no messages or other items that need to be passed along.

Ah, okay. It sounds to me like the username, email, and the code *are*
a message, but it's a matter of semantics for sure :D

> Granted - and I am using memcache as a queue in this case. I've done a
> lot of looking and can't find much else that will do what I need - and
> Joyent has memcache already installed.

Twitter uses a queue that talks the memcache protocol, but I can't
remember what it's called. Geoffrey Grosenbach recently posted a cool
blog post about message queues, too.

http://nubyonrails.com/articles/about-this-blog-beanstalk-messa...

> If anyone has a better suggestion, I'd love to hear it. This is the best
> I've been able to figure out on my own.

I think the main problem is that memcache just isn't intended for what
you're trying to do. The hash solution you came up with will get the
job done, but it's a bit of a hack (as we both seem to agree). If it
works though, there's not really any reason to change.

I think if you want to change, you'll need to find some way to pass
messages, but hopefully someone else will have a more creative solution.

Cheers,
Ben

Tony

2/14/2008 11:12:00 PM

Ben Bleything wrote:
> On Fri, Feb 15, 2008, Tony Garcia wrote:
>> Granted - and I am using memcache as a queue in this case. I've done a
>> lot of looking and can't find much else that will do what I need - and
>> Joyent has memcache already installed.
>
> Twitter uses a queue that talks the memcache protocol, but I can't
> remember what it's called. Geoffrey Grosenbach recently posted a cool
> blog post about message queues, too.
>
> http://nubyonrails.com/articles/about-this-blog-beanstalk-messa...

I read that. It's what pointed me toward memcache, actually. Again, part
of the problem is that I'm on a shared Joyent server, and I was trying
to work with what they had. Installing a beanstalk server isn't a very
good option for me at the moment.

>> If anyone has a better suggestion, I'd love to hear it. This is the best
>> I've been able to figure out on my own.
>
> I think the main problem is that memcache just isn't intended for what
> you're trying to do. The hash solution you came up with will get the
> job done, but it's a bit of a hack (as we both seem to agree). If it
> works though, there's not really any reason to change.

My biggest concern is scalability. I'm pretty sure I'm going to have to
change things before too long.

Thanks
--
Posted via http://www.ruby-....

romeda@gmail.com

2/15/2008 10:31:00 AM

On Feb 14, 3:11 pm, Tony Garcia <ton...@dslextreme.com> wrote:
> Ben Bleything wrote:
> > On Fri, Feb 15, 2008, Tony Garcia wrote:
>
> I read that. It's what pointed me toward memcache, actually. Again, part
> of the problem is that I'm on a shared Joyent server, and I was trying
> to work with what they had. Installing a beanstalk server isn't a very
> good option for me at the moment.
>
> >> If anyone has a better suggestion, I'd love to hear it. This is the best
> >> I've been able to figure out on my own.
>
> > I think the main problem is that memcache just isn't intended for what
> > you're trying to do. The hash solution you came up with will get the
> > job done, but it's a bit of a hack (as we both seem to agree). If it
> > works though, there's not really any reason to change.
>
> My biggest concern is scalability. I'm pretty sure I'm going to have to
> change things before too long.

FWIW, we (Twitter) were on Joyent and used Starling (our message queue
that speaks the memcache protocol) there with no problems. All you
need to get started is to do a "gem install starling" and run it with
write permissions to /var/spool/starling (you can specify a different
path since it's unlikely you can write to /var on joyent). Once it's
running, you can use the normal memcache library to connect to <server
ip>:22122 and use set() to put things in the queue and get() to take
them out again.

Hope that helps!

b.

Tony

2/15/2008 5:23:00 PM

Blaine Cook wrote:

>
> FWIW, we (Twitter) were on Joyent and used Starling (our message queue
> that speaks the memcache protocol) there with no problems. All you
> need to get started is to do a "gem install starling" and run it with
> write permissions to /var/spool/starling (you can specify a different
> path since it's unlikely you can write to /var on joyent). Once it's
> running, you can use the normal memcache library to connect to <server
> ip>:22122 and use set() to put things in the queue and get() to take
> them out again.
>
> Hope that helps!
>
> b.

I'll check into that - thanks!
--
Posted via http://www.ruby-....

comp.lang.ruby

Memcached: Dealing with unknown keys

Tony

Ben Bleything

Marcin Raczkowski

Tony

Ben Bleything

Tony

Marcin Raczkowski

Ben Bleything

Tony

romeda@gmail.com

Tony

x Login to ForumsZone