Asp Forum - fsync on stdout for mod_rewrite

Eric Anderson

12/11/2004 6:31:00 AM

I have a script that I want to ensure has flushed stdout after ever line
of output. I have $stdout.sync=true but when I tried to do $stdout.fsync
I get an Invalid Argument error. Not what I expect according to the docs.

My goal is that I have a script providing lookups for mod_rewrite in
Apache. It hands me the HTTP_HOST header on $stdin and I return the path
where it should look for a specific website on $stdout. It seems to work
well but every now and then it returns the wrong answer. My only two
possibilities that I can see for this problems are:
1) Apache is asking the script the next request before the first request
is answered. But I am using the RewriteLock directive so Apache should
have the requests synchronized.
2) My other option is that the buffer is not getting flushed. Then
another request comes it and both answers are outputted with the first
answer being the wrong answer.

The script seems to perform perfect when run from the command line. Just
not in Apache. Sometime is returns the right answer sometimes it returns
the wrong answer. Even when make the same request (i.e. refresh).

See http://httpd.apache.org/docs-2.0/mod/mod_re... if you are
curious on the mod_rewrite semantics. If posting the script would help I
can post that also.

I appreciate any pointers.

Eric

23 Answers

Glenn Parker

12/11/2004 1:12:00 PM

Eric Anderson wrote:
> I have a script that I want to ensure has flushed stdout after ever line
> of output. I have $stdout.sync=true but when I tried to do $stdout.fsync
> I get an Invalid Argument error. Not what I expect according to the docs.

fsync is only meaningful for files in a file system. You probably want
to use $stdout.flush, but if you set $stdout.sync = true then
$stdout.flush is already being done implicitly.

> If posting the script would help I
> can post that also.

Maybe describing a little bit more about the connection between Apache
and your script would help, too. Is a new script process created for
every URL? If so, then this is likely not a buffering issue. If not,
then you might be failing to reset some lingering state in the Ruby
interpreter between "calls" from Apache.

--
Glenn Parker | glenn.parker-AT-comcast.net | <http://www.tetrafoi...

Eric Anderson

12/11/2004 5:06:00 PM

Glenn Parker wrote:
> fsync is only meaningful for files in a file system. You probably want
> to use $stdout.flush, but if you set $stdout.sync = true then
> $stdout.flush is already being done implicitly.

I first had $stdout.flush. Then I switched to sync when I found out
about it. But according to the docs the operating system might no flush
it. I wasn't sure if it applied to $stdout but I wanted to issue the
command just in case.

> Maybe describing a little bit more about the connection between Apache
> and your script would help, too. Is a new script process created for
> every URL? If so, then this is likely not a buffering issue. If not,
> then you might be failing to reset some lingering state in the Ruby
> interpreter between "calls" from Apache.

I have looked through the script 100 times and from I can tell
everything should be ok for each request. The protocol between Apache
and the script is:

Apache send the key followed by a newline character to the stdin of the
script. In this case I am telling Apache to send the HTTP Host header.
My script then does some logic to determine what to return. It usually
is a path to the website being requested. The script is started when
Apache starts up and continues to run as long as Apache runs. The path
returned is followed by a newline character (the spec says you can also
send it the characters NULL). The docs give it an example in perl with
the following script:

My script is as follows. Any suggestions or improve or fix the problem
are greatly appreciated. You can probably guess the structure of the
database from the SQL statement. The idea is that it is supposed to
return the name that is the closest to what was passed in. So if you get
a request for foo.bar.com and you have a site called bar.com it returns
bar.com. If you get a request from www.baz.bar.com and you have a site
called baz.bar.com it will return baz.bar.com instead of just bar.com.
There are a couple exceptions. For example account.<any domain>.com
should return account.realsimplehosting.com. Also anything with svn in
it should just return domain name and not the entire path. Also there is
an effort to retry requests after failure. After 10 failures it will
just return error.realsimplehosting.com. The script is attached to this
email. I attached it so the lines won't get wrapped.
#!/usr/bin/ruby

require 'dbi'

# CONSTANTS
RSH_VIRTUAL = 1
RSH_WEB_PATH = "/virtual/#{RSH_VIRTUAL}/web/"
NOT_FOUND_PATH = "#{RSH_WEB_PATH}notfound.realsimplehosting.com"
ERROR_PATH = "#{RSH_WEB_PATH}error.realsimplehosting.com"
PORTAL_PATH = "#{RSH_WEB_PATH}?.realsimplehosting.com"
PORTAL_SPECIFIERS = ['webreports', 'database', 'email', 'account', 'cp']

# Put these in the global context
$database = nil
$stmt = nil

# For caching the most recent answer. We may start caching more than this so
# that popular websites get a quicker response but I figured at a min we need
# to cache the most recent since each request calls this lookup several times.
last_request = '';
last_answer = '';

# To protect against infinate retries
loop_protect = 10

def init_database
$database = DBI.connect 'DBI:Mysql:rsh', 'username', 'password'
sql = <<-SQL
SELECT
accounts.id AS account,
primary_domain.name AS path,
LENGTH( aliases.name ) AS size
FROM
accounts INNER JOIN
sites
ON accounts.id = sites.account INNER JOIN
domains AS primary_domain
ON sites.primary_domain = primary_domain.id
INNER JOIN
domains AS aliases
ON sites.id = aliases.site
WHERE
(? LIKE CONCAT( '%', aliases.name )) AND
((sites.expire_date IS NULL) OR
(sites.expire_date <= CURRENT_DATE()))
ORDER BY size DESC
SQL
$stmt = $database.prepare sql
end

init_database
$stdout.sync = true
$stdin.each do | request |
request.chomp!
catch (:done) do
if request == last_request
throw :done
end
last_request = request
last_answer = nil

# Does DBI think we are good to go?
init_database unless !$database.nil? && $database.connected? &&
$database.ping

begin
$stmt.execute request
if $stmt.rows == 0
$stmt.cancel
last_answer = NOT_FOUND_PATH
throw :done
end
row = $stmt.fetch
if request =~ /svn/i
last_answer = row[1]
$stmt.cancel
throw :done
end
PORTAL_SPECIFIERS.each do |key|
dev_key = "#{key}-dev"
re = Regexp.new key, true
dev_re = Regexp.new dev_key, true
last_answer = PORTAL_PATH.sub '?', key if
request =~ re
last_answer = PORTAL_PATH.sub '?', dev_key if
request =~ dev_re
end
last_answer = "/virtual/#{row[0]}/web/#{row[1]}" if
last_answer.nil? or request =~ /bypass/
$stmt.cancel
rescue # In case the library gives us an error
# Try reiniting the connection a couple of times.
# Then just start directing to the error site
unless loop_protect
last_request = ''
last_answer = ERROR_PATH
end
loop_protect -= 1 unless loop_protect == 0
init_database
retry
end
end
puts last_answer
end

Glenn Parker

12/11/2004 7:43:00 PM

Eric Anderson wrote:
> My script is as follows.

I'm not too familiar with embedding SQL in Ruby, so I may not be much
help here, but my spidey-sense tingles when I look at the way the
database connection is manipulated here.

I would try two debug modes, one where the database connection closed
and re-opened on every Apache request, and another mode where the script
is restarted for every Apache request. It might help narrow down the
problem to eliminate lingering state in the continuous connection as a
culprit.

--
Glenn Parker | glenn.parker-AT-comcast.net | <http://www.tetrafoi...

Dominik Werder

12/11/2004 9:13:00 PM

clear last_request after the puts so that not accidentially an old value
can be given to apache.. maybe overcautious, anyway.. :)
bye!
Dominik

Eric Anderson

12/12/2004 2:43:00 PM

Dominik Werder wrote:
> clear last_request after the puts so that not accidentially an old
> value can be given to apache.. maybe overcautious, anyway.. :)

last_request stores the last request for the next request. It is a basic
caching mechanism. For every web request this script is called a couple
of times because of the way mod_rewrite works. Therefore I want to cache
the most recent answer so I don't end up with a bunch of SQL queries for
each request. Also if a site is popular it will cut down on the number
of SQL requests although a more extensive caching would be better for
that. If last_request == request then we give the same answer as the
last request.

Eric

Eric Anderson

12/12/2004 3:01:00 PM

Glenn Parker wrote:
> I'm not too familiar with embedding SQL in Ruby, so I may not be much
> help here, but my spidey-sense tingles when I look at the way the
> database connection is manipulated here.

I'm just using straight ruby. The SQL is just a Ruby here-doc. The
database connection is just created once unless something goes wrong. If
something goes wrong it tries to re-init the connection. After 10 times
it gives up.

> I would try two debug modes, one where the database connection closed
> and re-opened on every Apache request, and another mode where the script
> is restarted for every Apache request. It might help narrow down the
> problem to eliminate lingering state in the continuous connection as a
> culprit.

I'll give that a try, but I don't think it is a state issue. If things
are getting flushed properly then I am leaning towards some kind of
concurrency problem even though I am using the rewrite lock. I need to
see if am using the prefork model in Apache or some threaded model.
Maybe if it is using the threaded model that is causing some problem in
relation to ruby even though the script is a separate process.

Thanks for your suggestions,

Eric

Thomas Dehn

4/9/2011 8:47:00 PM

On 04/09/2011 10:21 PM, judyorcarl wrote:
> On Apr 9, 11:04 am, Thomas Dehn<thomas-use...@arcor.de> wrote:
>> On 04/09/2011 02:32 PM, Adam Lea wrote:
>>
>>> Playing 5cm (better minor), 15-17 NT, three weak two's with 2C as the
>>> only strong bid, 20-22 2NT, what do you open with this hand after a pass
>>> on your right, and what is your planned rebid?
>>
>>> AQ32
>>> A
>>> A986
>>> AKQJ
>>
>>> Assume that if you open 2C, partner will respond 2D (0-2 controls), if
>>> you open 1C, partner will respond 1H.
>>
>> Open 2C, rebid 2NT over 2D, showing
>> "23-24 somewhat balanced"
>>
>> Thomas
>
> but of course you have 25 with slam chances in 3 suits.

23-24 is close enough to what you have.
Whereas a 1m opener might easily end
the auction quickly even though you have a slam
in some other suit, and after 2C 2D 3C,
responder will expect a hand that has lots
of tricks in clubs, and you then
might easily miss a 4-4 fit in spades.

Thomas

derek

4/10/2011 12:12:00 AM

On Apr 9, 5:21 pm, judyorcarl <judyorc...@verizon.net> wrote:
> On Apr 9, 11:04 am, Thomas Dehn <thomas-use...@arcor.de> wrote:
>
>
>
> > On 04/09/2011 02:32 PM, Adam Lea wrote:
>
> > > Playing 5cm (better minor), 15-17 NT, three weak two's with 2C as the
> > > only strong bid, 20-22 2NT, what do you open with this hand after a pass
> > > on your right, and what is your planned rebid?
>
> > > AQ32
> > > A
> > > A986
> > > AKQJ
>
> > > Assume that if you open 2C, partner will respond 2D (0-2 controls), if
> > > you open 1C, partner will respond 1H.
>
> > Open 2C, rebid 2NT over 2D, showing
> > "23-24 somewhat balanced"
>
> but of course you have 25 with slam chances in 3 suits.

That must be RGB counting - I can only see 24 pts.

judyorcarl

4/10/2011 11:40:00 AM

On Apr 9, 8:12 pm, derek <de...@pointerstop.ca> wrote:
> On Apr 9, 5:21 pm, judyorcarl <judyorc...@verizon.net> wrote:
>
>
>
>
>
> > On Apr 9, 11:04 am, Thomas Dehn <thomas-use...@arcor.de> wrote:
>
> > > On 04/09/2011 02:32 PM, Adam Lea wrote:
>
> > > > Playing 5cm (better minor), 15-17 NT, three weak two's with 2C as the
> > > > only strong bid, 20-22 2NT, what do you open with this hand after a pass
> > > > on your right, and what is your planned rebid?
>
> > > > AQ32
> > > > A
> > > > A986
> > > > AKQJ
>
> > > > Assume that if you open 2C, partner will respond 2D (0-2 controls), if
> > > > you open 1C, partner will respond 1H.
>
> > > Open 2C, rebid 2NT over 2D, showing
> > > "23-24 somewhat balanced"
>
> > but of course you have 25 with slam chances in 3 suits.
>
> That must be RGB counting - I can only see 24 pts.- Hide quoted text -
>
> - Show quoted text -

Add a point for 4 aces, said Goren. If you expect 4321 to work, you
must.

Carl

derek

4/10/2011 6:11:00 PM

On Apr 10, 8:39 am, judyorcarl <judyorc...@verizon.net> wrote:
> On Apr 9, 8:12 pm, derek <de...@pointerstop.ca> wrote:
>
>
>
> > On Apr 9, 5:21 pm, judyorcarl <judyorc...@verizon.net> wrote:
>
> > > On Apr 9, 11:04 am, Thomas Dehn <thomas-use...@arcor.de> wrote:
>
> > > > On 04/09/2011 02:32 PM, Adam Lea wrote:
>
> > > > > Playing 5cm (better minor), 15-17 NT, three weak two's with 2C as the
> > > > > only strong bid, 20-22 2NT, what do you open with this hand after a pass
> > > > > on your right, and what is your planned rebid?
>
> > > > > AQ32
> > > > > A
> > > > > A986
> > > > > AKQJ
>
> > > > > Assume that if you open 2C, partner will respond 2D (0-2 controls), if
> > > > > you open 1C, partner will respond 1H.
>
> > > > Open 2C, rebid 2NT over 2D, showing
> > > > "23-24 somewhat balanced"
>
> > > but of course you have 25 with slam chances in 3 suits.
>
> > That must be RGB counting - I can only see 24 pts
>
> Add a point for 4 aces, said Goren. If you expect 4321 to work, you
> must.

Not buying it - even if Goren said it. A 1NT bid made on 4 aces and
nothing else, is NOT a 17-count. I happily adjust for many features
in unbalanced hands, but adding points for features partner already
expects in narrowly defined NT ranges is asking for trouble.

comp.lang.ruby

fsync on stdout for mod_rewrite

Eric Anderson

Glenn Parker

Eric Anderson

Glenn Parker

Dominik Werder

Eric Anderson

Eric Anderson

Thomas Dehn

derek

judyorcarl

derek

x Login to ForumsZone