Asp Forum - Real World Scalability and Ruby - Top 20

Joseph

9/10/2006 3:27:00 AM

Folks,

After the long post regarding Joe's now infamous entry about Ruby, I
wondered, what have the really successful, scalable, big, websites /
web applications out there have used.

So I turned mostly to two sources, which while not perfect, are good
enough.

For popularity or raw scalability I used Alexa's ranking here:
http://www.alexa.com/site/ds/top_sites?ts_mode=global&...

And for webserver I either used Netcraft:
http://news.net...

or actually tried to figure out what the site used by looking at the
code or what the page used.

This is the list of the top 20 sites and what they are apparently
using, these days many of these sites hide what goes on inside, but
still we can guess, and yes I have given it my best guess in some
cases.

Here is the table, feel free to add more information to it since there
are still many gaps that I have identified with a question mark. The
first line mentions the site name and web server, the second line the
language or framework behind it.

1 Yahoo FreeBSD
PERL, PHP, Proprietary, C?

2 MSN Windows Server 2000/2003, Some Apache
ASP, ASP.NET, DLLs?

3 Google Linux based or unknown servers, probably modified FreeBSD or
Apache.
Python, Perl, PHP?, C, Proprietary, Java

4 Baidu.com Linux based unknown.
?

5. Qq.com Linux based unknown and Windows 2003.
?

6. MySpace Windows 2003 / 2000 some Linux unknowns too.
Coldfusion

7. sina.com.cn FreeBSD, Solaris 8, Linux based unknowns,
?

8. Yahoo Japan Like Yahoo at 1.

9. 163.com China FreeBSD, Linux based unknowns,
?

10 Live.com Windows 2003, Linux unknown servers
ASP, ASP.NET, DLLs?

11 eBay.com Windows 2000/2003
PERL?, C?, DLLs, Proprietary, more?

12. Sohu.com China Linux unknown servers
?

13. YouTube.com Linux unknown servers
?

14. Yahoo China Like 1

15. Microsoft Windows 2003 / 2000
ASP.net, ASP, DLLs?

16. Wikipedia FreeBSD, Linux unknown servers
PHP, PERL?

17. Amazon.com FreeBSD, Linux unknown servers, Solaris 8, Netware
PERL, Proprietary, more?

18. Orkut.com Linux unknown server
PHP?, PERL?

19. Blogger FreeBSD, Linux unknown servers
PHP?, PERL?

20. Google UK Like Google

INTERESTING FACTS

* Not a single significant "safe" Java J2EE in the top 20.
* Many proprietary variations with FreeBSD or Linux as the only common
ground
* Some .NET and Windows 2003 are indeed listed
* Arguably the biggest web application is MySpace which is based on
Coldfusion! Certainly not a "safe" choice by a long shot.

CONCLUSIONS

* Choosing one Framework or language over another seems to be mostly
irrelevant as long as you stick to the underlying technology: FreeBSD,
Linux Based server or Windows 2003 which appear consistently in the top
web sites again and again. Also although is not mentioned anywhere,
Oracle, MS SQL Server and MySQL are high up there in these rankings
too.
* Java and J2EE is by far absent from this list, this should tell us
all something.
* .Net is very present on the list, MS obviously is doing something
right. The progress Ruby is doing with Windows is encouraging.
* Choosing the best tools for the job can give you a big payout.
MySpace is Coldfusion based, this is risky, but gives you the ability
to write database web applications fast... and it has worked well. I
would say the risk was worth it.
* Not a single major Ruby or Ruby on Rails app cuts to this list yet,
but I see no reason why this would not happen eventually.

Food for thought fellows, any input on how would Ruby would ever get
there, or who runs what, would be appreciated.

Jose L. Hurtado
Web Developer
Toronto, Canada

20 Answers

Charles O Nutter

9/10/2006 3:52:00 AM

On 9/9/06, Joseph <jlhurtado@gmail.com> wrote:
> Folks,
>
> After the long post regarding Joe's now infamous entry about Ruby, I
> wondered, what have the really successful, scalable, big, websites /
> web applications out there have used.

It seems like you're just guessing at these. I don't know all of them,
but I know for certain that eBay is running a crapload of Java in
their backend. In fact, they used to be a solid ASP site (pre-.NET)
but switched to Java because the ASP stuff scaled horribly. IBM made a
big media event out of that a few years back. I mean really, the site
has Sun/Java branding right at the top...so it's probably safe to
assume Java's involved.

It's also extremely subjective to "look at what the site uses" because
I know for a fact many large Java-based sites use URLs showing
something other than ".jsp", for reasons that are perhaps obvious. And
there's another large chunk of sites that use PHP or .NET or what have
you for web-facing stuff while the vast majority of their apps are
actually backed by large Java clusters behind some variety of service
layer.

It would probably be better to leave the guesses off the list
completely and not try to draw any conclusions at all. Unless you
really know what these sites are using (a difficult prospect at best)
no conclusions are possible.

--
Contribute to RubySpec! @ www.headius.com/rubyspec
Charles Oliver Nutter @ headius.blogspot.com
Ruby User @ ruby.mn

Chad Perrin

9/10/2006 3:55:00 AM

On Sun, Sep 10, 2006 at 12:30:09PM +0900, Joseph wrote:
>
> Here is the table, feel free to add more information to it since there
> are still many gaps that I have identified with a question mark. The
> first line mentions the site name and web server, the second line the
> language or framework behind it.
>
> 1 Yahoo FreeBSD
> PERL, PHP, Proprietary, C?

Also Python and Common Lisp (though the Lisp codebase is not growing at
this point -- it's "legacy code" that is indispensable as long as they
keep their RTML templating system).

>
> 2 MSN Windows Server 2000/2003, Some Apache
> ASP, ASP.NET, DLLs?

I believe they're still using some FreeBSD systems at Hotmail, and all
of Windows is behind free unix firewalls through a proxy service.

>
> 3 Google Linux based or unknown servers, probably modified FreeBSD or
> Apache.
> Python, Perl, PHP?, C, Proprietary, Java
>
> 4 Baidu.com Linux based unknown.
> ?
>
> 5. Qq.com Linux based unknown and Windows 2003.
> ?
>
> 6. MySpace Windows 2003 / 2000 some Linux unknowns too.
> Coldfusion

Migrating to BlueDragon.NET, which uses .NET as the back end for
ColdFusion, last I checked.

>
> 7. sina.com.cn FreeBSD, Solaris 8, Linux based unknowns,
> ?
>
> 8. Yahoo Japan Like Yahoo at 1.
>
> 9. 163.com China FreeBSD, Linux based unknowns,
> ?
>
> 10 Live.com Windows 2003, Linux unknown servers
> ASP, ASP.NET, DLLs?
>
> 11 eBay.com Windows 2000/2003
> PERL?, C?, DLLs, Proprietary, more?
>
> 12. Sohu.com China Linux unknown servers
> ?
>
> 13. YouTube.com Linux unknown servers
> ?
>
> 14. Yahoo China Like 1
>
> 15. Microsoft Windows 2003 / 2000
> ASP.net, ASP, DLLs?
>
> 16. Wikipedia FreeBSD, Linux unknown servers
> PHP, PERL?

The Wikimedia Foundation (Wikipedia, Wikinews, et cetera) has to my
knowledge only ever had a grand total of one FreeBSD server, and it
wasn't really used in production. The servers are primarily running on
Fedora Core 3-5, with a couple of old Red Hat Linux and pre-Novell SuSE
Linux servers (unless those have been upgraded since I stopped working
there). The MediaWiki software is all PHP. MySQL is used for
databases. Thus, it's classic LAMP platform. There are some Perl and
Python scripts running about for various administrative tasks, but they
don't represent any kind of measurable percentage of traffic load.

There are a lot of squid proxies used for caching to serve pages faster.
There's some rudimentary load balancing (last I checked) that's handled
at least in part by in-house scripting.

The Wikimedia Foundation uses zero .NET or Java, in case you were
wondering.

>
> 17. Amazon.com FreeBSD, Linux unknown servers, Solaris 8, Netware
> PERL, Proprietary, more?
>
> 18. Orkut.com Linux unknown server
> PHP?, PERL?
>
> 19. Blogger FreeBSD, Linux unknown servers
> PHP?, PERL?
>
> 20. Google UK Like Google
>
> INTERESTING FACTS
>
> * Not a single significant "safe" Java J2EE in the top 20.
> * Many proprietary variations with FreeBSD or Linux as the only common
> ground
> * Some .NET and Windows 2003 are indeed listed
> * Arguably the biggest web application is MySpace which is based on
> Coldfusion! Certainly not a "safe" choice by a long shot.

While MySpace is (again, based on what I've last heard) migrating to a
NET foundation for its ColdFusion, it got to its current prominence
entirely on a ColdFusion 5 back-end, as far as I'm aware. Having never
been employed by MySpace, I of course cannot be as sure of this as I am
about Wikimedia Foundation information.

>
> Food for thought fellows, any input on how would Ruby would ever get
> there, or who runs what, would be appreciated.

It's also worth noting that Slashdot is Perl on Linux, I think via
Apache and MySQL (but don't quote me on that unless I'm right).

--
CCD CopyWrite Chad Perrin [ http://ccd.ap... ]
"The measure on a man's real character is what he would do
if he knew he would never be found out." - Thomas McCauley

Chad Perrin

9/10/2006 4:04:00 AM

On Sun, Sep 10, 2006 at 12:54:56PM +0900, Chad Perrin wrote:
>
> I believe they're still using some FreeBSD systems at Hotmail, and all
> of Windows is behind free unix firewalls through a proxy service.

Arrrgh, typo. That should read "all of Microsoft". Sorry.

--
CCD CopyWrite Chad Perrin [ http://ccd.ap... ]
"The ability to quote is a serviceable
substitute for wit." - W. Somerset Maugham

Joseph

9/10/2006 4:35:00 AM

Charles,

Good to know about eBay, as I said I am guessing, and getting input was
precisely what I wanted... so it seems at least one Java is out there,
are there more?

Thanks Charles!

Jose L. Hurtado
Web Developer
Toronto, Canada

Charles O Nutter wrote:
> On 9/9/06, Joseph <jlhurtado@gmail.com> wrote:
> > Folks,
> >
> > After the long post regarding Joe's now infamous entry about Ruby, I
> > wondered, what have the really successful, scalable, big, websites /
> > web applications out there have used.
>
> It seems like you're just guessing at these. I don't know all of them,
> but I know for certain that eBay is running a crapload of Java in
> their backend. In fact, they used to be a solid ASP site (pre-.NET)
> but switched to Java because the ASP stuff scaled horribly. IBM made a
> big media event out of that a few years back. I mean really, the site
> has Sun/Java branding right at the top...so it's probably safe to
> assume Java's involved.
>
> It's also extremely subjective to "look at what the site uses" because
> I know for a fact many large Java-based sites use URLs showing
> something other than ".jsp", for reasons that are perhaps obvious. And
> there's another large chunk of sites that use PHP or .NET or what have
> you for web-facing stuff while the vast majority of their apps are
> actually backed by large Java clusters behind some variety of service
> layer.
>
> It would probably be better to leave the guesses off the list
> completely and not try to draw any conclusions at all. Unless you
> really know what these sites are using (a difficult prospect at best)
> no conclusions are possible.
>
> --
> Contribute to RubySpec! @ www.headius.com/rubyspec
> Charles Oliver Nutter @ headius.blogspot.com
> Ruby User @ ruby.mn

e

9/10/2006 4:46:00 AM

Jose Hurtado wrote:
> Folks,
>
> <snip />

Many people seem to forget that the word 'scalability'
implies bidirectionality. I assert that Ruby scales
better than Java for most things:

Small----------------------------------Large
Java <-------->
Ruby <------------------------------->

Have a nice day.

--
Posted via http://www.ruby-....

Seth Thomas Rasmussen

9/10/2006 9:23:00 AM

Everybody talks about this "real" world as if it is different from the
one we all experience day to day.

I hope to one day experience this world, because I'm sure I'll be
completely ready to rule them all with my vast knowledge of what works
in the "real" world.

Here's to talking heads..

P.S. Scalability. And enterprise, don't forget enterprise. My latest
benchmarks show that under certain conditions, some numbers are
produced.

A. S. Bradbury

9/10/2006 10:07:00 AM

On Sunday 10 September 2006 04:54, Chad Perrin wrote:
> The Wikimedia Foundation uses zero .NET or Java, in case you were
> wondering.

This is really getting off topic, but Wikimedia do use Lucene for at least the
english search (compiled with GCJ however).

Alex

Chad Perrin

9/10/2006 10:51:00 AM

On Sun, Sep 10, 2006 at 07:06:36PM +0900, A. S. Bradbury wrote:
> On Sunday 10 September 2006 04:54, Chad Perrin wrote:
> > The Wikimedia Foundation uses zero .NET or Java, in case you were
> > wondering.
>
> This is really getting off topic, but Wikimedia do use Lucene for at least the
> english search (compiled with GCJ however).

Hmm. I'd forgotten about that.

It's kinda like splitting hairs, though -- which is why I didn't
remember it was technically written in Java.

--
CCD CopyWrite Chad Perrin [ http://ccd.ap... ]
"The first rule of magic is simple. Don't waste your time waving your
hands and hopping when a rock or a club will do." - McCloctnick the Lucid

David Vallner

9/10/2006 11:17:00 AM

Joseph wrote:
> [snip pulling arguments out of your pinky finger]

You forget intranets. Internal company webapps have to serve humongous
amounts of traffic on not really lavish hardware. Listing the Fortune 20
of websites which indeed CAN afford to "just throw more servers at it"
tells us precisely nothing at all about technology scalability.

Windows probably sees more use for company backends than you can imagine
on accounts of being easy to set up and work with up to a certain scale
when you really need automation instead of an underpaid student support
gimp.

Also, your method of research is laughable.

> * Choosing one Framework or language over another seems to be mostly
> irrelevant as long as you stick to the underlying technology: FreeBSD,
> Linux Based server or Windows 2003 which appear consistently in the top
> web sites again and again.

Oh yes. Only the three most mainstream server OSs appear in that list.
Surprise.

> * Java and J2EE is by far absent from this list, this should tell us
> all something.

Or not, since the list is worthless data.

> * .Net is very present on the list, MS obviously is doing something
> right. The progress Ruby is doing with Windows is encouraging.

Good marketing, IIS comes with Windows, more straightforward than Java
to do MVC and deployment with. Makes it an easier first choice nowadays.

> * Choosing the best tools for the job can give you a big payout.
> MySpace is Coldfusion based, this is risky, but gives you the ability
> to write database web applications fast... and it has worked well. I
> would say the risk was worth it.

Worstofmyspace.com begs to differ. I completely ignore the very
existence of MySpace except from tidbits on the aforementioned site, but
if it's remotely to be trusted, it's far from stable and reliable.

David Vallner

Joseph

9/10/2006 4:48:00 PM

David Vallner said:

> You forget intranets. Internal company webapps have to serve humongous
> amounts of traffic on not really lavish hardware. Listing the Fortune 20
> of websites which indeed CAN afford to "just throw more servers at it"
> tells us precisely nothing at all about technology scalability.

Aha... I would argue this is not true. Having worked for two large
corporations in the past with over 200,000 employees and huge
Intranets, I can assure you some of the very worst delays, and
website/web application design is behind closed doors of those
Intranets. And I have yet to find one internal webapp that reaches the
scalability of a public app... but then again, some government intranet
apps might truly be huge and similar to a major website traffic.

David said
>
> Also, your method of research is laughable.

OK... this is a very aggressive way of making a point isn't it David?
I am tempted to reply, but I will ignore the attack and suggest you
give us a better research method in under 1 hour that works and does
not make me us laugh, and then please do email us the results ; )

Now, don't come back and tell us you need a few thousand dollars, a
month and a research firm to find out anything about this subject, when
something can already be known in under one hour, and that was my
point, to try to shed some light into this subject, and to receive more
information from others who may probably know more than I do. I
believe the open source movement calls this collaboration, and it
works.

Best Regards,

Jose Hurtado
Web Developer
Toronto, Canada

comp.lang.ruby

Real World Scalability and Ruby - Top 20

Joseph

Charles O Nutter

Chad Perrin

Chad Perrin

Joseph

e

Seth Thomas Rasmussen

A. S. Bradbury

Chad Perrin

David Vallner

Joseph

x Login to ForumsZone