Surviving 100,000 instant Visitors on a Budget

Web Server Digg Day 1Web Server Traffic from Digg during day 1, outlining some important events Last Sunday I casually wrote a post about the most common pitfalls in photography.
I had written the post a long time ago, but I never put it on this site.

The story ended up on the front page of Digg, stayed there for nearly 12 hours and was picked up by a bunch of secondary sources, resulting in 100,000 Visitors within a day (most of them within the first 12 hours). Trying to keep my server up and alive, I learned a lot about what it takes to keep the server alive, hence I decided to slide a non photography related post in, hopefully giving everyone who finds themselves in a similar situation some points to consider.

On Sunday Usproblogger (Digital Photography Blog) submitted the story to Digg (probably using the button that my blog software puts below each post).
Initially I wasn't aware of what had happened and while I was watching a movie the post must have hit the front page of Digg.

My site is hosted on a Virtual Private Server (VPS) with 256MB guaranteed RAM and 1GB burstable RAM. I have tweaked my VPS significantly, turning off all non-essential demons and keeping only the things that are required for hosting. This article is not about tweaking a VPS machine; I assume you have already done it.

This website is powered by a dynamic content management software. Among the many benefits (visitors can comment, I can maintain a site with over 2000 pages easily), it has one major drawback. The web content is stored in a database. Each time someone views a page; a program is executed (php). That program gets the information from the database (MySQL), and literally builds the page for each and every visitor (enough of the basics, I am going to assume you know all this if you are having the same problem that I had).

With such limited resources and the amount of traffic Digg can send, my Server quickly ran out of breath and was unable to handle the traffic.
The server recovered somewhat, probably because the traffic dropped momentarily.
By the time I noticed what was happening, i was completely locked out of my system and had to reboot in order to regain control and do something about the situation.

What I did to keep the Server up

Static Cache

After the reboot I immediately enabled The Cache [1]. (Follow the reference for more information on this), basically replacing the database version with a static html page.
Since it was very late for me, I then went to bed. One big mistake I made at that point: I didn't cache any of the pages linked from the page that was being on Digg, but I had created a lot of links to example images, each being a full page served dynamically.

VPSINFOVPSINFO server information page

More Problems popping up

When I woke up in the morning, I opened up my Vpsinfo (Vpsinfo is a very handy tool to monitor your server performance) I was confronted with this nightmare (see picture).

I have enlarged some items of the screenshot for your convenience.

The first thing I noticed was all the red.

  • The server load was through the roof. The 3 bars indicate the average loads of the VPS during 1minute, 5 minutes and 15minutes averaging intervals. To put it simply: A load below 1 means you have more computing power then you need, a load above 1 means if you had a faster computer processes could execute faster (I know its a lame explanation)
  • I exceeded my guaranteed RAM (256MB) and used 644MB
  • My burstable Ram Resources were at the maximum (currently 1014MB out of 1024MB, but below you can see that I had many failures (almost 100,000) when it tried to go above 1GB)
  • Currently I had 857 network connections open and 176 apache web servers running
    I needed to get this number higher, as I could see that the server was at its limit and people were locked out
  • All the way on the bottom I could see that this simple php page took nearly 33s to generate, which is very unhealthy. Basically the parts of the site that wasn't cached would not be accessible. In fact when i tried, all I got was error messages telling me that I was out of memory
Vpsinfo has lots more useful information.

Image Outsourcing

At this moment, the only thing I could think of to reduce the load was getting rid of the images in the post. Each time someone opened the page (now just html), apache (web server) still had to perform multiple requests (hits), one for the html file and one for each image. Since I was pushing a lot of traffic at this time (so far over 50GB in about 8 hours), image load times would be longer, tying up valuable apache processes.
I decided to put all images to my SmugMug Pro Account (affiliate link), that powers my photo store. They basically offer unlimited storage and transfer for a very low price (considering what you get).
Most people run their entire site on Smugmug, but I like messing around with the server.
I copied all pictures over there and moved replaced the references in my html file.
In the first picture (Traffic Graph), you can see the immediate drop in Traffic. In fact at this point I was able to push even more visitors (although the graph looks like I got a huge drop in visitors, its just the traffic that dropped).

Caching secondary pages

Basically my server would still stay in the red zone, but I wouldn't exceed burstable resources anymore. That doesn't mean I wouldn't get any failures (as I am really only guaranteed 256MB and 1GB is just for emergencies). I did a preliminary check on Google Analytics and I noticed that a really significant amount of my visitors was clicking on the images and links in the article, generating a lot of hits on the database (I still had about 100 database queries per second, although I had cached the page).
Only after I cached all pages that the article linked to would my server return almost back to normal (the article was still on the front page). I should have thought about this earlier.

What I should have done instead

Even with all those fixes, the cost was high. I have another website on this server that was effected very much by this.
I should have momentarily redirected all Digg traffic through the Coral Content Distribution Network.
In fact I had already prepared part of the .htaccess file (careful, this is not tested):
RewriteCond %{HTTP_REFERER} !^http://(www\.)?yoursite\.com [NC]
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_USER_AGENT} !^CoralWebPrx
RewriteCond %{QUERY_STRING} !(^|&)coral-no-serve$
RewriteRule ^/url1.html$ [L]
The first two lines are optional. They basically just determine if someone came from an external source like Digg. Replace yoursite with the url of your web site and url1.html with the page you want to cache.

To redirect everything within a folder (also untested):
RewriteCond %{HTTP_USER_AGENT} !^CoralWebPrx
RewriteCond %{QUERY_STRING} !(^|&)coral-no-serve$
RewriteRule ^/images/(.*)$$1 [L]
Or to redirect every image file (again, not tested):
RewriteCond %{REQUEST_FILENAME} .*jpg$|.*gif$|.*png$ [NC]
RewriteCond %{HTTP_USER_AGENT} !^CoralWebPrx
RewriteCond %{QUERY_STRING} !(^|&)coral-no-serve$
RewriteRule ^(.*)$$1 [L]
On Corals site they have [R,L], but that will change the url in the browser !

Running two web servers at the same time.

Although there are countless sources on the Internet about Apache configuration, here are some helpful pointers of what I have and what worked for me. I am not an Apache expert:
Timeout 200
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 3
MinSpareServers 5
MaxSpareServers 15
StartServers 15
MaxClients 250
MaxRequestsPerChild 10000 # I changed this because of memory leaks
Anyways, since Apache is resource intensive and since each request spawns a new apache process (even the tiniest image like the little Digg button), I have considered installing Boa as a secondary web server for images. The basic idea is to have boa listen on port 8080 and redirect all image traffic to port 8080.
A redirect can be executed much faster by apache and the process will become available for other users faster, while boa is a single threading high throughput server that can easily handle images and only has a minimum size footprint.
The redirect would probably look something like this:
RewriteCond %{REQUEST_URI} .*\.(gif|png|jpg)$ 
RewriteRule ^/(.*) http://%{HTTP_HOST}:8080/$1 [P]

The Fallout

Second Day

Web Server Traffic Day 2Web Server Traffic Day 2 (50,000 Visitors)I had to reboot the server again, due to some log file problems (see next chapter).
During the second day, the article hit the front page of Reddit.
The combined traffic of Reddit and a few other sources (the floor) was much lower due to the cache, but at times I had just as many people visiting my site as I had during the initial Digg.
Fortunately from then on my server didn't even break a sweat anymore (except huge transfer numbers). Inspecting this image, I wonder if Reddit had a brief downtime on Tuesday.

Apache Log File creation

There are quite a few things I still don't understand when it comes to web servers. My disk space is somewhat limited and the access logs that apache generates for traffic analysis grew to several gigabytes in size in a matter of hours.
Quoting the support team of Servint (where I buy my server and traffic):

When apache is not restarted the disk space

continues to be used, as the file descriptors are still open.

This is really annoying, as this means I had to restart at least the web server. Before the Digg I had several months of uptime (not a single restart). I will definitely have to play with some options in Cpanel to see if I can do something about this. I hit almost 100% of disk space before I had to reboot on the second day.


Alexa number 4 Movers and ShakersAlexa number 4 Movers and ShakersBeing the curious (and hopeful) type, I went over to Alexa and checked the Top 10 Movers and Shakers list. I couldn't believe my eyes when I saw myself on number 4 (probably just proofs that Digg users are tech savvy and have the Alexa Bar installed).
Anyways, I will keep this screenshot as a badge of honor, since this also means I have been able to keep my server up and running.

What next

Obviously I need to rethink my plan of putting my 800 Mega pixel San Francisco picture on the site any time soon, in case someone finds it :-)

Fourth Day

Traffic has picked up again and has been heavy during the entire day.


I get several brute force warnings from my brute force detection script every day. The script automatically blocks access at the firewall level. I am using this firewall.
Usually I check the IP's of the hackers and in about 80% of the cases they originate in China. The blocking usually happens very quickly and doesn't concern me too much. During the period of heavy traffic I had a total of about 5 ssh login brute force attacks.
Often I catch a single IP from China tying up over 1000 TCP connections at once (someone tries to download this entire website with a robot and does not throttle). Until I have figured out how to limit the amount of TCP connections a single IP can tie up, I will have to block the entire country of China from accessing this site.
I feel really bad about this, since people from China are already behind the Great Firewall of China, and now I am throwing up yet another wall, but I simply do not have the time and resource to deal with this on an individual basis. If such a robot hits my server during a Digg, I will literally lose thousands of legitimate visitors.
I will use the Zone file information found here and put it into my deny_hosts.rules file.

The Cache

A while ago I had implemented a cache system (when the site was on a shared host) that I haven't used much since.
But I have always kept the following lines in my .htaccess file:
 RewriteCond %{QUERY_STRING} =""
RewriteRule ^$ index.html [L]
RewriteCond /home/xxx/public_html/static%{REQUEST_URI} -f
RewriteCond %{QUERY_STRING} !^.+$
RewriteRule ^(.*)$ static/$1 [L]
(you might want to get rid of the first two lines, unless you are also caching the front page like me)

Put lines 3-5 into your .htacess file.

Lets say someone wants to see the url url1.html on your website.
Now line3 will check if that url exists in the static directory located in my public_html directory.
If the file exists, and the query string is not empty (which would indicate the index is requested), the request is redirected and the static file is shown to the visitor.
A visitor will not be able to see the difference, since the URL in the browser will not change.

Interesting tips

that what great informations i agree with what you are posting here too
i learn a lot from affiliate marketing
and i also earn from it too thank you

Alex wong
success blog
success blog


Thanks for the praise, but that was just a sidestep. The affiliate stuff is just to keep the servers running and glowing red hot so I can keep the site up without having to sell my camera :-)
But hey, if you are looking for a good VPS host, tell Servint I sent you and you will have contributed to keeping the blog up ;-)

Great Post

This is a great post, it is useful to see what a server can expect when it gets "dugg".

Just on the point of redirecting to Boa for all images. The RewriteRule you have used will force Apache to act as a proxy to the Boa server, which means that Apache will send the request on to Boa, wait for a response and then resend that response back to the client. You are in effect still tying up an Apache process and you are also adding the overhead of an additional HTTP request (albeit a local one) to the processing chain. You might get some benefits running a high throughput, single process web server like Boa but you are better off changing all your URLs in the generated pages to point directly to the second HTTP server, this way the client's browser creates the request directly to Boa and Apache is never invoked.

Once again, thanks for the article.


Thanks Sean for the valuable insight.

Wouldn't the Apache process immediately be freed up after the redirect? I was under the assumption that this would work the same way as redirecting all images to the Coral Servers, only you are redirecting to a secondary web server.

If I redirect to Coral, the Apache process would be free right away wouldn't it? The Browser is now going to request from Coral. So when I redirect to another port, the browser will then request the image from that port will it not?

I basically took the idea outlined here apache performance tuning and reversed it (since boa can't do redirects).

Well, I am not a professional in that area. I just wrote down some ideas.
Maybe I will try to benchmark this some time.


Great Tips & Informaions, Thanks.


Re: Boa

Alex, it depends on the options you pass to the RewriteRule command. Using the [P] option will tell Apache to act as a proxy which means it takes the initial URL browser, re-writes it based on your rule and sends the request to the new URL itself, then waits for the response and returns it to the initial client.

If you provide the [R] option to RewriteRule it will do as you describe in your last comment, send a redirect back to the browser after which the browser will send another request to the server using the new port. In this case [R] is better than [P] because the Apache process is freed up straight away. However it also means that for every image you will get 2 HTTP requests from the browser to the server, which could be costly in very high traffic situations.

The better solution would be to use URLs that point directly to the Boa server for all your images to be served by Boa, this results in just one request per image and Apache will never get invoked.

Good work though and your right, the only way to know for sure is the benchmark.

Wow. Thanks for clearing

Wow. Thanks for clearing this up Sean. I appreciate that you took the time to respond and set things straight.
There is only one small mistake, my name is Andre :-)

Re: Digg

Andre, Your page has hit the front page of Digg, and is still loading quite fast (I'd say extremely fast for a page with so much content). It's almost instantaneous like clicking Compared to blog pages (specially the ones based on Wordpress), and other private sites..which frequently dies, even before they hit the front page of Digg..I'd say, you've done an excellent job here.

Also, I was looking at your 'Travel Website'. Very nice, bookmarked it. The one thing that disappointed me, was I was looking for a link about you. Like you know, an 'About Us' link. Wanted to know more about you, and your travels. And the most important thing, how you afford to visit all over the world.

Well, going by this site of yours, and your travel website..and the ads, and you probably earn quite a decent figure from your picture sales...I'd like to think they contribute a lot to help you maintain this beautiful hobby of yours.

I'd like to thank Digg for introducing me to your excellent website. :)

I'm also looking for a way

I'm also looking for a way to limit the number of connections per ip, found two alternatives:

Via apache: You can use mod_limitipconn, its a apache module, downside is that people still can connect to your machine freely ...

Via iptables/netfilter : This is the one that I want to really use, but until now I'm not very sucessful since this one involves recompiling the kernel (I'm talking about linux)

Here is the link to it and in case that you have sucess implementing I would love to hear from you.

limit ip connections

To handle your problem with many connection from one single ip I would suggest installing the limitipconn to your apache server

Does one hell of a job stopping spiders and robots from thumping your server


Ren: Thanks for your kind words. You might be surprised, but my day job actually pays for all this, not the pictures. I hope that they will one day contribute a nice portion to my travels. Its just a matter of setting priorities for me.

I was looking for a link about you. Like you know, an 'About Us' link. Wanted to know more about you, and your travels.
I am curious why? I get a lot of emails from Students asking me the very same question.

Guga-NYC, Pantsuninja: Thanks for the links. I will check them out.

Apache Configuration

I'd say you need to read up a lot more on apache configuration.
There are some one line config changes that will allow you to limit the number of connection per IP address and there are compile time options for how many connections to handle per thread.

Lighttpd is great if you have limited resources, but being single threaded and 'Select' based it gets really bogged down under high load(A single thread checking for events on 100,000 sockets means high latency).

Great article, and your

Great article, and your methods seem to be working very well. This site loads in an instant even on the front page of Digg.

Which module are you using to get the .html extension on your drupal pages?

Like the tips

I have a bunch of optimizing things bookmarked, and I'll add this one too. The coral network thing confuses me though.. I remember having problems finding "usage guidelines" before, and I still can't locate it (current/future paying for the service, etc).


Jen: If you follow the link, there is a wiki somewhere explaining how to use Coral. Usage is free. The bandwidth is donated.
Maybe this wiki article will be of some use.
So far I have been able to get around using it though.

I know

OJ: I know. I have read a bit about lighttpd, Litespeed and thttpd a while ago, but since this site evolved over time through different CMS and since I use a static cache, hotlink protection and a bunch of other neat rewrites, my .htaccess file is rather extensive (e.g. I have a whole lot of 301 and links still point to the originals back from the day's when this was a static site). Modrewrite for my CMS would have to be transfered to support a clean url structure + i think CPANEL requires Apache anyways, meaning I would have to switch all accounts to directadmin.
All of these problems combined make me not want to switch.
Besides, if I can push 100k visitors with apache, I think it should be fine.

Thanks for the suggestion though. If I were to start a complete new site and wouldn't have all that history I might switch, but I don't have the time to mess with web servers for weeks to get everything transfered, set up a test environment at home and check all sites that are on this server ...

bandwidth for images

We've ran into a couple situations like this (but no digg homepage) with our site and I decided awhile back just to farm the hosting out to Pair networks on a high volume account. They do a pretty good job and keeping us up and running under most circumstances.

did your revenue follow the traffic? I've always heard traffic from digg doesn't result much for the bottom line in the short term but backlinks are always good.


Well, I am pretty happy with Smugmug. They don't charge that much and I have effectively unlimited storage and Bandwidth. Their Servers are amazing.

Revenue: Who cares. As long as it pays the bills for the site, I am blogging for fun only. If Digg ever costs me money, I would still be happy about the traffic, since that means a lot of people read what I am writing. I guess I am alone with this view, everyone asks about money.

Logfiles and bouncing httpd

When you move a file or delete it, any process that still has the file open will still write to it, and the file will remain on disk. Have a look at for more details. It's not a webserver thing, it's a unix thing.

If you don't want the file, just run

> /path/to/file

(the initial > is intentional). That will truncate the file without changing the inode.

If you want to keep it, move it out of the way and do a graceful restart

mv logfile logfile.1
apachectl -k graceful


I did this

Thanks Sean,

I actually did this:
cat /dev/null>/path/to/file
after finding out the exact same thing you were mentioning :-)

Limiting parallel connections in apache

Hey Andre,

I found another modlule for apache that's way better the limitipconn .... mod_cband is nice, easy to setup and works with vhosts as well, offer what limitipconn offers and more.

You can also have bandwith control, but most important is the number of connections per ip and number of requests.

Home page for cband

Tutorial on how to use it.

I think this one is far better, on the application level, than the other stuff around there.

Good luck,


That was quite a post. Too much for me to follow in some parts, but it gave me some great info nonetheless...especially that I need to learn a bit more if I ever get off shared hosting or before I get dugg too hard.

1000 cnx / IP = brut force attack? why?

What if this supposed chinese brut force attack correspond to a transparent http proxy, or some NAT mechanism? You may have forbidden "1000" legitimate users as well. Do IP based bandwith limiters consider people behind a router or a proxy?

1000 legitamate

I agree and if I had any choice I wouldn't do it. In fact, I am currently employing a per IP blocking with apf, but the amount of spam vs. legitamate users from some countries is overwhelming.

Free Capacity

Another alternative, of course, is to move your blog to a service like Blogger. While you give up a significant amount of flexibility, you never have to worry about capacity. Believe me when I tell you that 100K requests over the course of a day wouldn't have a problem. You'd probably have to bring in that many requests per hour (30 requests per second) before it was even noticeable to them.


But blogger has a bunch of drawbacks, the worst of which is that you don't even own your work and that somebody else can close your account for whatever reason, including all your work of many years. I don't think its wise to use a hosted blog, although I do so on the California Photo Scout myself. The limitations are pretty awful though.

how to increase visitors

how it is possible for my blog please give step wise directions what to do with site

There are plenty of forums

There are plenty of forums on the web that can help you drive more visitors. If your main concern is to give useful information and if you don't worry about visitors but write with passion traffic will follow. Most of all it takes patience.

wouldn't have a problem

Another alternative, of course, is to move your blog to a service like Blogger. While you give up a significant amount of flexibility, you never have to worry about capacity. Believe me when I tell you that 100K requests over the course of a day wouldn't have a problem. You'd probably have to bring in that many requests per hour (30 requests per second) before it was even noticeable to them.

Blogger is not an option

I need the flexibility and most importantly I own the content. This site represents a significant personal investment that I won't turn over to Google.


JESUS IS LORD, and believe in your ♥ that GOD raised HIM from the dead, thou shalt be saved, and thy house.(Refer to THE HOLY BIBLE: ROMANS 10:9 and ACTS 16:31b)

Have you received THE LORD GOD OF ISRAEL’S MOST BELOVED and ONLY BEGOTTEN SON, THE LORD JESUS CHRIST, to be your personal Lord and Savior?

If not, say this prayer: “LORD GOD, please forgive me of my sins. Thank YOU for YOUR Faithfulness in always being with me and loving me so much to send me YOUR MOST BELOVED and ONLY BEGOTTEN SON, THE LORD JESUS CHRIST, to die on the cross for me. I receive HIM now into my heart and life to be my Lord and Savior. Please send and fill me with YOUR HOLY SPIRIT. I pray this in the name of GOD, THE FATHER; GOD, THE SON; and GOD, THE HOLY SPIRIT. † Amen.”

“So let everyone in Israel know for certain that GOD has made this JESUS, whom you crucified, to be both Lord and Messiah!” ACTS 2:36

“For I tell you, you will not see me again until you say, BLESSED IS HE WHO COMES IN THE NAME OF THE LORD." MATTHEW 23:39

Visit and share this with family, friends, fans and followerst:

May you be blessed abundantly!

Thanks a lot for enjoying

Thanks a lot for enjoying this warning compute with me. I am appreciating it unyielding oft! Unranked sassy to somebody hot penalisation. Echt acquisition to the communicator! all the humanlike!
birthday wishes for boyfriend

internet marketing

this website show how to increase our visitor and Sounds interesting? Browse on for more information on this topic. You have reached the Right place we "Cash And Prices " show you how to achive it easily.

Future residents are within

Future residents are within a short driving distance to Ang Ko Kio Hub and Compass Point. With such a short drive to the city area as well as the orchard and bugis area, entertainment for your love ones and family will come at a stone’s throw away.