18 Nov 2008 10:59
For the last couple of weeks we are thinking of ways to make Wikidot.com faster, more responsive and more scalable. Personally I am fascinated with Amazon's web services, particularly :
- EC2 (Elastic Cloud Computing) — virtual servers on demand
- and S3 (Simple Storage Service) — storage space on demand
I can see those services as a natural environment for future Wikidot hosting, but this is not exactly what I wanted to write about.
I had a spare 10 minutes today to test the service and… works great. In a few words it works like that:
- You have some static files (e.g. like files from static.wikidot.com) and you want to provide quick, latency-minimized access to them for your users
- This introduces latency (speed of light is somehow limited to 300 000 km / s), e.g. from Germany you get 140ms delay when accessing such files
- CloudFront is a network of servers. They can distribute your content across the globe.
- So that our users from Germany do not need to fetch static files from USA, they can get them from the nearest CloudFront node right in Germany
- The quality difference is huge, can reduce the delay from 140ms to 5ms (literally!)
- Now when you have dozens of static files, this can improve the responsiveness dramatically.
And believe me, to set up an initial "distribution" it took me 15 minutes, which included learning all about the CloudFront, upgrading my S3Fox to support CloudFront. It took me futher 15 minutes to test the availability of my content from our servers in Poland, Germany and USA. Works perfectly so far.
Here is an example tcptraceroute to our distribution bucket from Germany, compared to the traceroute to the current static server in USA.
traceroute to d6c4mtj4tv0du.cloudfront.net (126.96.36.199), 30 hops max, 40 byte packets 1 static-ip-85-25-57-10.inaddr.intergenia.de (188.8.131.52) 0.047 ms 0.021 ms 0.022 ms 2 static-ip-85-25-95-129.inaddr.intergenia.de (184.108.40.206) 0.168 ms 0.180 ms 0.158 ms 3 static-ip-85-25-225-5.inaddr.intergenia.de (220.127.116.11) 0.177 ms 0.151 ms 0.191 ms 4 tge-2-4.498.bbr1.fra3.de.inetbone.net (18.104.22.168) 0.542 ms 0.522 ms 0.510 ms 5 22.214.171.124 (126.96.36.199) 0.723 ms 0.655 ms 0.658 ms 6 ae-2-79.edge4.Frankfurt1.Level3.net (188.8.131.52) 1.013 ms ae-1-69.edge4.Frankfurt1.Level3.net (184.108.40.206) 118.276 ms ae-4-99.edge4.Frankfurt1.Level3.net (220.127.116.11) 65.774 ms 7 (18.104.22.168) 1.136 ms 1.268 ms 1.078 ms 8 22.214.171.124 (126.96.36.199) 1.613 ms 1.581 ms 1.096 ms
traceroute to static.wikidot.com (188.8.131.52), 30 hops max, 40 byte packets 1 static-ip-85-25-57-10.inaddr.intergenia.de (184.108.40.206) 0.074 ms 0.029 ms 0.025 ms 2 static-ip-85-25-95-129.inaddr.intergenia.de (220.127.116.11) 0.171 ms 0.169 ms 0.130 ms 3 * * * 4 te1-1.cer03.dal01.dallas-datacenter.com (18.104.22.168) 123.063 ms 123.123 ms 123.122 ms 5 po3.dar02.dal01.dallas-datacenter.com (22.214.171.124) 123.348 ms 123.616 ms 123.592 ms 6 po2.fcr03.dal01.dallas-datacenter.com (126.96.36.199) 123.377 ms 123.369 ms 123.490 ms 7 188.8.131.52-static.reverse.softlayer.com (184.108.40.206) 123.359 ms 123.376 ms 123.397 ms
See the difference? ;-) 1 ms vs 123 ms. OK, this is because our server and the CF node are both in Frankfurt ;-), but our users in Europe and Asia should feel the huge difference!
The only problem now is that CloudFront is BETA which means it is not guaranteed to work very reliably. There is no SLA for it (yet), so I guess we will keep an eye on it.
In fact we have been thinking about using a CDN for quite a long time now, but could not find a good option for it. The beauty of CloudFront is that it is very easy to integrate with our system. One could do the following:
- Create a bucket in S3.
- Use S3Fox to create a "distribution" for the bucket, which means the content of the bucked will be distributed through CF.
- Use s3fs to mount the bucket on the server.
- Simply copy contents of the static server to the bucket (it is being accessed as a filesystem now).
- CF will automatically pull content from your bucket, so there is no need for manual sync.
- You can set up a CNAME for the distribution. We could use… static.wikidot.com ;-)
- When our users access static.wikidot.com, they will be automatically directed to the closest CF node.
This can be implemented in 1 hour.
Easy? You bet. The only trick for us right now is that CF does not handle HTTPS (SSL) connections, so we still need to keep static-ssl.wikidot.com on our primary servers.
One more note: it looks like DNS data for CloudFront (cloudfront.net) servers did not propagate yet and some of us have problems finding the cloudfront.net domain. This is likely a temporary issue and tomorrow everything will be fine. The problem we had was most likely caused by invalid routing tables, so it has nothing to do with CloudFront itself.
UPDATE: The implementation was not that easy as I thought and there were some quirks, but we managed to move all content from static.wikidot.com to CloudFront on Wed, 19 Nov. Together with some other optimizations some pages load twice as fast!. Will write about this later because this is such a huge improvement!
rating: 2, tags: amazon aws cdn cf cloudfront wikidot