Archive for category AWS
Have my auto update script mostly ready, this is a test!
Posted by Jon Zobrist in AutoScale, AWS, ELB on July 23, 2011
Hopefully the new server will start automatically behind the ELB, update to the latest, and join the pool!
Can’t wait to play with SQS on AWS!
Posted by Jon Zobrist in AWS, SQS on July 23, 2011
Read this interesting post about how coffee shop workers parallel applications in Amazon Web Services, with new types of CloudWatch metrics and how you can use them to Auto Scale your AWS stack. The post is at the AWS blog here http://aws.typepad.com/aws/2011/07/additional-cloudwatch-metrics-for-amazon-sqs-and-amazon-sns.html
Thanks for the great post!
Thinking about hosting a WordPress site on S3
I recently moved my Joomla backed consulting website completely to Amazon S3, and have been very happy with the results. I would like to do something similar for my personal blog site at jonzobrist.com, however I would like it to be more dynamic, or at least easily update-able.
For my Joomla site, I did a complete mirror to static html and then uploaded all of that to S3, in a bucket with the same name as the site’s (www.bluesun.net), and changed DNS to point to the CNAME for that bucket’s HTTP address. This involved running wget -r -k -E -p -U Mozilla http://www.bluesun.net, editing the files wget copied to all point at the right places for things like menus, etc, and then uploading the files to Amazon S3.
My goal here is to recreate that in a more automated way, so that I can have a main site that is dynamic, but most, if not all, of the content is served from a static repository on S3. The expected outcome I think will be to take a site that costs around $15-20/month and make it cost < $1 /month. And, if I get some huge surge of traffic, to handle the load gracefully, and scale into the many terabytes of serving up data affordably.
A few quick thoughts/notes;
First, if you don’t change permission on newly uploaded items on S3 they default to your default, which is usually no public access. However, if you upload a new version of a file, it keeps the permissions the previous version had.
Second, you cannot host a naked domain (in this case http://bluesun.net) on Amazon S3. This is more a limitation of the the standards that say you shouldn’t. It means that you need something to redirect your naked domain to your web server. A lot of people don’t do this at all, but I think it’s a good thing to do. I think the details of this limitation will actually come in handy in my hybrid dynamic/static WordPress site.
Third, it makes a lot of sense to compress objects, and setting the right headers on the object will, I believe, get S3 to automatically server it up in a way a browser can understand. Most of the things that make up web pages (HTML and javascript) are text based and compress very well. On the other side images used on the web are generally already very compressed.
Fourth, having a hybrid site means you will still have some dynamic objects and this will mean manually processing (or manually setting up automated processing) html files to separate dynamic from static content.
Fifth, I’m a huge fan of things like Google Analytics, which are hosted by Google, and only included in my site as a static snippet of code that pulls more code direct from their servers. I would love to have something similar for comments and other user generated content that messes up the static website paradigm. I think technologies like AJAX can really shine here.
Brief background, my site (jonzobrist.com) is a standard WordPress install, currently running on an EC2 Micro Instance running Ubuntu 10.04 with Apache/PHP/MySQL all running on one machine. It’s an EBS backed instance, and I snapshot the root volume. I don’t really make updates more than once or twice a week, and none of my content needs to be pushed live in any kind of urgent manner. That said, I use WP to Twitter to auto tweet new posts, so I need to be able to force an update, or handle not having new content on S3 gracefully. I don’t get a particularly large number of visitors, lately about 1,000 a month. My main motivation for doing this is to see if it can be done, so I can do it for other sites I support.
Here is a graphical representation of what I think it will look like when done.
Then I just need to push all the very static content to Cloud Front for CDN!
What do you think?
Moved my consulting website to Amazon S3
Posted by Jon Zobrist in AWS, HTML, internet, S3 on June 26, 2011
It’s a Joomla site, but I rarely have updated it, so I just made a static mirror of it with wget, then uploaded it to S3!
Amazing how easy it is. I want to make either a WordPress plugin, or a set of scripts so I can keep my WordPress site dynamic locally (like a stage and master copy), and then when I want to push updates have it update a static directory, and put files in S3. Ideally this would also push to Cloud Front.
Now my website is up all the time!
In addition, my web site is cheap to run, requires no server (other than a core http://bluesun.net/ redirect, which many DNS hosts will do for free)
Plus how secure is that? Static HTML files on S3? You can download them, but that’s about it, unless you’re trying to hack Amazon, and good luck with that.
I can’t wait to see what it costs for the very few visitors I get to get things straight from S3, and in the future, CloudFront.
Amazon keeps doing it again and again! AWS route53
Posted by Jon Zobrist in AWS, internet, Route53 on June 6, 2011
Amazon almost makes me laugh whenever I sign up my small business for one of their “Amazon Web Services” (AWS).
This time, it’s their new DNS hosting called “Amazon Route53″
Checkout the screen shot of the pricing. Seriously? $1/month plus a whopping $.50 per BILLION queries. Seriously? A Billion? Almost 1 in 10 people on the planet would have to make a single DNS request to dent my pocket book a whopping half a dollar?
Well, Amazon will indeed have wrested begrudgingly my half dollar from my hand that first month I get my billion queries worth.
Delete lots of EC2 EBS volumes
Posted by Jon Zobrist in AWS, Bash, EBS, Programming on May 23, 2011
Here is a quick script to delete lots of EBS volumes on Amazon EC2.
The output from something like
ec2-describe-instances i-6fc30f01 | grep ‘^BLOCKDEVICE’ | awk ‘{ print $3 }’ > delete-these-volumes.txt
Then run
./delete-volumes.sh delete-these-volumes.txt
Can be fed into this script (you may want to edit the file first, I accept no responsibility for you deleting your volumes, with or without my script.
Download the gzipped script here.
Example output from me deleting 60 real volumes:
My EC2 EBS RAID-0 creation script
Posted by Jon Zobrist in AWS, Bash, EBS, EC2, Linux, Linux Support - Servers, Programming on May 4, 2011
I want to write up a full howto/tutorial, but have not had the time.
Here is the script that I referred to in my post at http://www.jonzobrist.com/2011/04/16/ec2-raid0-on-linux-with-ebs-volumes-notes/
This program is distributed in the hope that it will be useful but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License HERE for more details.
http://www.jonzobrist.com/files/setup-ec2-raid-0.zip
http://www.jonzobrist.com/files/setup-ec2-raid-0.sh.gz
This script needs an AWS command line setup, and may need some minor tweaking if you’re not running a Ubuntu server.
Ubuntu’s latest AMI’s are available for 10.04 here.
Please feel free to submit patches, comments, or questions.
Thanks to everyone whose helpful posts online and in the AWS forums helped me with this script. I could not have done it with out your generous sharing of work.
Enjoy!
Getting started with Chef.
Posted by Jon Zobrist in AWS, Chef, EC2, Linux, Linux Support - Servers on April 21, 2011
I write a lot of scripts, configure a lot of systems, and do a lot of snapshot related backup/restore/sync stuff on servers in house, and in the cloud.
Chef is a Ruby-based configuration management engine. There is a good intro here.
I set up a Chef Server. There’s a good overview here.
I followed the instructions to setup a Chef server from this Wiki on Opscode.
I also referred to the information from Grig Gheorghiu’s blog here.
Now that my server is up, I’m hoping to port my server configurations to Chef, with heavy reliance on EC2 and hopefully UEC.
EC2 RAID0 on Linux with EBS volumes notes
Posted by Jon Zobrist in AWS, EBS, EC2 on April 16, 2011
I’m almost done with my publish / post of my script that auto creates, formats, and mounts arbitrary sized RAID-0 arrays.
I wanted to share some things about running RAID0 under Linux on Amazon AWS/EC2 with EBS volumes.
- If you reboot and an array comes up inactive due to missing members, don’t panic, try rebooting again. Sometimes EBS volumes aren’t ready when the array is assembled. I’ve only seen this once and on an instance with 33 drives spanning 4 arrays and several direct EBS volumes.
- I don’t set my volumes to auto mount. I prefer to reboot as little as possible, and don’t mind starting up things. EBS volumes in large RAID’s seem to cause problems and can hang your boot. If you need them automatic, try adding a delayed mount script.
- EBS volumes on RAID don’t really care what order or drive letters they are assigned to, they can use the software meta-information to assemble correctly.
- Use AWS CloudWatch and get lots of great disk metrics. Also setup alerts for bad conditions for disks.
- Watch your dmesg/kernel output for kernel panics. It seems these happen when there is a large amount of things to be written to disk, and cause server load to spike suddenly.
- Remember that on Linux, setting the user mountable flag in /etc/fstab will automatically mount the partition with no-exec. You must specify exec in the options in fstab.
- Be careful to avoid having things try to use your partition when it’s not mounted. The easiest thing is to make sure it’s mount point folder is empty, and owned by root.
- Don’t forget that unix/Linux can happily mount a new volume over a folder with data in it, and that data will appear gone until the volume is unmounted. This can play tricks with file handles. If you launch a program that opens a file handle and keeps it open, and then mount a volume over top of the directory or parent, most of the time the program will continue to access the file that seems lost but is just clobbered.
I’m sure there’s more. Feel free to post questions or e-mail me questions jon@jonzobrist.com
I hope to finish my post to auto-create EBS volumes, attach them to an instance, assemble them into an array, format it, and add an entry to fstab for it.


