Archive for category AWS

Capturing users IP addresses in Apache httpd and Tomcat logs behind an ELB

When an Elastic Load Balancer handles a connection it sends it’s own (internal/private/10.x) address instead of the clients. It sends the clients along with the request as X-Forwarded-For. To log this you need to log X-Forwarded-For instead of the source IP.

Here are 2 links discussing the problem, the first covers a basic Apache & Tomcat setup, but the second one has a point about direct access getting not logged and has an Apache httpd specific solution.

http://blog.kenweiner.com/2009/09/amazon-elb-capturing-client-ip-address.html

http://blog.grahampoulter.com/2011/10/how-to-log-client-ip-from-apache-behind.html

Thanks @grahampoulter and @kweiner

 

 

, , , , ,

No Comments

My Comments on ‘Cheap Hosting Amazon AWS’ Blog Post

My Comments on the post

“Cheap Hosting Case Study: You Get What You Pay For (And Sometimes Not At All)”

These are all interesting points that anyone using AWS, and especially Spot Instances should know. The service provider (Amazon AWS) should not be blamed for the user mis-understanding how their service works.

There are 3 things about Spot instances that everyone should know, that Amazon is less than clear about on their product pages, and I found out at the AWS Summit in SF.
1. You have your instance only for as long as your price is higher than the current spot price, and this is measured much more frequently than the bid interval of 1 hour. So you could launch an instance for up to $.15/hr it could run for 30 minutes, and then be terminated unexpectedly when the spot price is over $.15/hr
2. People can, and do, bid HIGHER than the on-demand price for spot instances to get priority machine allocation. This means they get priority OVER all other spot prices, AND over “normal” on demand users.
3. Spot instances are terminated without a safety net, all data/jobs/processes on them need to be designed to be re-entrant, and fail-able. The mentality for typical spot instance usage is, there is a need, a spot instance is launched, it checks out a job from a queue (like SQS, but frequently a custom setup), it runs, *IF* it finishes, it returns success, *IF* it does not finish, another instance picks up the job.
The better your application and plan meet the functionality of spot instances the more you can work the system and save a lot.
Design and plan for failure, and you will love the cloud.
Design and plan like you have a physical server, and you will complain that the cloud let you down.

, , , , , , , , ,

No Comments

Reading : Peecho Minimizing Downtime on Amazon AWS

 

What the guys over at Peecho have done is cool. Similar to the path I’ve taken with auto-scaling and machine updating from S3.

I am going to add some of their ideas, and will incorporate that in my upcoming posts with scripts to do it.

Check it out on their blog – http://www.peecho.com/blog/minimizing-downtime-on-amazon-aws.html

 

, , , , ,

No Comments

Your IE new tab has been hijacked (but not by me)

The short version is that you are at my site because someone hijacked your new or private tab in Internet Explorer.

Please do not blame me, my web servers have been slammed with lots of traffic from this and it will cost me money, and I did not cause it.

Here is how to fix it. Below is a detailed description of what I think happened.

You will need to use the windows registry editor to fix it, or you can download this registry file, and double click on it.

Open Regedit and go to : HKLM\Software\Microsoft\Internet Explorer\AboutURLs
On the right pane double click on the tabs value and change it to : res://ieframe.dll/tabswelcome.htm.

I use Amazon Web Services to host this blog. Part of those services is a load balancer called an Enterprise Load Balancer (ELB).

I point my site at an amazon name (www-jonzobrist-com-954435911.us-east-1.elb.amazonaws.com) and Amazon handles the IP addresses and networking. The upside is I get great scalability for very low cost. Sometimes people set their load balancer incorrectly, and point a hostname (in this case gg.blogpear.com) directly at one of the IP addresses in their load balancer pool. This is wrong because Amazon can change at any time which IP address gets assigned to which load balancer, and they do not guarantee you will ever get that IP back. Someone, who is probably a malicious hacker type, hijacked your browser tab for either new tab or private tab in your Internet Explorer browser. They pointed it at a DNS name gg.blogpear.com, and that DNS name at an IP address on Amazon’s ELB. Somehow Amazon gave me that IP for my www.jonzobrist.com pool, so I got all the traffic. This killed my web servers quickly, and took me most of the day to recover from. I did so initially by setting up rules to return a quick 403 – permission denied error to all the requests. Then as I investigated it further, I figured out (I think) what happened. So now, you get redirected to this page, and hopefully you will get your computer cleaned up and we can all move on without too much trouble.

I recommend you also get a virus scanner, something like Avast, which is free for the non pro version. Download it from Avast.

I would also recommend you download and use Google Chrome and Mozilla Firefox, as they are both more secure (and generally better) web browsers.

I hope this helps!

-Jon Zobrist <jon@jonzobrist.com> http://www.jonzobrist.com/

, , , , , , ,

No Comments

Snowed in? 15 Thinks you should be using online CCOD – 9.6.2011

There are a ton of cool things to do on the Internet. New doors are open to everyone. I’m surprised how often we take it for granted that everyone is in on the latest trend in tech. Here is my humble addition to a list of things that I think people should be using online.

 

1. Twitter – News *stream*, or should I say FLOOD. Follow smart people, get smart (filtered) news and info. Want to blow your news mind? Get tweetdeck and put in a search for any hot topic. (Don’t follow #earthquake unless you want to feel constant fear).

2. Facebook – Connect with your family and friends. Be benign on Facebook! The Internet is public, immortal, and Facebook does hate your privacy.

3. Amazon AWS/EC2 - What you don’t need a virtual server? You sure about that? Not for your blog? Not even if it scales infinitely? Not even if it’s free?

4. WordPressJoomla and Drupal are cool, but WordPress is the king of the web page CMS.

5. Gmail – Seriously, stop deleting your email, get a gmail account. Use your own domains (Google Apps is still free for < 10 users).

6. Google Docs – If you haven’t had 10 people all editing the same spreadsheet at the same time you have not Cloud’d it up.

7. Cloud Music (Google Music, Amazon Music Locker, iCloud, Soundcloud, Spotify) – This is new, try them all out, find new music, sync your own.

8. Google – Search done right. Everyone has been playing catchup for a while now, and I’m sure that one day they will, but until then, google.com

9. Snopes – The Internet means rapid access to information sharing, but many people share false information. Sites like snopes.com

10. Shopping - Deal sites like slickdeals.net, fatwallet.com, woot and more track deals as they happen, often with good comments on how to maximize them. The people on some of these sites are mad geniuses when it comes to getting the most for your buck.

11. Skype – Everyone has it, get on and video chat your friends in other countries for free. Ride this one until Microsoft torpedos it, and we all move to Google Chat, which you should be on already via your gmail account.

12. Linux – If you are even slightly technically inclined, Linux opens the door to you (for free) to everything from high end movie effects to  computer forensics. Get started with a Live CD from Ubuntu (Your computer is probably 64-bit, and you probably want the desktop version – You can boot the CD and use Linux without doing anything to your computer), and NO it does not run Office or any Windows program, but it does run thousands of cool programs.

13. Photo sites (Picasa, Flickr, Smugmug) There is no reason you should be burning a photo CD to send to your friends and family. Get an upload utility, and start putting your photos on the ‘net. You don’t have to share them, and I would highly recommend NOT sharing them publicly unless they are very public information. I do not post pictures with faces in them without permission from the person owning the face, and, in general, don’t do this.

14. Education (Khanadademy, Alison.com, MIT Open Courseware, Instructables, k12) – There are too many to name, and pretty much access to infinite information is it’s own education. Don’t think that just because a skill isn’t directly computer related that you can’t learn howto do it online, and for maybe for free.

15. Wikipedia - What is a wikipedia? Well, a wiki is a website that anyone can edit the pages of, so, Wikipedia is an encyclopedia that anyone can edit. Not always right, but rarely uninformative.

 

Well, I hope this helps. Please send me your lists or additions (comment below, or email to jon@jonzobrist.com).

 

, , , , , ,

No Comments

I’m wondering if I should have my install script clone common Linux OS’s like Red Hat/CentOS and Debian/Ubuntu?

So, I have a script that can run from init. You configure it from a config file, set it to start on boot via normal init, and then you can run it on to snapshot your data to Amazon S3 anytime.

It supports directories/users/permissions as well as full MySQL dumps, only specific database MySQL dumps, and only specific table per database dump.

It goes both ways, run it with /etc/init.d/bluesun-setup.sh start and it deletes the local directories, and downloads all the updated ones from Amazon S3.

If you run it with /etc/init.d/bluesun-setup.sh updateS3 # it will push all the same configured files to Tar/GZ files on S3.

I need to polish it a bit more, but it’ll be free/open source here soon.

Any feedback is appreciated.

, , , , , , , ,

No Comments

Have my auto update script mostly ready, this is a test!

Hopefully the new server will start automatically behind the ELB, update to the latest, and join the pool!

, , , , , ,

1 Comment

Can’t wait to play with SQS on AWS!

Read this interesting post about how coffee shop workers parallel applications in Amazon Web Services, with new types of CloudWatch metrics and how you can use them to Auto Scale your AWS stack. The post is at the AWS blog here http://aws.typepad.com/aws/2011/07/additional-cloudwatch-metrics-for-amazon-sqs-and-amazon-sns.html
Thanks for the great post!

, , ,

No Comments

Thinking about hosting a WordPress site on S3

I recently moved my Joomla backed consulting website completely to Amazon S3, and have been very happy with the results. I would like to do something similar for my personal blog site at jonzobrist.com, however I would like it to be more dynamic, or at least easily update-able.

For my Joomla site, I did a complete mirror to static html and then uploaded all of that to S3, in a bucket with the same name as the site’s (www.bluesun.net), and changed DNS to point to the CNAME for that bucket’s HTTP address. This involved running wget -r -k -E -p -U Mozilla http://www.bluesun.net, editing the files wget copied to all point at the right places for things like menus, etc, and then uploading the files to Amazon S3.

My goal here is to recreate that in a more automated way, so that I can have a main site that is dynamic, but most, if not all, of the content is served from a static repository on S3. The expected outcome I think will be to take a site that costs around $15-20/month and make it cost < $1 /month. And, if I get some huge surge of traffic, to handle the load gracefully, and scale into the many terabytes of serving up data affordably.

A few quick thoughts/notes;

First, if you don’t change permission on newly uploaded items on S3 they default to your default, which is usually no public access. However, if you upload a new version of a file, it keeps the permissions the previous version had.

Second, you cannot host a naked domain (in this case http://bluesun.net) on Amazon S3. This is more a limitation of the the standards that say you shouldn’t. It means that you need something to redirect your naked domain to your web server. A lot of people don’t do this at all, but I think it’s a good thing to do. I think the details of this limitation will actually come in handy in my hybrid dynamic/static WordPress site.

Third, it makes a lot of sense to compress objects, and setting the right headers on the object will, I believe, get S3 to automatically server it up in a way a browser can understand. Most of the things that make up web pages (HTML and javascript) are text based and compress very well. On the other side images used on the web are generally already very compressed.

Fourth, having a hybrid site means you will still have some dynamic objects and this will mean manually processing (or manually setting up automated processing) html files to separate dynamic from static content.

Fifth, I’m a huge fan of things like Google Analytics, which are hosted by Google, and only included in my site as a static snippet of code that pulls more code direct from their servers. I would love to have something similar for comments and other user generated content that messes up the static website paradigm. I think technologies like AJAX can really shine here.

Brief background, my site (jonzobrist.com) is a standard WordPress install, currently running on an EC2 Micro Instance running Ubuntu 10.04 with Apache/PHP/MySQL all running on one machine. It’s an EBS backed instance, and I snapshot the root volume. I don’t really make updates more than once or twice a week, and none of my content needs to be pushed live in any kind of urgent manner. That said, I use WP to Twitter to auto tweet new posts, so I need to be able to force an update, or handle not having new content on S3 gracefully. I don’t get a particularly large number of visitors, lately about 1,000 a month. My main motivation for doing this is to see if it can be done, so I can do it for other sites I support.

Here is a graphical representation of what I think it will look like when done.

Diagram of a static WordPress site on Amazon S3

Then I just need to push all the very static content to Cloud Front for CDN!

What do you think?

, , , , , , , , ,

No Comments

Moved my consulting website to Amazon S3

It’s a Joomla site, but I rarely have updated it, so I just made a static mirror of it with wget, then uploaded it to S3!

http://www.bluesun.net

Amazing how easy it is. I want to make either a WordPress plugin, or a set of scripts so I can keep my WordPress site dynamic locally (like a stage and master copy), and then when I want to push updates have it update a static directory, and put files in S3. Ideally this would also push to Cloud Front.

Now my website is up all the time!

In addition, my web site is cheap to run, requires no server (other than a core http://bluesun.net/ redirect, which many DNS hosts will do for free)

Plus how secure is that? Static HTML files on S3? You can download them, but that’s about it, unless you’re trying to hack Amazon, and good luck with that.

I can’t wait to see what it costs for the very few visitors I get to get things straight from S3, and in the future, CloudFront.

, , , , , , , ,

No Comments

Easy AdSense by Unreal