Edward Capriolo’s (@edwardcapriolo) post may be better titled “Myth Busters: Opts edition. The Misleading Appearance of Amazon AWS Costs.”

http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/myth_busters_ops_editition_is

Edward, you are absolutely correct. The cost of servers on AWS is more than the cost of servers in real life.

Your final conclusion is absolutely incorrect.

In fairness you are attempting an apples to apples comparison, and concluding that apples are better than oranges.

I suggest you consider apples to oranges comparison and see that the cloud (specifically Amazon’s AWS) is not the sour apples you’re comparing it to.

So, if you compared features on the servers, and, even more so, if you change your application to take advantage of the AWS cloud, the cloud will absolutely crush your comparison in price and scale.

Things that you won’t have on your servers for $175k.

1. Atomic-multi data center sub second volume snapshots. EBS volumes rock. Snapshots persistent to S3 are amazing.
2. Global redundancy. You pay $2k/month for your data center, I’m guessing if it gets hit by a meteor you’re SOL. With AWS for far less than $2k/month we can recover to either coast of the US, Ireland, Singapore, Tokyo or Sao Paolo in < 1 hour.
3. Elasticity. They named their platform Elastic Cloud for a reason. You bought 20 servers. How long did that take? A week, a month, a day? On Amazon it took 2 minutes. Need 20 more, 2 more minutes. Don’t want to watch your Cassandra cluster for load and pre-order servers? Setup an auto scaling group, have 10-60 nodes based on average CPU, or any other metric you want. You don’t only get elastic load scaling, if your app can wait you get elastic pricing. Don’t care when your job runs, just needs to be sometime between midnight and 6am? Game the spot instance market, save a ton, 50-90%. Have a resource that is usually idle but sometimes needs 60GB RAM? Pay for a micro/medium instance and scale it to a 4XL whenever your spikes are.

This is all just with EC2, server virtualization.

If you added the components necessary to do this with your own hardware your price would be 4X what it is on AWS.

Now, let’s talk about where you can save 10X. Things AWS excels in that you did not even mention.

1. S3. I know your local SATA drives or SAN are cheaper. But are they designed for 11 9′s of redundancy? Compare that cost. Are they secure and globally accessible? Do they have virtually unlimited bandwidth to your alternate site/customers? Can you just keep growing them and only paying fer allocated space?
2. Bandwidth. You did not even mention this. We went from a traditional 5MBPS commit on a dual 100MBPS ethernet for $800/month 95% billing scam, to no upper limit for burst, and tens of dollars per month based on a fair, actual usage at pennies per GB.
3. Actual cloud apps. Ditch your MySQL database and use simpleDB or the SSD based DynamoDB. Get infinite scale, price per actual data used, and built in redundancy.
4. Support staff. As an IT person this one pains me a bit, until I recall how many bad IT departments I’ve experienced. How much are you paying the monkey who maintains those 20 servers? Your developers do it? What if they could just concentrate on coding? All of this costs more than your straight hardware comparison.
5. Opportunity cost. You own those servers. Microsoft or Apple or Google or someone completely new comes up with a new cloud paradigm, I can migrate in days if not less from AWS. There is no option for dealing with one of the most consistent paradigms of our age – Change is inevitable.
6. Development flexibility. You did not price the likely necessary qa and dev servers. Right off the top, that doubles or triples your price if you need a clone of production for test or development. On AWS you automatically clone your running production and test your continuous deploys on real, identical data and setups. It takes some work, but once you’re there it’s a million times better, and you never have to hear “It worked on QA” again! And what about new development? You or one of your engineers wants to ‘try out’ something new. How much is that server? AWS it’s pennies to tens of dollars to let people play with wild new configurations.
7. Growth. Already mentioned in the elastic part, but consider how much happier your boss is if sales hit the hockey stick and he didn’t hear complaints of any operations issues, vs. calling you on vacation to yell about “Everything being down” right at the moment you made it big.

There are 2 instances where I think you should not drop everything you are doing and migrate to AWS.

1. If you are very data heavy (PB) you should burst to the cloud, get a 10gb cross connect to AWS.
2. If you are cpu heavy (>80% cpu/server avg) you should burst cpu to the cloud.

I did the same comparisons you did when I first evaluated AWS, and am so glad my boss urged me to try it out. And, yes, it seemed scary and more expensive, but has turned out easy and far less expensive.

Everyone else should refactor their applications and move to the cloud. If you haven’t already started you are behind in the game.

And you can quote me on that.

-Jon

Related posts:

1. My response to “Is the Cloud Undermining Enterprise Disaster Recovery” @aryakanetworks [No] Disagree completely with your assessment of Cloud Computing Disastery Recovery....
2. Reading : Peecho Minimizing Downtime on Amazon AWS   What the guys over at Peecho have done is...
3. My Comments on ‘Cheap Hosting Amazon AWS’ Blog Post My Comments on the post “Cheap Hosting Case Study: You...

Related posts brought to you by Yet Another Related Posts Plugin.

Old google pricing is WAS the cheapest per GB at $.25/GB/Year.
Microsoft SkyDrive is second at $.50/GB/Year, and the New Google Drive pricing is third with tiers mostly around $.60/GB/Year

Here is the link to my “Public Google Spreadsheet With Cloud Storage Options Breakdown
http://bit.ly/IPlANs

There are plenty of good articles out there comparing the features of each cloud drive, as many provide nice add ons.
Here are a few

Mashable Tech - Google Officially Launches Google Drive

http://mashable.com/2012/04/24/google-drive/

Telegraph – Google Drive: iCloud, Dropbox, SkyDrive and Box comparison

http://www.telegraph.co.uk/technology/google/9223583/Google-Drive-iCloud-Dropbox-SkyDrive-and-Box-comparison.html

The Guardian – Google Drive versus Dropbox and the rest: cloud storage compared

http://www.guardian.co.uk/technology/blog/2012/apr/25/google-drive-cloud-storage-compared

CNN - How does Google Drive compare to the competition?

http://www.cnn.com/2012/04/25/tech/web/compare-competition-google-drive/index.html

Lifehacker’s article, “Drag-and-Drop To Automatically Encrypt Files in Google Drive Using Automator on Mac” http://lifehacker.com/google-drive/

And they discuss it extensively on Twit.tv’s podcast [which totally rocks] – This Week in Google episode 143 http://twit.tv/show/this-week-in-google/143

So many cloud storages, still too many files, and still too little bandwidth…. My ~3.5TB of files would take 169 Days 21 Hours 47 Minutes 44 Seconds to upload at 2 Mbit/Sec…

Related posts:

1. Guest Blogger Brad Zobrist – How he would implement a Bitcasa First I think it’s good to clarify what I understand...
2. My response to “Is the Cloud Undermining Enterprise Disaster Recovery” @aryakanetworks [No] Disagree completely with your assessment of Cloud Computing Disastery Recovery....

Related posts brought to you by Yet Another Related Posts Plugin.

Of course you need your ELB command line tools, but you also need the IAM Cli tools if you are using an SSL certificate.

There is more detail here for SSL certificates https://makandracards.com/makandra/1673-change-update-ssl-certificate-for-amazon-elastic-load-balancer

You will need your ELB load balancer’s name, find it with elb-describe-lbs

1. Remove the old port if there is one already (In this example it’s 80 and 443)

./elb-delete-lb-listeners my-inthinc-com-oregon --lb-ports 80

-or- for HTTPS

./elb-delete-lb-listeners my-inthinc-com-oregon --lb-ports 443

2. Add the new port (using your SSL cert name found from iam-servercertlistbypath)

./elb-create-lb-listeners my-inthinc-com-oregon --listener "lb-port=443,instance-port=80,protocol=http"

-or- for HTTPS

./elb-create-lb-listeners my-inthinc-com-oregon --listener "lb-port=443,instance-port=8080,protocol=https,cert-id=arn:aws:iam::322191361670:server-certificate/www.example.com"

That’s it!

You can’t do this in the current AWS Management Console.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Often times you manage similar users across various servers and they have the same public keys between servers.

Here is a quick script to gather all the users local public keys (the ~/.ssh/id_rsa.pub) as well as their externally used keys (the ~/.ssh/authorized_keys)

I published this on github, here.

#!/bin/bash
#
# Author : Jon Zobrist <jon@jonzobrist.com>
# Homepage : http://www.jonzobrist.com
# License : BSD http://en.wikipedia.org/wiki/BSD_license
# Copyright (c) 2012, Jon Zobrist
# All rights reserved.

# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice, this
# list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# Purpose : This script aims to gather all public ssh keys on a server and put them in a directory, with appropriate names
# Usage : gather-public-ssh-keys.sh [Directory]

if [ "${1}" ]
then
OUTPUT_DIR="${1}"
else
OUTPUT_DIR="./pubkeys"
fi

mkdir -p ${OUTPUT_DIR}
echo "Writing keys to ${OUTPUT_DIR}"

HOME_DIR="/home"
CHOWN_USER="root:root"
CHMOD_PERMS="400"
KEYFILES="id_rsa.pub id_dsa.pub identity.pub id_ecdsa.pub authorized_keys"

for USER in $(/bin/ls -1 ${HOME_DIR})
do
for KEY in ${KEYFILES}
do
if [ -f "${HOME_DIR}/${USER}/.ssh/${KEY}" ]
then
FILE=${OUTPUT_DIR}/${USER}-${KEY}
echo "${USER} has public keys, copied to ${FILE}"
touch ${FILE}
chown ${CHOWN_USER} ${FILE}
chmod ${CHMOD_PERMS} ${FILE}
cp ${HOME_DIR}/${USER}/.ssh/authorized_keys ${FILE}
else
echo "${USER} has no public keys"
fi
done
done

Download the script here : gather-public-ssh-keys.gz plain text : gather-public-ssh-keys.sh

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Need Java? Pissed that Ubuntu dropped official Sun JDK from their Partner Repository?
Use
https://launchpad.net/~ferramroberto/+archive/java
Need this to add-apt-repository
apt-get install -y python-software-properties
sudo add-apt-repository ppa:ferramroberto/java
apt-get update
apt-get -y install sun-java6-jdk
java -version
#Should return something like
java version “1.6.0_26″
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

#Setup the Kafka user, install Kafka

I’m using 4 nodes, their IP’s are 192.168.1.50,51,52,53, all run Kafka and Zookeeper. The Zookeeper Admin guide recommends an odd number for the ensemble so majority rule is less likely to be split evenly/split brain style. although I did get an error from one of them urging me to run either Kafka or Zookeeper in an odd number of nodes, which I’m not sure the reasoning behind.

groupadd -g 3320 kafka
useradd -m -d /usr/local/kafka -s /bin/bash -u 3320 -g 3320 kafka
wget http://people.apache.org/~nehanarkhede/kafka-0.7.0-incubating/kafka-0.7.0-incubating-src.tar.gz
tar -zxvf kafka-0.7.0-incubating-src.tar.gz
cd kafka-0.7.0-incubating-src
mv * ~kafka/
chown -R kafka:kafka ~kafka
chown -R kafka:kafka ~kafka/*

su kafka

echo ‘declare -x JAVA_OPTS=”-Xmx3600M -Xms256M”‘ >> ~/.bashrc
echo ‘declare -x PATH=”${HOME}/bin:${PATH}”‘ >> ~/.bashrc
mkdir -p ~/zookeeper ~/logs ~/run/logs
#Configure Kafka
#setup myid file, myid has to match the server’s setting in ~kafka/config/zookeeper.properties
#This will be different on each server (1,2,3,4)
echo “2″ > /usr/local/kafka/zookeeper/myid
#Setup Kafka’s config file, needs direct pointers to all servers in the Kafka cluster on the zk.connect setting

 brokerid=2
 port=9092
 num.threads=8
 socket.send.buffer=1048576
 socket.receive.buffer=1048576
 max.socket.request.bytes=104857600
 log.dir=/usr/local/kafka/logs
 num.partitions=1
 log.flush.interval=10000
 log.default.flush.interval.ms=1000
 log.default.flush.scheduler.interval.ms=1000
 log.retention.hours=168
 log.file.size=536870912
 log.cleanup.interval.mins=1
 enable.zookeeper=true
 zk.connect=192.168.1.50:2181,192.168.1.51:2181,192.168.1.52:2181,192.168.1.53:2181
 zk.connectiontimeout.ms=1000000

#Setup Zookeeper’s config file, needs server.X for each node along with IP:port:port
vi config/zookeeper.properties

dataDir=/tmp/zookeeper
 clientPort=2181
 maxClientCnxns=0
 tickTime=2000
 dataDir=/usr/local/kafka/zookeeper/
 initLimit=5
 syncLimit=2
 server.1=192.168.1.50:2888:3888
 server.2=192.168.1.51:2888:3888
 server.3=192.168.1.52:2888:3888
 server.4=192.168.1.53:2888:3888

#Setup Kafka’s producer.properties

 broker.list=1:192.168.1.50:9092,2:192.168.1.51:9092,3:192.168.1.52:9092,4:192.168.1.53:9092
 zk.connect=192.168.1.50:2181,192.168.1.51:2181,192.168.1.52:2181,192.168.1.53:2181
 producer.type=sync
 compression.codec=0
 serializer.class=kafka.serializer.StringEncoder

#Setup Kafka’s consumer.properties

 zk.connect=192.168.1.50:2181,192.168.1.51:2181,192.168.1.52:2181,192.168.1.53:2181
 zk.connectiontimeout.ms=1000000
 groupid=test-consumer-group

#Start it all, the docs say to use something like djb’s daemon tools, but for now I just run these in a screen, as user kafka

screen -R zookeeper
cd
~/bin/zookeeper-server-start.sh ~/config/zookeeper.properties
CTRL+A D

screen -R kafka
cd
~/bin/kafka-server-start.sh ~/config/server.properties
CTRL+A D

Check zookeeper

telnet localhost 2181
ruok

#expect back imok

#Send some Kafka messages
#Start Producer

bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic test
This is a test
This is more testing

#Shutdown producer
CTRL+D
#Start consumer
#Receive the messages

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

#Expect to see
This is a test
This is more testing
References
http://incubator.apache.org/kafka/quickstart.html
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html
http://zookeeper.apache.org/doc/r3.4.3/recipes.html
http://zookeeper.apache.org/doc/trunk/zookeeperJMX.html

Download Kafka from
http://incubator.apache.org/kafka/downloads.html

Related posts:

1. Capturing users IP addresses in Apache httpd and Tomcat logs behind an ELB When an Elastic Load Balancer handles a connection it sends...
2. I’m wondering if I should have my install script clone common Linux OS’s like Red Hat/CentOS and Debian/Ubuntu? So, I have a script that can run from init....
3. HowTo Quick Setup a Minecraft Server My kids have dove into the game Minecraft and one...

Related posts brought to you by Yet Another Related Posts Plugin.

I’ve been meaning to writeup a detailed account of the Asynchronous binlog replication I’ve been running for almost 2 years, and it’s never the right time, so I’m going to put this short version up, and see if anyone is interested.

I plan to post the Bash scripts on github soon, if you want them before I do, feel free to email me jon@jonzobrist.com

Let me address the awful acronym soup!

HA = High Availability

DW = Data Warehousing

DR = Disaster Recovery

And, of course, you probably know our good friend, MySQL.

The problem this setup tries to address is how to gain better uptime (our HA part), be prepared to handle the wide array of disasters (DR), and use some of this data shipping legwork to feed our data sales team and clients (DW), all while keeping things affordable on Amazon’s cloud computing platform, AWS.

I do not claim that this is the best way to do things, and there are some caveats I detail below.

If you merely want one region backups of your database, I highly recommend you checkout Eric Hammond’s excellent tool ec2-consistent-snapshot.

If you want multi-region (or multi-country) disaster recovery, perhaps a setup like the one I’ve made would be best.

Or you may have a use for asynchronous database replication that this fits.

Overview of Async Binlog shipping

Let’s go through it in a nice, ordered list.

Setup :

1. Production MySQL cluster has the normal master/slave replication strategy
2. A full locking backup is taken and manually restored onto a permanent EBS volume
1. Note – This EBS volume has a full install of MySQL server
3. The binlog from when the locking backup was taken is replayed from the log position onward
4. List files are setup on the volume indicating which files are already replayed and which are new and ready to be replayed
5. The remaining binlog files are processed, leaving a MySQL server updated from when the latest binlogs were transferred
6. Data is imported into the Data Warehouse
7. MySQL is shut down, the volume is unmounted and detached, the instance is then terminated
8. A snapshot of the volume is taken within AWS. This copies any changed data and puts it into S3.

Now that the volume is setup, we setup an automated job that runs daily that updates the volumes :

1. The EBS volume is mounted on the server running the job
2. S3 bucket is checked for new files, anything new is downloaded, decrypted and placed onto the EBS volume
3. The volume is unmounted, a new instance is launched
4. Once the instance is up and running, the volume is attached to it, mounted, any dependencies for MySQL are installed, and MySQL server is started
5. The decrypted binlogs are played back in order until they are all completed
6. Data is imported into the Data Warehouse
7. MySQL is stopped, the volume unmounted and snapshotted
8. The instance is terminated

Caveats :

• No temp tables – There are certain operations, largely involving the use of MySQL temp tables, that are not very asynchronous replication friendly. These kinds of things break my setup and require a resync with a full backup. I believe the goal for fixing this is to have a bit of pre-processing on the binlogs to either concatenate them all into one (which would lessen, but not eliminate the problem), or to be aware of things that are replication unfriendly and not execute them until it has the missing parts.
• GPG is using real encryption – If you lose your keys, you lose your encrypted data.
• All of the uses of Amazon’s services incur costs, that you will have to pay-  This should go without saying, but remember, YOU are responsible for putting files into AWS, and these scripts and methods are merely a tool.
• Backups are not encrypted everywhere. If you have sensitive data you may way to use an encrypted volume on the EBS volumes.

Related posts:

1. I’m wondering if I should have my install script clone common Linux OS’s like Red Hat/CentOS and Debian/Ubuntu? So, I have a script that can run from init....
2. Script to rotate the MySQL General Query Log You want general query logging in, but don’t want to...
3. Thinking about hosting a WordPress site on S3 I recently moved my Joomla backed consulting website completely to...

Related posts brought to you by Yet Another Related Posts Plugin.

Disagree completely with your assessment of Cloud Computing Disastery Recovery. With Amazon EC2 we have < 1 hour fail over for complete region loss, we can durably snapshot our volumes with a few seconds of a pause. We can test our DR for < $100. If a Disaster may affect the primary and secondary site, it’s trivial to add a third, or a fourth, or a fifth. If you have enough data and want to be sure you don’t have to rely on the Internet, Amazon Direct Connect (1Gbps $.30/hr, 10Gpbs $2.25/hr) DR is one of the key benefits to Cloud Computing. Should’t WAN optimization help you effectively use Cloud Computing?
Original URL of article  http://www.aryaka.com/2012/02/22/is-the-cloud-undermining-enterprise-disaster-recovery

Related posts:

1. Reading : Peecho Minimizing Downtime on Amazon AWS   What the guys over at Peecho have done is...
2. Thinking about hosting a WordPress site on S3 I recently moved my Joomla backed consulting website completely to...
3. My Comments on ‘Cheap Hosting Amazon AWS’ Blog Post My Comments on the post “Cheap Hosting Case Study: You...

Related posts brought to you by Yet Another Related Posts Plugin.

I have not run a Windows desktop since the 90s. I migrated from Windows 95 to FreeBSD with Enlightenment (e) and never looked back. Over the years I gradually shifted to Linux with e, every now and then switching to GNOME, KDE, or Fluxbox. These were always short lived, and I always ended up on e again. I was sad when e missed a few major chances to update or add new things, and the next version seemed like an awesome set of libraries, but never materialized as a usable desktop (yet). I would try out the new e every now and then. And then I found Beryl, which made GNOME much more usable, and shiny pretty flashy all at once. During the chaos of the Beryl/Compiz departure/re-joining, I would alternate between old e and GNOME/Beryl/Compiz. Eventually Compiz stabilized and GNOME/Compiz was the desktop for me for years. It worked, I was happy, and nothing else seemed to be even plausibly better.

Then one day I became rudely awakened that my favorite Linux Distro (Ubuntu) had gone down some horrible path and decided to make a new, completely lame, desktop called Unity. Being used to the open source world, and the total amount of control I didn’t worry about it. I gave Unity a few hours research, a few weeks trial in use, decided it was a pile of steaming shit, and moved back to my GNOME/Compiz. After upgrading my home Ubuntu to the latest/greatest version I was given another rude awakening. NO MORE GNOME/Compiz. W-T-F. Sure, I could go back and build it myself, but I have grown lazy (not to mention busy), and if I wanted LFS I would have LFS. I did a little research, downgraded my Ubuntu and went happily back to GNOME/Compiz on 10.04. Then, a co-worker who had run into much the same problem told me that Linux Mint had “solved” the GNOME 3/Unity dilemma rather aptly. As Mint had been the version of Linux I recommend to anyone for desktop use, and only did not use since I wanted my Linux to  match the Linux I used in production as closely as possible, I was hopeful. They had a GNOME 3 hackup with Compiz to act much like the GNOME/Compiz I had known and loved. I quickly jumped ships to Mint, and have been happy there for the past few months.

The problem with Mint, however, is that they have diverged from some of the basic functionality I am used to in Linux. I’m sure I could spend the time and “fix” this, but the fact these minor issues were not a simple option in the control panel, and that they had deviated from “standard” Linux (read every other version I have used ever), was not a good sign. The best illustration of this is that when I highlight something in Linux with a quick slick and drag with my mouse, it is immediately and awesomely put onto my middle click clip board. I use this probably 300 times a day, and it is the main reason I ever touch my mouse.

Thus my Linux world has been shattered. Now, I’m not crazy enough to consider running Windows on an actual production server, and my deepest sympathy to those of you who have made this error, but I do need a functional, working, usable, desktop. And Linux has consistently been in decline in this area. It’s probably worth mentioning that other areas of Linux have been annoying over the last few years, kernel “taint”? Really? Grow up guys, if I want binary close sourced drivers, I don’t need a nag that my junk is now ‘tainted’, and I don’t really give a shit if you’re hungup on it. In addition, there have been a boatload of security problems userspace to kernel that have made Linux security less defensible, and this was one of the key areas for it’s widespread adoption on servers. Throw in the occasional kernel panic bug, and Linux doesn’t get let into the club house of bomb proof OS’s like FreeBSD and openBSD.

So, being of sound mind and reasonable technical aptitude, I decided that Mac OS X was the way to go. I already have MacBooks, Mac Mini’s, iMacs, iPhones, iPods, iPads, I ought to run OS X on my main workstation. Sadly I am not retarded enough to throw money away on a Mac Desktop, even though they are very nice, so I decided to go the Hackintosh route. This is a very good route, and I have bought more than all versions of OS X from Apple. I got a USB drive and hit the forums on http://tonymacx86.com/ and decided I was already in luck, having a Gigabyte motherboard, an Nvidia graphics card, plenty of RAM and a fast CPU. I installed my first Hackintosh and was up and running in less than an hour. It was awesome. I was excited. This was a good progress. But, no sound. I searched forums, and tried a few things, but, again, lazy and busy. Having no sound via my HDMI connection to my home TV/stereo was a deal breaker. I wasn’t ready to admit it, so I tried installing my hackintosh on my work computer. It worked much less well. Intel CPU, intel chipset, but not Gigabyte, and not working easily. I went back to Linux, and decided to hold off on change for now.

Mint wasn’t bad, and I had a lot to get done. I swapped computers to one that would more likely work as a hackintosh. But, it was already running Windows, and I was more busy than normal. I decided to give ole Windows 7 a try again. I’ve used it, not hated it, and generally ignored it. I grabbed my usual program suites and was off to the races in less than an hour. I had forgotten the vast array of available software that I did not get on a Linux or even Mac platform. Skype on Mac is pretty good, but the Linux version is a steaming pile, and both seem to lag a fair amount behind Skype on Windows. As Skype is one of my major methods of production communication, this was a big check in the plus column for Windows. After setting up cygwin  (I need grep, it’s what I do), Putty, Winscp, Chrome, Firefox, gvim, Dropbox, Evernote (way better on Windows), 7-zip, Truecrypt, GIMP, LibreOffice, python, Strawberry Perl (where did this come from? Awesomeland?), pidgin, pidgin-otr, and a few tweaks to my command setup, I was completely back in business. The new ways Windows shows off multiple windows is much better than how it used to be. I still haven’t setup any multiple desktops or window groups, but I’m hopeful that won’t be too hard. After this I found there are a lot of FOSS packages out there for Windows.

I spent some time getting cygwin opensshd to allow me to remotely connect via a secure bash terminal. In the end this was actually fairly simple, but did not seem to be well documented anywhere I could find. I vow to make a post with big pictures titled “Step by step cygwin openssh sshd servers howto”. Now, I can remote into my Windows boxes via ssh with public key authentication and run both bash and Windows commands from the cygwin shell.

One of the most exciting things about coming back to Windows is Microsoft PowerShell. I follow a few people on Twitter who are always posting cool PowerShell scripts, and I wished I had access to this new, exciting, tool. I hope to soon have ssh->bash/cmd/powershell to all of my windows boxes. I admit I really do run a few Windows servers, but only because I have to.

So, with a deep sigh of parting, I have to say “Goodbye Linux on my desktop, you were awesome, and I believe you are still so. I hope to come back someday.”. At the same time “Helllloooooo Windows, what’s up you sexy devil? When did you get all grown up? New abilities? A *real* shell? SQL like data access in your shell? Open source programs? Let’s get it on!”

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

When an Elastic Load Balancer handles a connection it sends it’s own (internal/private/10.x) address instead of the clients. It sends the clients along with the request as X-Forwarded-For. To log this you need to log X-Forwarded-For instead of the source IP.

Here are 2 links discussing the problem, the first covers a basic Apache & Tomcat setup, but the second one has a point about direct access getting not logged and has an Apache httpd specific solution.

http://blog.kenweiner.com/2009/09/amazon-elb-capturing-client-ip-address.html

http://blog.grahampoulter.com/2011/10/how-to-log-client-ip-from-apache-behind.html

Thanks @grahampoulter and @kweiner

Related posts:

1. Reading : Peecho Minimizing Downtime on Amazon AWS   What the guys over at Peecho have done is...

Related posts brought to you by Yet Another Related Posts Plugin.

You want general query logging in, but don’t want to keep those pesky query log files around?

I generally setup the MySQL user and run this from cron.

Here’s my script to do so, here’s a gzip’d version, and here is a zipped version

#!/bin/bash
#setup a user in mysql with the RELOAD priviledge
#grant RELOAD on *.* to reloader@'localhost' identified by 'PASSWORD';
#flush privileges;

user="reloader"
password="PASSWORD"
LOG="${HOME}/mysqld/mysqld.log"
LOGARCHIVE="${HOME}/logarchive"
NEW_LOG="${LOGARCHIVE}/mysqld-`date +%F-%s`.log"
KEEP_MIN=15
MAX_LOAD="5.0"
RUNFILE="${HOME}/rotate-general-log.pid"

if [ -f ${RUNFILE} ]
 then
  echo "Runfile ${RUNFILE} exists, exiting at `date`"
  ${HOME}/bin/page.sh "Runfile ${RUNFILE} exists, exiting at `date`"
  exit 1
 else
  echo "$$" > ${RUNFILE}
fi

#if [ $(echo "`cut -f1 -d ' ' /proc/loadavg` < ${MAX_LOAD}" | bc) -eq 1 ];
if [ $(echo "`cut -f1 -d ' ' /proc/loadavg` < ${MAX_LOAD}" | bc) -eq 1 ];
 then
  echo "System load less than ${MAX_LOAD}, proceeding [DEBUG]"
else
  echo "log rotate delayed, due to system load > ${MAX_LOAD}"
  ${HOME}/bin/page.sh "log rotate delayed, due to system load > ${MAX_LOAD}"
  /bin/rm ${RUNFILE}
  exit 1
fi

MYSQLD_PID=`pgrep mysqld`

if [ ! "${MYSQLD_PID}" ]
 then
  echo "Mysqld is NOT running, paging and exiting at `date`"
  ${HOME}/bin/page.sh "NO MySQLD on `hostname` at `date`"
  /bin/rm ${RUNFILE}
  exit 1
else
  echo "Mysqld running at `date`, PID ${MYSQLD_PID}, continuing"
fi

if [ -d "${LOGARCHIVE}" ]
 then
  echo "Moving general log at `date`"
  /bin/mv ${LOG} ${NEW_LOG}
  touch ${LOG}
  chmod og-rwx ${LOG}
  echo "Flushing general log at `date`"
  mysqladmin -u ${user} -p --password="${password}" flush-logs
  echo "Done flushing general log at `date`"
  echo "Gzipping ${NEW_LOG} log at `date`"
  gzip ${NEW_LOG}
  echo "Done gzipping ${NEW_LOG} log at `date`"
  echo "Size is `du -sh ${LOGARCHIVE}` pre-clean"
  echo "Cleaning ${LOGARCHIVE}, deleting files older than ${KEEP_MIN} minutes at `date`"
  find ${LOGARCHIVE} -iname "*.gz" -mmin +${KEEP_MIN} -print -exec /bin/rm {} \;
  echo "Done cleaning ${LOGARCHIVE}, at `date`"
  echo "Size is `du -sh ${LOGARCHIVE}` post-clean"
else
 echo "Missing logarchive dir ${LOGARCHIVE}"
  ${HOME}/bin/page.sh "Missing logarchive dir ${LOGARCHIVE}"
 /bin/rm ${RUNFILE}
 exit 1
fi

/bin/rm ${RUNFILE}
echo "Done at `date`"

Related posts:

1. I am always cautious with this on my clipboard.. su mysql cd rm -Rf data/* innodb/* tmp/* logs/* mysqld/*...

Related posts brought to you by Yet Another Related Posts Plugin.