When an Elastic Load Balancer handles a connection it sends it’s own (internal/private/10.x) address instead of the clients. It sends the clients along with the request as X-Forwarded-For. To log this you need to log X-Forwarded-For instead of the source IP.

Here are 2 links discussing the problem, the first covers a basic Apache & Tomcat setup, but the second one has a point about direct access getting not logged and has an Apache httpd specific solution.

http://blog.kenweiner.com/2009/09/amazon-elb-capturing-client-ip-address.html

http://blog.grahampoulter.com/2011/10/how-to-log-client-ip-from-apache-behind.html

Thanks @grahampoulter and @kweiner

Related posts:

1. Reading : Peecho Minimizing Downtime on Amazon AWS   What the guys over at Peecho have done is...

Related posts brought to you by Yet Another Related Posts Plugin.

I want to write up a full howto/tutorial, but have not had the time.

Here is the script that I referred to in my post at http://jonzobrist.com/2011/04/16/ec2-raid0-on-linux-with-ebs-volumes-notes/

This program is distributed in the hope that it will be useful but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License HERE for more details.

http://jonzobrist.com/files/setup-ec2-raid-0.zip

http://jonzobrist.com/files/setup-ec2-raid-0.sh.gz

This script needs an AWS command line setup, and may need some minor tweaking if you’re not running a Ubuntu server.

Ubuntu’s latest AMI’s are available for 10.04 here.

Please feel free to submit patches, comments, or questions.

Thanks to everyone whose helpful posts online and in the AWS forums helped me with this script. I could not have done it with out your generous sharing of work.

Enjoy!

Related posts:

1. S3-du.sh script to get bucket size on Amazon AWS S3 Here is my script s3-du.sh that I wrote to determine...
2. EC2 RAID0 on Linux with EBS volumes notes   I’m almost done with my publish / post of...
3. Getting started with Chef. I write a lot of scripts, configure a lot of...

Related posts brought to you by Yet Another Related Posts Plugin.

I write a lot of scripts, configure a lot of systems, and do a lot of snapshot related backup/restore/sync stuff on servers in house, and in the cloud.
Chef is a Ruby-based configuration management engine. There is a good intro here.

I set up a Chef Server. There’s a good overview here.

I followed the instructions to setup a Chef server from this Wiki on Opscode.

I also referred to the information from Grig Gheorghiu’s blog here.

Now that my server is up, I’m hoping to port my server configurations to Chef, with heavy reliance on EC2 and hopefully UEC.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

No idea why this isn’t the default on the latest Ubuntu, as it is for every other service in /etc/pam.d/

Edit /etc/pam.d/su

Find, and uncomment the line that says

# session    required   pam_limits.so

Changes take effect on use, so you’ll have to re-su to get this to take effect.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Often times you need to recover a disk manually, as the automatic disk check (fsck) is not willing to risk data deletion. I have yet to find an instance where answering no to it’s (fsck’s) many requests to fix things has helped in getting data back, or in getting a server back up and running.

With that in mind, I pretty much always run fsck -y on a filesystem when it’s unwilling to mount due to an unclean mount (hard reboot/power failure/etc). This has always been done with ext3 or resierfs journaling file systems, and I’ve yet to notice any data loss, although the risk always exists.

As with any tip online, you proceed at your own risk.

So the standard procedure to recover a disk that won’t fsck on it’s own is simple, during it’s boot process it detects the drive was unmounted uncleanly (this is the default, and every ext2 or 3 drive is marked unclean when it is mounted, and only marked unclean when it is unmounted, the theory being that if it’s not unmounted then there is potentially lost or corrupt data.), and it starts running fsck on it. It will usually give you a progress indicator, and if fsck finds any problems it drops you to a shell, where you can manually repair the disk, with fsck -y /dev/XXX.

However, it seems, that often if the drive is a Linux software RAID partition, then it simply reboots, and trying to boot into single user mode, or with the drive in read only mode, all fail, and the system either loops a failing fsck, or reboots, or both, ad nausea m.

To the rescue is any Linux live CD. Lately, I prefer a Ubuntu install in live mode, but previously to that I really liked Knoppix. Boot it up, get to a command line, and follow these steps.

Overview is :

1. Boot up and get a root command line, (sudo su / su / single user mode)
2. Scan the RAID devices and build mdadm.conf file
3. Assemble each RAID device
4. fsck -y each RAID device
5. Reboot into normal / non-live CD mode
6. Success!

Most newer Linux store mdadm.conf in /etc/mdadm/mdadm.conf, some will have it in /etc/mdadm.conf, and some simply won’t have it.

Use ls to find which one your live CD is using, and then make sure to redirect the output there. This will assume it’s /etc/mdadm/mdadm.conf

Once booted, at the command prompt, as root, type (assuming you’re using either RAID-0 or RAID-1, and you have 3 partitions)

modprobe md
modprobe raid1
modprobe raid0
mdadm --examine --scan >> /etc/mdadm/mdadm.conf
mdadm /dev/md2 --assemble
mdadm /dev/md1 --assemble
mdadm /dev/md0 --assemble
cat /proc/mdstat
#verify your RAID arrays are all disks complete (should show UU)
#now check each one
fsck -y /dev/md0
fsck -y /dev/md1
fsck -y /dev/md2
#now reboot and cross your fingers it all comes up good
reboot

Some notes:
If the disk isn’t heavily used, or you are using ext3 or resierfs, then you stand a decent chance of not losing any data with a fsck -y of the file system.

If the power was lost, or the machine locked up you may lose the last little bit of data, even on a journaled file system.

RAID-1 is awesome, and this can be done with a software RAID-1 even if one of the drives is failed. You can also just mount the disk without RAID, and operate as if it never was in an array.

You can do other maintenance on the drives, such as mount them (mount /dev/md2 /mnt/md2) and modify/copy/backup data/etc instead of, or in addition to, fsck’ing them.

I’ve had the best luck with ext3 and resierfs. I’ve read bad things about XFS, JFS, and other file systems.

I would love for Linux to support ZFS, as I’ve played with it on Open Solaris, and Sun’s VM of their storage appliance, and it seems nice.

Enjoy!

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.