Archive for category Linux
Capturing users IP addresses in Apache httpd and Tomcat logs behind an ELB
Posted by Jon Zobrist in Apache, AWS, ELB, Linux Support - Servers on January 6, 2012
When an Elastic Load Balancer handles a connection it sends it’s own (internal/private/10.x) address instead of the clients. It sends the clients along with the request as X-Forwarded-For. To log this you need to log X-Forwarded-For instead of the source IP.
Here are 2 links discussing the problem, the first covers a basic Apache & Tomcat setup, but the second one has a point about direct access getting not logged and has an Apache httpd specific solution.
http://blog.kenweiner.com/2009/09/amazon-elb-capturing-client-ip-address.html
http://blog.grahampoulter.com/2011/10/how-to-log-client-ip-from-apache-behind.html
Thanks @grahampoulter and @kweiner
Script to rotate the MySQL General Query Log
Posted by Jon Zobrist in Bash, Linux, MySQL Server Support, Programming on October 24, 2011
You want general query logging in, but don’t want to keep those pesky query log files around?
I generally setup the MySQL user and run this from cron.
Here’s my script to do so, here’s a gzip’d version, and here is a zipped version
#!/bin/bash
#setup a user in mysql with the RELOAD priviledge
#grant RELOAD on *.* to reloader@'localhost' identified by 'PASSWORD';
#flush privileges;
user="reloader"
password="PASSWORD"
LOG="${HOME}/mysqld/mysqld.log"
LOGARCHIVE="${HOME}/logarchive"
NEW_LOG="${LOGARCHIVE}/mysqld-`date +%F-%s`.log"
KEEP_MIN=15
MAX_LOAD="5.0"
RUNFILE="${HOME}/rotate-general-log.pid"
if [ -f ${RUNFILE} ]
then
echo "Runfile ${RUNFILE} exists, exiting at `date`"
${HOME}/bin/page.sh "Runfile ${RUNFILE} exists, exiting at `date`"
exit 1
else
echo "$$" > ${RUNFILE}
fi
#if [ $(echo "`cut -f1 -d ' ' /proc/loadavg` < ${MAX_LOAD}" | bc) -eq 1 ];
if [ $(echo "`cut -f1 -d ' ' /proc/loadavg` < ${MAX_LOAD}" | bc) -eq 1 ];
then
echo "System load less than ${MAX_LOAD}, proceeding [DEBUG]"
else
echo "log rotate delayed, due to system load > ${MAX_LOAD}"
${HOME}/bin/page.sh "log rotate delayed, due to system load > ${MAX_LOAD}"
/bin/rm ${RUNFILE}
exit 1
fi
MYSQLD_PID=`pgrep mysqld`
if [ ! "${MYSQLD_PID}" ]
then
echo "Mysqld is NOT running, paging and exiting at `date`"
${HOME}/bin/page.sh "NO MySQLD on `hostname` at `date`"
/bin/rm ${RUNFILE}
exit 1
else
echo "Mysqld running at `date`, PID ${MYSQLD_PID}, continuing"
fi
if [ -d "${LOGARCHIVE}" ]
then
echo "Moving general log at `date`"
/bin/mv ${LOG} ${NEW_LOG}
touch ${LOG}
chmod og-rwx ${LOG}
echo "Flushing general log at `date`"
mysqladmin -u ${user} -p --password="${password}" flush-logs
echo "Done flushing general log at `date`"
echo "Gzipping ${NEW_LOG} log at `date`"
gzip ${NEW_LOG}
echo "Done gzipping ${NEW_LOG} log at `date`"
echo "Size is `du -sh ${LOGARCHIVE}` pre-clean"
echo "Cleaning ${LOGARCHIVE}, deleting files older than ${KEEP_MIN} minutes at `date`"
find ${LOGARCHIVE} -iname "*.gz" -mmin +${KEEP_MIN} -print -exec /bin/rm {} \;
echo "Done cleaning ${LOGARCHIVE}, at `date`"
echo "Size is `du -sh ${LOGARCHIVE}` post-clean"
else
echo "Missing logarchive dir ${LOGARCHIVE}"
${HOME}/bin/page.sh "Missing logarchive dir ${LOGARCHIVE}"
/bin/rm ${RUNFILE}
exit 1
fi
/bin/rm ${RUNFILE}
echo "Done at `date`"
Snowed in? 15 Thinks you should be using online CCOD – 9.6.2011
Posted by Jon Zobrist in AWS, Communications, internet, Linux, Technology, Wordpress on September 6, 2011
There are a ton of cool things to do on the Internet. New doors are open to everyone. I’m surprised how often we take it for granted that everyone is in on the latest trend in tech. Here is my humble addition to a list of things that I think people should be using online.
1. Twitter – News *stream*, or should I say FLOOD. Follow smart people, get smart (filtered) news and info. Want to blow your news mind? Get tweetdeck and put in a search for any hot topic. (Don’t follow #earthquake unless you want to feel constant fear).
2. Facebook – Connect with your family and friends. Be benign on Facebook! The Internet is public, immortal, and Facebook does hate your privacy.
3. Amazon AWS/EC2 - What you don’t need a virtual server? You sure about that? Not for your blog? Not even if it scales infinitely? Not even if it’s free?
4. WordPress – Joomla and Drupal are cool, but WordPress is the king of the web page CMS.
5. Gmail – Seriously, stop deleting your email, get a gmail account. Use your own domains (Google Apps is still free for < 10 users).
6. Google Docs – If you haven’t had 10 people all editing the same spreadsheet at the same time you have not Cloud’d it up.
7. Cloud Music (Google Music, Amazon Music Locker, iCloud, Soundcloud, Spotify) – This is new, try them all out, find new music, sync your own.
8. Google – Search done right. Everyone has been playing catchup for a while now, and I’m sure that one day they will, but until then, google.com
9. Snopes – The Internet means rapid access to information sharing, but many people share false information. Sites like snopes.com
10. Shopping - Deal sites like slickdeals.net, fatwallet.com, woot and more track deals as they happen, often with good comments on how to maximize them. The people on some of these sites are mad geniuses when it comes to getting the most for your buck.
11. Skype – Everyone has it, get on and video chat your friends in other countries for free. Ride this one until Microsoft torpedos it, and we all move to Google Chat, which you should be on already via your gmail account.
12. Linux – If you are even slightly technically inclined, Linux opens the door to you (for free) to everything from high end movie effects to computer forensics. Get started with a Live CD from Ubuntu (Your computer is probably 64-bit, and you probably want the desktop version – You can boot the CD and use Linux without doing anything to your computer), and NO it does not run Office or any Windows program, but it does run thousands of cool programs.
13. Photo sites (Picasa, Flickr, Smugmug) There is no reason you should be burning a photo CD to send to your friends and family. Get an upload utility, and start putting your photos on the ‘net. You don’t have to share them, and I would highly recommend NOT sharing them publicly unless they are very public information. I do not post pictures with faces in them without permission from the person owning the face, and, in general, don’t do this.
14. Education (Khanadademy, Alison.com, MIT Open Courseware, Instructables, k12) – There are too many to name, and pretty much access to infinite information is it’s own education. Don’t think that just because a skill isn’t directly computer related that you can’t learn howto do it online, and for maybe for free.
15. Wikipedia - What is a wikipedia? Well, a wiki is a website that anyone can edit the pages of, so, Wikipedia is an encyclopedia that anyone can edit. Not always right, but rarely uninformative.
Well, I hope this helps. Please send me your lists or additions (comment below, or email to jon@jonzobrist.com).
I’m wondering if I should have my install script clone common Linux OS’s like Red Hat/CentOS and Debian/Ubuntu?
Posted by Jon Zobrist in AWS, Bash, EC2, Linux, Programming on August 22, 2011
So, I have a script that can run from init. You configure it from a config file, set it to start on boot via normal init, and then you can run it on to snapshot your data to Amazon S3 anytime.
It supports directories/users/permissions as well as full MySQL dumps, only specific database MySQL dumps, and only specific table per database dump.
It goes both ways, run it with /etc/init.d/bluesun-setup.sh start and it deletes the local directories, and downloads all the updated ones from Amazon S3.
If you run it with /etc/init.d/bluesun-setup.sh updateS3 # it will push all the same configured files to Tar/GZ files on S3.
I need to polish it a bit more, but it’ll be free/open source here soon.
Any feedback is appreciated.
I am always cautious with this on my clipboard..
Posted by Jon Zobrist in Linux, MySQL Server Support on June 15, 2011
su mysql
cd
rm -Rf data/* innodb/* tmp/* logs/* mysqld/*
mysql_install_db –defaults-file=${HOME}/etc/my.cnf
${HOME}/libexec/mysqld –defaults-file=${HOME}/etc/my.cnf > ${HOME}/logs/init.log 2>&1 &
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql
HowTo Quick Setup a Minecraft Server
Posted by Jon Zobrist in Games, Linux, Macintosh, Minecraft on June 5, 2011
My kids have dove into the game Minecraft and one of the first things they wanted was their own servers that they could customize and create lots of world.
We’re now running 5 servers on my Linux box. I don’t notice a performance drop in using it for anything, and each one is set to use 1GB of RAM, fortunately I have 8 GB…
This should be a quick howto that will work on Linux and Macintosh as long as you have screen and java.
I run the Minecraft servers under screen, so I can disconnect and reconnect to them, and they’ll keep running if my connection stops.
Quick setup :
(I use -R instead of -S to start and name the screen, since it will resume the same screen if it’s already running)
mkdir minecraft
cd minecraft
Edit a server.properties file, here’s a sample one :
Go to http://www.minecraft.net/download.jsp and copy the link that says “minecraft_server.jar” (it will update based on version, so copy it each time)
wget -O minecraft_server.jar http://www.minecraft.net/download/minecraft_server.jar?v=1307312614783
(that link will change with each version)
Now start your server
screen -R minecraft
java -Xmx1024M -Xms1024M -jar minecraft_server.jar nogui
Now type CTRL+a d
This will disconnect you from your screen.
To reconnect run
screen -R minecraft
Now connect to your computer from minecraft, you may need to connect by IP address and port.
Running multiple servers on the same host is as easy as changing the port and setting up another directory, and the screen instances.
This is just a simple setup assuming you have a Mac or a Linux box already running.
I hope to cover using a Dynamic DNS service like DynDNS.com , opening the ports in your router so your friends can connect from outside.
It may also help to find out your public IP address, you can go to http://jonzobrist.com/ip.php
I wonder if you could run a minecraft server with only 512MB RAM, would be perfect for the free tier of Amazon AWS..
Ubuntu codenames, my own cheat sheet, rant, and link
Posted by Jon Zobrist in Linux on June 1, 2011
I love Ubuntu.
I use it everywhere. I love the Debian based dpkg, the default installs, the Amazon AWS compatible EUC cloud.
I’m also a big fan of cool names, I like having printers and domains and whatever on some big, geeky, internal naming schema, like everything relates to Norse mythology or Star Trek.
But, I CANNOT STAND the code names for Ubuntu projects being used to refer to the version. I’m running Ubuntu 9.04, or 8.04 LTS, or 10.04 LTS, etc. I am not running Ubuntu Honey Badger, or whatever code name they developed under.
This is only annoying when I’m trying to find instructions about how to do something and people say things like ’Well, I tried this and it worked on Hoary’. WTF is a Hoary? Please, don’t tell me, I don’t care.
So, for my own sanity I am posting this rant, and linking to the page with a list of Development Codes for Ubuntu, as I’m sure, going forward, I’m going to need it.
https://wiki.ubuntu.com/DevelopmentCodeNames
And for my quick use, the versions I use most:
8.04 Hardy Heron
9.04 Jaunty Jackalope
10.04 Lucid lynx
11.04 Natty Narwhal
Quick remote SMTP page script in Python
Posted by Jon Zobrist in Linux, Programming, Python on May 29, 2011
Disclaimer : You own what you do with this script, and are responsible for it. This script could cause problems with SMTP / mail server administrators, and you should be sure to get any use approved. I make no claims about the scripts fitness for any specific use.
I have several servers that are not allowed to communicate with the outside world. Often times this helps a lot, but it can be a pain when trying to send e-mail alerts. Combine that sometimes I’m not root on these boxes so I can’t simply change the postfix/exim/qmail/sendmail default relay server to something.
The work around is this script, which uses a hard coded SMTP server, so you’ll need to edit the script to change the from, to, and the smtp servers IP.
I haven’t set it to take arguments, as I rarely re-use it for different things. Let me know if you’d benefit from that and I’ll throw them in.
Here is the script in gzipped format
Here is the code of the script :
#!/usr/bin/python
#Author : jon@jonzobrist.com
#License : BSD/public/freeware
import smtplib
import sys
def prompt(prompt):
return raw_input(prompt).strip()
fromaddr = “noreply@example.com”
#toaddrs = ['userA@example.com','userB@example.com','Phone1@txt.att.net','Phone2@txt.att.net','userC@example.com']
toaddrs = ['userA@example.com']
subject = “[ALERT] Alert from localhost”
msg = (“From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n”
% (fromaddr,toaddrs,subject))
msg = msg + sys.argv[1]
server = smtplib.SMTP(‘server.ip.or.hostname’)
#server.set_debuglevel(1)
server.sendmail(fromaddr, toaddrs, msg)
server.quit()
Enjoy!
My EC2 EBS RAID-0 creation script
Posted by Jon Zobrist in AWS, Bash, EBS, EC2, Linux, Linux Support - Servers, Programming on May 4, 2011
I want to write up a full howto/tutorial, but have not had the time.
Here is the script that I referred to in my post at http://jonzobrist.com/2011/04/16/ec2-raid0-on-linux-with-ebs-volumes-notes/
This program is distributed in the hope that it will be useful but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License HERE for more details.
http://jonzobrist.com/files/setup-ec2-raid-0.zip
http://jonzobrist.com/files/setup-ec2-raid-0.sh.gz
This script needs an AWS command line setup, and may need some minor tweaking if you’re not running a Ubuntu server.
Ubuntu’s latest AMI’s are available for 10.04 here.
Please feel free to submit patches, comments, or questions.
Thanks to everyone whose helpful posts online and in the AWS forums helped me with this script. I could not have done it with out your generous sharing of work.
Enjoy!
Getting started with Chef.
Posted by Jon Zobrist in AWS, Chef, EC2, Linux, Linux Support - Servers on April 21, 2011
I write a lot of scripts, configure a lot of systems, and do a lot of snapshot related backup/restore/sync stuff on servers in house, and in the cloud.
Chef is a Ruby-based configuration management engine. There is a good intro here.
I set up a Chef Server. There’s a good overview here.
I followed the instructions to setup a Chef server from this Wiki on Opscode.
I also referred to the information from Grig Gheorghiu’s blog here.
Now that my server is up, I’m hoping to port my server configurations to Chef, with heavy reliance on EC2 and hopefully UEC.
-
You are currently browsing the archives for the Linux category.
Follow me on Google+
