Archive for category Programming
Script to rotate the MySQL General Query Log
Posted by Jon Zobrist in Bash, Linux, MySQL Server Support, Programming on October 24, 2011
You want general query logging in, but don’t want to keep those pesky query log files around?
I generally setup the MySQL user and run this from cron.
Here’s my script to do so, here’s a gzip’d version, and here is a zipped version
#!/bin/bash
#setup a user in mysql with the RELOAD priviledge
#grant RELOAD on *.* to reloader@'localhost' identified by 'PASSWORD';
#flush privileges;
user="reloader"
password="PASSWORD"
LOG="${HOME}/mysqld/mysqld.log"
LOGARCHIVE="${HOME}/logarchive"
NEW_LOG="${LOGARCHIVE}/mysqld-`date +%F-%s`.log"
KEEP_MIN=15
MAX_LOAD="5.0"
RUNFILE="${HOME}/rotate-general-log.pid"
if [ -f ${RUNFILE} ]
then
echo "Runfile ${RUNFILE} exists, exiting at `date`"
${HOME}/bin/page.sh "Runfile ${RUNFILE} exists, exiting at `date`"
exit 1
else
echo "$$" > ${RUNFILE}
fi
#if [ $(echo "`cut -f1 -d ' ' /proc/loadavg` < ${MAX_LOAD}" | bc) -eq 1 ];
if [ $(echo "`cut -f1 -d ' ' /proc/loadavg` < ${MAX_LOAD}" | bc) -eq 1 ];
then
echo "System load less than ${MAX_LOAD}, proceeding [DEBUG]"
else
echo "log rotate delayed, due to system load > ${MAX_LOAD}"
${HOME}/bin/page.sh "log rotate delayed, due to system load > ${MAX_LOAD}"
/bin/rm ${RUNFILE}
exit 1
fi
MYSQLD_PID=`pgrep mysqld`
if [ ! "${MYSQLD_PID}" ]
then
echo "Mysqld is NOT running, paging and exiting at `date`"
${HOME}/bin/page.sh "NO MySQLD on `hostname` at `date`"
/bin/rm ${RUNFILE}
exit 1
else
echo "Mysqld running at `date`, PID ${MYSQLD_PID}, continuing"
fi
if [ -d "${LOGARCHIVE}" ]
then
echo "Moving general log at `date`"
/bin/mv ${LOG} ${NEW_LOG}
touch ${LOG}
chmod og-rwx ${LOG}
echo "Flushing general log at `date`"
mysqladmin -u ${user} -p --password="${password}" flush-logs
echo "Done flushing general log at `date`"
echo "Gzipping ${NEW_LOG} log at `date`"
gzip ${NEW_LOG}
echo "Done gzipping ${NEW_LOG} log at `date`"
echo "Size is `du -sh ${LOGARCHIVE}` pre-clean"
echo "Cleaning ${LOGARCHIVE}, deleting files older than ${KEEP_MIN} minutes at `date`"
find ${LOGARCHIVE} -iname "*.gz" -mmin +${KEEP_MIN} -print -exec /bin/rm {} \;
echo "Done cleaning ${LOGARCHIVE}, at `date`"
echo "Size is `du -sh ${LOGARCHIVE}` post-clean"
else
echo "Missing logarchive dir ${LOGARCHIVE}"
${HOME}/bin/page.sh "Missing logarchive dir ${LOGARCHIVE}"
/bin/rm ${RUNFILE}
exit 1
fi
/bin/rm ${RUNFILE}
echo "Done at `date`"
Thinking about hosting a WordPress site on S3
I recently moved my Joomla backed consulting website completely to Amazon S3, and have been very happy with the results. I would like to do something similar for my personal blog site at jonzobrist.com, however I would like it to be more dynamic, or at least easily update-able.
For my Joomla site, I did a complete mirror to static html and then uploaded all of that to S3, in a bucket with the same name as the site’s (www.bluesun.net), and changed DNS to point to the CNAME for that bucket’s HTTP address. This involved running wget -r -k -E -p -U Mozilla http://www.bluesun.net, editing the files wget copied to all point at the right places for things like menus, etc, and then uploading the files to Amazon S3.
My goal here is to recreate that in a more automated way, so that I can have a main site that is dynamic, but most, if not all, of the content is served from a static repository on S3. The expected outcome I think will be to take a site that costs around $15-20/month and make it cost < $1 /month. And, if I get some huge surge of traffic, to handle the load gracefully, and scale into the many terabytes of serving up data affordably.
A few quick thoughts/notes;
First, if you don’t change permission on newly uploaded items on S3 they default to your default, which is usually no public access. However, if you upload a new version of a file, it keeps the permissions the previous version had.
Second, you cannot host a naked domain (in this case http://bluesun.net) on Amazon S3. This is more a limitation of the the standards that say you shouldn’t. It means that you need something to redirect your naked domain to your web server. A lot of people don’t do this at all, but I think it’s a good thing to do. I think the details of this limitation will actually come in handy in my hybrid dynamic/static WordPress site.
Third, it makes a lot of sense to compress objects, and setting the right headers on the object will, I believe, get S3 to automatically server it up in a way a browser can understand. Most of the things that make up web pages (HTML and javascript) are text based and compress very well. On the other side images used on the web are generally already very compressed.
Fourth, having a hybrid site means you will still have some dynamic objects and this will mean manually processing (or manually setting up automated processing) html files to separate dynamic from static content.
Fifth, I’m a huge fan of things like Google Analytics, which are hosted by Google, and only included in my site as a static snippet of code that pulls more code direct from their servers. I would love to have something similar for comments and other user generated content that messes up the static website paradigm. I think technologies like AJAX can really shine here.
Brief background, my site (jonzobrist.com) is a standard WordPress install, currently running on an EC2 Micro Instance running Ubuntu 10.04 with Apache/PHP/MySQL all running on one machine. It’s an EBS backed instance, and I snapshot the root volume. I don’t really make updates more than once or twice a week, and none of my content needs to be pushed live in any kind of urgent manner. That said, I use WP to Twitter to auto tweet new posts, so I need to be able to force an update, or handle not having new content on S3 gracefully. I don’t get a particularly large number of visitors, lately about 1,000 a month. My main motivation for doing this is to see if it can be done, so I can do it for other sites I support.
Here is a graphical representation of what I think it will look like when done.
Then I just need to push all the very static content to Cloud Front for CDN!
What do you think?
Moved my consulting website to Amazon S3
Posted by Jon Zobrist in AWS, HTML, internet, S3 on June 26, 2011
It’s a Joomla site, but I rarely have updated it, so I just made a static mirror of it with wget, then uploaded it to S3!
Amazing how easy it is. I want to make either a WordPress plugin, or a set of scripts so I can keep my WordPress site dynamic locally (like a stage and master copy), and then when I want to push updates have it update a static directory, and put files in S3. Ideally this would also push to Cloud Front.
Now my website is up all the time!
In addition, my web site is cheap to run, requires no server (other than a core http://bluesun.net/ redirect, which many DNS hosts will do for free)
Plus how secure is that? Static HTML files on S3? You can download them, but that’s about it, unless you’re trying to hack Amazon, and good luck with that.
I can’t wait to see what it costs for the very few visitors I get to get things straight from S3, and in the future, CloudFront.
Quick remote SMTP page script in Python
Posted by Jon Zobrist in Linux, Programming, Python on May 29, 2011
Disclaimer : You own what you do with this script, and are responsible for it. This script could cause problems with SMTP / mail server administrators, and you should be sure to get any use approved. I make no claims about the scripts fitness for any specific use.
I have several servers that are not allowed to communicate with the outside world. Often times this helps a lot, but it can be a pain when trying to send e-mail alerts. Combine that sometimes I’m not root on these boxes so I can’t simply change the postfix/exim/qmail/sendmail default relay server to something.
The work around is this script, which uses a hard coded SMTP server, so you’ll need to edit the script to change the from, to, and the smtp servers IP.
I haven’t set it to take arguments, as I rarely re-use it for different things. Let me know if you’d benefit from that and I’ll throw them in.
Here is the script in gzipped format
Here is the code of the script :
#!/usr/bin/python
#Author : jon@jonzobrist.com
#License : BSD/public/freeware
import smtplib
import sys
def prompt(prompt):
return raw_input(prompt).strip()
fromaddr = “noreply@example.com”
#toaddrs = ['userA@example.com','userB@example.com','Phone1@txt.att.net','Phone2@txt.att.net','userC@example.com']
toaddrs = ['userA@example.com']
subject = “[ALERT] Alert from localhost”
msg = (“From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n”
% (fromaddr,toaddrs,subject))
msg = msg + sys.argv[1]
server = smtplib.SMTP(‘server.ip.or.hostname’)
#server.set_debuglevel(1)
server.sendmail(fromaddr, toaddrs, msg)
server.quit()
Enjoy!
LCOD 5.29.2011 – 2 quick TCP port check scripts in Python
Posted by Jon Zobrist in Programming, Python on May 29, 2011
Disclaimer : These scripts could be considered malicious and should only be used to test servers with permission from the server administrator. I accept no responsibility for your use of these scripts, and make no warranty about their usefulness either express or implied.
I often find myself troubleshooting someone’s network connectivity, or one of my servers ability to receive connections or mass connections.
For this, nmap is invaluable, but sometimes it’s easier to just send someone a script to run, especially in larger companies where things like nmap, and other port scanners are frowned on.
So, here are 2 scripts that take the same arguments HOST PORT.
They are tcpcheck.py (here is the gzipped version), and tcpcheck-bulk.py (here is the gzipped version).
tcpcheck.py makes 1 connection to a TCP port and reports success or failure.
Example usage and output :
./tcpcheck.py jonzobrist.com 80
Success. Connected to jonzobrist.com on port: 80
./tcpcheck.py jonzobrist.com 81
Failure. Cannot connect to jonzobrist.com on port: 81
tcpcheck-bulk.py is hard wired to attempt 1500 connections to a TCP port and reports success/failures.
Example usage and output is :
./tcpcheck-bulk.py jonzobrist.com 80
Done with jonzobrist.com on port: 80
Done. Failures : 0 Successes : 1500
This script was very useful when we tried to determine server performance over a link that seemed to be killing our connections.
Please feel free to use it, and share if it helped.
Delete lots of EC2 EBS volumes
Posted by Jon Zobrist in AWS, Bash, EBS, Programming on May 23, 2011
Here is a quick script to delete lots of EBS volumes on Amazon EC2.
The output from something like
ec2-describe-instances i-6fc30f01 | grep ‘^BLOCKDEVICE’ | awk ‘{ print $3 }’ > delete-these-volumes.txt
Then run
./delete-volumes.sh delete-these-volumes.txt
Can be fed into this script (you may want to edit the file first, I accept no responsibility for you deleting your volumes, with or without my script.
Download the gzipped script here.
Example output from me deleting 60 real volumes:
My First Android App, Hello Android!
Posted by Jon Zobrist in Android Hardware, Android Software on May 20, 2011
My EC2 EBS RAID-0 creation script
Posted by Jon Zobrist in AWS, Bash, EBS, EC2, Linux, Linux Support - Servers, Programming on May 4, 2011
I want to write up a full howto/tutorial, but have not had the time.
Here is the script that I referred to in my post at http://jonzobrist.com/2011/04/16/ec2-raid0-on-linux-with-ebs-volumes-notes/
This program is distributed in the hope that it will be useful but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License HERE for more details.
http://jonzobrist.com/files/setup-ec2-raid-0.zip
http://jonzobrist.com/files/setup-ec2-raid-0.sh.gz
This script needs an AWS command line setup, and may need some minor tweaking if you’re not running a Ubuntu server.
Ubuntu’s latest AMI’s are available for 10.04 here.
Please feel free to submit patches, comments, or questions.
Thanks to everyone whose helpful posts online and in the AWS forums helped me with this script. I could not have done it with out your generous sharing of work.
Enjoy!
15 Incredible WordPress Plugins
Posted by Jon Zobrist in PHP, Wordpress on April 16, 2011
I love WordPress, and I am constantly impressed by the quality and functionality in both WordPress and the many plugins that are out there. I just found this awesome list of WordPress Plugins at sitesketch101.com.
http://www.sitesketch101.com/15-incredible-wordpress-plugins-you-need
Thanks Nicholas Cardot @ Site Sketch 101!

