Archive for category Bash
Script to rotate the MySQL General Query Log
Posted by Jon Zobrist in Bash, Linux, MySQL Server Support, Programming on October 24, 2011
You want general query logging in, but don’t want to keep those pesky query log files around?
I generally setup the MySQL user and run this from cron.
Here’s my script to do so, here’s a gzip’d version, and here is a zipped version
#!/bin/bash
#setup a user in mysql with the RELOAD priviledge
#grant RELOAD on *.* to reloader@'localhost' identified by 'PASSWORD';
#flush privileges;
user="reloader"
password="PASSWORD"
LOG="${HOME}/mysqld/mysqld.log"
LOGARCHIVE="${HOME}/logarchive"
NEW_LOG="${LOGARCHIVE}/mysqld-`date +%F-%s`.log"
KEEP_MIN=15
MAX_LOAD="5.0"
RUNFILE="${HOME}/rotate-general-log.pid"
if [ -f ${RUNFILE} ]
then
echo "Runfile ${RUNFILE} exists, exiting at `date`"
${HOME}/bin/page.sh "Runfile ${RUNFILE} exists, exiting at `date`"
exit 1
else
echo "$$" > ${RUNFILE}
fi
#if [ $(echo "`cut -f1 -d ' ' /proc/loadavg` < ${MAX_LOAD}" | bc) -eq 1 ];
if [ $(echo "`cut -f1 -d ' ' /proc/loadavg` < ${MAX_LOAD}" | bc) -eq 1 ];
then
echo "System load less than ${MAX_LOAD}, proceeding [DEBUG]"
else
echo "log rotate delayed, due to system load > ${MAX_LOAD}"
${HOME}/bin/page.sh "log rotate delayed, due to system load > ${MAX_LOAD}"
/bin/rm ${RUNFILE}
exit 1
fi
MYSQLD_PID=`pgrep mysqld`
if [ ! "${MYSQLD_PID}" ]
then
echo "Mysqld is NOT running, paging and exiting at `date`"
${HOME}/bin/page.sh "NO MySQLD on `hostname` at `date`"
/bin/rm ${RUNFILE}
exit 1
else
echo "Mysqld running at `date`, PID ${MYSQLD_PID}, continuing"
fi
if [ -d "${LOGARCHIVE}" ]
then
echo "Moving general log at `date`"
/bin/mv ${LOG} ${NEW_LOG}
touch ${LOG}
chmod og-rwx ${LOG}
echo "Flushing general log at `date`"
mysqladmin -u ${user} -p --password="${password}" flush-logs
echo "Done flushing general log at `date`"
echo "Gzipping ${NEW_LOG} log at `date`"
gzip ${NEW_LOG}
echo "Done gzipping ${NEW_LOG} log at `date`"
echo "Size is `du -sh ${LOGARCHIVE}` pre-clean"
echo "Cleaning ${LOGARCHIVE}, deleting files older than ${KEEP_MIN} minutes at `date`"
find ${LOGARCHIVE} -iname "*.gz" -mmin +${KEEP_MIN} -print -exec /bin/rm {} \;
echo "Done cleaning ${LOGARCHIVE}, at `date`"
echo "Size is `du -sh ${LOGARCHIVE}` post-clean"
else
echo "Missing logarchive dir ${LOGARCHIVE}"
${HOME}/bin/page.sh "Missing logarchive dir ${LOGARCHIVE}"
/bin/rm ${RUNFILE}
exit 1
fi
/bin/rm ${RUNFILE}
echo "Done at `date`"
I’m wondering if I should have my install script clone common Linux OS’s like Red Hat/CentOS and Debian/Ubuntu?
Posted by Jon Zobrist in AWS, Bash, EC2, Linux, Programming on August 22, 2011
So, I have a script that can run from init. You configure it from a config file, set it to start on boot via normal init, and then you can run it on to snapshot your data to Amazon S3 anytime.
It supports directories/users/permissions as well as full MySQL dumps, only specific database MySQL dumps, and only specific table per database dump.
It goes both ways, run it with /etc/init.d/bluesun-setup.sh start and it deletes the local directories, and downloads all the updated ones from Amazon S3.
If you run it with /etc/init.d/bluesun-setup.sh updateS3 # it will push all the same configured files to Tar/GZ files on S3.
I need to polish it a bit more, but it’ll be free/open source here soon.
Any feedback is appreciated.
Thinking about hosting a WordPress site on S3
I recently moved my Joomla backed consulting website completely to Amazon S3, and have been very happy with the results. I would like to do something similar for my personal blog site at jonzobrist.com, however I would like it to be more dynamic, or at least easily update-able.
For my Joomla site, I did a complete mirror to static html and then uploaded all of that to S3, in a bucket with the same name as the site’s (www.bluesun.net), and changed DNS to point to the CNAME for that bucket’s HTTP address. This involved running wget -r -k -E -p -U Mozilla http://www.bluesun.net, editing the files wget copied to all point at the right places for things like menus, etc, and then uploading the files to Amazon S3.
My goal here is to recreate that in a more automated way, so that I can have a main site that is dynamic, but most, if not all, of the content is served from a static repository on S3. The expected outcome I think will be to take a site that costs around $15-20/month and make it cost < $1 /month. And, if I get some huge surge of traffic, to handle the load gracefully, and scale into the many terabytes of serving up data affordably.
A few quick thoughts/notes;
First, if you don’t change permission on newly uploaded items on S3 they default to your default, which is usually no public access. However, if you upload a new version of a file, it keeps the permissions the previous version had.
Second, you cannot host a naked domain (in this case http://bluesun.net) on Amazon S3. This is more a limitation of the the standards that say you shouldn’t. It means that you need something to redirect your naked domain to your web server. A lot of people don’t do this at all, but I think it’s a good thing to do. I think the details of this limitation will actually come in handy in my hybrid dynamic/static WordPress site.
Third, it makes a lot of sense to compress objects, and setting the right headers on the object will, I believe, get S3 to automatically server it up in a way a browser can understand. Most of the things that make up web pages (HTML and javascript) are text based and compress very well. On the other side images used on the web are generally already very compressed.
Fourth, having a hybrid site means you will still have some dynamic objects and this will mean manually processing (or manually setting up automated processing) html files to separate dynamic from static content.
Fifth, I’m a huge fan of things like Google Analytics, which are hosted by Google, and only included in my site as a static snippet of code that pulls more code direct from their servers. I would love to have something similar for comments and other user generated content that messes up the static website paradigm. I think technologies like AJAX can really shine here.
Brief background, my site (jonzobrist.com) is a standard WordPress install, currently running on an EC2 Micro Instance running Ubuntu 10.04 with Apache/PHP/MySQL all running on one machine. It’s an EBS backed instance, and I snapshot the root volume. I don’t really make updates more than once or twice a week, and none of my content needs to be pushed live in any kind of urgent manner. That said, I use WP to Twitter to auto tweet new posts, so I need to be able to force an update, or handle not having new content on S3 gracefully. I don’t get a particularly large number of visitors, lately about 1,000 a month. My main motivation for doing this is to see if it can be done, so I can do it for other sites I support.
Here is a graphical representation of what I think it will look like when done.
Then I just need to push all the very static content to Cloud Front for CDN!
What do you think?
Delete lots of EC2 EBS volumes
Posted by Jon Zobrist in AWS, Bash, EBS, Programming on May 23, 2011
Here is a quick script to delete lots of EBS volumes on Amazon EC2.
The output from something like
ec2-describe-instances i-6fc30f01 | grep ‘^BLOCKDEVICE’ | awk ‘{ print $3 }’ > delete-these-volumes.txt
Then run
./delete-volumes.sh delete-these-volumes.txt
Can be fed into this script (you may want to edit the file first, I accept no responsibility for you deleting your volumes, with or without my script.
Download the gzipped script here.
Example output from me deleting 60 real volumes:
My EC2 EBS RAID-0 creation script
Posted by Jon Zobrist in AWS, Bash, EBS, EC2, Linux, Linux Support - Servers, Programming on May 4, 2011
I want to write up a full howto/tutorial, but have not had the time.
Here is the script that I referred to in my post at http://jonzobrist.com/2011/04/16/ec2-raid0-on-linux-with-ebs-volumes-notes/
This program is distributed in the hope that it will be useful but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License HERE for more details.
http://jonzobrist.com/files/setup-ec2-raid-0.zip
http://jonzobrist.com/files/setup-ec2-raid-0.sh.gz
This script needs an AWS command line setup, and may need some minor tweaking if you’re not running a Ubuntu server.
Ubuntu’s latest AMI’s are available for 10.04 here.
Please feel free to submit patches, comments, or questions.
Thanks to everyone whose helpful posts online and in the AWS forums helped me with this script. I could not have done it with out your generous sharing of work.
Enjoy!
S3-du.sh script to get bucket size on Amazon AWS S3
Posted by Jon Zobrist in AWS, Bash, FreeBSD, Linux, Programming, S3 on March 19, 2011
Here is my script s3-du.sh that I wrote to determine how big a bucket is on Amazon S3. I’m sure there are other ways to skin this cat, but I wrote this and figured I’d share it. Feel free to use it under any free license.
Requires you to have a working s3ls and your Amazon AWS credentials setup already. I use Tim Kays AWS tools.
Links to the file as text, as gzip, as zip.
#!/bin/bash
if [ "${1}" ]
then
NUM=0
COUNT=0
for N in `s3ls ${1} | awk ‘{print $11}’ | grep [0-9]`
do
NUM=`expr $NUM + $N`
((COUNT++))
done
KB=`expr ${NUM} / 1024`
MB=`expr ${NUM} / 1048576`
GB=`expr ${NUM} / 1073741824`
echo “${COUNT} files in bucket ${1}”
echo “${NUM} B”
echo “${KB} KB”
echo “${MB} MB”
echo “${GB} GB”
else
echo “Usage : ${0} s3-bucket”
exit 1
fi
LCOD – 11.17.09 – Simple page/alert email script
Posted by Jon Zobrist in Bash, Linux, Linux Command of the Day, Programming on November 17, 2009
Here’s a quick page script for *nix boxes with mail.
I grabbed the mail / file redirect from
http://theos.in/shell-scripting/send-mail-bash-script/
#!/bin/bash
#Author : dougnaka@gmail.comThis e-mail address is being protected from spam bots, you need JavaScript enabled to view it
#Description : Quick alert/page script, edit EMAIL="" to include a comma separated list of emails to send to
#Usage : page.sh "Short or long message"
if [ "${1}" ]
then
SUBJECT="[ALERT] ${1}"
EMAIL=" someone@example.com"
EMAILMESSAGE="/tmp/`date +%s`-message"
echo "ALERT: at `date`" > ${EMAILMESSAGE}
echo "${1}" >> ${EMAILMESSAGE}
echo "From `hostname`" >> ${EMAILMESSAGE}
mail -s "${SUBJECT}" "${EMAIL}" < ${EMAILMESSAGE}
/bin/rm ${EMAILMESSAGE}
else
echo "Usage: ${0} MESSAGE"
exit 1
fi
LCOD 11.14.07 – Quick RAID check script for cron
Posted by Jon Zobrist in Bash, Linux, Linux Command of the Day, Programming on November 14, 2007
This is a quick script that will let you run the new raid checks, or repair, or pause a running raid check. It doesn’t do e-mails or anything, and from what I read on this thread it’s something mdadm needs to be set to watch for. I ran the check on a RAID and received no output, so I guess it was ok?
I will update the script as I find out more. I’d like to add a simple escalation from check to rebuild? (I think it does this anyways), and an e-mail status report, or even a syslog call saying I ran all things look good.
#!/bin/bash
###############
# Quick script to use raid functionality
# Author : Jon Zobrist < jon@bluesun.netThis e-mail address is being protected from spam bots, you need JavaScript enabled to view it >
# http://www.bluesun.net/
# License : LGPL
# Version : 1.0
# Date : 11.14.07
# install/notes:
# put it somewhere, /root/bin/ is where I prefer
# put this in your root crontab to run from cron monthly
# @monthly /root/bin/checkraids check > /root/logs/cron.log 2>&1
###############
case "$1" in
check)
for raid in /sys/block/md*/md/sync_action
do
echo check > $raid
done
;;
repair)
for raid in /sys/block/md*/md/sync_action
do
echo repair > $raid
done
;;
pause)
for raid in /sys/block/md*/md/sync_action
do
echo idle > $raid
done
;;
status)
for raid in /sys/block/md*/md/sync_action
do
echo "******************************"
echo $raid `cat $raid`
echo ""
done
;;
*)
echo "Usage : $0 check|repair|pause|status"
;;
esac
-
You are currently browsing the archives for the Bash category.
Follow me on Google+
