Posts Tagged backup
Guest Blogger Brad Zobrist – How he would implement a Bitcasa
Posted by bradzobrist in Backups, internet, Startups, Technology, Uncategorized on September 14, 2011
First I think it’s good to clarify what I understand Bitcasa is trying to do and if I was their architect how I would do it.
This of course is all guessing and speculation.
They are trying to store all clients data for ever and cache all recently accessed data locally & predictively cache other data. All while expiring from the cache old infrequently accessed data.
Features they have or will be implementing are:
- All data is backed up to a cloud. All data is deduplicated with all other client data
- All data is NOT accessible by any other client or them
- Files and / or folders can be shared seamlessly with other users.
- Local hard drive space is used as a cache by predictive algorithms
Here is how I would implement each one of these to create an overall architecture similar to what I think Bitcasa is providing. At a high level I would follow the process of deduplicating all data at the block level across ALL clients, have each client encrypt (with client’s key) the reference to what blocks that make up their files and folders, store those encrypted references in a users file database or key->value store, compress and encrypt (with Bitcasa’s master key) new unknown blocks, then write those blocks back to the Bitcasa cloud storage. Bitcasa would retain a master hash table of all known blocks and each clint would send a list of it’s blocks and then Bitcasa master block hash table would respond and tell the client to only send new, unknown, blocks back to the storage compressed and encrypted.
So breaking it down here is how the address each feature:
- All data is backed up to a cloud & All data is deduplicated with all other client data.
All un-encrypted blocks (not files) are hashed and those hashes are sent back to the master hash table at Bitcasa and then you get a list of what “new” blocks need to be backed up and duplicate blocks can be thrown away.
- All data is NOT accessible by any other client.
The next level is to take the blocks / hashes that make up a file and create an encrypted user client key hash of what blocks make up that hash. You may even need to take the block level down to a sector level to get away files that could fit in a single block or segment size.
- Files and / or folders can be shared seamlessly with other users.
Because file / folder references are only stored with an encrypted reference to what blocks make up those files you simple need to give the new client the list of those blocks.
- Local hard drive space is used as a cache by predictive algorithms
An additional piece of code is monitoring what files / hashes / blocks are being accessed and knows if they’re cached locally or need to be pulled remotely. I believe the predictive part is where most of their patents are but unfortunately we won’t be able to find out about them for ~18 months, read Bitcasa gets an early start on IP acquisition.
Well, that’s what I think?
Thoughts?
MySQL backup sript – more info
Posted by Jon Zobrist in Linux, MySQL Server Support on April 2, 2007
Please email me updates you make and I will post them here. My email is kgb@bluesun.net
More “in depth” installation:
passwd mysqldump
mysql -p
GRANT select,lock tables on *.* to mysqldump@localhost identified by ‘p@ssword’;
flush privileges;
su – mysqldump
mkdir ~/bin
mkdir ~/logs
mkdir ~/bkup
cd ~/bin
wget http://www.submarinefund.com/backup_mysql/backup_mysql
chmod u+x backup_mysql
vi backup_mysql
make changes to config
– change passwd to the same one you used on the
GRANT instead of p@ssword
– change num_days_keep to the number of days you want to keep around
– change all the commands to match your system, these
should be the same on most linux systems, BSD’s may
need to change some paths
test the script
add a line to your crontab like this
0 3 * * * /home/mysqldump/bin/backup_mysql >> /home/mysqldump/logs/cron.log 2>&1
MySQL backup script
Posted by Jon Zobrist in Linux, MySQL Server Support on April 1, 2007
I wrote this script because I needed to have each table in all databases backed up separately.
If you have hundreds or thousands of tables and and want to back them up separately then this script could help. It also names and rolls files by unix timestamp, and allows you to specify
how long a file is kept, regardless of number of backups made. So, say you want 2 days of backups, and you make one every hour (48 total) it will auto roll them after 2 days, suppose then you manually run the script to create a backup before some important change or event, this will still keep that backup for 2 days.
Here’s my script to backup mysql
To install it on a linux/freebsd box just edit the top parts for configuration. If you make a user mysqldump with home directory of /home/mysqldump and a mysql password of p@assword, make a /home/mysqldump/logs and /home/mysqldump/bkup directories, this should just work. Hack it up, it’s GPL’d.
Please email me updates you make and I will post them here. If you have questions you can email me, or post them on my LCoD message board. My email iskgb@bluesun.net, but I don’t really check it that much 
More “in depth” installation:
adduser mysqldump
passwd mysqldump
mysql -p
GRANT select,lock tables on *.* to mysqldump@localhost identified by ‘p@ssword’;
flush privileges;
su – mysqldump
mkdir ~/bin
mkdir ~/logs
mkdir ~/bkup
cd ~/bin
wget http://www.submarinefund.com/backup_mysql/backup_mysql
chmod u+x backup_mysql
vi backup_mysql
make changes to config
- change passwd to the same one you used on the
GRANT instead of p@ssword
- change num_days_keep to the number of days you want to keep around
- change all the commands to match your system, these
should be the same on most linux systems, BSD’s may
need to change some paths
test the script
add a line to your crontab like this
0 3 * * * /home/mysqldump/bin/backup_mysql >> /home/mysqldump/logs/cron.log 2>&1
LCOD – 8.10.05 – How to net backup/restore using tar|ssh|dd
Posted by Jon Zobrist in Linux, Linux Command of the Day on August 10, 2005
Ok, so you read the dd LCOD and tried it out, but you have a 250GB drive and only about 10GB data.. so you’d prefer if you only read the files that were actually there, and not your 100GB of deleted data.
Well, here’s what I do. On the computer to be backed up I boot a linux live cd (such as knoppix from knopper.net), and mount the disks, on say, /mnt/disk and since I use gentoo I have /boot seperate and mount it on /mnt/disk/boot, so I only have to take 1 image/backup.
Then I make sure networking is up.. it all goes something like this
| Code: |
| mkdir /mnt/disk mount /dev/hda3 /mnt/disk mount /dev/hda1 /mnt/disk/boot cd /mnt/disk tar -zcvf – * | ssh user@192.168.0.X dd of=/home/user/backup.tar.gz |
Then we get the prompt to accept the key, put in our password and wait. The v in the -zcvf will give us verbose output, if you just want to wait you can omit it and not see each file as it goes. This will create a tar archive of all the files in that directory, gzip it and put it to standard out, which redirects to ssh to 192.168.0.X as user “user”, and then pipes it to the dd which puts it into the file /home/user/bakcup.tar.gz
Now, on to the restore part… So you have this image, and you want to put it onto another computer.. Well this is kind of like installing gentoo, so boot to your trusty linux cd, open a console and follow these steps..
create your partitions using fdisk (here’s an fdisk howto http://www.tldp.org/HOWTO/Partition/partition-5.html)
Now I’m going to assume you’ve made /dev/hda1 as your /boot /dev/hda2 as your swap and /dev/hda3 as your / (root) partition.
To format your disks use the mkfs commands like this
| Code: |
|
mke2fs /dev/hda1 mkreiserfs /dev/hda3 mkswap /dev/hda2 |
now that you have your disks setup, lets mount them, its just like above is we use /mnt/disk as the mount point, and run our restore, which is just like the above command but reversed… note you could probably use cat instead of dd… and a > instead of dd on the creating side above..
| Code: |
| mkdir /mnt/disk mount /dev/hda3 /mnt/disk mkdir /mnt/disk/boot mount /dev/hda1 /mnt/disk/boot swapon /dev/hda2 cd /mnt/disk ssh user@192.168.0.X dd if=/home/user/backup.tar.gz | tar -zxvpf - |
note: the – at the end of the the tar command means standard in/out
So now you’ve got your data restored exactly as the system was at last shutdown, on a new file system. If you changed your disk layout, you’ll want to update /etc/fstab and the final thing we need to do is reinstall grub so the system boots to it. To do that we need to chroot, just like a gentoo install
assuming your disks are still mounted on /mnt/disk
| Code: |
|
chroot /mnt/disk /bin/bash grub root (hd0,0) setup (hd0) |
Assuming your disk is 0,0 (first IDE drive, slice is /boot)
You can use the tab key in grub to see what disks are available, so type root (hd
and hit tab
and it’ll say 0,1,2 or something telling you which disks are available
That’s it, now exit the shell and reboot like this
| Code: |
|
exit cd / umount /mnt/disk/boot umount /mnt/disk swapoff /dev/hda2 reboot |
don’t forget to remove the CDROM when you reboot (knoppix will eject it and ask you to hit enter once its out)
Now you should get your old OS booted back up, FYI if you’re imaging lots of systems you may want to edit your config files for your network if you’re not using DHCP (/etc/conf.d/net on gentoo) so the machines don’t all try to come up with the same IP address.
When imaging systems with server services, specifically SSH, is that you should change your server key, in fact, it’d be best if you deleted your keys BEFORE you created your image, so that when the server boots up it’ll generate its own keys upon starting SSH. Having your SSH private keys in any type of unsecure location is a BAD idea…
to delete your keys simply
| Code: |
| /bin/rm /etc/ssh/*key* |
Follow me on Google+
