Shell Script to Backup Files Locally Using Rsync

Written by BinnyVA on May 9, 2008 – 11:39 pm -

Terminal

All programmers have their own customized backup solutions. I have six. Yes, six! Five to backup files and one to backup database tables. And I am not counting version control or other backup systems built into the tools I use. Anyway, in the first post of the shell scripts series, let me introduce you to my Rsync based local backup solution.

Configuration File

This script reads a configuration file named ‘rsyncbackup.config’ and backups all the folders specified in that file to another location in the same system. This configuration file must be in the same folder as the perl script. The configuration file format is given below…

#Notes - do NOT include the last '/' at the end of the source folders
~/Scripts
~/Documents

#################### Web files ####################
# My Sites
#Folder to backup		#Folder to which it should be backuped to
/var/www/html/Sites		Htdocs/
/var/www/html/Projects	Htdocs/

If there is just one column in a line, that folder will be backuped to “<backup folder>/<folder name>“. Let say that my backup destination folder is ‘/var/Backup/Rsync‘. So the first line, ‘~/Scripts‘ will copy the contents of ‘~/Scripts‘ to ‘/var/Backup/Rsync/Scripts

If a line in the configuration file has two columns, then an extra folder will be created with the name provided in the second column. For example, the line ‘/var/www/html/Sites     Htdocs/‘ will create a backup of ‘/var/www/html/Sites‘ in ‘/var/Backup/Rsync/Htdocs/Sites

And if you have not guessed it already, all lines that begin in a # are comments and will be ignored.

The Script

There is the perl script that automates the rsync calls…

#!/usr/bin/perl

#EDIT THIS LINE
$backup_folder = '/var/Backup/Rsync'; #Final '/' must NOT be there.

use File::Basename;
my $config_file = dirname($0) . "/rsyncbackup.config";
my @all_locations = removeComments(getFileContents($config_file));

chdir($backup_folder) or die("Cannot go to folder '$backup_folder'");

foreach my $folder_locations (@all_locations) {
	my($folder,$backup_location) = split(/\s+/,$folder_locations);

	print "Backing up $folder to $backup_location ... ";
	`rsync -a $folder $backup_folder/$backup_location`;
	print "Done\n";
}

sub getFileContents {
	my $file = shift;
	my @lines;
	if(!open (FILE,$file)) {
		die("Can't open '$file': $!");
	} else {
		@lines=<FILE>;
		close(FILE);
	}
	return @lines;
}

sub removeComments {
	my @lines = @_;

	@cleaned = grep(!/^\s*#/, @lines); #Remove Comments
	@cleaned = grep(!/^\s*$/, @cleaned); #Remove Empty lines

	return @cleaned;
}

Execute this script using the command ‘perl RsyncBackup.pl’. In my system I have created an alias ‘bk’ for this script. I recommend that you make a similar alias if you take backups regularly(extremely recommended).

Backing up is done using rsync – so its faster than a simple ‘cp’ as only the modified and new files are copied.


Tags: , , , , ,
Posted in Programming, Scripting, Shell Scripts | 4 Comments »

Shell Script Language – Use Perl, Not Bash

Written by BinnyVA on May 2, 2008 – 11:40 pm -

To me, a shell script is a script that automates repetitive tasks. But that is not the ‘official’ definition. Wikipedia has this definition

A shell script is a script written for the shell, or command line interpreter, of an operating system.

I use a definition that defines the purpose of the script – while the others prefer a definition that defines the technology used. I am not going to claim that my definition is better than the other definition – that’s pointless. Besides, even I think that the ‘other’ definition is the right one. But I will try to show you the advantages of my approach.

The Purpose

I like to automate things(in other words, I’m lazy). So I have a nice little collection of custom shell scripts. But there is a huge barrier to writing shell scripts – the language used. Traditionally shell scripts are written in a language provided by the shell – like bash or tcsh. That is the problem – these languages are Bad – with a capital ‘B’.

The Problem

To people who are accustomed to decent languages, these shell languages will seem clunky – or even evil. But since most shell scripts are small, most people don’t mind the torture.

In bash, the control flow commands seem to be thrown in as a after thought rather than something that’s built into the language. If you don’t believe me, compare the ‘if’ loop of bash and Perl.

Bash

This code checks wether the variable ‘$value’ has the value 0 – if so it prints the message ‘No Value’

if [ $value -eq 0 ] ; then
	echo "No Value"
fi

Perl

The same code in Perl…

if($value == 0) {
	print "No Value";
}

Of course, Perl experts will go for if(!$value) or even unless($value) – but that’s not the point. See how better the code looks in Perl. Yeah, even I am surprised to hear those words – Perl is considered by many to be an ‘ugly’ language. But when compared to shell languages, Perl is a gem(sorry about the pun – couldn’t resist).

The Solution

The solution is simple – don’t use a shell language to write your shell scripts – use a high level scripting language like Perl. Or Python, Ruby, PHP, Tcl or even JavaScript.

I still use bash to write shell scripts – but if the shell script has an if condition(other than a simple argument check), I use a higher language – usually Perl.

Advantages of using a Shell Language

  • Its supported in even the tinest linux distros – even in embedded systems.
  • Its the method preferred by the majority – I hope this will change soon.
  • Command calls look more natural in a shell language.

Advantages of using a High Level Language

  • Better Code
  • Easy to maintain
  • Faster development
  • Libraries provide a lot of functionality
  • Easier to port to different platforms.

So why am I telling you all this? Two reasons…

One, the next time you are going to write a shell script, I want you to choose a high level language rather than using bash.

The second reason is that now that my series on Linux MP3 Players are over, I am going to take a small break from desktop posts and write on more ‘linuxy’ topics. And one of those topic is Shell Scripting. So in the future posts, I am going to share some of my shell scripts with you. So when I publish a Perl script and call it a shell script, I don’t want you to get confused .

Update: This article is translated to Serbo-Croatian language.


Tags: , , , , ,
Posted in Command Line, Programming, Scripting | 42 Comments »

mpg123/mpg321 – The Command Line MP3 Players

Written by BinnyVA on February 25, 2008 – 1:25 am -

Terminal

So far we looked at the GUI MP3 Players for linux – like Amarok, Exile, XMMS etc. Most linux users need only that – but there are some people who want a simpler system – command line players. The two top players in this area are mpg123 and mpg321.

mpg123

mpg123 is a fast, free, minimalist, console MPEG audio player software program for UNIX and Linux operating systems. The development on this project ceased for a long time – but now it has a new maintainer.

mpg321

mpg321 is a clone of mpg123.

Installation

Most distros have mpg321 in their repositories(not all have mpg123) – so just log in as root and use these commands to install it…
yum install mpg321 – For RedHat, Fedora, CentOS etc.
OR
apt-get install mpg321 – For Debian, Ubuntu, etc.

Using mpg321

Using mpg321 is straight forward. Since it has no GUI, just go to the folder with the song and just execute the command…

mpg321 MP3_FILE_NAME

If you want to play all the songs in a folder, you have to create a list of files in that folder. Then you have to specify this list as an argument for the mpg321 command. These commands can get the job done…

Create a list of all MP3 Files in the current folder…

find . -name "*.mp3"|sed 's/\.\///;'|sort>List.lst

Use this list as the play list for mpg321

mpg123 --list List.lst

Why Use Command Line Players?

If we have a number of shiny GUI players, why should one use a command line player? Unless you are an anti-GUI command line guru, chances are you don’t need it. But still, it has its uses.

For example, if you have to restart the X Server often for some reason, then mpg321 is the best player for you. All other player will stop playing when X server is down.

This actually happend to me – when I bought my second LCD screen, it took me some time to configure it. And that meant editing the xorg.conf file and restarting the X Server to see if it worked. At that time, I opened a virtual terminal(Ctrl+Alt+F1) and opened mpg321 with a bunch of MP3s in the list. It kept playing even if the X server is down. This is propably the reason that I was still sane after around six hours of XOrg configuration. :-)


Tags: , , , , ,
Posted in Audio, Command Line, Tools | 11 Comments »

How to Mount ISO/MDF Images in Linux

Written by BinnyVA on May 21, 2007 – 12:51 pm -

Harddisk

The mount command in Linux is a very powerful command. It can be used to mount any thing from a Samba mount to a ISO image file. The ability of the mount command to read ISO files makes it very useful for me.

My brother has a habit of saving DVD films as a image file. I believe he uses Alcohol 120% to do this. If he wants to view the Film, he ‘mount’ this image using Daemon Tools. He uses windows to do this. After he is done, he hands me this image. I can mount this image without any special tools in Linux – I just need the mount command.

mount -t iso9660 -o loop <Image_File> <Mount_Point>

Mounting Example

mount -t iso9660 -o loop /home/binnyva/Films/300.iso /mnt/Image

The ‘-t’ option specifies the filetype – this is optional.

This command works with both ISO and MDF images.


Posted in Command Line, Tools | 25 Comments »

Internet Connection Sharing using iptables

Written by BinnyVA on April 16, 2007 – 4:26 pm -

Networking

It is very easy to setup an internet connection sharing in Linux system using iptables. This method can be used to share an internet connection from a Linux system(I used Fedora Core 6, but it should work on other distributions that support iptables). Another method of doing this is using a proxy server like squid.

Enable IP forwarding

Run as root

sysctl -w net.ipv4.ip_forward=1

To enable it in system startup, edit the file /etc/sysctl.conf and set

net.ipv4.ip_forward = 1

iptables

Run command as root

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
service iptables save

Configuration

The configuration should be like this

Router

Connected to the internet provider
IP : 192.168.1.1

Internet Connected System

eth0 (LAN Card 1)

Connected to router

  • IP : 192.168.1.10
  • Netmask : 255.255.255.0
  • Gateway : 192.168.1.1 (IP of the router)

eth1 (LAN Card 2)

Connected to the other system

  • IP : 192.168.0.20 (Not the same network as the first card)
  • Netmask : 255.255.255.0
  • Gateway : 192.168.1.1 (IP of the router)

Second System

LAN Card connected to the first system

  • IP : 192.168.0.30
  • Netmask : 255.255.255.0
  • Gateway : 192.168.0.20 (IP of the second Card in the first system)

Disclaimer

Linux networks is not a subject I am an expert on. So take my advice with a pinch of salt. The above procedure worked for me. YMMV.

Status

Some results of various commands are shown here. Check to see if it matches the result on your system.

# iptables -t nat -L POSTROUTING
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  anywhere             anywhere
# cat /proc/sys/net/ipv4/ip_forward
1
# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Posted in Command Line, Fedora, Networking, Tutorials | 32 Comments »

fish(Friendly Interactive Shell)

Written by BinnyVA on April 8, 2007 – 1:00 pm -

fish(Friendly Interactive Shell) is a new shell for Linux. I tried it out and have decided to dump bash for fish. Try it out – you will come to the same conclusion as well.

Features of fish

Syntax Coloring

The shell colorizes the commands as you type them – if it is a valid command it will have a green color. For example, lets say I want to see my network interfaces. Recently, I had a lot of use for that – but that is another post. So, I type ifconfig into the shell. When I am at ‘ifco’, the shell will be like this…

Ifco - Typing completion

When I have completed the command, ifconfig, the shell will be like this…

Completed

Strings, matching etc. are also highlighted as you type.

Syntax Highlighting

Tab Completion

You would not think that this is a new feature. Bash has tab completion. Even Windows XP’s DOS terminal has tab completion. But fish’s tab completion is no ordinary tab completion – think of it as tab completion on steroids.

fish’s tab completion implements a feature that I really needed – tab completion for subcommands. Subcommand is the command line argument that is given to some programs. For example,

yum update gimp
cvs commit file.php

In the first example, yum is the command an ‘update‘ is the subcommand. Just type ‘yum upd’ and press Tab to complete the command. Similarly in the second case cvs is the command and commit is the subcommand.

Many other completions are also supported…

  • Commands, both builtins, functions and regular programs.
  • Environment variable names (Eg. $HOME).
  • Usernames for tilde(~) expansion.
  • Filenames, even on strings with wildcards such as ‘*’, ‘**’ and ‘?’.
  • Job id, job name and process names for process expansion. This is very useful when using kill.

Enough talking. You can download the fish shell from their official website. For Red Hat/Fedora Core users, this command will do the trick.

yum install fish

If you are a debian or Ubuntu user, use this command

apt-get install fish


Tags: , , ,
Posted in Command Line, Tools | 5 Comments »