LinDesk

I have a small but very useful site called txt. Its a collection of code snippets, Linux commands and other such tidbits of information. The posts will be minimal and often crude. The purpose of that site is just to help me remember the information. To make it easier for me to look up the commands later.

For example, say that I need to remove all subversion information from a folder. That means deleting the ‘.svn’ folders in the current folder and all the folders under it. I have used this command before and have saved it to the txt site. So all I have to do is look up the tag that may have this command - in this case, the svn tag.

Soon, I get the page about removing SVN data from a folder. Nifty, huh?

The purpose of this post is two fold. First, I want to introduce you to my txt site - and hopefully get more traffic for that site. Second, and more important purpose is to get you intrested in this idea - to get a similar site for yourself. I saved a lot of time by using that site. Because its online, I was able to use my commands even when away from my computer. And its helpful for others as well.

You can see the reasons for creating the txt site here…

• Saving Code Snippets - Part 1
• Saving Code Snippets - Part 2

If you want to create a similar site, just go to WordPress and register for a new site. Trust me, its easy. If you have a similar system, please let me know - post a comment.

So far, we covered backing up files locally and over a network. Now let let see how to backup databases(only mysql supported - yet).

The script uses ‘mysqldump’ command to backup the data. That means that the backups are in the SQL dump format. The dumps of all the databases that are backed up are compressed and stored in the destination folder. They will be named in this format - YYYY-MM-DD.tar.gz.

Configuration File

This script reads a configuration file named ‘dbbackup.config’ and backups all the databases specified in that file to another location in the same system. This configuration file must be in the same folder as the perl script. The configuration file format is given below…

Data
Project_Nexty
App_activecollab
# Unwanted_DB - commented - will not be backedup
binco
binnyva

The Perl Script

#!/usr/bin/perl
# Backups all the databases specified in the dbbackup.config file

$backup_folder = '/var/Backup/Special/Databases'; #EDIT THIS LINE

use File::Basename;
my $config_file = dirname($0) . "/dbbackup.config";
my @databases = removeComments(getFileContents($config_file));

chdir($backup_folder) or die("Cannot go to folder '$backup_folder'");

my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
$year += 1900;
$mon++;
#Zero padding
$mday = '0'.$mday if ($mday<10);
$mon = '0'.$mon if ($mon<10);

my $folder = "$year-$mon-$mday";
mkdir($folder) or die("Cannot create a folder called '$folder'");

foreach my $database (@databases) {
	next if ($database eq '');
	chomp($database);

	my $table = '';
	if(index($database,' ')+1) { #Get just 1 table in the database - if there is a ' '(space) in the db name
		my @parts = split(' ',$database);
		$database = $parts[0];
		$table = $parts[1];
	}

	print “Backing up $database … “;

 	my $file = $database;
 	$file .= ‘_’ . $table if($table ne ”);
 	$file .= “.sql”;

 	`mysqldump -u root $database $table > $folder/$file`;

	print “Done\n”;
}
print “Compressing the folder … “;
`tar -czf $folder.tar.gz $folder/`;
print “Done\nRemoving Folder … “;
`rm -rf $folder`;
print “Done\n\n”;

sub getFileContents {
	my $file = shift;
	open (FILE,$file) || die(”Can’t open ‘$file’: $!”);
	my @lines=<FILE>;
	close(FILE);

	return @lines;
}

sub removeComments {
	my @lines = @_;

	@cleaned = grep(!/^\s*#/, @lines); #Remove Comments
	@cleaned = grep(!/^\s*$/, @cleaned); #Remove Empty lines

	return @cleaned;
}

If you need, you can set this script as a cron job - this will make sure that you don’t have to worry about the backup.

This script will backup the specified files to another computer on your network. You can also use this to send your files to a remote server. This script compliments the last Rsync Backup script. Its possible to combine both the script together, I prefer to keep them separate.

The Setup

For this to work, you need to have a password-less login system over ssh. You should configure the remote system to accept your credentials by giving your public key to the remote server. If you are not sure how to do that, just leave a comment and I’ll make a post on how to set it up.

The configuration file is the same format as the one used in the last Rsync script. But in this case, the file name will be ‘rsyncnetworkbackup.config‘.

The Code

#!/usr/bin/perl

#The folder on the remote system that must be used to store the data
$backup_folder = '/home/neo/Backup'; #Final '/' must NOT be there.
# The user for whom we have set up the key based login
$backup_user = 'neo';
# The IP address/domain name of the remote system.
$backup_server = '192.168.0.30';

use File::Basename;
my $config_file = dirname($0) . "/rsyncnetworkbackup.config";
my @all_locations = removeComments(getFileContents($config_file));

foreach my $folder_locations (@all_locations) {
	my($folder,$backup_location) = split(/\s+/,$folder_locations);

	print "Backing up $folder to $backup_location ... ";
	`rsync -avze ssh $folder $backup_user\@$backup_server:\"$backup_folder/$backup_location\"`;
	print "Done\n";
}

sub getFileContents {
	my $file = shift;
	my @lines;

	open (FILE,$file) || die("Can't open '$file': $!");

	@lines=<FILE>;
	close(FILE);
	return @lines;
}

sub removeComments {
	my @lines = @_;

	@cleaned = grep(!/^\s*#/, @lines); #Remove Comments
	@cleaned = grep(!/^\s*$/, @cleaned); #Remove Empty lines

	return @cleaned;
}

All programmers have their own customized backup solutions. I have six. Yes, six! Five to backup files and one to backup database tables. And I am not counting version control or other backup systems built into the tools I use. Anyway, in the first post of the shell scripts series, let me introduce you to my Rsync based local backup solution.

Configuration File

This script reads a configuration file named ‘rsyncbackup.config’ and backups all the folders specified in that file to another location in the same system. This configuration file must be in the same folder as the perl script. The configuration file format is given below…

#Notes - do NOT include the last '/' at the end of the source folders
~/Scripts
~/Documents

#################### Web files ####################
# My Sites
#Folder to backup		#Folder to which it should be backuped to
/var/www/html/Sites		Htdocs/
/var/www/html/Projects	Htdocs/

If there is just one column in a line, that folder will be backuped to “<backup folder>/<folder name>“. Let say that my backup destination folder is ‘/var/Backup/Rsync‘. So the first line, ‘~/Scripts‘ will copy the contents of ‘~/Scripts‘ to ‘/var/Backup/Rsync/Scripts‘

If a line in the configuration file has two columns, then an extra folder will be created with the name provided in the second column. For example, the line ‘/var/www/html/Sites     Htdocs/‘ will create a backup of ‘/var/www/html/Sites‘ in ‘/var/Backup/Rsync/Htdocs/Sites‘

And if you have not guessed it already, all lines that begin in a # are comments and will be ignored.

The Script

There is the perl script that automates the rsync calls…

#!/usr/bin/perl

#EDIT THIS LINE
$backup_folder = '/var/Backup/Rsync'; #Final '/' must NOT be there.

use File::Basename;
my $config_file = dirname($0) . "/rsyncbackup.config";
my @all_locations = removeComments(getFileContents($config_file));

chdir($backup_folder) or die("Cannot go to folder '$backup_folder'");

foreach my $folder_locations (@all_locations) {
	my($folder,$backup_location) = split(/\s+/,$folder_locations);

	print "Backing up $folder to $backup_location ... ";
	`rsync -a $folder $backup_folder/$backup_location`;
	print "Done\n";
}

sub getFileContents {
	my $file = shift;
	my @lines;
	if(!open (FILE,$file)) {
		die("Can't open '$file': $!");
	} else {
		@lines=<FILE>;
		close(FILE);
	}
	return @lines;
}

sub removeComments {
	my @lines = @_;

	@cleaned = grep(!/^\s*#/, @lines); #Remove Comments
	@cleaned = grep(!/^\s*$/, @cleaned); #Remove Empty lines

	return @cleaned;
}

Execute this script using the command ‘perl RsyncBackup.pl’. In my system I have created an alias ‘bk’ for this script. I recommend that you make a similar alias if you take backups regularly(extremely recommended).

Backing up is done using rsync - so its faster than a simple ‘cp’ as only the modified and new files are copied.

To me, a shell script is a script that automates repetitive tasks. But that is not the ‘official’ definition. Wikipedia has this definition…

A shell script is a script written for the shell, or command line interpreter, of an operating system.

I use a definition that defines the purpose of the script - while the others prefer a definition that defines the technology used. I am not going to claim that my definition is better than the other definition - that’s pointless. Besides, even I think that the ‘other’ definition is the right one. But I will try to show you the advantages of my approach.

The Purpose

I like to automate things(in other words, I’m lazy). So I have a nice little collection of custom shell scripts. But there is a huge barrier to writing shell scripts - the language used. Traditionally shell scripts are written in a language provided by the shell - like bash or tcsh. That is the problem - these languages are Bad - with a capital ‘B’.

The Problem

To people who are accustomed to decent languages, these shell languages will seem clunky - or even evil. But since most shell scripts are small, most people don’t mind the torture.

In bash, the control flow commands seem to be thrown in as a after thought rather than something that’s built into the language. If you don’t believe me, compare the ‘if’ loop of bash and Perl.

Bash

This code checks wether the variable ‘$value’ has the value 0 - if so it prints the message ‘No Value’

if [ $value -eq 0 ] ; then
	echo “No Value”
fi

Perl

The same code in Perl…

if($value == 0) {
	print "No Value";
}

Of course, Perl experts will go for if(!$value) or even unless($value) - but that’s not the point. See how better the code looks in Perl. Yeah, even I am surprised to hear those words - Perl is considered by many to be an ‘ugly’ language. But when compared to shell languages, Perl is a gem(sorry about the pun - couldn’t resist).

The Solution

The solution is simple - don’t use a shell language to write your shell scripts - use a high level scripting language like Perl. Or Python, Ruby, PHP, Tcl or even JavaScript.

I still use bash to write shell scripts - but if the shell script has an if condition(other than a simple argument check), I use a higher language - usually Perl.

Advantages of using a Shell Language

• Its supported in even the tinest linux distros - even in embedded systems.
• Its the method preferred by the majority - I hope this will change soon.
• Command calls look more natural in a shell language.

Advantages of using a High Level Language

• Better Code
• Easy to maintain
• Faster development
• Libraries provide a lot of functionality
• Easier to port to different platforms.

So why am I telling you all this? Two reasons…

One, the next time you are going to write a shell script, I want you to choose a high level language rather than using bash.

The second reason is that now that my series on Linux MP3 Players are over, I am going to take a small break from desktop posts and write on more ‘linuxy’ topics. And one of those topic is Shell Scripting. So in the future posts, I am going to share some of my shell scripts with you. So when I publish a Perl script and call it a shell script, I don’t want you to get confused .

So far we looked at the GUI MP3 Players for linux - like Amarok, Exile, XMMS etc. Most linux users need only that - but there are some people who want a simpler system - command line players. The two top players in this area are mpg123 and mpg321.

mpg123

mpg123 is a fast, free, minimalist, console MPEG audio player software program for UNIX and Linux operating systems. The development on this project ceased for a long time - but now it has a new maintainer.

• mpg123 Homepage
• Wikipedia Page

mpg321

mpg321 is a clone of mpg123.

• Homepage
• Wikipedia Page

Installation

Most distros have mpg321 in their repositories(not all have mpg123) - so just log in as root and use these commands to install it…
yum install mpg321 - For RedHat, Fedora, CentOS etc.
OR
apt-get install mpg321 - For Debian, Ubuntu, etc.

Using mpg321

Using mpg321 is straight forward. Since it has no GUI, just go to the folder with the song and just execute the command…

mpg321 MP3_FILE_NAME

If you want to play all the songs in a folder, you have to create a list of files in that folder. Then you have to specify this list as an argument for the mpg321 command. These commands can get the job done…

Create a list of all MP3 Files in the current folder…

find . -name "*.mp3"|sed 's/\.\///;'|sort>List.lst

Use this list as the play list for mpg321

mpg123 --list List.lst

Why Use Command Line Players?

If we have a number of shiny GUI players, why should one use a command line player? Unless you are an anti-GUI command line guru, chances are you don’t need it. But still, it has its uses.

For example, if you have to restart the X Server often for some reason, then mpg321 is the best player for you. All other player will stop playing when X server is down.

This actually happend to me - when I bought my second LCD screen, it took me some time to configure it. And that meant editing the xorg.conf file and restarting the X Server to see if it worked. At that time, I opened a virtual terminal(Ctrl+Alt+F1) and opened mpg321 with a bunch of MP3s in the list. It kept playing even if the X server is down. This is propably the reason that I was still sane after around six hours of XOrg configuration.

The mount command in Linux is a very powerful command. It can be used to mount any thing from a Samba mount to a ISO image file. The ability of the mount command to read ISO files makes it very useful for me.

My brother has a habit of saving DVD films as a image file. I believe he uses Alcohol 120% to do this. If he wants to view the Film, he ‘mount’ this image using Daemon Tools. He uses windows to do this. After he is done, he hands me this image. I can mount this image without any special tools in Linux - I just need the mount command.

mount -t iso9660 -o loop <Image_File> <Mount_Point>

Mounting Example

mount -t iso9660 -o loop /home/binnyva/Films/300.iso /mnt/Image

The ‘-t’ option specifies the filetype - this is optional.

This command works with both ISO and MDF images.

It is very easy to setup an internet connection sharing in Linux system using iptables. This method can be used to share an internet connection from a Linux system(I used Fedora Core 6, but it should work on other distributions that support iptables). Another method of doing this is using a proxy server like squid.

Enable IP forwarding

Run as root

sysctl -w net.ipv4.ip_forward=1

To enable it in system startup, edit the file /etc/sysctl.conf and set

net.ipv4.ip_forward = 1

iptables

Run command as root

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
service iptables save

Configuration

The configuration should be like this

Router

Connected to the internet provider
IP : 192.168.1.1

Internet Connected System

eth0 (LAN Card 1)

Connected to router

• IP : 192.168.1.10
• Netmask : 255.255.255.0
• Gateway : 192.168.1.1 (IP of the router)

eth1 (LAN Card 2)

Connected to the other system

• IP : 192.168.0.20 (Not the same network as the first card)
• Netmask : 255.255.255.0
• Gateway : 192.168.1.1 (IP of the router)

Second System

LAN Card connected to the first system

• IP : 192.168.0.30
• Netmask : 255.255.255.0
• Gateway : 192.168.0.20 (IP of the second Card in the first system)

Disclaimer

Linux networks is not a subject I am an expert on. So take my advice with a pinch of salt. The above procedure worked for me. YMMV.

Status

Some results of various commands are shown here. Check to see if it matches the result on your system.

# iptables -t nat -L POSTROUTING
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  —  anywhere             anywhere

# cat /proc/sys/net/ipv4/ip_forward
1

# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

fish(Friendly Interactive Shell) is a new shell for Linux. I tried it out and have decided to dump bash for fish. Try it out - you will come to the same conclusion as well.

Features of fish

Syntax Coloring

The shell colorizes the commands as you type them - if it is a valid command it will have a green color. For example, lets say I want to see my network interfaces. Recently, I had a lot of use for that - but that is another post. So, I type ifconfig into the shell. When I am at ‘ifco’, the shell will be like this…

When I have completed the command, ifconfig, the shell will be like this…

Strings, matching etc. are also highlighted as you type.

Tab Complition

You would not think that this is a new feature. Bash has tab completion. Even Windows XP’s DOS terminal has tab completion. But fish’s tab completion is no ordinary tab completion - think of it as tab completion on steroids.

fish’s tab completion implements a feature that I really needed - tab completion for subcommands. Subcommand is the command line argument that is given to some programs. For example,

yum update gimp
cvs commit file.php

In the first example, yum is the command an ‘update‘ is the subcommand. Just type ‘yum upd’ and press Tab to complete the command. Similarly in the second case cvs is the command and commit is the subcommand.

Many other completions are also supported…

• Commands, both builtins, functions and regular programs.
• Environment variable names (Eg. $HOME).
• Usernames for tilde(~) expansion.
• Filenames, even on strings with wildcards such as ‘*’, ‘**’ and ‘?’.
• Job id, job name and process names for process expansion. This is very useful when using kill.

Enough talking. You can download the fish shell from their official website. For Red Hat/Fedora Core users, this command will do the trick.

yum install fish

If you are a debian or Ubuntu user, use this command

apt-get install fish