Archive for the ‘HowTo’s’ Category

Simple bash variable security with OpenSSL

November 17th, 2010 No comments

Lets say you have a script that requires a number of variables to operate such as a database connection that requires a server, username and password. It’s usually a good idea to keep all this config in one place but dumping this in a plain text file is usually a bad idea.
The example below allows you to place application level config into a decently encrypted file and keep it all behind one master password.
To do all this we need two files. The first is obviously the encrypted variable config itself. The second is your script file which needs to decrypt the config file and perform whatever action is needed.

In this example I’m going to assume we are connecting to a remote database. Most protocols require three pieces of data to pull this off: a server, a username and a password. We are going to store these three pieces of data in a file called config.plain, encrypt this file (making config.aes), extract its variables and finally connect to our remote database.

I’m going to use the following example config.plain file:


So we need to encrypt our config.plain file and turn it into config.aes (AES being the encryption standard we are using). The following example is cheerfully swiped from the example given over at Tombuntu.

>openssl aes-256-cbc -a -salt -in config.plain -out config.aes

So if we wanted to reverse this and decrypt the config.aes file on the command line we could run:

>openssl aes-256-cbc -d -a -salt -in config.aes

NOTE: For some reason the OpenSSL CLI doesn’t obey the Unix standard of specifying ‘-‘ as a file name to output to the console. Omitting the ‘-out’ argument will instead output to STDOUT.

Putting it all together

The config.plain example file:


The config.aes file (same as above but after encryption using the password ‘password’):


The actual script file which does the decryption and performs the final actions on the data.
We are going to accept the master password from the command line (so we would run the below as ‘script password’ – assuming our script was called ‘script’ and the password was ‘password’). This is not the most secure method available since you can see the launch method of all applications running on the system using ‘ps’. If you want more security I would instead store your password in a system variable and use that instead of reading from the command line. If you wish to do this you can simply strip out all the lines that do error checking and use ‘KEY’ as a system variable to store the master password.

# USAGE: script 
# Do something with the associated config.aes script in a reasonably secure way
# Example config.plain content:
# Command to encrypt config.plain -> config.aes
# openssl aes-256-cbc -a -salt -in config.plain -out config.aes

# Do some housekeeping to check we launched with the requisite number of arguments
if [ "$#" -ne 1 ]; then
	echo "USAGE: $0 "
	exit 1

# Import the variables from our decrypted stream
STREAM=`openssl aes-256-cbc -d -pass "pass:$KEY" -a -salt -in config.aes`
if [ "$?" -ne 0 ]; then
	echo "Invalid password specified to decrypt configuration file. Aborting."
	exit 1
source <(echo "$STREAM")

echo "Server: $SERVER"
echo "Username: $USER"
echo "Password: $PASS"
Categories: HowTo's Tags:

Interactive file merging with Unison

August 13th, 2009 3 comments

After trying in vein to find decent documentation on the Merge functionality of Unison I eventually gave in and invested an hour or two trying out different Linux based merge programs in order to find something Unison, the merge program and I were happy with.

So onto the diff-merging candidates:

VimDiff (part of Vim)

Doesn’t work too well with Unison. I can’t get gnome-terminal (my terminal of choice) to execute synchronously so I was forced to fall back on XTerm instead.

After using XTerm I immediately hit upon a problem: there is no sensible way of outputting the results of a merge in VimDiff. Which is a pity because I do prefer VIM as an editor but this lack of functionality is annoying (yes I could have used a wrapper script but I’m trying to keep this simple).


Very pretty but lacks features such as ‘Use Left-hand-side-file for all changes’ and some of the other filtering options provided by KDiff3.

Also, like, VimDiff, does not provide any sensible output mechanism of the eventually merged file.


Not as good looking as Meld but provides pretty much everything I want in a merge program including the all important merged file output that Unison requires.

Can get a little annoying though with its insentient pop-up boxes. I wish there were a ‘Dont show this option again’ option to each. KDE is usually quite good with this so I’m not sure why KDiff3 managed to escape this unspoken standard.


So how to use KDiff3 with Unison? First install kdiff3:

apt-get install kidff3

Then open your Unison profiles .prf file and insert the following:

merge = Name * -> kdiff3 CURRENT1 CURRENT2 -L1 'Local File' -L2 'Remote File' -o NEW -m
confirmmerge = true

If you are comfortable with KDiff3 you could remove the ‘confirmmerge’ line which forces Unison to double check you actually want to accept the changes.

The downside to the above interactive merging process is there is no sensible way to preview the merge in Unison (the ‘View Merge’ button in the confirmation dialogue). For now though its preferable to not having this functionality at all.

Categories: HowTo's Tags: , ,

Converting Null

April 24th, 2009 2 comments

Simple shell script fix this week – convert all newlines to null characters.

Since most Linux shells have a major problem with doing this in a sensible way, the following (on my box called simply ‘0’) takes any input (either from files or via a pipe) and converts the characters.

# From:
# Author: Matt Carter 
while (<>) {

So now commands like:

find -type f | 0 | xargs -0 | mplayer

work perfectly (in the above case to play all files in directories recursively.

You may well ask why I don’t use the simple ‘-print0’ argument for the find program – because it doesn’t work when using later pipes.

For example if i wanted to sort the above, ‘sort’ would see the null symbol as a regular character.

So ‘0’ works nicely in this case:

find -type f | 0 | sort | xargs -0 | mplayer

Or even:

find -type f | 0 | shuffle | xargs -0 | mplayer

To play all files recursively, in random order. See the article on shuffle for the source code of that filter.

Categories: Fixes, HowTo's Tags: , , ,

Shuffling shell input

April 17th, 2009 2 comments

While the Fisher-Yates shuffle sounds like 1920’s dance move, it is the standard method for shuffling larger arrays of data.

The below operates like a simple Unix filter taking an array of data (separated by newlines) and returning a quickly sorted output.

# Usage: cat file | shuffle
# From:
# Return a shuffled version of the input stream
# Author: Matt Carter 

sub fisher_yates_shuffle {
	my $array = shift;
	my $i;
	for ($i = @$array; --$i; ) {
		my $j = int rand ($i+1);
		next if $i == $j;
		@$array[$i,$j] = @$array[$j,$i];

while(<>) {
	push @input, $_;

print join("\n",@input);

For example:

ls | shuffle

Returns a randomized list of files in the current directory.

Categories: HowTo's Tags: ,

Easy Apache VirtualHost config

March 15th, 2009 1 comment

Like most of the word I use Apache to serve up both my live and test websites.

The downside to having a box for development though is the constant config of apache to host these sites, most of which require absolute links (e.g. ‘/css/style.css’) to function.

The Apache snippet below sets up dynamic sub-domains which can be changed without reloading Apache every time you add, remove or update anything something. Simply drop your sites inside /var/www whenever you want a new subdomain. Alternatively you can symlink if they are hosted elsewhere on the box.

Its pretty easy to install, simply paste the below in your /etc/apache2/apache.conf file…

# Easy virtual host config
# See

DocumentRoot /var/www/
ServerAdmin admin@localhost
Options +FollowSymLinks -Indexes
LogLevel debug
ServerAlias localhost
ServerName localhost
RewriteEngine  on
RewriteLogLevel 3
RewriteCond    %{HTTP_HOST}  ^localhost
RewriteRule    ^(.*)$        /www/$1 [L]
RewriteCond    %{HTTP_HOST}  ^www.*
RewriteRule    ^(.*)$        /www/$1 [L]
RewriteCond    %{HTTP_HOST}  ^(.*)\.localhost
RewriteRule    ^(.*)$        /%1/$1 [L]

Now add your servers to the /etc/hosts file:      server.localhost

Now any directory you place in the /var/www directory automatically becomes a subdomain. You can now visit http:///server.localhost as a fully fledged webhost.

If you want to use another name than the rather boring ‘localhost’ simply replace the text in both text pastes above.

If only there were a way of removing that /etc/hosts hack it would mean zero config from then on. If anyone has any ideas do let me know. The addition to the hosts file seems to need applying if you are working from the same box the Apache server resides on. If you arn’t and the box is a local dedicated server its not needed at all and domains can be added and removed as necessary.

Categories: HowTo's Tags: , ,