Posts Tagged ‘script’

A Mageia based Firewall with auto_inst and lots of other stuff like chrooted squid

2013/01/07

I’ve been working on renewing some of my systems, and as I now moved fully to Mageia version 2, I’ve worked on tooling my installation of my firewall.

It’s still not fully as I wanted it to be, but is already worth sharing s well as some comments around the distribution usage.

First, I used a PXE install of Mageia with auto_inst. On my PXE server, I used the following config for PXElinux:
label pxe
kernel k/m2
append initrd=i/m2.img ramdisk_size=512000 root=/dev/ram3 kickstart=http://x.y.z.k/pub/ks/www/guerrero.pl automatic=met:http,int:eth1,ser:w.y.z.k,dir:/pub/mageia/distrib/2/i586,netw:dhcp

Nothing special here, just following the doc. Well, which doc could you say ? The one I just added to the Mageia wiki from the Mandriva wiki, itself from the Mandrka version. Remember, auto_inst was Mandriva’s best kept secret ! Hopefully, it will change with Mageia !

Now the secret sauce is in the guerrero.pl file, which is the auto_inst config.
Here is mine:
#!/usr/bin/perl -cw
#
# $Id$
#
#
# You should check the syntax of this file before using it in an auto-install.
# You can do this with 'perl -cw auto_inst.cfg.pl' or by executing this file
# (note the '#!/usr/bin/perl -cw' on the first line).
$o = {
'timezone' => {
'ntp' => '0.pool.ntp.org',
'timezone' => 'Europe/Paris',
'UTC' => 1
},
'services' => [
'acpid',
'crond',
'fusioninventory-agent',
'gpm',
'msec',
'network',
'network-up',
'ntpd',
'numlock',
'partmon',
'postfix',
'resolvconf',
'rsyslog',
'shorewall',
'squid',
'sshd'
],
'security_user' => 'bruno_at_musique-ancienne.org',
'default_packages' => [
'acpi',
'acpid',
'apache',
'basesystem',
'drakxtools-curses',
'ethtool',
'fusioninventory-agent',
'gpm',
'grub',
'iptraf',
'kernel-server-latest',
'locales-fr',
'lshw',
'lsof',
'mondo',
'msec',
'nss',
'ntpd',
'numlock',
'openssh-server',
'openssh-client',
'pam_abl',
'pam_cgroup',
'postfix',
'rsyslog',
'squid',
'squidguard',
'shorewall',
'shorewall-doc',
'strace',
'sudo',
'tcpdump',
'tmpwatch',
'traceroute',
'tshark',
'vim-enhanced',
'vlock',
'wget',
],
'users' => [
{
'icon' => 'default',
'realname' => 'administrator',
'uid' => undef,
'groups' => [],
'name' => 'administrator',
'shell' => '/bin/bash',
'gid' => undef,
'pw' => '$2xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
}
],
'locale' => {
'country' => 'FR',
'IM' => undef,
'lang' => 'fr',
'langs' => {
'fr' => 1
},
'utf8' => 1
},
'net' => {
'zeroconf' => {},
'network' => {
'NETWORKING' => 'yes',
'GATEWAY' => 'x.y.z.k',
'CRDA_DOMAIN' => 'FR',
'FORWARD_IPV4' => 'false'
},
'autodetect' => {},
'network::connection::ethernet' => {},
'resolv' => {
'DOMAINNAME' => 'nameserver',
'dnsServer' => 'x.y.z.k',
'DOMAINNAME2' => 'search',
'dnsServer2' => 'musique-ancienne.org',
},
'wireless' => {},
'ifcfg' => {
'eth0' => {
'BROADCAST' => '',
'isUp' => 1,
'BOOTPROTO' => 'dhcp',
'isPtp' => '',
'NETWORK' => '',
'HWADDR' => undef,
'DEVICE' => 'eth0',
'METRIC' => 10
}
},
'type' => 'network::connection::ethernet',
'net_interface' => 'eth0',
'PROFILE' => 'default'
},
'authentication' => {
'shadow' => 1,
'blowfish' => 1
},
'partitions' => [
{
'fs_type' => 'ext4',
'mntpoint' => '/',
'size' => 1138567
},
{
'fs_type' => 'swap',
'mntpoint' => 'swap',
'size' => 4038086
},
{
'fs_type' => 'ext4',
'mntpoint' => '/usr',
'size' => 6165190
},
{
'fs_type' => 'ext4',
'mntpoint' => '/var',
'size' => 8283384
},
{
'fs_type' => 'ext4',
'mntpoint' => '/tmp',
'size' => 542289
},
# Put the one extending lst
{
'fs_type' => 'ext4',
'mntpoint' => '/var/spool/squid',
'size' => 20283384,
'ratio' => 100,
},
],
'partitioning' => {
'auto_allocate' => 1,
'clearall' => 1,
'eraseBadPartitions' => 1
},
'superuser' => {
'pw' => '$2xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
'realname' => 'root',
'uid' => '0',
'shell' => '/bin/bash',
'home' => '/root',
'gid' => '0'
},
'security' => 'secure',
'mouse' => {
'EmulateWheel' => undef,
'synaptics' => undef,
'name' => 'Any PS/2 & USB mice',
'device' => 'input/mice',
'evdev_mice' => [
{
'device' => '/dev/input/by-id/usb--event-mouse',
'HWheelRelativeAxisButtons' => '7 6'
}
],
'evdev_mice_all' => [
{
'device' => '/dev/input/by-id/usb--event-mouse',
'HWheelRelativeAxisButtons' => '7 6'
}
],
'type' => 'Universal',
'nbuttons' => 7,
'Protocol' => 'ExplorerPS/2',
'wacom' => [],
'MOUSETYPE' => 'ps/2'
},
'interactiveSteps' => [
],
'autoExitInstall' => '0',
'no_suggests' => 1,
'mkbootdisk' => 0,
'isUpgrade' => 0,
'excludedocs' => 0,
'miscellaneous' => {
'numlock' => 1,
},
'keyboard' => {
'GRP_TOGGLE' => '',
'KEYBOARD' => 'us'
},
'postInstall' => '
cd /root
wget http://x.y.z.t/pub/ks/www/post-install.sh
chmod 755 ./post-install.sh
./post-install.sh 2>&1 | tee /dev/tty7 | tee /var/log/post-install.log
',
};

First, that doesn’t completely install a minimal Mageia. For now, due to plymouth (from ML feedback) it adds a lot of X11 packages which shouldn’t be required. Even adding the no_suggests (not documented on the Mandriva wiki, but now on the Mageia version ;-) didn’t fully solved the problem, even if it improved stuff. I now have a compliant install with 387 packages – after my postinstall phase removed most of what was not needed.

So what does my postinstall ?
Here it is:

#!/bin/bash
#
# $Id$
#
# Common conf for all zones
# Idempotent PostInstall script

echo "Common final setup"
echo "---------------"

echo "Allow remote access for sshd"
grep -Eq '^sshd:' /etc/hosts.allow 2> /dev/null
if [ $? -ne 0 ]; then
echo "sshd: LOCAL, musique-ancienne.org, x.y.z.t" >> /etc/hosts.allow
fi

# Temporary hack before overwrite by cb
grep -Eq '^ssh' /etc/shorewall/rules.drakx
if [ $? -ne 0 ]; then
echo "ACCEPT net fw tcp 22 -" >> /etc/shorewall/rules.drakx
fi

echo "Allow sudo access for administrator"
grep -Eq '^administrator' /etc/sudoers
if [ $? -ne 0 ]; then
echo "Defaults:administrator !requiretty" >> /etc/sudoers
echo "administrator ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
else
perl -pi -e 's/^administrator.*/administrator ALL=(ALL) NOPASSWD:ALL/' /etc/sudoers
perl -pi -e 's/^Defaults:administrator.*/^Defaults:administrator !requiretty/' /etc/sudoers
fi

echo "Allow cron access to administrator"
grep -Eq '^administrator' /etc/cron.allow 2> /dev/null
if [ $? -ne 0 ]; then
echo "administrator" >> /etc/cron.allow
fi

echo "Allow shutdown access to administrator"
grep -Eq '^administrator' /etc/shutdown.allow 2> /dev/null
if [ $? -ne 0 ]; then
echo "administrator" >> /etc/shutdown.allow
fi
FWDIR=`echo ~administrator`
echo "administrator setup"
mkdir -p $FWDIR/.ssh
chmod 700 $FWDIR/.ssh
cat > $FWDIR/.ssh/authorized_keys < /var/spool/cron/root
echo "0 3 * * * /usr/local/bin/rc.upd" >> /var/spool/cron/root
chmod 600 /var/spool/cron/root

echo "Postfix alias"
perl -pi -e "s/root:\s*postfix/root: bruno_at_musique-ancienne.org/" /etc/postfix/aliases
postalias /etc/postfix/aliases

echo "Manages long reboot of dhcp server just in case"
# Not necessarily used BTW
cat > /etc/dhclient-eth0.conf < /etc/dhclient-eth1.conf <> /var/spool/cron/root
fi
grep -Eq "mkcommon" /var/spool/cron/root
if [ $? -ne 0 ]; then
echo "30 4 * * * /usr/local/bin/mkcommon" >> /var/spool/cron/root
fi
grep -Eq "mk$h" /etc/rc.local
if [ $? -ne 0 ]; then
echo "/usr/local/bin/mk$h" >> /etc/rc.local
fi
grep -Eq "mkcommon" /etc/rc.local
if [ $? -ne 0 ]; then
echo "/usr/local/bin/mkcommon" >> /etc/rc.local
fi
echo "Setup administrator passwd"
echo "XXXXXXXXXXXXXXXXXXXXX" | passwd --stdin fwadmin
echo "Setup root passwd"
echo "XXXXXXXXXXXXXXXXXXXXX" | passwd --stdin root

echo "Start specific postinstall for machine $h"
wget http://x.y.z.t/pub/ks/www/post-install-$h.sh
chmod 755 post-install-$h.sh
z=`grep -E "^#[ ]*ZONE:" post-install-$h.sh`
zone=`echo $z | cut -d: -f2`
# Doing the zone first
wget http://x.y.z.t/pub/ks/www/post-install-$zone.sh
chmod 755 post-install-$zone.sh
echo "Start specific postinstall for zone $zone"
./post-install-$zone.sh
echo "End specific postinstall for zone $zone"
# Then the machine
./post-install-$h.sh
echo "End specific postinstall for machine $h"
echo "End common postinstall"
echo "Now you can run cb -m $h to distribute content"

Seems complex, but isn’t that much. What it does roughly is opening enough security on a machine configured with security level of “secure” or 5 for msec to have an administrator account allowed to connect on it remotely with ssh and use sudo automatically (scripting purposes), cron and shutdown, configure mail redirection, distribution and machine update via cron, password setup and the launch of other scripts, depending on the zone in which the machine is (that postinstall script is common to many installed machines) and the machine itself.

So what does the zone post install script in addition ?. Here it is again:

#!/bin/bash
#
# $Id$
#
# Common conf for DMZ Zone
#

# Idempotent PostInstall script

echo "DMZ final setup"
echo "---------------"

echo "DNS setup"
cat > /etc/resolv.conf <> /etc/postfix/main.cf
if [ _"$kickstart" = _"" ]; then
/etc/init.d/postfix restart
fi

echo "NTP conf"
perl -pi -e 's/^server.*/server 0.pool.ntp.org/' /etc/ntp.conf
echo "0.pool.ntp.org" > /etc/ntp/step-tickers
if [ _"$kickstart" = _"" ]; then
/etc/init.d/ntpd restart
fi

cat > /etc/sysconfig/network < /etc/hostname << EOF
$h.musique-ancienne.org
EOF
if [ _"$kickstart" = _"" ]; then
/etc/init.d/network restart
fi

So basically, network and some services (ntp, smtp) setup. Stuff that every machine in that zone should get.
Now the final script run is the one for that specific machine, :


#!/bin/bash
#
# $Id$
#
# KEEP THAT COMMENT INTACT - USED FOR COMMON DMZ/LAN CONF
#
# ZONE:dmz
#

# Idempotent PostInstall script for guerrero

machine=`basename $0 .sh | cut -d- -f3`

echo "$machine final setup"
echo "--------------------"

echo "Rotate on a year"
perl -pi -e "s/rotate \d+/rotate 52/" /etc/logrotate.conf /etc/logrotate.d/*

DSK=`df | grep -E ' /$' | grep /dev | awk '{print $1}' | sed 's/[0-9]*$//'`
echo "Tuning File Systems"
tune2fs -c 0 -i 0 -m 1 ${DSK}1 # /
tune2fs -c 0 -i 0 -m 1 ${DSK}6 # usr
tune2fs -c 0 -i 0 -m 1 ${DSK}7 # var
tune2fs -c 0 -i 0 -m 0 ${DSK}8 # squid
tune2fs -c 0 -i 0 -m 1 ${DSK}9 # tmp

echo "$machine static network configuration"
# Affect the static address to $machine
cat > /etc/sysconfig/network-scripts/ifcfg-eth1 < /etc/sysconfig/network-scripts/ifcfg-eth0 <> /etc/sysctl.conf
else
perl -pi -e 's/net.ipv4.ip_forward[\s]*=.*/net.ipv4.ip_forward = 1/' /etc/sysctl.conf
fi
sysctl -p

if [ _"$kickstart" = _"" ]; then
/etc/init.d/network restart
fi

echo "Secure the system with msec"
perl -pi -e 's/BASE_LEVEL=.*/BASE_LEVEL=secure/' /etc/security/msec/security.conf

echo "cleanup extra packages installed on Mageia 2"
urpme --auto iw libmcpp0 libx11-common libxaw7 libxcomposite1 libxfixes3 libxcursor1 libxi6 libxinerama1 libxkbfile1 libxpm4 libxtst6 libxxf86dga1 libxxf86misc1 libxxf86vm1 mandi wireless-regdb x11-font-alias x11-font-cursor-misc xli xmodmap x11-font-encodings x11-data-xkbdata x11-data-bitmaps libdmx1 libmnl0 libnl3 rgb sessreg x11-font-misc-misc

So outside fixing the network conf after install (addresses, routing and topology), and tuning the file systems, adjuting the logrotate, it really secure the system with msec by changing the level in /etc/security/msec/security.conf which doesn’t seem to be done correctly by the auto_inst setup I used.

And finally it removes these ackages that I do not want on such a system hardened, a,d which resist to the no_suggests option ! Hopefully, Mageia 3 won’t have that issue anymore (will test later on this month the beta of Mageia 3)

But that’s not all ! As you’ve probably seen, some other scripts are invoked on the system, through cron or /etc/rc.local. This is where I really transform that gneric system into a firewall and a my proxy.

A first script invoked on all my system (mkcommon) does that for the moment:

#!/bin/bash
#
# $Id$
#
# Common setup for systems
#
# Script is idempotent
echo "Re-activate sysrq"
grep -Eq '^kernel.sysrq' /etc/sysctl.conf
if [ $? -ne 0 ]; then
echo "kernel.sysrq = 1" >> /etc/sysctl.conf
else
perl -pi -e 's/kernel.sysrq[\s]*=.*/kernel.sysrq = 1/' /etc/sysctl.conf
fi
/sbin/sysctl -p

grep 'll=' /etc/bashrc
if [ $? -ne 0 ]; then
echo "alias ll='ls -lia'" >> /etc/bashrc
fi

I like keeping control through the keyboard of the system so reactivate what msec desactivate for sysrq. And that’s also an easy way to add aliases, or all other common stuf you may want.

The other one, does the conf for the system:

#!/bin/bash
#
# $Id$
#
# Setup squid in a chrooted environment
# requires usage of the chroot directive in squid.conf
# Cf: http://wiki.squid-cache.org/ConfigExamples/ChrootJail
#
export CHROOTDIR=/var/spool/squid
SQUID=squid
SQGID=squid
# Script is idempotent

#
# Setup a global chrooted environment (normally a separated script expanded here)
#
# Script is idempotent
if [ _"$CHROOTDIR" = _"" ]; then
echo "Variable CHROOTDIR is not defined so unable to run mkchrootbase"
exit -1
fi
if [ "`echo $CHROOTDIR | cut -c1`" != "/" ]; then
echo "Variable CHROOTDIR doesn't start with / so unable to run mkchrootbase"
exit -1
fi
if [ "$CHROOTDIR" = "/" ]; then
echo "Variable CHROOTDIR is / so unable to run mkchrootbase"
exit -1
fi
rm -rf $CHROOTDIR/var/log $CHROOTDIR/var/run $CHROOTDIR/etc $CHROOTDIR/lib $CHROOTDIR/usr $CHROOTDIR/dev $CHROOTDIR/tmp
#
echo "Creating base chroot content"
install -v -m 755 -o root -g root -d $CHROOTDIR
install -v -m 1777 -o root -g root -d $CHROOTDIR/tmp
install -v -m 755 -o root -g root -d $CHROOTDIR/var/log/
install -v -m 755 -o root -g root -d $CHROOTDIR/var/run/
install -v -m 755 -o root -g root -d $CHROOTDIR/dev
cp -a /dev/null /dev/zero /dev/random /dev/urandom $CHROOTDIR/dev
install -v -m 755 -o root -g root -d $CHROOTDIR/etc
cp -a /etc/resolv.conf /etc/nsswitch.conf /etc/hosts /etc/localtime $CHROOTDIR/etc/
install -v -m 755 -o root -g root -d $CHROOTDIR/lib
cp -a /lib/libnss_dns* $CHROOTDIR/lib/

echo "Creating squid chroot content"
install -v -m 755 -o $SQUID -g $SQGID -d $CHROOTDIR/var/spool/squid
install -v -m 755 -o $SQUID -g $SQGID -d $CHROOTDIR/var/log/squid
install -v -m 755 -o $SQUID -g $SQGID -d $CHROOTDIR/etc/squid
cp -a /etc/squid/* $CHROOTDIR/etc/squid
install -v -m 755 -o root -g root -d $CHROOTDIR/usr/share/squid
cp -a /usr/share/squid/{icons,errors} $CHROOTDIR/usr/share/squid
install -v -m 755 -o root -g root -d $CHROOTDIR/usr/lib/squid
cp -a /usr/lib/squid/* $CHROOTDIR/usr/lib/squid/
chown ${SQUID}:$SQGID $CHROOTDIR/var/run
install -v -m 755 -o root -g root -d $CHROOTDIR/usr/bin
cp -a /usr/bin/squidGuard $CHROOTDIR/usr/bin
cp -a `/usr/sbin/mindi --locatedeps /usr/bin/squidGuard | sort -u` $CHROOTDIR/lib

# This is to make systemd happy
ln -sf $CHROOTDIR/var/run/squid.pid /var/run/

# Secure squid properly
grep -Eq squid /etc/security/msec/perm.local 2> /dev/null
if [ $? -ne 0 ]; then
cat >> /etc/security/msec/perm.local << EOF
/var/log/squid/ squid.squid 750
/var/spool/squid/var/log/squid/ squid.squid 750
/var/log/squid/* squid.squid 640
/var/spool/squid/var/log/squid/* squid.squid 640
EOF
msec
fi

echo "Setup of the squidGuard conf..."
sqg=`ls -d /usr/share/squidGuard*`
install -v -m 755 -o root -g root -d $CHROOTDIR/$sqg
ln -sf $sqg /usr/share/squidGuard
rm -rf $CHROOTDIR/usr/share/squidGuard
cd $sqg
rm -f blacklists.tar.gz
wget http://dsi.ut-capitole.fr/blacklists/download/blacklists.tar.gz
if [ $? -eq 0 ]; then
rm -rf blacklists
echo "Extracting the blacklists..."
tar xfz blacklists.tar.gz
echo "Generating the DBs for squidGuard..."
squidGuard -b -d -C all -c /etc/squid/squidGuard.conf
chown -R ${SQUID}:$SQGID blacklists
fi
echo "Copying squidGuard content..."
rm -rf $CHROOTDIR/usr/share/squidGuard
cp -a /usr/share/squidGuard $CHROOTDIR/usr/share/squidGuard

So this one is the trickiest one. It does the chroot environment in order to run squid in it. I may move to an LXC container later on, but that was what I previoulsy had, and thought it was still a valid approach.
The problem I found with tthis is with systemd. Colleagues could say I’m not found of it, but this was the first time I really had to interact erioulsy with that new init apporahc, and I’m less tan happy of the move :-(

systemd is hard to understand, hard to debug (ok jouis it so darmn complicated !) and doesn’t understand the chroot approach as I wanted to do it here. I had to add the trick around the copy of /var/run pid file to make it happy. ANd even with that, when I restart the squid process with systemd very often it fails, leave some processes. In any case I do not use restart anymore, but just stop, then start, in order to minimize issues. SysVinit wasn’t having all these problems. Which may lead me to consider LXC or that type of setup after all.

Squidguard is automatically updated in this conf with the latest content from the University of Toulouse which does a great job to propose their conf files.

Finally I use a tool I developed to maintain all this. casparbuster is my small distribution tool, to propagate the various conf files that are still needed on the system. Now that ssh is up and running, with an account able to use it and become root, then I use an SVN controlled environment in order to store and manage all the relevant conf files (such as my shorewall files e.g.) and I can very easily distribute them to my target systems. I just do cb -m firewall, and voila, all my files are there, process relaunched, and system ready to work ! But enough for this article, very long already, these details are left for a new one I hope to write soon.

Most of that was part of my Christmas activities, and I now have a new shiny low power machine (but still powerfull) managing our security and Internet access. Took time, but happy with the results !

md2mb.pl improved to better support migration from Kmail to Thunderbird

2012/04/11

It’s just incredible how the Open Source world works. You could never believe it before being yourself part of it.

It started with a simple poor small script I wrote for myself to support the migration of my kids to Thunderbird from KMail. And I blogged about it, as I thought it could be interesting as I didn’t found myself a working tool to do it.

It turned out it was the most read article of this blog ! Grumph ! I thought people could be interested by MondoRecue or Continuous Packaging, my bad ;-)

Of course, when I have something to do, I prefer to write a script. Because old guy like me know perfectly you’ll have to do the stuff twice, and thus a script is useful !! And it was prooved again when I migrated my last kid with it again. And the script was improved at the same time. But I knew there were still some flaws, as one colleague found it as well, and had an issue with it (some mails not seen correctly even if imported in subfolders) !

So last month I received feedback from another user, who took that script and improved it greatly, fixing btw the above mentioned problem. And so that small script has now as many contributors as Mondorescue and twice as many as project-builder.org.
Edward Baudrez did a great job to improve the script, adding lots of CLI options capabilities to make it more versatile, fixing mail import in subfolders, error message handling, adding POD doc, …

I still have one or two improvements I’ll discuss with him to try making it even better, but that’s already quite an interesting piece of code !

So as it’s now even more useful than before, those interested by that migration should download and test it in their environment, and continue to send patches ;-) and make me more mazaed by the power of our community, for large as well as small stuff !

CDDBeditor

2011/01/30

I’ve been upset these last monthes by the fact that my usual applications which were allowing me to publish on FreeDB CDs content were not working anymore. I was using kscd from the KDE project, but the latest version with KDE 4.x is just not interesting anymore, with that feature having disappeared.

Same for audex which doesn’t provide it. Grip is Gnome based and doesn’t allow for category edition.

Too bad as it’s an area where a graphical tool is very interesting in general.

So, I decided to write a small tool to help me doing that. So here is CDDBeditor. Not a nice and fancy tool. But it provides to me what I need: CDDB data edition and re-send by mail.

It’s perl based (always a good way to lear), based on the CDDB_get CPAN module for the CDDB features, and also uses Newt, as a way to learn how to use this environment. It could be very seful for future versions of project-builder.org or even MondoRescue.

I use it since August 2009, and to my great surprise, I remarked that I was among the best contributors to Freedb.org with it !! More over, as I’m alone to use it (well I think so at least ;-)) I now know I created 274 entries in FreeDB last year with it. Still far from all the CD for which I got an entry already entered previously by others in FreeDB,but my small stone to the whole wall.

Migrating from KMail to Thunderbird

2011/01/10

This week-end, after passing too much time to try to understand contact management in KMail (which was working quite well in KDE 3.x but is completely flawed IMHO in KDE 4.x), I decided with my wife it was time after 8+ years to try another mail reader.

Thunderbird was of course the first candidate, due to the familiarity she had with Firefox, Outlook (at school) and the fact it’s not that different from KMail anyway in its behavior and interface. I must say I was impressed to see my wife just say that she didn’t mind changing Mail reader, in order to get lacking features. Probably her familiarity with Open Source has also created an open mind for computing programs, and the fact she already uses multiple tools without major issue, makes her willing to try new ones.

So I started to work migrating her data from KMail to TB. I found a small program in order to help md2mb.pl. However, it was far from being complete, and I decided to rewrite a new one, based on the ideas from the other one, in order to have a better migration support. So here is a new md2mb.pl which hopefully will be useful for others as well.


#!/usr/bin/perl -w
#
# Program to import a maildir kmail environement into a thunderbird one.
# (c) Bruno Cornec under the GPLv2.

use strict;
use File::Find;
use File::Copy;
use File::Basename;
use File::Path;

my $cmd="formail";
# CHANGE AS YOU WISH
my $oldroot = "/beatrice/.Mail.sav";
my $newroot = "/beatrice/.thunderbird/30cq2rn3.default/Mail/Local Folders/";
# Is the newroot a file (1) or a dir (0)
my $nrisfile = 0;
my $debug = 0;
# END CHANGE

$debug++ if ((defined $ARGV[0]) && ($ARGV[0] eq "-v"));
print "DEBUG MODE, not doing anything, just printing\n" if ($debug);
if ($debug) {
print "CMD1: mkdir -p $newroot\n" if ((not -d "$newroot") && (not $nrisfile));
} else {
mkpath("$newroot",0, 0755) if ((not -d "$newroot") && (not $nrisfile));
}
system("$cmd /dev/null 2>/dev/null") == 0 or die "cannot find formail on your \$PATH!\nAborting";

find(\&md2mb,($oldroot));

sub md2mb {

if (-f $File::Find::name) {
return if (
($File::Find::name =~ /\.ids$/) ||
($File::Find::name =~ /\.sorted$/) ||
($File::Find::name =~ /\.index$/) ||
($File::Find::name =~ /\/cur\//) ||
($File::Find::name =~ /\/new\//) ||
($File::Find::name =~ /\/tmp\//));
}
if (-d $File::Find::name) {
return if (
($File::Find::name =~ /\/cur$/) ||
($File::Find::name =~ /\/new$/) ||
($File::Find::name =~ /\/tmp$/));
}
if ($debug) {
print "CURR: $File::Find::name\n";
}
my $destname = $File::Find::name;
$destname =~ s|^$oldroot||;
$destname =~ s|\.([[:alnum:]éèçàù\s]*)\.directory|$1.sbd|g;
if ($debug) {
print "DEST: $destname\n";
}
my $cdir = dirname("$newroot/$destname");
my $outputfile="$newroot/$destname";
$outputfile="$newroot" if ($destname =~ /^\s*$/);
if (-d $File::Find::name) {

my @files = (,);

if (@files) {
if ($debug) {
print "CMD2: mkdir -p $cdir\n" if (not -d "$cdir");
} else {
mkpath("$cdir",0, 0755) if (not -d "$cdir");
}
}
foreach my $file (@files) {
next unless -f $file; # skip non-regular files
next unless -s $file; # skip empty files
next unless -r $file; # skip unreadable files
$file =~ s/'/'"'"'/; # escape ' (single quote)
# NOTE! The output file must not contain single quotes (')!
my $run = "cat '$file' | $cmd >> '$outputfile'";
if ($debug) {
print "CMD3: $run\n";
} else {
print "Copying maildir content from $File::Find::name to $outputfile\n";
system($run) == 0 or warn "cannot run \"$run\".";
}
}
}
if (-f $File::Find::name) {
if ($debug) {
print "CMD2: mkdir -p $cdir\n" if (not -d "$cdir");
print "CMD3: cp $File::Find::name $cdir\n";
} else {
mkpath("$cdir",0, 0755) if (not -d "$cdir");
copy($File::Find::name,$cdir);
print "Copying mailbox content from $File::Find::name to $outputfile\n";
}
}
}

Warning: this code doesn’t work from the blog as wordpress makes some modifications, so download it from here

Run it first with a -v option in order to see what will be done. Be careful as I’ve just fixed the problems I had on my side, but probably not all of them. Especially, take care of file names with “‘” in them. they won’t work well.

So migration has been done, and of course more re-organization of the mailboxes. But now everything works fine, including contact management (that were imported using the LDIF exchange format).

And my wife has already sent a couple of mails, so I guess it will not take her long to be very familiar with TB.

Once again this shows all the power of FLOSS: freedom of choice, no info hidden, open format, and voilà.

Loving trac, hating spammers

2009/08/21

I love the tool called trac which allow development teams to easily follow the life of their project, manege Bug Report and Enhancement requests, wiki, link to SVN, ….

But I hate spammers who are stupid enough to think that creating a bunch of iki pages on my trac will help them sell more.

I tried to find an easy way to avoid them doing that. In trac 0.10 it doesn’t seem so easy. So I removed wiki page creation, even for authenticated person. They will have to ask me to create the page first.

And then I wrote that smal script to remove all the pages from trac. Feel free to adapt and reuse. Fight against spammers is a good one !

#!/bin/bash
# Provided under the GPL v2
#
if [ "$1" = "-f" ]; then
        doit=true
else
        doit=false
fi

# replace with your list of trac DB
for pj in /mondo/*/trac/* ; do
        for p in `trac-admin $pj wiki list | 
            awk '{print $1}' | 
            grep -E '_download$|_buy_|_buy$'`; do
                echo "Deleting $p..."
                if [ $doit = "true" ]; then
                        trac-admin $pj wiki  remove $p
                fi
        done
done

Launch it first to see what it will do. Then relaunch with -f to see the spam wiki pages disappear !
I hate spammers.

But now I love my new shiny trac again ;-


Follow

Get every new post delivered to your Inbox.

Join 100 other followers