G-Loaded Journal

Software that detects violations of open source licenses

George Notaras — Fri, 16 Apr 2010 14:57:24 +0000

The rules make the game. You take out the rules, the game goes up in smoke. I think it’s still fine if someone takes advantage of any inconsistencies between the rules to win the game, but cheating is completely unacceptable.

In the free software world, it’s neither the maker of the software nor the market, via supply and demand, that set the rules of the game, but software licenses. That’s why today I was excited to learn about the existence of a software that can determine whether code, that has been released under a free/open-source license, has been used in another software by performing binary analysis, examination of files in binary form. The excitement is not because of some sort of personal benefit from this software. It’s just good to know that it is possible to develop tools that can defend open-source code, even if they have to analyze the derived binary file. At the moment the tool supports:

Automated extraction of the version and configuration of BusyBox
Extraction of file systems
Automated checking for the Linux kernel
Brute force scanning of firmware
Feeding known information through a knowledgebase

This means that, currently, it can detect the kernel and busybox in the firmware of electronic devices. I have no clue about its effectiveness and, possibly, I won’t be trying it any time soon, but, if you are interested, make sure you read more information about the features of the Binary Analysis Tool and check out the code.

Software that detects violations of open source licenses, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License. Terms and conditions beyond the scope of this license may be available at www.g-loaded.eu.

Some Preliminary Redmine Customizations

George Notaras — Wed, 14 Apr 2010 15:55:33 +0000

I’ve been using Redmine for several months now. During this period of time I had the chance to evaluate most of its features. Although I still consider it one of the most feature-rich project management web applications currently available, using it on a production website revealed some weaknesses in the anti-spam area. Truth is the application lacks anti-spam features, but, on the other hand, such features most probably belong to the plugin area instead of the application core, so noone can put the blame on the core developers for not making the app spam-proof. Apart from this, I also realized that some of its templates need some improvement in some cases. Time permitting, I’ll be working on these things in the upcoming weeks and release the customizations in the form of patches that can be applied directly on the latest official release of Redmine.

Some Preliminary Redmine Customizations, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License. Terms and conditions beyond the scope of this license may be available at www.g-loaded.eu.

PHP Interactive Interpreter

George Notaras — Tue, 13 Apr 2010 13:16:42 +0000

The PHP interpreter supports running it in interactive mode by using the --interactive (short equivalent: -a) command-line switch. Running an interactive PHP shell can be useful when you need to quickly try code snippets. But, for this mode to be fully functional, PHP has to be compiled with readline support. Unfortunately, on CentOS PHP has not been compiled this way.

You can check the PHP compile-time configuration switches, by invoking the following command:

php --info | grep -i 'configure command'

While searching for a solution about this, I ran across an interesting project, phpa. From the developer:

phpa is an interactive command line shell for PHP.

It is a replacement for the interactive mode of the PHP interpreter (php -a), hence the name.

This software is public domain.

Before using it, make sure php-readline is installed:

yum install php-readline

I downloaded phpa and set the executable bit as shown below:

wget -O - http://david.acz.org/phpa/phpa.txt > ~/bin/phpa
chmod +x ~/bin/phpa

I also had to fix the bad shebang:

sed -i 's#/usr/local/bin/php#/usr/bin/php#' ~/bin/phpa

Finally, an interactive PHP shell! Now running code snippets from the PHP manual is fun:

[rocky@arena ~]$ phpa
PHP 5.1.6 (cli) (Jan 13 2010 17:05:38) [Linux]
>>> $arr = array("somearray" => array(6 => 5, 13 => 9, "a" => 42));
>>>
>>> echo $arr["somearray"][6];    // 5
5
>>> echo $arr["somearray"][13];   // 9
9
>>> echo $arr["somearray"]["a"];  // 42
42
>>>

Enjoy your new PHP interactive interpreter!

PHP Interactive Interpreter, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License. Terms and conditions beyond the scope of this license may be available at www.g-loaded.eu.

Using a switch to prevent system shutdown/reboot/suspend

A change of plans regarding a web-based VCS manager

George Notaras — Mon, 12 Apr 2010 16:06:41 +0000

About a month ago, I had finally decided to start writing a simple web-based VCS manager. This is a project I had been thinking about for the last 2 years as a solution for source-code management at CodeTRAX.org, but I never really started development. The initial plan included support (repository creation only) for the most popular open-source version control systems, like subversion, mercurial, git, and bazaar. The VCS manager, actually a Django application, should be able to create source code repositories and manage user permissions on a per repository basis. It would also include an authentication/authorization backend for httpd, compatible with mod_wsgi‘s access control mechanisms, in order to provide controlled access to the repositories over HTTP or HTTPS. During the last days, after exploring several of the Mercurial features, I am quite certain that Mercurial includes all the necessary features to meet any requirement of CodeTRAX. So, I seriously consider dropping support for any other VCS except hg and try to make the VCS manager more hg specific. Now I realize that this should be the approach from the beginning. The needs are quite specific, so supporting several version control systems was completely unnecessary.

A change of plans regarding a web-based VCS manager, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License. Terms and conditions beyond the scope of this license may be available at www.g-loaded.eu.

Started using rdiff-backup on Windows

George Notaras — Thu, 08 Apr 2010 15:08:14 +0000

rdiff-backup is my favorite tool for keeping backups on Linux systems. Today, I started using it on a Windows 7 desktop as well. At first, I used rdiff-backup from the cygwin project, but I soon noticed that, if I did anything else that required moderate hard disk access on the computer on which the cygwin rdiff-backup version was already running, my system suffered from random lockups. For example, if I tried to open Thunderbird, my computer froze for several minutes. I finally decided to try the native Windows release of rdiff-backup and all this strange behavior is now gone. My computer experiences no lock-ups and is usable while rdiff-backup is backing up some subdirectories of my homedir. I am not really sure what is the problem with the cygwin version. However, I intend to investigate this when I have the time, because I use cygwin for several tasks and I’d like to know about any possible incompatibility issues with current Windows versions.

Started using rdiff-backup on Windows, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License. Terms and conditions beyond the scope of this license may be available at www.g-loaded.eu.

From Subversion to Mercurial

George Notaras — Thu, 08 Apr 2010 12:33:59 +0000

I’ve been using the Subversion version control system during the last 2-3 years. Although a VCS has never been a top priority for me, it seems there is always room for version control. I ended up using it for almost everything, from simple to more complex scripts and even system configuration files in some cases. A few weeks ago I decided to switch to Mercurial. I had read about Mercurial for the first time in some mailing list messages written by Giorgos Keramidas and also in several posts about Mercurial on his blog, but it was only a few months ago when I started experimenting with it for real. Although I use many of its features, my general use of version control systems is rather basic, so do not expect any sophisticated reasons why I have decided to switch:

It’s open-source software, written in Python. This is very important for me as there is not much room in my head right now for other programming languages.
Easily extensible. If I ever need a special feature I will have the opportunity to implement it by writing a Python extension.
I got rid of those .svn directories.
I can use mod_wsgi and all its good features to provide repository access over HTTP/HTTPS.
I find the concept of a “distributed version control system” to be real progress in the VCS area.

Truth is I will not stop using Subversion, but I will definitely choose Mercurial instead of SVN for any new project I start.

From Subversion to Mercurial, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License. Terms and conditions beyond the scope of this license may be available at www.g-loaded.eu.

mod_wsgi incompatible with mod_python

George Notaras — Wed, 07 Apr 2010 09:00:58 +0000

This is a quick note that mod_wsgi daemon processes are now incompatible with mod_python. Upgraded mod_wsgi earlier today in CentOS and saw the following note. I guess this limitation has been around for a while before affecting CentOS/RHEL:

#################################################################################
# mod_python and mod_wsgi compatibility note
#################################################################################
# mod_wsgi will deadlock if run in daemon mode while mod_python is enabled
# do not enable both mod_python and mod_wsgi if you are going to use the
# WSGIDaemonProcess directive
# In previous version of mod_wsgi, apache would segfault when both mod_wsgi
# and mod_python were enabled.  This update does not guarantee that will not
# happen.
#################################################################################
# Do not enable mod_python and mod_wsgi in the same apache process.
#################################################################################

Keep this in mind if you plan to mix both modules under the same Apache instance.

Update: As Graham pointed out in this comment, there is no issue if mod_python 3.3.1 is used. That’s great news because I would miss the flexibility of mod_python when it comes to request processing.

mod_wsgi incompatible with mod_python, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License. Terms and conditions beyond the scope of this license may be available at www.g-loaded.eu.

Permanently remove deleted posts and topics in bbPress

George Notaras — Wed, 07 Apr 2010 06:51:50 +0000

Permanently removing deleted posts and topics in bbPress can be cumbersome. bbPress does not provide a decent way to completely remove deleted content from the database. I am not sure why… In my case, these posts mostly contain spam and there is absolutely no reason to keep them in the database any more. As it usually happens when I try to administer bbPress, I ended up executing SQL queries at the MySQL prompt. I hereby publish these SQL statements.

First delete all posts in deleted topics:

DELETE FROM bb_posts WHERE post_id IN (
    SELECT post_id FROM (
        SELECT DISTINCT(p.post_id)
        FROM bb_posts AS p LEFT JOIN bb_topics AS t ON p.topic_id=t.topic_id
        WHERE t.topic_status=1
    ) AS bb_posts_in_deleted_topics
);

Then remove all deleted posts:

DELETE FROM bb_posts WHERE post_status=1;

Finally, remove all deleted topics:

DELETE FROM bb_topics WHERE topic_status=1;

In the bbPress administration panel, you can now use the “Tools” section to recount posts.

Permanently remove deleted posts and topics in bbPress, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License. Terms and conditions beyond the scope of this license may be available at www.g-loaded.eu.

Partition Misalignment Slows Down 4096-Byte Sector Hard Disks

George Notaras — Sun, 28 Mar 2010 23:32:01 +0000

I just read this very interesting article about the new 4096-byte sector hard disks, like the new Western Digital Caviar Green drives, and the impact a partition misalignment might have on the drive’s write performance.

The problem most likely to hit you with one of these drives is very slow write performance. This is caused by improper logical-to-physical sector alignment. OS’s like Linux use 4K blocks (or multiples of 4K) to store data, which matches well with the physical sector. However, nothing restricts you from creating a partition that starts on an odd-numbered 512-byte logical sector. This misalignment causes a performance hit since the drive has to read and rewrite the 4K sectors with whatever 512-byte slices changed.

The article author continues with a test he performed on two identical WD Green drives containing one partition. On the first drive he used fdisk defaults for the partition boundaries, while on the second drive he manually aligned the partition. The results show that a misaligned drive can be up to 3 times slower than a properly aligned one.

…I formatted both drives using the command “time mke2fs /dev/sdc1″ (and sdd1).

/dev/sdc, which was aligned, took 5m 45.716s to format.
/dev/sdd, which was not aligned, took 19m 53.609s to format.

That’s a difference of greater than a factor of three!

The article author conducted several other tests. This is a very good read. Read the full article.

Partition Misalignment Slows Down 4096-Byte Sector Hard Disks, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License. Terms and conditions beyond the scope of this license may be available at www.g-loaded.eu.

Script for Apache Error Report

George Notaras — Sun, 28 Mar 2010 17:26:47 +0000

The last incident with the php-cgi errors as a result of a bad PHP script made me re-evaluate the daily reports I receive from the server. I realized that a report about the httpd errors that have occured during the previous day, including all virtualhosts, is more important than I had initially thought. Such a report would have brought the problem to my attention much earlier and could also point me to the right direction while I was trying to figure out what was the cause of it. Fortunately, I found a nice error report generator for this purpose.

The script is called scanerrlog and has some quite nice features. I highly recommend you check it out. In order to run it you will also need an extra Python module, jaxml. Download both packages and extract scanerrlog.py and jaxml.py to the same directory. Then you can call scanerrlog.py using the following cronjob.

#! /usr/bin/env bash
#
# Error Report for httpd
#
# Requires:
#   - scanerrlog - http://www.librelogiciel.com/software/ScanErrLog/action_Presentation
#   - jaxml - http://www.librelogiciel.com/software/jaxml/action_Presentation
#
 
cat /var/log/httpd/error_log \
    /var/www/vhosts/*/log/error_log \
    | python /usr/local/bin/scanerrlog.py -f text \
    | /bin/mail -s "`hostname` - Apache Error Report" root@localhost
 
exit 0

Some notes:

Make sure that this cronjob runs before log rotation takes place.
The above shell script concatenates the error_log files from the main web server and all virtualhosts. Edit the paths to reflect your server configuration and virtualhost layout.
Edit the path to scanerrlog.py.

Scanerrlog has some quite nice features that let you further customize the report. Make sure you read the help message.

Script for Apache Error Report, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License. Terms and conditions beyond the scope of this license may be available at www.g-loaded.eu.

G-Loaded Journal

Software that detects violations of open source licenses

Related Articles

Some Preliminary Redmine Customizations

Related Articles

PHP Interactive Interpreter

Related Articles

A change of plans regarding a web-based VCS manager

Related Articles

Started using rdiff-backup on Windows

Related Articles

From Subversion to Mercurial

Related Articles

mod_wsgi incompatible with mod_python

Related Articles

Permanently remove deleted posts and topics in bbPress

Related Articles

Partition Misalignment Slows Down 4096-Byte Sector Hard Disks

Related Articles

Script for Apache Error Report

Related Articles