• export the documentation into multiple HTML files using make html and then send the relevant html files one-by-one to the printer, which is insanely inconvenient, or
• export the documentation to LaTeX format and then use a tex-to-pdf converter in order to merge the documentation files into a single file, which I would finally send to the printer. It should be noted that in this case, the tex-to-pdf converter had some other dependencies which summed up to 130+MB. Compressed. That’s insanely too much for converting the documentation to single-file form!!!

So, I ended up using grep to scan the sphinx source code for expressions like single, single file etc. I soon hit singlehtml! Fortunately, there is a builder, named singlehtml, that produces a single HTML document for documentation that is split into multiple files. Hooray!!

I used the sphinx-build utility directly to generate a single HTML document containing the entire documentation. After changing to the documentation’s root directory, I issued the command:

sphinx-build -b singlehtml . zzz

I finally had a single HTML file at zzz/index.html, which I sent to the printer and got some nice documentation in-print.

Related Articles

• Using a CUPS printer from command line
• Print to CUPS printer instances
• Print a Man Page
• Effective data wiping with a single complete overwrite
• Another way to create a text file

Update: Thanks to sharkic‘s feedback, this guide has now been improved by providing complete instructions on how to use wget and curl with the Rapidshare API. See the new sections at the end of the article.

The python implementation of a Rapidshare downloader

Instead of writing a wrapper script around wget or curl, I decided to go ahead with a pure Python Rapishare downloader, which works with both free and pro accounts. The script is called rsapiget.

Here is the code:

#! /usr/bin/env python
# -*- coding: utf-8 -*-
#
#  rsapiget - A simple command-line downloader that supports the Rapidshare API.
#
#  Homepage: http://www.g-loaded.eu/rsapiget-download-rapidshare-api/
#
#  Copyright (c) 2010 George Notaras, G-Loaded.eu, CodeTRAX.org
#
#  Licensed under the Apache License, Version 2.0 (the "License");
#  you may not use this file except in compliance with the License.
#  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.
#
 
# Configuration BEGIN
LOGIN = ''
PASSWORD = ''
USE_SSL = False
VERIFY_MD5SUM = False
# Configuration END
 
__version__ = '0.1.0'
 
import sys
import os
import urllib
import subprocess
import time
try:
    import hashlib
    md5 = hashlib.md5
except ImportError:
    import md5
    md5 = md5.new
 
def info(msg):
    sys.stdout.write('%s\n' % msg)
    sys.stdout.flush()
 
def error(msg):
    sys.stderr.write('%s\n' % msg)
    sys.stderr.flush()
    sys.exit(1)
 
def transfer_progress(blocks_transfered, block_size, file_size):
    percent = float((blocks_transfered * block_size * 100) / file_size)
    progress = float(blocks_transfered * block_size / 1024)
    downspeed = (float(blocks_transfered * block_size) / float(time.time() - starttime)) / 1024
    sys.stdout.write("Complete: %.0f%% - Downloaded: %.2fKb - Speed: %.3fkb/s\r" % (percent, progress, downspeed))
    sys.stdout.flush()
 
def download(source, target):
    global starttime
    starttime = time.time()
    filename, headers = urllib.urlretrieve(source, target, transfer_progress)
    sys.stdout.write('Complete: 100%\n')
    sys.stdout.flush()
    for ss in headers:
        if ss.lower() == "content-disposition":
            filename = headers[ss][headers[ss].find("filename=") + 9:]  # 9 is len("filename=")=9
    urllib.urlcleanup()     # Clear the cache
    return filename
 
def verify_file(remote_md5sum, filename):
    f = open(filename, "rb")
    m = md5()
    while True:
        block = f.read(32384)
        if not block:
            break
        m.update(block)
    md5sum = m.hexdigest()
    f.close()
    return md5sum == remote_md5sum
 
def main():
    if len(sys.argv) != 2:
        error('Need Rapidshare link as argument')
 
    file_link = sys.argv[1]
 
    try:
        rapidshare_com, files, fileid, filename = file_link.rsplit('/')[-4:]
    except ValueError:
        error('Invalid Rapidshare link')
    if not rapidshare_com.endswith('rapidshare.com') or files != 'files':
        error('Invalid Rapidshare link')
 
    if USE_SSL:
        proto = 'https'
        info('SSL is enabled')
    else:
        proto = 'http'
 
    if VERIFY_MD5SUM:
        info('MD5 sum verification is enabled')
 
    info('Downloading: %s' % file_link)
 
    if filename.endswith('.html'):
        target_filename = filename[:-5]
    else:
        target_filename = filename
    info('Save file as: %s' % target_filename)
 
    # API parameters
 
    params = {
        'sub': 'download_v1',
        'fileid': fileid,
        'filename': filename,
        'try': '1',
        'withmd5hex': '0',
        }
 
    if VERIFY_MD5SUM:
        params.update({
            'withmd5hex': '1',
            })
 
    if LOGIN and PASSWORD:
        params.update({
            'login': LOGIN,
            'password': PASSWORD,
            })
 
    params_string = urllib.urlencode(params)
 
    api_url = '%s://api.rapidshare.com/cgi-bin/rsapi.cgi' % proto
 
    # Get the first error response
    conn = urllib.urlopen('%s?%s' % (api_url, params_string))
    data = conn.read()
    #print data
    conn.close()
 
    # Parse response
    try:
        key, value = data.split(':')
    except ValueError:
        error(data)
    try:
        server, dlauth, countdown, remote_md5sum = value.split(',')
    except ValueError:
        error(data)
 
    # Wait for n seconds (free accounts only)
    if int(countdown):
        for t in range(int(countdown), 0, -1):
            sys.stdout.write('Waiting for %s seconds...\r' % t)
            sys.stdout.flush()
            time.sleep(1)
        info('Waited for %s seconds. Proceeding with file download...' % countdown)
 
    # API parameters for file download
 
    dl_params = {
        'sub': 'download_v1',
        'fileid': fileid,
        'filename': filename,
        }
 
    if LOGIN and PASSWORD:
        dl_params.update({
            'login': LOGIN,
            'password': PASSWORD,
            })
    else:
        dl_params.update({
            'dlauth': dlauth,
            })
 
    dl_params_string = urllib.urlencode(dl_params)
 
    download_link = '%s://%s/cgi-bin/rsapi.cgi?%s' % (proto, server, dl_params_string)
 
    downloaded_filename = download(download_link, target_filename)
 
    if VERIFY_MD5SUM:
        if remote_md5sum.lower() == 'not found':
            info('Remote MD5 sum is not available. Skipping MD5 sum verification...')
        elif downloaded_filename:
            if verify_file(remote_md5sum.lower(), downloaded_filename):
                info('Downloaded and verified %s' % downloaded_filename)
            else:
                error('The downloaded file could not be verified')
        else:
            error('Will not verify. File not found: %s' % downloaded_filename)
 
    info('Operation Complete')
 
if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        error('\nAborted')

Save the code in a file called: rsapiget.py

Usage is very simple:

python rsapiget.py <rapidshare_link>

There are some configuration options at the top of the script you may need to check out:

• LOGIN, PASSWORD: If you are a registered Pro user, set your username and password here. Being a pro user you will never have to wait for the download to start. Otherwise leave blank.
• USE_SSL: Set to True to force the client to communicate with the rapidshare servers over an encrypted connection. Note that, according to the docs, this is more expensive in terms of Rapidshare points, so it is disabled by default.
• VERIFY_MD5SUM: If this is set to True, the downloaded file’s integrity will be verified. The docs say that this results in more API calls than not using md5 verification, so this is disabled by default as well.

Although the old article has a small download server implementation in BASH, I haven’t tested whether the latter works with this client or not.

Please note that this script is work in progress and I might update the code in the following days. So, check back often for updates.

Download from Rapidshare API using wget

All credit for this method goes to sharkic (see comments).

I admit that when I was checking out the API, I had completely overlooked the withcookie option of the getaccountdetails_v1 subroutine. Also, I was not aware that it is now possible for free users to have an account with Rapidshare.

So, to sum sharkic‘s feedback up, here is how it is done. The following information requires that you have signed up with Rapidshare. Of course downloading files using wget and the following instructions requires a Rapidshare Pro account.

First, save the cookie data. This has to be done once:

wget -q -O - \
    --post-data="sub=getaccountdetails_v1&withcookie=1&login=LOGIN&password=PASSWORD" \
    https://api.rapidshare.com/cgi-bin/rsapi.cgi \
    | grep cookie | cut -d '=' -f 2 > .rapidshare_cookie

Substitute LOGIN and PASSWORD with your Rapidshare account’s username and password.

Now, you can download files using:

wget --no-cookies --header="Cookie: enc=`cat .rapidshare_cookie`" http://rapidshare.com/files/XXXXXXXX/test.zip

Download from Rapidshare API using curl

The curl method is a derivative of the wget method. First save the cookie with:

curl --data "sub=getaccountdetails_v1&withcookie=1&login=LOGIN&password=PASSWORD" \
    https://api.rapidshare.com/cgi-bin/rsapi.cgi \
    | grep cookie | cut -d '=' -f 2 > .rapidshare_cookie

Substitute LOGIN and PASSWORD with your Rapidshare account’s username and password.

Now, you can download files using:

curl -L -O --cookie "enc=`cat .rapidshare_cookie`" http://rapidshare.com/files/XXXXXXXX/test.zip

Enjoy! Your feedback is welcome.

Related Articles

• Use wget or curl to download from RapidShare Premium
• VeriTAR – Verify checksums of files within a TAR archive
• Upgraded to WordPress 2.3.1
• Using the mod_dav_svn SVNParentPath directive with multiple authz files
• Search for a string in multiple files

But first, let’s install some dependencies. I assume that you will try the code on a machine aimed for testing, so I won’t be using the system’s package manager but instead use easy_install to install the needed Python modules. Other commands like pip could be used as well. If you do not want to mess with your system-wide python installation, just use virtualenv and pip, but I won’t go into the details about how to use these tools in the current document.

Install the needed dependencies:

easy_install pycrypto
easy_install pyasn1
easy_install pam
easy_install twisted

Make sure you have gcc installed, because it will be needed to complete the installation of Twisted and PyCrypto.

I have created a project for this code at Bitbucket, called RapidSSH. This will make it easier to share code and ideas about SSH server implementations in Python using Twisted.

Filename: scripts/rapidsshd_unix.py

#!/usr/bin/env python
#
# This file is part of rapidssh - http://bitbucket.org/gnotaras/rapidssh/
#
# rapidssh - A set of Secure Shell (SSH) server implementations in Python
#            using Twisted.conch, part of the Twisted Framework.
#
# Copyright (c) 2010 George Notaras - http://www.g-loaded.eu
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#
# Initially based on the sshsimpleserver.py kindly published by:
# Twisted Matrix Laboratories - http://twistedmatrix.com
#
 
import sys
import pam
 
from twisted.conch.unix import UnixSSHRealm
from twisted.cred import portal
from twisted.cred.credentials import IUsernamePassword
from twisted.cred.checkers import ICredentialsChecker
from twisted.cred.error import UnauthorizedLogin
from twisted.conch.checkers import SSHPublicKeyDatabase
from twisted.conch.ssh import factory, userauth, connection, keys, session
from twisted.internet import reactor, defer
from zope.interface import implements
from twisted.python import log
 
# Logging
# Currently logging to STDERR
log.startLogging(sys.stderr)
 
# Server-side public and private keys. These are the keys found in
# sshsimpleserver.py. Make sure you generate your own using ssh-keygen!
 
publicKey = 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAGEArzJx8OYOnJmzf4tfBEvLi8DVPrJ3/c9k2I/Az64fxjHf9imyRJbixtQhlH9lfNjUIx+4LmrJH5QNRsFporcHDKOTwTTYLh5KmRpslkYHRivcJSkbh/C+BR3utDS555mV'
 
privateKey = """-----BEGIN RSA PRIVATE KEY-----
MIIByAIBAAJhAK8ycfDmDpyZs3+LXwRLy4vA1T6yd/3PZNiPwM+uH8Yx3/YpskSW
4sbUIZR/ZXzY1CMfuC5qyR+UDUbBaaK3Bwyjk8E02C4eSpkabJZGB0Yr3CUpG4fw
vgUd7rQ0ueeZlQIBIwJgbh+1VZfr7WftK5lu7MHtqE1S1vPWZQYE3+VUn8yJADyb
Z4fsZaCrzW9lkIqXkE3GIY+ojdhZhkO1gbG0118sIgphwSWKRxK0mvh6ERxKqIt1
xJEJO74EykXZV4oNJ8sjAjEA3J9r2ZghVhGN6V8DnQrTk24Td0E8hU8AcP0FVP+8
PQm/g/aXf2QQkQT+omdHVEJrAjEAy0pL0EBH6EVS98evDCBtQw22OZT52qXlAwZ2
gyTriKFVoqjeEjt3SZKKqXHSApP/AjBLpF99zcJJZRq2abgYlf9lv1chkrWqDHUu
DZttmYJeEfiFBBavVYIF1dOlZT0G8jMCMBc7sOSZodFnAiryP+Qg9otSBjJ3bQML
pSTqy7c3a2AScC/YyOwkDaICHnnD3XyjMwIxALRzl0tQEKMXs6hH8ToUdlLROCrP
EhQ0wahUTCk1gKA4uPD6TMTChavbh4K63OvbKg==
-----END RSA PRIVATE KEY-----"""
 
 
class PamPasswordDatabase:
    """Authentication/authorization backend using the 'login' PAM service"""
    credentialInterfaces = IUsernamePassword,
    implements(ICredentialsChecker)
 
    def requestAvatarId(self, credentials):
        if pam.authenticate(credentials.username, credentials.password):
            return defer.succeed(credentials.username)
        return defer.fail(UnauthorizedLogin("invalid password"))
 
 
class UnixSSHdFactory(factory.SSHFactory):
    publicKeys = {
        'ssh-rsa': keys.Key.fromString(data=publicKey)
    }
    privateKeys = {
        'ssh-rsa': keys.Key.fromString(data=privateKey)
    }
    services = {
        'ssh-userauth': userauth.SSHUserAuthServer,
        'ssh-connection': connection.SSHConnection
    }
 
# Components have already been registered in twisted.conch.unix
 
portal = portal.Portal(UnixSSHRealm())
portal.registerChecker(PamPasswordDatabase())   # Supports PAM
portal.registerChecker(SSHPublicKeyDatabase())  # Supports PKI
UnixSSHdFactory.portal = portal
 
if __name__ == '__main__':
    reactor.listenTCP(5022, UnixSSHdFactory())
    reactor.run()

Some notes:

• The server uses the very same public and private keys as found in the sshsimpleserver.py script. Make sure you generate new keys using ssh-keygen if you plan to run this on a public server (although this not recommended).
• Twisted included a module for PAM authentication, twisted.cred.pamauth, but unfortunately I could not locate one of its dependencies to make it work. So, I used the excellent python-pam to create a custom PAM authenticator class.
• I can guess what you might thought when you read about this code:
  

  Python is a cross-platform programming language. Twisted runs on Windows. So, is this a solution for running a SSH server on Win32?

  Well, at the moment it won’t run because internally it uses some Python modules that are available on UNIX platforms only. But, I intend to investigate the possibility of running this on Windows since I already need something like that.

We can now enjoy our Python SSH server. Run as root:

python scripts/rapidsshd_unix.py

From another machine, connect to the server using an ssh client:

ssh -p 5022 rocky@arena

If you have deployed your public key in the ~/.ssh/authorized_keys file on the remote machine (arena), you should be able to authenticate using the public key:

ssh -p 5022 -i /path/to/private.key rocky@arena

You should get output like the following on the server:

[root@arena ~]# python ssh.py
2010-03-26 21:30:05+0000 [-] Log opened.
2010-03-26 21:30:05+0000 [-] __main__.UnixSSHdFactory starting on 5022
2010-03-26 21:30:05+0000 [-] Starting factory <__main__.UnixSSHdFactory instance at 0xb7a0b0cc>
2010-03-26 21:30:13+0000 [__main__.UnixSSHdFactory] disabling diffie-hellman-group-exchange because we cannot find moduli file
2010-03-26 21:30:13+0000 [SSHServerTransport,0,192.168.0.172] kex alg, key alg: diffie-hellman-group1-sha1 ssh-rsa
2010-03-26 21:30:13+0000 [SSHServerTransport,0,192.168.0.172] outgoing: aes256-ctr hmac-sha1 none
2010-03-26 21:30:13+0000 [SSHServerTransport,0,192.168.0.172] incoming: aes256-ctr hmac-sha1 none
2010-03-26 21:30:14+0000 [SSHServerTransport,0,192.168.0.172] NEW KEYS
2010-03-26 21:30:14+0000 [SSHServerTransport,0,192.168.0.172] starting service ssh-userauth
2010-03-26 21:30:17+0000 [SSHService ssh-userauth on SSHServerTransport,0,192.168.0.172] rocky trying auth none
2010-03-26 21:30:17+0000 [SSHService ssh-userauth on SSHServerTransport,0,192.168.0.172] rocky trying auth publickey
2010-03-26 21:30:17+0000 [SSHService ssh-userauth on SSHServerTransport,0,192.168.0.172] rocky trying auth publickey
2010-03-26 21:30:17+0000 [SSHService ssh-userauth on SSHServerTransport,0,192.168.0.172] rocky authenticated with publickey
2010-03-26 21:30:17+0000 [SSHService ssh-userauth on SSHServerTransport,0,192.168.0.172] starting service ssh-connection
2010-03-26 21:30:17+0000 [SSHService ssh-connection on SSHServerTransport,0,192.168.0.172] got channel session request
2010-03-26 21:30:17+0000 [SSHChannel session (0) on SSHService ssh-connection on SSHServerTransport,0,192.168.0.172] channel open
2010-03-26 21:30:17+0000 [SSHChannel session (0) on SSHService ssh-connection on SSHServerTransport,0,192.168.0.172] pty request: xterm (24L, 80L, 0L, 0L)
2010-03-26 21:30:17+0000 [SSHChannel session (0) on SSHService ssh-connection on SSHServerTransport,0,192.168.0.172] getting shell

This first release of the RapidSSH project exists solely for demonstration purposes. Don’t get fooled by the small amount of code. Information about how to put the pieces together is scarce and it required a lot trial&error and source code reading. The above implementation contains none of the customizations I had in mind. These will be done as soon as I find some time to do so. Until then it will be nice to hear about your experiences or implementations you have worked on.

Related Articles

• Use Python to get the web page data in Epiphany
• Epiphany Python Console – Open New Tab
• Python IRC Bot
• Python Crash Course
• Set up the VNC Server in Fedora

[This post has been updated]

[Update #1]: This limitation also exists on the ping command, which runs with setuid access rights. (Thanks Stephane)

[Update #2]: After Chris Hallman reported that the original code actually works on Windows only, I made some changes to it and publish the updated code below.

The following code is a pure Python implementation of the ping command. I originally found it in the source code tree of pylucid in a subdirectory where the developers keep various code snippets.

I had originally tested the code under Microsoft Windows as this was the OS I had available at the moment. Chris Hallman noticed that the code does not work under GNU/Linux. After spending some hours trying to figure out what was wrong it, I realized that the part of the code that caused the failure on Linux was the use of the time.clock() function instead of time.time(). The clock() method works differently under Windows and Linux.

I fixed the 2007 implementation and hereby publish the updated code(as python-ping). Feel free to test it and report any issues.

Needed: test on Solaris.

#!/usr/bin/env python
 
"""
    A pure python ping implementation using raw socket.
 
 
    Note that ICMP messages can only be sent from processes running as root.
 
 
    Derived from ping.c distributed in Linux's netkit. That code is
    copyright (c) 1989 by The Regents of the University of California.
    That code is in turn derived from code written by Mike Muuss of the
    US Army Ballistic Research Laboratory in December, 1983 and
    placed in the public domain. They have my thanks.
 
    Bugs are naturally mine. I'd be glad to hear about them. There are
    certainly word - size dependenceies here.
 
    Copyright (c) Matthew Dixon Cowles, <http://www.visi.com/~mdc/>.
    Distributable under the terms of the GNU General Public License
    version 2. Provided with no warranties of any sort.
 
    Original Version from Matthew Dixon Cowles:
      -> ftp://ftp.visi.com/users/mdc/ping.py
 
    Rewrite by Jens Diemer:
      -> http://www.python-forum.de/post-69122.html#69122
 
    Rewrite by George Notaras:
      -> http://www.g-loaded.eu/2009/10/30/python-ping/
 
    Revision history
    ~~~~~~~~~~~~~~~~
 
    November 8, 2009
    ----------------
    Improved compatibility with GNU/Linux systems.
 
    Fixes by:
     * George Notaras -- http://www.g-loaded.eu
    Reported by:
     * Chris Hallman -- http://cdhallman.blogspot.com
 
    Changes in this release:
     - Re-use time.time() instead of time.clock(). The 2007 implementation
       worked only under Microsoft Windows. Failed on GNU/Linux.
       time.clock() behaves differently under the two OSes[1].
 
    [1] http://docs.python.org/library/time.html#time.clock
 
    May 30, 2007
    ------------
    little rewrite by Jens Diemer:
     -  change socket asterisk import to a normal import
     -  replace time.time() with time.clock()
     -  delete "return None" (or change to "return" only)
     -  in checksum() rename "str" to "source_string"
 
    November 22, 1997
    -----------------
    Initial hack. Doesn't do much, but rather than try to guess
    what features I (or others) will want in the future, I've only
    put in what I need now.
 
    December 16, 1997
    -----------------
    For some reason, the checksum bytes are in the wrong order when
    this is run under Solaris 2.X for SPARC but it works right under
    Linux x86. Since I don't know just what's wrong, I'll swap the
    bytes always and then do an htons().
 
    December 4, 2000
    ----------------
    Changed the struct.pack() calls to pack the checksum and ID as
    unsigned. My thanks to Jerome Poincheval for the fix.
 
 
    Last commit info:
    ~~~~~~~~~~~~~~~~~
    $LastChangedDate: $
    $Rev: $
    $Author: $
"""
 
 
import os, sys, socket, struct, select, time
 
# From /usr/include/linux/icmp.h; your milage may vary.
ICMP_ECHO_REQUEST = 8 # Seems to be the same on Solaris.
 
 
def checksum(source_string):
    """
    I'm not too confident that this is right but testing seems
    to suggest that it gives the same answers as in_cksum in ping.c
    """
    sum = 0
    countTo = (len(source_string)/2)*2
    count = 0
    while count<countTo:
        thisVal = ord(source_string[count + 1])*256 + ord(source_string[count])
        sum = sum + thisVal
        sum = sum & 0xffffffff # Necessary?
        count = count + 2
 
    if countTo<len(source_string):
        sum = sum + ord(source_string[len(source_string) - 1])
        sum = sum & 0xffffffff # Necessary?
 
    sum = (sum >> 16)  +  (sum & 0xffff)
    sum = sum + (sum >> 16)
    answer = ~sum
    answer = answer & 0xffff
 
    # Swap bytes. Bugger me if I know why.
    answer = answer >> 8 | (answer << 8 & 0xff00)
 
    return answer
 
 
def receive_one_ping(my_socket, ID, timeout):
    """
    receive the ping from the socket.
    """
    timeLeft = timeout
    while True:
        startedSelect = time.time()
        whatReady = select.select([my_socket], [], [], timeLeft)
        howLongInSelect = (time.time() - startedSelect)
        if whatReady[0] == []: # Timeout
            return
 
        timeReceived = time.time()
        recPacket, addr = my_socket.recvfrom(1024)
        icmpHeader = recPacket[20:28]
        type, code, checksum, packetID, sequence = struct.unpack(
            "bbHHh", icmpHeader
        )
        if packetID == ID:
            bytesInDouble = struct.calcsize("d")
            timeSent = struct.unpack("d", recPacket[28:28 + bytesInDouble])[0]
            return timeReceived - timeSent
 
        timeLeft = timeLeft - howLongInSelect
        if timeLeft <= 0:
            return
 
 
def send_one_ping(my_socket, dest_addr, ID):
    """
    Send one ping to the given >dest_addr<.
    """
    dest_addr  =  socket.gethostbyname(dest_addr)
 
    # Header is type (8), code (8), checksum (16), id (16), sequence (16)
    my_checksum = 0
 
    # Make a dummy heder with a 0 checksum.
    header = struct.pack("bbHHh", ICMP_ECHO_REQUEST, 0, my_checksum, ID, 1)
    bytesInDouble = struct.calcsize("d")
    data = (192 - bytesInDouble) * "Q"
    data = struct.pack("d", time.time()) + data
 
    # Calculate the checksum on the data and the dummy header.
    my_checksum = checksum(header + data)
 
    # Now that we have the right checksum, we put that in. It's just easier
    # to make up a new header than to stuff it into the dummy.
    header = struct.pack(
        "bbHHh", ICMP_ECHO_REQUEST, 0, socket.htons(my_checksum), ID, 1
    )
    packet = header + data
    my_socket.sendto(packet, (dest_addr, 1)) # Don't know about the 1
 
 
def do_one(dest_addr, timeout):
    """
    Returns either the delay (in seconds) or none on timeout.
    """
    icmp = socket.getprotobyname("icmp")
    try:
        my_socket = socket.socket(socket.AF_INET, socket.SOCK_RAW, icmp)
    except socket.error, (errno, msg):
        if errno == 1:
            # Operation not permitted
            msg = msg + (
                " - Note that ICMP messages can only be sent from processes"
                " running as root."
            )
            raise socket.error(msg)
        raise # raise the original error
 
    my_ID = os.getpid() & 0xFFFF
 
    send_one_ping(my_socket, dest_addr, my_ID)
    delay = receive_one_ping(my_socket, my_ID, timeout)
 
    my_socket.close()
    return delay
 
 
def verbose_ping(dest_addr, timeout = 2, count = 4):
    """
    Send >count< ping to >dest_addr< with the given >timeout< and display
    the result.
    """
    for i in xrange(count):
        print "ping %s..." % dest_addr,
        try:
            delay  =  do_one(dest_addr, timeout)
        except socket.gaierror, e:
            print "failed. (socket error: '%s')" % e[1]
            break
 
        if delay  ==  None:
            print "failed. (timeout within %ssec.)" % timeout
        else:
            delay  =  delay * 1000
            print "get ping in %0.4fms" % delay
    print
 
 
if __name__ == '__main__':
    verbose_ping("heise.de")
    verbose_ping("google.com")
    verbose_ping("a-test-url-taht-is-not-available.com")
    verbose_ping("192.168.1.1")

I decided not to publish the updated code on CodeTRAX, where I normally publish any programming-related stuff, but leave it on G-Loaded instead.

Related Articles

• Sockets Programming In Python
• Descramble Passwords from gftp Bookmarks using Python
• Python IRC Bot
• Use Python to get the web page data in Epiphany
• Python Crash Course

 
char *
gftp_descramble_password (const char *password)
{
  const char *passwordpos;
  char *newstr, *newpos;
  int error;
 
  if (*password != '$')
    return (g_strdup (password));
 
  passwordpos = password + 1;
  newstr = g_malloc (strlen (passwordpos) / 2 + 1);
  newpos = newstr;
 
  error = 0;
  while (*passwordpos != '\0' && (*passwordpos + 1) != '\0')
    {
      if ((*passwordpos & 0xc3) != 0x41 ||
          (*(passwordpos + 1) & 0xc3) != 0x41)
        {
          error = 1;
          break;
        }
 
      *newpos++ = ((*passwordpos & 0x3c) << 2) |
                  ((*(passwordpos + 1) & 0x3c) >> 2);
 
      passwordpos += 2;
    }
 
  if (error)
    {
      g_free (newstr);
      return (g_strdup (password));
    }
 
  *newpos = '\0';
  return (newstr);
}

The following is the Python port of the above code. The person who did the porting wished to remain nameless. The exact answer I got when I had asked about how I should give him credit for the code was:

please don’t attach my name to it. it’s horrible, awful code. Consider it public domain, do with it as you wish.

Python port:

 
def gftp_descrable_password(password):
    """gftp password descrambler
 
    This code has been released in the Public Domain by the original author.
 
    """
    if not password.startswith('$'):
        return password
    newpassword = []
    pwdparts = map(ord, password)
    for i in range(1, len(pwdparts), 2):
        if ((pwdparts[i] & 0xc3) != 0x41 or
            (pwdparts[i+1] & 0xc3) != 0x41):
            return password
        newpassword.append(chr(((pwdparts[i] & 0x3c) << 2) +
                               ((pwdparts[i+1] & 0x3c) >> 2)))
    return "".join(newpassword)

The only thing I have added to the above code is the docstring so that the function can be reused without licensing issues.

The above snippet has not been thoroughly tested, but it seems to work fine.

As for the bookmark converter, I am not interested in doing it any more.

Related Articles

• Smart Bookmarks in Epiphany
• Python SSH Server for UNIX Systems using Twisted.conch
• ping.py – Python Implementation of the ping command
• Python Crash Course
• Sockets Programming In Python

One way to resolve the issue mentioned above is to inject our own authorization code into the relevant phase of the HTTP request processing. What this authorization code is going to do is to dynamically construct the path to the requested repository’s authz file, parse the authz file, determine the level of access the user may get in accordance to the HTTP request type and, finally, return an OK or an error code back to Apache.

Tonight I’ve spent several hours on this. Although, the appropriate approach would be to patch mod_authz_svn, I am not that excited about such an approach as I would need several days of hard work to modify an existing apache module. Instead, I have almost finished an authorization handler and an authz file parser (only basic syntax is supported), both written in python, which can be used by apache with the help of mod_python.

This is not finished yet. As you might have assumed I need this piece of code for Project CodeTRAX. Although, I had decided not to release anything related to that project until it is finished, I might make an exception and release this piece of python code in order to get some valuable feedback about if and how secure it is.

Related Articles

• Strange mod_dav_svn error
• Search for a string in multiple files
• A change of plans regarding a web-based VCS manager
• VeriTAR – Verify checksums of files within a TAR archive
• rsapiget downloads files using the new Rapidshare API

The HTTP Protocol specification does not specify a maximum URL or path length, so the implementation of such limits is left to the developers of HTTP software.

The limits described above do not apply to POST HTTP requests. The data of such requests is completely contained into their body and not in the headers, therefore it is very unlikely that the Host header or the path exceed even Internet Explorer’s limits. Note that, if you do not use a HTML form or a high-level API to generate the POST request, it is required that you set the Content-Type header to be application/x-www-form-urlencoded. You can check HTTP headers using curl.

Below, you will find the headers and the body of an example POST request:

POST /cgi-bin/test.cgi HTTP/1.1
Host: www.example.org
Content-Type: application/x-www-form-urlencoded
Content-Length: 41
lastname=Smith&firstname=John&phone=12345

In Python, you can generate such a urlencoded body in the following way:

>>> param = { 'firstname':'John', 'lastname':'Smith', 'phone':'12345' }
>>> import urllib
>>> print urllib.urlencode(param)
lastname=Smith&firstname=John&phone=12345

This is exactly the same as generating the data of a GET request, but without any restrictions regarding the length, which is explicitly stated in the Content-Length header. Of course, you can use any programming language you like. This was just an example.

Related Articles

• HTTP headers
• HTTP Status Codes
• ModSecurity Overview

Creation of the TAR archive and the MD5 sums file

In the following example it is assumed that the files to backup reside in the myfiles/ subdirectory, the name of the tar archive will be mybackup.tar and the name of the file containing the md5sums will be mybackup.md5.

$ tar -cvpf mybackup.tar myfiles/ \
    | xargs -I '{}' sh -c "test -f '{}' && md5sum '{}'" \
    | tee mybackup.md5

Some notes:

• You can use any tar switch for the creation of the archive except -C. If you need to change to another directory, do it using cd or else no md5 sums will be recorded.
• Make sure that you include the -v (–verbose) switch when invoking tar, as the paths need to be printed to stdout in order to be processed by xargs.
• In the xargs statement, the -I ‘{}’ part indicates that the '{}' string will be replaced by the path that is passed to xargs through the pipe.
• The sh -c “test -f ‘{}’ && md5sum ‘{}’” does two things: tests if the path ('{}') is a file and calculates the md5 sum for it.
• In the last part, tee is used in order to print the md5sum to the stdout and also to the mybackup.md5 file.

When this operation ends, you will end up with two files: mybackup.tar and mybackup.md5.

Special thanks to:

* Anvil for the suggestion to use bash -c "...test goes here..." stuff.
* Giorgos Keramidas for the improvement he suggested, so that the md5 sum calculation is not limited to regular files only:

sh -c "test -d '{}' || md5sum '{}'"

VeriTAR will verify the md5 sums of regular files only, so either test you use when creating the TAR archive, it is still fine.

VeriTAR – Tar archive verification

VeriTAR [Veri(fy)TAR] is a command-line utility that verifies the md5 sums of files within a tar archive. Due to the tar (‘ustar‘) format limitations the md5 sums are retrieved from a separate file and are checked against the md5 sums of the files within the tar archive. The process takes place without actually exctracting the files.

It works with corrupted tar archives. The program carries on to the next file within the archive skipping the damaged parts. At the moment, this relies
on Python’s tarfile module internal functions.

VeriTAR is written in Python.

Works with compressed TAR archives (gzip or bz2).

• VeriTAR Development Website and Bug Tracking
• Downloads

Provided that you have used the method above (or any other method) in order to create a file with the md5 sums together with the tar archive, you can easily verify the contents of the archive with veritar.

$ veritar mybackup.tar mybackup.md5

Please not that veritar’s output and command line switched need some work, but for now it does the job.

Veritar is released under the Apache License version 2.

It is completely unsupported, but you can still get community support at our software forums. This is also the place where you can inform me about any bugs.

Known issues

1. Multi-volume tar archives are not supported at the moment
2. Tar archives in which the metadata of the first archived file has been corrupted cannot be processed due to a limitation in the tarfile Python module at the time of writing
3. Although the checksum of any algorithm, md5, sha1, crc(crc32), could be used, the current alpha version is not very flexible.
4. It may crash on damaged archives on older python versions.

Related Articles

• Verify a burned CD/DVD image on Linux
• Choosing a format for data backups – tar vs cpio
• How to extract RPM or DEB packages
• Error when using old run/bin installers under Linux
• Cheap Biometrics – Use Keystroke Dynamics to Identify and Verify Users

A while ago I had written a script that would automate the WordPress upgrade process. Such a program could have been a simple shell script, just a few lines of code. However, I decided to write a more flexible script in Python instead, which can keep backups, perform a thorough upgrade of the website’s platform, keep the files I need to keep etc. A script that takes good care of everything while upgrading is also a good idea because it minimizes downtime while performing upgrades. Today was the first time to test it. After the upgrade, everything seems to be where it should. The script is in pre-alpha state and despite the fact that I used it on my website, it is not ready for release yet. Below is the output of the upgrade on G-Loaded:

# python wpupgrade.py 2.3.1
info: ++ Performings Tests. Please wait...
info: Initial configuration checks - OK
info: Retrieval of WordPress database settings from wp-config.php - OK
info: Checking existence of mysqldump - OK
info: ++ Starting data backup. Please wait...
info: Database backup - OK
info: Filesystem backup - OK
info: Data backup Complete
info: ++ Downloading wordpress archive. Please wait...
info: Download and Verification Complete
info: ++ Removing old WordPress files and directories. Listing follows:
info: Removed DIR: wp-admin
info: Removed DIR: wp-includes - PRESERVED wp-includes/languages
warning: --> DIR wp-content/cache CANNOT BE REMOVED - NOT CRITICAL - REMOVE MANUALLY IF POSSIBLE
info: Removed FILE: wp-commentsrss2.php
info: Removed FILE: wp-rss.php
info: Removed FILE: wp-rdf.php
info: Removed FILE: wp-login.php
info: Removed FILE: wp-app.php
info: Removed FILE: wp-settings.php
info: Removed FILE: wp-comments-post.php
info: Removed FILE: wp-mail.php
info: Removed FILE: wp-rss2.php
info: Removed FILE: wp-pass.php
info: Removed FILE: wp-blog-header.php
info: Removed FILE: wp-register.php
info: Removed FILE: wp-feed.php
info: Removed FILE: wp-trackback.php
info: Removed FILE: wp-cron.php
info: Removed FILE: wp-links-opml.php
info: Removed FILE: wp-config-sample.php
info: Removed FILE: wp-atom.php
info: Removed FILE: index.php
info: Removed FILE: xmlrpc.php
info: Removed FILE: readme.html
info: Removed FILE: license.txt
info: WordPress files removal complete
info: ++ WordPress Upgrade
info: Writing new files. Please wait...
info: File Upgrade Complete
info: ++ Starting Cleanup
info: Nothing - TODO
info: Finish Upgrade by visiting: http://www.yoursite.org/wp-admin/upgrade.php

Related Articles

• Upgraded to WordPress Coltrane
• Upgraded to WordPress 2
• Highly Exploitable Code Planted into WordPress 2.1.1
• Server upgraded to Fedora 6
• WordPress 2.5 – Plugin Compatibility

Related Articles

• Tab Session Management extension for Epiphany
• Epiphany Python Console – Documentation
• Tab Links extension for the Epiphany browser
• Featured Epiphany Plugins
• More On cElementTree