Neil Schemenauer's Web Log

Using Git to checkout the Python source code

Tue, 23 Feb 2010 19:06:13 GMT

The Python developers have decided to move from Subversion to Mercurial. However, that's taking some time to implement. For those who want a DVCS now, there are Git repositories on svn.python.org that allow efficient use of Git. I wrote a set of instructions on the Python wiki. The initial setup is a little complicated but day to day use is convenient, IMHO.

Why would you want to use Git? I can only speak for myself, but I was an early adopter of Subversion. I lead the push while at the MEMS Exchange to switch from CVS to Subversion (version 0.17.0 in 2002, I believe) . At the time, CVS's extremely poor support for moving and renaming files was hampering our ability to refactor our source code. Subversion was a definite improvement and is still a decent choice for an internally developed project. It's model of svn up doing a rebase of local changes is intuitive for most users.

Subversion still has a number of deficiencies for serious developers. For one, the handling of multiple branches of development is awkward enough that many users avoid branches (I understand SVN is getting better in this regard). Less notable but just as importantly, IMHO, it lacks many tools for dealing with changes as patches. Often I would dump a change using svn diff, edit the patch, and the apply the new patch to a different branch. Since moving to Git, I rarely have to resort to manual patch editing (instead using tools like git rebase, git format-patch, git add -i, git cherry-pick, etc).

As a tip to someone dipping their toe in the Git waters, it is critical to understand the difference between merging changes and rebasing changes. Git sort of defaults to merging but in the vast majority of cases you want to rebase instead. As an example, for one of my projects I've probably done hundreds of rebase operations and only about three merges (those three are because we have a long lived branch for a variant of the software that can't be done using configuration files). When I speak of "merge" I don't mean what Git calls a "fast-forward merge" since that's not really a merge a all. Instead, the destination branch is a strict subset of the source branch and changes are just being appending to the head version. Subversion defaults to doing rebasing rather than merging. When you run svn up your local changes are being rebased on the HEAD revision from the repository.

Git definitely has a learning curve. When I first started using it I was often confused. However, after about a week I realized it was much better than any other VCS that I had experienced. I now consider a DVCS with similar capabilities an essential developer's tool, just like a good text editor.

Quixote 2.7b2 beta released

Mon, 01 Feb 2010 02:14:17 GMT

I released another Quixote beta about a week ago. I think it finally fixes the problem cased by Python 2.6's breakage of the ihooks module. What happened was that the when relative imports were implemented, the ihooks module got forgotten. If you use the 2.6 ihooks module and any package uses a relative import, you lose.

I've fixed the ihooks module in the SVN trunk and it will be fixed in Python 2.7. Quixote 2.7b2 works around the problem by shipping with it's own ihooks module.

H1N1 flu takes off

Wed, 28 Oct 2009 21:05:56 GMT

Take a look at the Google Flu trend graph. The Canada Health FluWatch report has a similar graph. Saskatchewan is starting to roll out the vaccine but I suspect it will be too late to help most people.

BTW, can the media please stop calling it "swine flu"? The CDC says "further study has shown that this new virus is very different from what normally circulates in North American pigs. It has two genes from flu viruses that normally circulate in pigs in Europe and Asia and bird (avian) genes and human genes."

Also interesting is that the best guess is that it's contagious one day before symptoms are shown. Based on that, I think it would be prudent for the government to shutdown schools and universities for a week or two. That would be a small cost in exchange to helping to avoid overloading the health care system.

No need to panic but given that fact that this virus attacks lung tissue, some caution would be prudent. Please consider getting vaccinated.

Low cost parametric CAD software

Tue, 01 Sep 2009 19:30:13 GMT

SolidWorks is cool but the price tag is sometimes difficult to justify. Alibre Design looks like a great deal at $99. Based on this YouTube video, I'm impressed by the focus of the company. I think it is important to decide what your software will focus on doing well, rather than trying to make it all things to all people.

Update: After a little research, I decided the deal was too good to pass up. Alibre Design really does cover most of the commonly used functionality of SolidWorks or AutoDesk Inventor. I suspect they are having the promotion because the next major version of the software is due to be released soon. Serious users will pay full price for the new version so it's a clever promotional strategy.

Poor performance of Radeon X800 (R430) with X

Fri, 24 Jul 2009 23:35:31 GMT

I've got a Radeon X800 GTO card in my desktop machine. It's an older card but it still has pretty good 3D gaming performance (it came out of the box with all pipelines enabled). I bought it a few years ago based on an ArsTechnica system building guide. I've been irritated for a while by it's poor performance in X but never bothered to look into the cause.

Yesterday I was tinkering driver problems for a Radeon 9600 HD (or some similar recent, cheap card). When using Ubuntu's fast user switching (i.e. multiple instances of X), performance goes to pot. It turns out that the driver can only use DRI and therefore acceleration on the first X instance. It's likely that the limitation will be fixed in an upcoming "radeon" or "radeonhd" driver version.

Anyhow, that tinkering session gave me a hint to my poor X800 performance. I suspected it was related to the XAA/EXA options. Sure enough, a peak at the Xorg.0.log revealed:

(==) RADEON(0): Using XAA acceleration architecture
...
(II) RADEON(0): XAA Render acceleration unsupported on Radeon
 9500/9700 and newer. Please use EXA instead.
(II) RADEON(0): Render acceleration disabled

Adding Option "AccelMethod" "EXA" to the Device section of the xorg.conf configuration file produced a huge speedup. The performance before was not awful but the improvement was dramatic. The configuration of X has gotten vastly easier over the years due to sensible defaults but it looks like they missed this one.

Repairing a Seagate ST3500320AS 500 GB drive

Fri, 24 Jul 2009 21:19:01 GMT

I recently had a Seagate 500GB drive die on me. The Linux kernel produced the following messages:

ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata4.00: failed to read native max address (err_mask=0x1)
ata4.00: HPA support seems broken, skipping HPA handling
ata4.00: ATA-8: ST3500320AS, AD14, max UDMA/133
ata4.00: 0 sectors, multi 0: LBA NCQ (depth 0/32)
ata4.00: configured for UDMA/133 (device error ignored)
scsi 3:0:0:0: Direct-Access     ATA      ST3500320AS      AD14 PQ: 0 ANSI: 5
sd 3:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
sd 3:0:0:0: [sdb] 0 512-byte hardware sectors: (0 B/0 B)

The computer BIOS reported the drive as having 0 size. After much searching I finally figured out that I was facing what is known as the "Seagate 7200.11 0 LBA" problem. Essentially, a firmware bug causes the drive to get into a hung state. The only way to repair it is to use the 3.3V CMOS level pins connected to the serial interface of the drive controller and issue some (potentially dangerous) low level commands.

A good description of the fix procedure as been compiled by some owners of the drives. Seagate have not handled the bug well considering that apparently many drives are affected. The fix is not recommended for those unfamiliar with electronics as you can easily brick your drive.

If you want to attempt it, the trickiest part is finding a RS232 level converter that can work with 3.3 volts. You need a chip like the MAX232. I have a MAX232N on a AVR board but according to the datasheet it only handles 5 volts. I made a level converter for the MAX232 TX (out) pin using a N-channel MOSFET I had kicking around. You only need to convert the MAX232 output since the input will sense 0-3.3V logic okay. The MOSFET level converter circuit I used is very elegant. If you are going to attempt this, double and triple check that you have the RX and TX pins connected correctly (there is much confusion on the labeling of these pins). You want the T?OUT pin of the MAX232 (via the level converter) going to the RX pin of the drive (the rightmost pin).

After you have successfully run the commands, you need to upgrade the drive firmware to prevent the bug from happening again. My drive went from firmware version AD14 to SD1A. The good news about this procedure is that the data on the drive is intact.

My setup is shown below (MacGyver would be proud).

BBC: Obama wants to block AIG bonuses

Tue, 17 Mar 2009 01:58:13 GMT

It's progress that Obama thinks it's a problem but he should focus on the real issue instead of chasing after secondary effects. First, the US Treasury should have gotten voting shares instead of preferred shares for their cash infusion (despite the cries from risk-loving bank owners that "nationalization" is evil and un-American).

The second and much deeper issue is that corporations have to be made accountable to investors. The current corporate model just does not work. I don't have statistics but I would guess that the majority of the general public who own stock do not exercise their votes. That allows the board and the management to pay each other huge salaries and generally screw everyone else.

The third issue is that regulation of the financial industry is needed, despite what some conservatives might say. Businesses that are "too big to fail" are obviously either too big to exist or they must to heavily regulated to ensure they don't blow up. As someone who does not like big government, I would vote for the "split them up" strategy rather than for regulation.

Debian vote and voting methods

Sun, 14 Dec 2008 23:43:26 GMT

Debian has a pretty important vote happening right now. The ballot is complicated, some would say even poorly written. I agree with Daniel Burrows that the including both super-majority options and simple-majority options on a single ballot is problematic. It would have been better to have two votes, I think. Using Daniel's example, the first ballot would be

[ ] Choice 1: Omelette [3:1]
[ ] Choice 2: Quiche [3:1]
[ ] Choice 3: Further Discussion

If option 3 wins then the next ballot should be:

[ ] Choice 1: Toast
[ ] Choice 2: Further Discussion

While the Debian ballot is complicated, I don't think it requires people to vote strategically (which would be a failure of the voting system). As long as most people rank the "delay Lenny" option below the "don't delay" options then the "delay" option will not win. Thankfully we don't use first-past-the-post because this is a classic vote splitting ballot.

As a final note, I wish approval voting would gain more traction. The Condorcet method is obviously better than plurality but it's probably just too complicated to ever see wide spread adoption. Approval voting can be easily be done with a pencil and paper and still gives nice results. The Canadian Wheat Board recently had an election where they used IRV. IRV sounds like a good idea but it's more complicated that approval voting and has some nasty behavior in certain cases.

Storing password hashes

Mon, 08 Dec 2008 17:06:05 GMT

Update for 2010-07-25: The original code here was vulnerable to a timing attack. The old version of the code used == to compare the hashed passwords. I've updated the code to use a constant time compare. Bewarned there very well could be more bugs in this code.

This is a quick note on hashing user passwords. I fear that some people might still be doing this wrong. They might be storing plain-text passwords (yikes!) or they might use a hash that is vulnerable to brute force attacks. The following is an example of the algorithm I'm using. Note that for new code sha1 should be replaced by a stronger hash function (e.g. sha256) and perhaps _HASH_ROUNDS should be adjusted.

def constant_time_compare(a, b):
    """A constant time comparison function for two strings. Returns
    True if the strings are equal.
    """
    result = 0 if len(a) == len(b) else 1
    for x, y in zip(a, b):
        result |= ord(x) ^ ord(y)
    return result == 0

class User(Persistent):
   """
    Instance attributes:
      password_hash : str | None
        if None then logins are disabled, see _encrypt() for details on hashing
      password_salt : str
        a noonce 
    """

    _SALT_LENGTH = 32 # 256 bits
    _HASH_ROUNDS = 2**16

    ...

    def _encrypt(self, password):
        """(password : str | unicode) -> str
        """
        # salt and stretch password as recommended by Ferguson & Schneier in
        # "Practical Cryptography"
        if isinstance(password, unicode):
            password = password.encode('utf-8')
        h = ''
        assert self._HASH_ROUNDS > 0
        for i in range(self._HASH_ROUNDS):
            h = hashlib.sha1(h + self.password_salt + password).digest()
        return h

    def set_password(self, password):
        self.password_salt = os.urandom(self._SALT_LENGTH)
        self.password_hash = self._encrypt(password)

    def validate_password(self, password):
        """(password : str) -> bool
        """
        if self.password_hash is None:
            return False
        else:
            return constant_time_compare(self._encrypt(password),
                                         self.password_hash)

There is a useful discussion on stackoverflow.com along with some sample Python code. Note that using HMAC instead of concatenation doesn't buy any security if you assume the salt is not secret. Also, the random_bytes() function in the sample code is lame since it's a poor nonce.

Sound setup on ASUS m2n-e SLI board (CM6501)

Wed, 26 Nov 2008 18:13:55 GMT

Audio has got to be one of the most frustrating parts of setting up Linux. It looks like finally the situation is improving with most distributions adopting pulseaudio as a middle layer. Still, getting sound working on my cheap ASUS M2N-E board proved to be difficult and I wasted way too much time tinkering with it. Hopefully this post will save someone from my misery.

The root of the problem seems to be how ALSA handles the CM6501 (C-Media Electronics USB audio) chip. Try as I might, there seems to be no way to get sound out of the lime colored connector. According to the manual that is either "Line Out" or "Front Speaker Out". I began to make progress after running:

speaker-test -c 6

Aside from the lime connector being apparently dead, the channel mapping does not match the ASUS manual. In 4 channel mode:

ALSA Channel	ALSA Name	Plug
1/2	Front	None
3/4	Rear	gray

In 6 channel mode:

ALSA Channel	ALSA Name	Plug
1/2	Front	None
3/4	Rear	orange
5/6	Center/LFE	gray

The most straightforward way of getting some sound working is to configure pulseaudio with the following one line in /etc/pulseaudio/default.pa:

load-module module-alsa-sink device=hw:0 sink_name=front_stereo channels=4 channel_map=aux0,aux1,front-left,front-right

I see I'm not the only one having problems with the CM6501 and I guess Linux is not solely to blame. In retrospect, I should have bought a decent PCI sound card years ago. Anyone got recommendations?