djc: thinking in writing

Single-source Python 2/3 doctests

Somewhere in 2009, I took over maintenance of CouchDB-Python from Christopher Lenz. While maintenance has slowed down over the years, since the core libraries work well and the CouchDB API has been quite stable, I still feel responsible for the project (I also still use it in a bunch of places). This being a Python project, it always felt like it would have to be ported to Python 3 sooner or later. Since it's working with a fairly deep HTTP API (as in, it uses a large subset of the protocol, with extensive hacking of httplib/http.client), the changes needed in string/bytes handling are quite involved.

My first serious attempt started in November of 2012, as evidenced from some old patches that I have lying around in mq repositories. I picked it back up again about a year later, until I had most of the tests passing, save for one specific category: the doctests. Specifically, the problem I had was with unicode literals (like u'str'). For Python 2.7 doctests, I needed the unicode annotation to pass the test. In Python 3, all strings are unicode; while unicode literals can be used in source code in Python 3.3 and later, the repr() of a string always lacks the unicode annotation. This resulted in lots of test failures like this:

======================================================================
FAIL: client (couchdb)
Doctest: couchdb.client
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.3/doctest.py", line 2154, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for couchdb.client
  File "./couchdb/client.py", line 8, in client

----------------------------------------------------------------------
File "./couchdb/client.py", line 15, in couchdb.client
Failed example:
    doc['type']
Expected:
    u'Person'
Got:
    'Person'
----------------------------------------------------------------------
File "./couchdb/client.py", line 17, in couchdb.client
Failed example:
    doc['name']
Expected:
    u'John Doe'
Got:
    'John Doe'

While these simple cases might have been easy to fix some other way (e.g. by printing the value instead of just asking for the representation), other cases would be significantly harder to fix that way. Here's one example:

----------------------------------------------------------------------
File "./couchdb/mapping.py", line 343, in couchdb.mapping.Document.items
Failed example:
    sorted(post.items())
Expected:
    [('_id', 'foo-bar'), ('author', u'Joe'), ('title', u'Foo bar')]
Got:
    [('_id', 'foo-bar'), ('author', 'Joe'), ('title', 'Foo bar')]

After asking around on the Python 3 porting mailing list, Lennart Regebro (the author of the Porting to Python 3 book) kindly pointed me to the relevant section of his book, but it didn't contain any great suggestions for this particular problem. It took me a few months to get back into it, but I started looking into the doctest APIs yesterday, and managed to figure out a fairly clean solution:

class Py23DocChecker(doctest.OutputChecker):
  def check_output(self, want, got, optionflags):
    if sys.version_info[0] > 2:
      want = re.sub("u'(.*?)'", "'\\1'", want)
      want = re.sub('u"(.*?)"', '"\\1"', want)
    return doctest.OutputChecker.check_output(self, want, got, optionflags)

As it turns out, the doctest API is pretty well-designed, so it allows you to pass in your own OutputChecker object. As its name indicates, this is the bit of code that compares the actual output and the expected output of a given example. By slightly processing the expected value when running on Python 3, we can make sure that actual and expected output match on both versions. Use it like this:

doctest.DocTestSuite(mod, checker=Py23DocChecker())

Fixing these test failures has cleared the way (along with some other fixes) for a Python 3-compatible CouchDB-Python release soon. I hope this will enable other projects to start moving in the direction of 3.x; at the very least, it should significantly lower the barrier for my own projects to start using Python 3.

No Close Buttons

Or, How To Turn Your Firefox userChrome.css Hack Into a Neat Little Restart-Less Add-on, in Five Delightfully Simple Steps.

For as long as I can remember, I've had a small number of tweaks set in my Firefox profile's about:config. One of these was the browser.tabs.closeButtons pref, which I had set to 2. By default, Firefox shows a little close button on the right of every tab, but since I pretty much always use a keyboard shortcut to close tabs, these little buttons aren't that helpful, and they end up obscuring parts of their tab's titles. Setting the value to 2 removes all of the buttons.

In Firefox 31 (to be released to the general public in about 12 weeks), this preference has been removed, leading me to look into other ways of removing the buttons. A commenter on the bug noted the CSS required to remove the buttons again, saying that this could be added to the userChrome.css file in a profile. However, I don't really like that solution, since it would require me to port the fix to every computer I use, and it would be easy for me to lose it. Instead, I wanted to put it in an add-on, which would make it easy for me to install on other computers, in addition to being relatively easy to find. As an added benefit, others can benefit from the same add-on.

The result of this is the No Close Buttons add-on, which I put up on AMO yesterday. It was promptly reviewed by a friendly reviewer from the Dutch community, so that it can be installed without trouble. However, it's currently restricted to Firefox 31 and later, since I figured people on earlier versions wouldn't need it. Because it took me a while to piece together everything for what I thought should be a well-documented process (turning simple chrome CSS hacks into an add-on), I figured I'd document the process here.

First off, create a file called install.rdf:

<?xml version="1.0" encoding="utf-8"?>
<!-- This Source Code Form is subject to the terms of the Mozilla Public
   - License, v. 2.0. If a copy of the MPL was not distributed with this
   - file, You can obtain one at http://mozilla.org/MPL/2.0/. -->
<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:em="http://www.mozilla.org/2004/em-rdf#">
  <Description about="urn:mozilla:install-manifest">
    <em:id>no-close-buttons@xavamedia.nl</em:id>
    <em:version>0.1</em:version>
    <em:type>2</em:type>
    <em:bootstrap>true</em:bootstrap>
    <em:unpack>false</em:unpack>
    <!-- Firefox -->
    <em:targetApplication>
      <Description>
        <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id>
        <em:minVersion>31.0a1</em:minVersion>
        <em:maxVersion>31.0</em:maxVersion>
      </Description>
    </em:targetApplication>
    <!-- Front End MetaData -->
    <em:name>No Close Buttons</em:name>
    <em:description>Remove close buttons from tabs</em:description>
    <em:creator>Dirkjan Ochtman</em:creator>
  </Description>
</RDF>

Second, add a file named chrome.manifest, with a single line:

content              no-close-buttons        content/

Third, add some JavaScript code to register and unregister the stylesheet to a file named bootstrap.js:

/* This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/. */

'use strict';

var sss = Components.classes["@mozilla.org/content/style-sheet-service;1"].getService(Components.interfaces.nsIStyleSheetService);
var ios = Components.classes["@mozilla.org/network/io-service;1"].getService(Components.interfaces.nsIIOService);
var uri = ios.newURI('chrome://no-close-buttons/content/style.css', null, null);

function startup(data, reason) {
    sss.loadAndRegisterSheet(uri, sss.USER_SHEET);
}

function shutdown(data, reason) {
    sss.unregisterSheet(uri, sss.USER_SHEET);
}

Fourth, add your CSS, to the file named in your bootstrap.js. It should go inside the content directory you've already referenced in the manifest.

.tab-close-button { display: none !important; }

Fifth, and finally, if you zip up the four resulting items (install.rdf, chrome.manifest, bootstrap.js and the content directory), rename the resulting file so that its extension is xpi, and drop it into the extensions directory in your Firefox profile, Firefox should prompt you to install your new add-on!

I cobbled together all these bits and pieces by looking at a small add-on by Benjamin Smedberg (who has a number of UI-related add-ons up on AMO), some code from Stack Overflow, and MDN pages on bootstrapped add-ons and chrome registration. I put the result up on GitHub, any feedback is most welcome.

Giving in to Git(Hub)

Published on 2014-03-30 by Dirkjan Ochtman in tech, code

Last month, I moved most of my code from Bitbucket to GitHub:

As a former Mercurial developer, this feels like an admission of defeat. Most of hg's user interface still seems superior to Git's, even if Git was quicker to get the branching model right. The Mercurial code base, in many ways, is a testament to how approachable a Python application can be, and the extension possibilities stemming from writing a few Python functions seem far more attractive than Git's apparent hodge-podge of C, shell and Perl. It's good that people at Mozilla and Facebook are starting to talk more about hg's advantages, though.

While I wanted to learn Git sooner, the lack of usability made me mostly avoid it until about 8 months ago, when I became a CouchDB committer and thus could no longer escape. Two months ago, I also got a new job where Git is the primary VCS, so I've been diving in. Obviously, it's a pretty great VCS, but some aspects of the (command-line) user interface are still baffling to me. This has been written about in plenty of places, so I won't go point-to-point here. And I'll have to admit that many commands are starting to be ingrained in muscle memory, to the point that I sometimes use Git-like commands in places where I use still hg.

However, now that I have basic usage down such that my lack of experience with Git is no longer a limiting factor, the network effect values from Git (and GitHub, specifically) outweigh my usability concerns. The GitHub UX feels more polished (and seems to receive more attention) than Bitbucket's, and makes me quite happy to use it. I also feel that the community on GitHub is quite a bit larger than on Bitbucket, which could make my projects more accessible (see also this account from Eli Bendersky). I've already gathered some stars (mostly for Persona-TOTP, so far) over the past six weeks; I hope that's just the start.

Changing your OS X Mavericks user icon without iPhoto

Published on 2013-10-24 by Dirkjan Ochtman in tech

I wanted to update the user icon/picture for my OS X user (which may include the iCloud/Apple ID picture as well), but it turned out to be harder than I thought. Here's to hoping this post may help others who run into the same problem. tl;dr: use iCloud's web app to upload the new picture for your own Contacts entry.

Update (2013-11-02): on Twitter, both Christopher Lenz and Justin Mayer pointed out that you can just drag and drop an image onto the System Preferences panel. I thought I'd tried that, but apparently not! Still, I wonder if that UI is sufficiently ingrained that discoverability is not important.

Update (2013-11-25): Hugh Hosman, via email, points out that you can also drop an image into /Library/User Images if you have super user privileges.

Like any person who values 0-day upgrades more than their system's stability, I recently upgraded to OS X Mavericks. Going into the Users & Groups preferences panel, double-clicking my current picture provided me with 6 possible options:

  • Defaults: a sample of pictures provided by Apple
  • Recents: contains the current picture, but no others
  • iCloud: is apparently connected to my iCloud Photo Stream
  • Faces: a selection based on the iCloud Photo Stream
  • Camera: take a new picture from my laptop camera
  • Linked: appears to have something to do with my Contacts

In other words, there was no way here to simply link in a JPEG. Apparently, the way to get pictures into the Photo Stream is either through an iOS device (probably through the Camera app) or via Apple's iPhoto or Aperture photo software, neither of which I own (though iPhoto is apparently free for everyone who buys a new machine from now on). I did some Googling, which yielded precisely zero useful results; apparently, using a JPEG was still supported under Mountain Lion, and no one had documented this problem yet. (One of the more promising venues appeared to be the Apple StackExchange site Ask Different.)

But, I figured it out:

  • Go to the web interface for iCloud
  • Go to the Contacts interface
  • Find your own Contacts entry
  • Click "Edit"
  • Click the picture
  • Click "Choose Photo..."

You can now upload the picture. Now, you should be able to go back to the Users & Groups panel and select the uploaded picture from the Linked list of pictures.

Not a great user experience, but at least it works.

A nanomsg presentation

Update (2013-10-17): slides and video are now available.

For Software Freedom Day 2013, which is on Wednesday, the 18th of September, I will give a presentation about nanomsg at the Centrum Wiskunde & Informatica (the Center for Mathematics & Computer Science) at the Amsterdam Science Park. If you're in the neighborhood and/or interested in nanomsg, come visit!

nanomsg: simple smart sockets

nanomsg is a socket library that provides several common communication patterns to help build distributed systems. It aims to make the networking layer fast, scalable, and easy to use. Implemented in C, it works on a wide range of operating systems with no further dependencies.

This talk will give a short history of the nanomsg project, an explanation of the value provided by nanomsg in building distributed systems, and a demonstration of some key features.

Are We Meeting Yet?

For a few months now, I've worked on a little single-file web thingy: Are We Meeting Yet? (AWMY for short). Here are two example URLs:

Gervase Markham kindly wrote about it on his blog after I recommended it for a Firefox development meeting, which made me think I should write about it here.

What it is

AWMY is a tool to communicate event (meeting) times to geographically dispersed and therefore timezone-challenged audiences. This means it displays date/time values in (a) an original timezone, (b) the UTC timezone and (c) the user's local timezone, with a title or description and a countdown timer.

Critically, it supports recurring meetings in a way that a single URL will show the next meeting in the series no matter when it's loaded into the browser. This makes it a good fit for use in automatically generated meeting announcements. Currently, the only supported repeating modes are weekly and bi-weekly.

One of the design goals is to have nice-looking URLs; ideally, you can understand the meeting date/time from the URL even without clicking the link. For now, hacking the URL is the only way to create a new event page; this should be easy in most cases. I hope to add a form to make it even easier sometime soon.

Timezone support is based on the venerable Olson timezone database. I've put some thought into handling events near daylight savings transitions and tried to put in some warnings, but it's probably not perfect yet. At least weekend events close to daylight savings transitions should be somewhat rare.

The domain name was chosen because it fits in with a Mozilla meme (e.g. fast, pretty, small, popular, flash and probably others); I couldn't come up with a better alternative that was also still available. This one will hopefully be memorable at least for some part of the intended audience.

How to use it

In the current iteration, the page accepts a maximum of 5 arguments:

  • A timezone: a subset of Olson timezones are accepted and can be referenced in a few different forms. Only the continent timezones are accepted (e.g. "America/Los_Angeles", "Europe/Amsterdam"), plus the "UTC" timezone. The continent is optional (and left out in the canonical versions). A space can be used where underscores are used in timezone names.
  • A date: an ISO 6801-formatted date, like "2013-08-26". A three-letter weekday abbreviation also works here (like "Mon"), but it will emit a warning if used without the weekly repeating mode.
  • A time: ISO 6801-formatted 24-hour time, like "15:30".
  • A repeating mode: currently "w" for weekly or "b" for bi-weekly.
  • A title: any text.

If no timezone is provided, it's assumed to be UTC. Some examples:

Why

I got started based on some discussion on the mozilla-governance mailing list. Most Mozilla meetings are coordinated based on the timezone for the Mozilla HQ, in California. For many non-US participants, it's easier if meeting times are communicated in UTC, because they know their own UTC offset. However, this would change actual local meeting times based on daylight savings, which is a bit of a pain for recurring meetings. Therefore, it makes more sense to keep the reference meeting time in a timezone that has daylight savings, on the premise that most people live in zones that use mostly similar daylight savings schedules.

Some tools exist: for example, here's a timeanddate.com link use for a Firefox developer meeting. Although timeanddate.com has most of the information available from AWMY, it's provided in a much more cluttered fashion. Personally, I find it quite hard to visually parse that page to find the data I need. Of course, it does provide other useful features that AWMY does not currently offer.

I've also seen everytimezone.com used for this kind of thing; here's an example. It does provide the user with a sense of context, which is probably useful when you want to see what meeting times make sense in timezones you care about. For the purpose of communicating a single meeting time, it feels rather unfocused.

The user experience for these tools doesn't work well for this use case, so I thought I might be able to do better. On top of that, the other tools don't appear to handle recurring meetings. Having a stable URL for a series of events is useful when you want to point to a meeting time from many different places, but having to update each pointer every week is kind of a drag. Thus was born AWMY.

Future plans

At the top of my to do list is a feature to combine event series. This is mostly inspired by CouchDB meetings, which take place at alternating 13:00 UTC and 19:00 UTC times to accomodate people in different timezones. My current implementation strategy is to have a "merge" flag that signals another meeting series, such that two bi-weekly events series can be joined together.

As mentioned before, friendlier UI to build new events is one of my other priorities. A few form elements could go a long way, though I probably want a slightly more polished experience. I'll also have to figure out how to make dealing with series easy, in particular when working with the merging feature.

It would make sense to add a few other repeating modes, in particular "3rd Wednesday of each month"-like functionality. Offering ICS downloads would be nice. I would like each page to show the next meeting instance, if only as an indication that you're dealing with a recurrent event.

Because there's no server side component, I really want to keep all state in the URLs. On the other hand, I also want readable URLs. These goals don't always align well, so balancing them is an interesting act. I'm thinking about a way to generate alternative URLs that aren't as readable, but significantly shorter.

Wrapping up

I hope this will be a useful tool for the open source community (and anybody else who has a use for it). I'd be interested to hear your thoughts on what features would be most useful to add. If you want to contribute some code, that would be even better; check it out via the Bitbucket project. All feedback is welcome!

A Persona interview

Published on 2013-07-26 by Dirkjan Ochtman in tech, mozilla

I have recently been contributing to Mozilla's Persona project, which is an awesome way to make authentication easier for sites and their users. They kindly published an interview with me, which I reproduce here in full for archival purposes.

Introduction

Over the past year, Dirkjan Ochtman has been a consistent, constructive voice in the Persona community. His involvement has helped ensure that we stay true to Mozilla’s mission of open, transparent, and participatory innovation.

More impressively, Persona’s new backgroundColor feature is the direct result of Dirkjan’s efforts.

We hope this interview highlights his contributions and inspires others to get involved.

From the rest of us at Mozilla, thank you.

Who are you?

I’m Dirkjan Ochtman, a 30-year old software developer living in Amsterdam. I work for a financial startup by day; in my free time, I contribute to a bunch of open source projects, like Mercurial, Python, Gentoo Linux and Apache CouchDB. I also started a few things of my own.

Have you contributed to Mozilla projects in the past? How did you get involved in Persona?

I started using Firefox almost ten years ago, and I’d been watching Mozilla before that. The Mozilla mission of an open Internet resonates with me, so I tend to try and find stuff around the edges of the project where I can help. This year, I also became a Mozilla Rep.

I find BrowserID/Persona compelling because I hate having to register on different sites and make up passwords that fit (often inane) security requirements. And you just know that many sites store passwords insecurely, leaking sensitive information when they get hacked. Persona allows me to authenticate with my email address and a single password; no more guessing which username I used. I trust Mozilla’s password storage to be much more secure than the average Internet site, and because Persona is open source, I can verify that it is.

In addition to setting up Persona sign in on a small community site I run, I’ve also implemented my own Python-based Identity Provider. This means that when I use Persona, I control my own login experience. My Identity Provider uses Google Authenticator, so now I don’t have to remember any passwords at all.

The documentation for building an Identity Provider was scattered and incomplete, so I helped improve that. From that work, I got to know some of the great people who work on Identity at Mozilla.

What have you hacked on recently?

There has been a long-standing issue that the Persona dialog contained too much Mozilla branding and did not sufficiently emphasize the individual websites that users were signing into. There was an issue about this on Github, but I seem to remember complaints on the mailing list from even longer ago.

Of course, I prefer to use Persona over Facebook Connect or Twitter, so I decided to see if I could fix some of these issues. Luckily one of the Persona developers, Shane Tomlinson, was available to work on this at roughly the same time.

To improve the branding balance, we first de-emphasized the Persona branding. I focused on allowing websites to specify a background color for the Persona dialog. This is important because it can make the dialog feel much more “at home" on a site. We had to work out some tricks to ensure that text stayed readable regardless of the background color specified.

What was that experience like?

It was great. I had no previous experience with Node.js, but getting the application up and running was easy. I got basic backgroundColor support working in a few hours, but it took a few nights to tweak things and write tests. Fortunately, Shane is also based in Europe, so we could easily work together. When Shane showed our work on the mailing list, response from the other developers was very positive.

It would be really great if this helps drive Persona adoption amongst large websites.

Any plans for future contributions?

I’ll probably stay involved for the foreseeable future. Now that I know what I’m doing with the dialog, I would like to help out with further improvements to the login flow and website API. I’m also very interested in stabilization and/or standardization of the Identity Provider API.