For instance:

• the datetime class lives in the datetime module; careful with your imports, you will want both!
• the datetime module has a time class, but there’s also a time module
• the documentation talks at different places about “time tuples” and also about “time.struct_time” objects; they’re the same thing
• to convert a unix timestamp to a time tuple representing UTC time, you use time.gmtime(); for the reverse conversion, you use calendar.timegm() (this is the only place you’ll use the calendar module)
• to get now you use:
  • time.localtime() if you want a timetuple
  • time.time() if you want a unix timestamp
  • datetime.now() if you want a datetime object
  • time.gmtime() if you want a timetuple representing UTC time
  • datetime.utcnow() if you want a datetime object representing UTC time
  • a crowbar if necessary
• to build a datetime object from a time-tuple, you can set the fields one-by-one… or you can slice the time-tuple and unpack the list: datetime(*a_time_tuple[:6])

In other words, there’s a lot of fiddly detail to remember. So I made a chart.

Click through for (non-blurry) PDF.

It’s formatted for a4 paper and prints nicely. I used the incredible LaTeX drawing system TikZ. (On my screen the arrows have a little glitch at start and finish, but it doesn’t appear in my printout. Possibly a TikZ bug? Not sure.)

Note that it doesn’t show everything you can do; apart from the more egregious omissions (like timezone handling on datetime objects, which actually seems quite straight-forward but didn’t fit nicely into the design), I didn’t include arrows for nonsensical operations. For example you could feed a UTC time-tuple into time.mktime() (which takes a localtime time-tuple to a unix timestamp), but the result would be nonsense: the number of seconds since a date similar to, but probably not quite the same as, the unix epoch.

First the blank screen. From these instructions I added the option nomodeset for the install, and that went swimmingly. But of course on normal startup I had the same problem; I don’t even know what nomodeset means, but should I be setting it as a grub option or something? No, the answer is simpler than that: plug in a second (DVI) monitor.¹ Assuming that doesn’t have any driver problems, you’ll be able to see what you’re doing and can move on to the next hitch…

Wireless. Without a driver, wireless is completely disabled; the network manager applet doesn’t even see it as turned off, it’s simply absent. Solution: plug in a network cable instead. Duh. (It took quite some time for this blindingly obvious insight to hit.)

So at that point I had internet access and could see the screen. But these aren’t “solutions”, they’re temporary workarounds; from a bit of googling, the next step looks like it’s going to be tracking down drivers, uninstalling and reinstalling packages, and tweaking configuration settings ad nauseum.

Nope. System -> Administration -> Hardward Drivers, it found the missing two correctly, they activated without a hitch, and now I’m good to go (the extra monitor and network cable can go back into storage).

Notes:

1. I happen to have one lying around because my previous laptop is a netbook.

Neato, huh?

Pretty funny if the association is familiar. Otherwise, probably pretty incomprehensible. [via]

This, on the other hand, is.

I’ve used GNU screen for ages to keep my working environment set up the way I like it, wherever I am. I have a screen session permanently running on my office linux box, which I ssh to from wherever I happen to be.

That all changed this week, when my branch of our research institute moved to a new building. My linux box was taken offline and loaded into a truck; when it arrived at its new home, due to various cock-ups among the administration, it had no network access. (Word is they’re not going to let us ssh in anyway, sigh.)

So I’ve been playing around trying to set up my eee pc as a useful working environment. Step 1 was just copying all my configs from the desktop over to the netbook, and running everything from there instead. Imagine my surprise when some applications did not work the same way as they used to!

Specifically, screen has gained a purely awesome addition: screen-profiles provides something like the mode bar in emacs, configurable to add all sorts of goodies (battery monitor, wifi strength, lots of bits and pieces).

My other new discovery, which rather overshadows the first, is StumpWM. If screen and emacs had a baby, and brought it up to be a window manager, that would be stumpwm.

It’s a window manager written in lisp, like emacs,¹ and like emacs it’s configurable on-the-fly by lisp hacking.² Also like emacs it’s highly keyboard-driven, but more along the lines of screen: there’s a prefix key that diverts input to the stumpwm keymaps. (Actually I never realised before how similar the emacs and screen models are here; the only real difference is that screen gets away with only one keymap, hence only one prefix command. Stumpwm lets you define more if you need to, naturally.) Like screen, and unlike most window managers, you only see whatever you’re using at the moment (although you can tile windows the same way you can split emacs frames).³

I started messing around with stumpwm because the display of the eee is so tiny, and I was frustrated with the amount of space being wasted on menu bars and panels and whatever.⁴ I didn’t get very far at first, because I was trying to do everything by hand: shut down the Gnome display manager and restart X, pointing it at stumpwm. Of course all sorts of things stopped working, most importantly audio. But then I followed XSteve’s instructions for adding stumpwm as a session type under ubuntu, and magically everything worked.⁵

By “everything”, I mean everything. Audio works (I’m listening to Bettye Lavette in Amarok right now). Two-finger touch-pad scrolling works. I can split a stumpwm screen into two panes, pull firefox into one pane and emacs into the other, mouse-select text in firefox and drag and drop it into emacs. It not only works, it shows the text being dragged (otherwise I never would have tried the experiment). Global shortcut keys defined in Gnome work (so I can control the volume Amarok plays at).

I am a complete convert: stumpwm is fantastic.

I have two complaints.

The first is that I want better ways to switch windows: a list across all groups, that works like ido in emacs: filter the list of possible targets by any sort of match, rather than just prefix-matching and tab-completion.⁶

My second complaint is that both stumpwm and ido are written in (dialects of) lisp. Meaning that it’s conceivable that someone could hack bits of one into the other. Meaning that I’m awfully tempted to give it a try, instead of working on my dissertation. Shame on you, developers, for not protecting me from my instincts.

Notes:

1. Well, stumpwm is written in what emacs would call an inferior brand of lisp…
2. Yes, you can change the source code of your window manager while it’s running. And yes, once you can do this you’ll discover all sorts of reasons why you might want to.
3. And just like screen, now that screen-profiles exists, there’s a configurable mode line.
4. Emacs with no scrollbars and the font set to 8pt fits 80 characters twice, in a two-column split, which is magical for LaTeX. Viewing the pdf in Okular is less pleasant.
5. XSteve is the author of the psvn package (which adds excellent subversion support to emacs), among many other software projects. His config tips are worth checking out too.
6. Ido (‘interactively do things’) is another recent discovery; I’m completely sold on buffer-switching, and the fact that it lets me prefer .tex over .aux when opening files might be enough of a draw card to force me to get used to the way it treats backspace and tab.

(Thanks to Micha — I discovered this while helping him tidy up his texmf tree. And yes, I guess this is less down to (La)TeX and more down to my distribution, details jargon technicalities disclaimer.)

For instance, you can scan in somebody’s photocopy of a paper from 1978 using the office printer, which emails you a pdf of the scan, and run it through this file to get a single page of the original on each page of the pdf:

\documentclass{article}
\usepackage{pgffor}
\usepackage{pdfpages}
 
\begin{document}
 
% Paper starts on a right-hand page of the original book,
% we don't need the final page of the paper before it.
\includepdf[pages=1,
            angle=90,
            trim={0 0 0 135mm}
            ]{Kamp78SemanticsVersusPragmatics.pdf}
 
% For each double-page spread, rip out the left hand then
% the right hand side. The trim was set by eye.
\foreach \page in {2,...,17} {
  \includepdf[pages=\page,
              angle=90,
              trim={0 135mm 0 0}
              ]{Kamp78SemanticsVersusPragmatics.pdf}
  \includepdf[pages=\page,
              angle=90,
              trim={0 0 0 135mm}
              ]{Kamp78SemanticsVersusPragmatics.pdf}
}
 
\end{document}

(This also gives me a chance to show off try out my new code-formatting plugin. Pretty!)

Just like there’s a contrast between compiled and interpreted programming languages, there’s a contrast between compiled and interpreted ebook formats.² Pdf is at the compiled end, html is at the interpreted end. The two extremes are good at different things (just like compiled and interpreted languages); I’ll give some examples below, which point to why the reactions to the suvudu release were so extreme. The analogy with compiler technology suggests a third alternative: a level of intermediate code which is cheap to interpret but contains more compiled-in information than html and its relatives. As a concrete application, I’ll argue that language-specific discretionary hyphenation belongs in this layer.

The first complaint about pdf as an ebook format is always that it doesn’t support reflow: the page dimensions are fixed, and if they don’t match the size of the screen you’re reading on, that’s too bad.³ Html, on the other hand, is delivered to your computer as an unbroken stream of text and it’s up to your web browser to decide how to split that text into lines that fit on your screen. (I’m ignoring css; it’s a separate technology from html proper and badly-written css tends to make html pages that behave more like pdf anyway.)

I could go on talking about ‘pdf-style’ and ‘html-style’ ebook formats, but there’s a more principled distinction available: pdf is a compiled format and html is an interpreted format. By that I mean that the bulk of the layout work for displaying a pdf document is done when the document is created (‘compiled in’ to the file you download), while the bulk of the layout work for html is done when the document is viewed (‘interpreted online’).

Where the work gets done is also where the decisions get made — that is, where the control is. Here’s a reason a publisher might prefer to produce compiled rather than interpreted texts: they get to make all the decisions about how those texts appear. If a textbook is produced in both paper and ebook versions, it might be helpful if the page numbering is the same for both.⁴ Comics, too, are unlikely to benefit much from interpreted delivery: in drawng a strip or a page spread the author will probably want to control all the decisions that might be left up to the interpreter in a more text-heavy document.

So far our analogy hasn’t done much for us. Now I want to put it to work, by introducing another reason to prefer compiled over interpreted documents: typographic quality.

Go have a look at Dario Taraborelli’s page The Beauty of LaTeX; in particular, the section on line breaks, justification and hyphenation (I’ll use ‘justification’ as shorthand for all three of these processes). Taraborelli’s point is different to mine: he’s comparing different kinds of document compilers. And in fact some wysiwyg systems do do the clever and complicated things TeX does to get justification right; Adobe InDesign apparently has this feature, and for all I know the latest Microsoft Word might include it also. It’s unlikely, though, that we’ll get this kind of quality in an interpreted document.⁵

Justification as TeX does it is a complex process of simultaneously adjusting all the lines in a paragraph, to get the best possible overall quality. This is expensive, among other reasons because it can involve backtracking: an early tentative decision about the first line of the paragraph might have to be reversed if it would force things to turn out badly on later lines. The more globally sensitive the algorithm is, the more expensive it gets (should it notice page breaks and avoid widows and orphans?) but the higher the typographic quality of the result.⁶ Another thing making TeX’s justification expensive is that it invokes automatic hyphenation. Doing this right involves knowing which language the document is written in: different languages allow hyphenation at different points. It involves a complex system of (language-dependent) hyphenation rules, and some mechanism for marking by hand cases where these do not apply. All this is expensive not in processor time (as a backtracking line-break algorithm is) but in the complexity of the system itself. TeX, as a document compiler, can include hyphenation rules for many different languages and call on the particular one that is needed for the particular document being compiled. The same flexibility and scope is hard to imagine in a document interpreter that might have to run under severe hardware restrictions; it is no surprise that your web browser doesn’t even attempt hyphenation.

So here’s a reason why both publishers and readers might prefer compiled over interpreted documents. Publishers, because it lets them produce documents of high typographic quality — or more exactly, because it gives them control over the typographic quality of the documents they produce. A lot of design effort may go into a textbook, with clever placement of different elements on the page (main text, side notes, boxed ‘expert information’ sections, pointers to related material and so on), careful choice of fonts and colour schemes and what-have-you. A publisher might easily refuse to put that work to waste in the electronic version of the book, by giving control over to the interpreter doing the display. There is rather less evidence, I’m afraid, for readers who appreciate typographic quality. The suvudu reaction shows that quite clearly: these folk wanted the content, and forget about whether the form is pretty or not.⁷ That’s the interpreted-ebook market; high-quality typesetting is for the compiled-ebook market. I hope I’m not the only person in that market… (Feedbooks seem to be using LaTeX to generate pdfs with custom page dimensions, from free sources such as Project Gutenberg. Unfortunately the typographic quality is still low; there’s no way their offerings are a replacement for a LaTeX production that has had some human design attention. Still it’s a definite cut above html or plain text. If only they offered the LaTeX sources as well…)

I promised some concrete payoff from the compiled/interpreted analogy, and here it is. There’s a way to get some of the benefits of both approaches, in language design: you compile your handwritten programs to intermediate code, which gets interpreted on the platform the program runs on. The idea is the the compiler part of the process can do expensive things like type-checking and optimisation, while the interpreter can be tightly matched to the runtime environment. We shouldn’t stretch the analogy too far, but here’s a suggestion for something that belongs in the intermediate code level: discretionary hyphens.

Suppose we had a compiler that did TeX-style hyphenation analysis: it knew what language the document was written in, and it used hyphenation patterns for that language to find all the valid hyphenation points in the document. The intermediate-code level would be something like html, but with those hyphenation points marked with ‘discretionary hyphens’. And the interpreter level could use that hyphenation information to improve the quality of on-the-fly justification (reflow) based on changing the display area or font size; I imagine that just adding reliable hyphenation would already improve the line-by-line justification enormously.

What was the point again?

I’ve wandered, it’s true. Here are the main points. Compiled and interpreted formats (static design like pdf and reflowable like html) are good at different things, just like compiled and interpreted languages. Compiled formats put the control in the hands of the document designer, and give high typographic quality — at least when the designer puts the effort into making this happen! Interpreted formats respond to the details of the environment in which they’re viewed (display area, the user’s preferred fonts and font sizes, etc.). On the downside, compiled formats are inflexible (in particular, viewing them on devices with different display areas is usually unpleasant) while interpreted formats tend to produce low typographic quality and enforce boring design. The analogy suggests that ‘intermediate code’ might be a way to increase the typographic quality of interpreted formats, and descretionary hyphenation is a specific example where this could work.

Notes:

1. This was a great marketing move: they took several sf/f series that have proven popularity, and gave away the first book in each. Way to get new readers hooked: “First time’s free…”
2. Sure, this is more general than ebooks. Let’s keep this concrete and specific though, otherwise I’ll end up drowning in abstraction and generality.
3. Actually pdf can do some sort of reflow, these days. Nobody understands or uses it yet though.
4. My argument for pages instead of an unbroken column of text… will have to wait for another day.
5. A Word file is a compiled document, but it’s one you can edit. The analogy only goes so far.
6. Only to a point, presumably: we are automating the process of aesthetic judgement, which —I hope!— can only be approximated.
7. As an aside, interpreted formats tend to be easier to convert losslessly than compiled ones, as anyone who has tried to copy-paste from pdf will attest. That’s another reason those folk may have preferred an interpreted format: it’s easier to share across different devices, by making the conversions as needed.

DRM for digital music cannot succeed.

This idea is slowly getting taken seriously, partly due to overwhelming evidence and partly due to the efforts of activists like Cory Doctorow. Doctorow has a nice analogy, which you can read here, showing why DRM is a basically broken idea. Stripped of its rhetorics (which are the point, by the way; if you’re unconvinced by my exposition I definitely recommend his) the idea is this: DRM provider Alice tries to hide a piece of content from Carol (who isn’t supposed to get access to it) while revealing it to Bob (who is). Only no DRM system can prevent Bob and Carol being the same person (a criminal intending to sell copies of Your Band’s Latest Album buys it legitimately; now the DRM has to let him get it on his Bob days but not let him get it on his Carol days).

There’s more subtlety to this idea than Doctorow admits, though. There are a bunch of factors that make digital music particularly un-DRM-able (to coin a barbarism). Here’s a list of factors ranging from highly technological to highly social (the middle entries are sort of a mixture of both):

• audio output makes pipelining easy
• cds and the mp3 format are de-facto standards
• digital music gets played on general-purpose computers
• enormous numbers of people listen to music digitally
• there are social assumptions that swapping music is ok
• there’s a general perception of record labels as exploitative

Let’s take them in order.

Pipelining

Doctorow’s analogy assumes that Bob and Carol both want the same thing from their file. For music that’s a pretty safe bet, because there’s basically a single bottom line: Bob (the legit listener) needs to be able to get the music out; for reasons we’ll come back to in a moment, he almost certainly needs to be able to get it out digitally (not just as audio output). But that means Carol can take that output and pipeline it straight back in as the input to something else: a cd burner, her mail client, a file-sharing program, or whatever. (Actually even just having audio output would let her do this to some extent, but she would lose some of the information in the original, even with the best-quality gear.)

De-facto standards

Cds and the mp3 format are de-facto standards for digital music. That means attempts to crack DRM can be pretty tightly focussed: get the music off a protected cd, convert it to mp3, release it. And cds are pretty much impossible to DRM effectively, because the same physical technology (a cd drive) gets used both for data and for music cds, when they get played on …

General-purpose computers

A cd can’t really tell when it spins up in your computer’s drive whether it’s being pipelined into an excellent music program or into an mp3 ripper. I said above that Carol would lose quality in re-recording from audio output, but if she can put the cd into her laptop she doesn’t even have to do this. She just has to trick whatever DRM system is on the CD into believing that she’s only accessing the data for the purposes of playing the music; once she has the data, she can do whatever she likes with it. And on a computer that lets you install your own software (your laptop or pc, that is, but not your iPod — at least not without some serious extra hassle) that’s going to be very easy indeed.

Enormous numbers of users

I said “not without some serious hassle”; in the case of music, that doesn’t actually matter so much. So many people listen to digital music that there’s bound to be someone out there willing to put in the time and effort to rip Your Band’s Latest Album, even if you make it difficult. Having cds and playing-on-your-pc as a de-facto standard makes this a lot easier, of course.

Social attitudes

The last two points are pretty fuzzy. It seems to me, based on the people I know and the stuff I read online, that these are prevalent attitudes; there might be quite some self-selection involved here though. That said, I see two attitudes: the assumption that swapping music among friends is basically ethically ok, and the assumption that little if any of the money we pay for music gets back to the artists. These attitudes are pretty clearly going to make folk more comfortable with sharing music without paying for it, whether that’s ‘technically’ illegal or not. It doesn’t even matter if the perception of record labels as exploitative is true: the assumption that this is the case makes people more comfortable with ripping them off, since it’s only fair turnabout.

I don’t think I’ve said anything surprising so far. But let’s see how these observations carry over to ebooks.

No pipelining

This is the most important point where the analogy between music and ebooks breaks down. There is no equivalent to ‘audio output’ for ebooks. You can always, without anything more complicated than a piece of cable and the right plugs, pipeline your music if you can play it. The equivalent low-tech solution for an ebook would be photographing a screen displaying each page in turn.

There’s another post hidden in that observation, about typographical standards and the differences between pdf and on-the-fly adjustable technologies like html, mobipocket and so on. But the basic point is: what the reader needs to see is not necessarily what the reproducer needs to get at.

That’s partly because there are as yet…

No de-facto standards

That’s not really true: there are too many de-facto standards, which is as good as having none at all. There are a plethora of physical reading devices out there, all with different screen dimensions and technical qualities (e-ink, backlit screens, link-based navigation and what-have-you). Content that is right for one won’t necessarily be right for another — both in the trivial sense that a file for the Amazon Kindle won’t be readable on a Sony Reader, and in the less trivial sense that a pdf carefully formatted for the dimensions of the iLiad won’t be pleasant to read on your netbook or laptop.

The lack of standards also means there isn’t a central clearing-house for DRM-cracking efforts the way there is with music: rip the cd to mp3 and you’re done, but get ahold of a pdf and you have to convert it to half-a-dozen other formats before you’ve satisfied everyone. Tools like calibre are going to change that, eventually, but they haven’t gotten there yet.

Special-purpose devices

Part of that lack of standardisation comes from the variety of physical devices being used to read ebooks. The most important of those devices are dedicated ebook-reading systems like the Kindle or the iLiad. Amazon’s Kindle is a closed device: you’re not supposed to be able to run your own programs on it.¹ And, contra Doctorow, a DRM scheme based on such a device can work. Again, it won’t work for music, because even plain audio output can be pipelined; if you make the output quality too bad to use for re-recording then the device is no good for listening to music any more. But that’s not the case with an ebook reader: the output is an image on the screen, and there’s no way to turn that into something as versatile as the original file was if the device doesn’t let you get access to that file.

Limited numbers of users

If enough people were using ebooks, even this wouldn’t matter. We would have pirated editions consisting entirely of photographs of authorised editions. It happened to Harry Potter, it could happen to everyone!

Well yeah, except that it won’t. There are too many books and too few people who could be bothered to perform this incredibly tedious work. (Unfortunately for DRM, incredibly popular titles like the Harry Potter series will continue to attract this kind of insane effort, and those are exactly the ones that ought otherwise to give the largest returns for the publishers. But it’s not something for them to be worried about in the general case.)

Different attitudes?

Finally, many attitudes to books are rather different at this point to attitudes to music. This is again pretty fuzzy and probably extremly personally biased. It’s also very much based on attitudes to paper books, which might not carry over to ebooks at all. For instance, paper books are for lending; you lend cds too, but the cd you lend can be copied while the paper book can’t.² I doubt this attitude will carry over to ebooks, because by now we’re completely used to the idea that data can be copied: I won’t want to ‘lend’ you my ebook because there’s no non-DRM reason why I should be deprived of it while you’re reading it. Again my personal feeling (which is probably not representative of any significant market) is that publishers don’t get the same assumptions of evildoing that record labels do. Here I don’t see any reason why electronic publishing should change matters; on the other hand if major publishers get DRM-crazy and end up associated with the likes of the RIAA, their credibility is going to take a hit (at least, and to belabour point one final time, among the people who are already writing and reading blog posts about DRM).

What does it all mean?

So I hope you’re convinced that ebooks are not like digital music, and that DRM being completely braindead for the one does not automatically mean it will be braindead for the other. In particular, the fact that you order your Kindle ebook through the Kindle, download it using the Kindle’s own network connection, and view it on the Kindle screen without ever having access to the file itself, means that DRM is technologically feasible on that platform. (Bob gets to read; Carol doesn’t get to copy.)

Yet there is hope! There are so many hardware options, and so many competing standards, that restricting to Kindle-only (or Sony-only, or…) for the sake of DRM will be a bad business move for publishers unless Amazon manages to strangle all the competing alternatives before they get serious market share. So long as you can get the original DRM-locked files onto a general-purpose computer, the DRM is eventually going to collapse: Bob needs access, but Carol can dress up as Bob. So we need to lobby for the ability to share ebooks across multiple devices, to make our own backups, whatever else gives us an excuse to get the file out of that device and onto our own computers. If ebooks are sold under the assumption that this is possible (and if enough people are reading them to make cracking the DRM a fun hobby for someone), these DRM schemes will eventually crumble.

What still has me seriously worried, I have to admit, is that I don’t yet see a sensible alternative to DRM-based restrictions that still gives me high-quality ebooks to read and that will be attractive to publishers. A music junkie who will fill 150G with mp3s is going to carry on beyond 150G. I’m not sure that’s the case with books: one can only read so much in a lifetime, and 150G might be pushing that limit. A swapping session with a new friend might give me a couple of days worth of new music, including a couple of albums that send me looking for more stuff by the same people. The same session with someone with a nice library might mean that I never have to buy anything by Gene Wolfe: not many people have complete album collections for any but their favourite bands; I bet though that everyone who reads beyond the first book of a series will accumulate all the others if they’re easily available. It could keep me reading for the next two or three months. That has to be a scary thought for publishers.

The most obvious alternative to DRM is simply to keep some titles non-digital, or to provide them but in deliberately low-quality versions to give an incentive to buy ‘the real thing’. I don’t like that alternative any more than I like DRM: I want my ebooks to be of high typographic quality. Next ebook post (which almost certainly will be after read-an-ebook week is over, perhaps long over) will be about this issue, and (again rather depressingly) about how often this is not the case: demonstrably so at present, and I think likely also for quite a while to come.

Notes:

1. The iLiad, on the other hand, is largely open-source. I own one and it’s a fun thing to play with, as well as good for reading books on. Go the iLiad! Apart from the name, which is quite frankly embarrassing.
2. Yes, it can, but not easily and the result is not nearly as pleasant an object as the original. It still happens among students for textbooks, though, that’s for sure.

My hosting service Dreamhost puts several users on a single machine. That means if someone uses up lots of cpu resources with a hungry process, the others suffer. So Dreamhost monitors processes and kills the ones that seem to be making trouble.

I’d read lots of folk saying that a WordPress blog can be resource-hungry, and that it’s smart to install a caching plugin to slim things down a bit. Wp-cache gets a lot of good press, so I installed it.

Unfortunately it has a bug. So the caching module that I set up to ensure that Dreamhost wouldn’t need to clobber my processes… made Dreamhost clobber my processes. Sigh.

Probably I don’t need caching anyway. Please don’t slashdot this post.