Drink The Lemonade
Dave Cridland posted to maemo-developers again today, spuring me to spend my lunch time checking out his email client instead of doing something productive like eating. It's very interesting.
Basically he edits the RFCs for Lemonade, which is a IETF standard for email on mobile devices. Basically it is a set of extensions for IMAP and SMTP that improve their performance over high-latancy low-bandwidth connections (like bad wi-fi or GPRS), such as forward-without-download and efficient re-synchronisation commands. Existing extensions such as a useful IDLE implementation are also used to push new mail notifications to the client.
To test the ideas in the real world there is Polymer. Polymer is a simple IMAP client that uses the Lemonade extensions as much as possible, so is ideal for mobile use. There is more: it uses ACAP to store the settings on the Internet (Dave also provides free ACAP accounts) so it's configure once, run anywhere. I've just tried it on my laptop, and it works nicely.
But it gets better. There is also Telomer, an email client using Lemonade and designed for Maemo. I've just tried it out, it may be seriously lacking in features at the moment, but it works. Features can be written over time, but it takes serious thought to get something as fast and usable as Polymer on mobile devices. I'll quote Dave:
That all said, once it's started, it connects to the mailserver over GPRS in about 10 seconds and pulls up the summary listing in 15, with a mailbox size of 2,882 messages. With a mailbox with 33,732 messages in, it's a little slower at around 25 seconds. That's still way faster than a desktop client on a LAN, of course, unless that desktop client is Polymer - in which case it'll seem remarkably slow.
This is what we need for a mobile email client. I know I've found my email client of choice on my new N800.
NP: Wish You Were Here, Pink Floyd
Posted by Chris Cunningham at Tue Jan 23 14:11:38 2007:
Tinymail Tinymail Tinymail. Tinymail? Tinymail Tinymail. Tinymail Tinymail Tinymail Tinymail. Tinymail! Tinymail Tinymail, Tinymail Tinymail Tinymail. Tinymail: Tinymail. Tinymail Tinymail Tinymail? Tinymail...
- Chris
Posted by Luis at Tue Jan 23 14:12:28 2007:
Posted by Ross at Tue Jan 23 14:19:20 2007:
- Chris
Luis: M-Box is the best email server, as it was written by the same people. Cyrus is pretty much Lemonade compliant though, if you can't have that use Dovecot. Sadly Courier doesn't support the extensions.
Posted by Philip Van Hoof at Tue Jan 23 14:19:52 2007:
But regretfully it only does IMAP and it doesn't make it possible to read E-mail offline, which is most of what an E-mail client is all about.
Which makes me wonder why one would not simply use a web interface like Squirrel, IMP or Roundcube with the browser that nearly all mobiles get shipped with (or GMail, Hotmail and yahoo)? Those web pages (and for some of those webmail softwares, AJAX requests) are not going to be significantly larger than receiving the summary directly from the IMAP.
The very reason for existance of tinymail is the offline features. Getting mime parts of a message and summary information from an IMAP server is by far the most easy part of the game.
The RFC's he's influcencing, however, are very interesting indeed.
Posted by Philip Van Hoof at Tue Jan 23 14:22:34 2007:
Posted by Ross at Tue Jan 23 14:30:37 2007:
Which makes me wonder why one would not simply use a web interface like Squirrel, IMP or Roundcube with the browser that nearly all mobiles get shipped with (or GMail, Hotmail and yahoo)? Those web pages (and for some of those webmail softwares, AJAX requests) are not going to be significantly larger than receiving the summary directly from the IMAP.
The very reason for existance of tinymail is the offline features. Getting mime parts of a message and summary information from an IMAP server is by far the most easy part of the game.
The RFC's he's influcencing, however, are very interesting indeed.
Sigh.
Message caching to do offline reading is just a feature missing from Polymer. I don't care if Camel can do offline mail, when Telomer in Python on my 770 can connect and download headers from my mail server faster than Evolution on my desktop can.
Saving 4 bytes per message header in memory isn't what the user wants, as Camel is light enough now. They want a new IMAP implementation in Camel that supports all of the Lemonade profiles, so that using Camel over slow links isn't painful -- because its not exactly fast over 8mbit ADSL.
Posted by Philip Van Hoof at Tue Jan 23 14:48:47 2007:
Message caching to do offline reading is just a feature missing from Polymer. I don't care if Camel can do offline mail, when Telomer in Python on my 770 can connect and download headers from my mail server faster than Evolution on my desktop can.
Saving 4 bytes per message header in memory isn't what the user wants, as Camel is light enough now. They want a new IMAP implementation in Camel that supports all of the Lemonade profiles, so that using Camel over slow links isn't painful -- because its not exactly fast over 8mbit ADSL.
Then again, improvements to Camel are being made for this reason. A lot bandwidth consumption improvements have happened to camel-lite. A lot more will happen in future. This is not going to be unique for either tinymail nor Polymer nor Telomer.
Receiving less than the actual visible information itself is not really possible (except when compressing things, of course, which is not something impossible or something that isn't being done already by multiple E-mail solutions).
Next to all that are by far not all IMAP servers supporting the Lemonade features yet. And this isn't very likely going to change soon (imagine the amount of people currently using the IMAP features of Microsoft Exchange).
But don't get me wrong, Ross. Those IMAP protocol extensions are interesting and they will go into camel (or, camel-lite at least). Piece by piece.
But they don't necessarily require online-only access with the IMAP server.
ps. That "it's just a feature part", might surprise you a little bit in complexity. But of course, everything is doable and possible.
Posted by André at Tue Jan 23 15:21:59 2007:
Receiving less than the actual visible information itself is not really possible (except when compressing things, of course, which is not something impossible or something that isn't being done already by multiple E-mail solutions).
Next to all that are by far not all IMAP servers supporting the Lemonade features yet. And this isn't very likely going to change soon (imagine the amount of people currently using the IMAP features of Microsoft Exchange).
But don't get me wrong, Ross. Those IMAP protocol extensions are interesting and they will go into camel (or, camel-lite at least). Piece by piece.
But they don't necessarily require online-only access with the IMAP server.
ps. That "it's just a feature part", might surprise you a little bit in complexity. But of course, everything is doable and possible.
+1 for a new IMAP implementation in Camel. Does Camel support the Idle extension by now? IMAP is the reason I'm staying with Thunderbird since IMAP in Evo is a mess.
Posted by Philip Van Hoof at Tue Jan 23 15:40:49 2007:
Posted by Dave Cridland at Tue Jan 23 16:04:17 2007:
"[...] he edits the RFCs for Lemonade [...]"
Well, not quite. I edit a couple of the drafts - nobody ever edits RFCs as such. There are other editors and authors aside from me, and there's lots of documents I don't edit at all.
Posted by André at Tue Jan 23 16:21:36 2007:
Well, not quite. I edit a couple of the drafts - nobody ever edits RFCs as such. There are other editors and authors aside from me, and there's lots of documents I don't edit at all.
+1 for a new IMAP implementation in Camel. Does Camel support the Idle extension by now? IMAP is the reason I'm staying with Thunderbird since IMAP in Evo is a mess.
Posted by André at Tue Jan 23 16:28:12 2007:
Sorry for duplication. (Just hit Reload.) Feel free to delete above.
Philip: I filed and commented on some bugs in Evo that were mostly rejected by fejj. Apart from this I unfortunetly don't have the time for active development besides @work but would love to test. :)
Posted by Dave Cridland at Tue Jan 23 16:29:43 2007:
Philip: I filed and commented on some bugs in Evo that were mostly rejected by fejj. Apart from this I unfortunetly don't have the time for active development besides @work but would love to test. :)
Philip: Just for laughs, Exchange actually does support some Lemonade features. For instance, it's one of the very few implementations of ESMTP's BINARYMIME and CHUNKING, both of which are needed, and neither of which are well supported by the majority of servers.
You're certainly right in saying that there aren't many Lemonade capable servers right now - but what would be the point, when there's only one Lemonade capable client? And yet Cyrus IMAP is already full RFC4550, Oryx Mailsystems provides both IMAP and ESMTP, Isode provides both too. I know of other major developers working on Lemonade support too.
Now, offline access. There is absolutely no point whatsoever in downloading all your email to a mobile device. My email wouldn't fit on a 770, for starters. Making it available offline, on the other hand, is definitely useful - and in fact, if you shut down WiFi while using Telomer, you can look at messages it has cached. All I have left to do on that front is asynchronous mailbox synchronization and some mailbox listing caching. It really is just a feature. The forward without download took a couple of months to get right, on the other hand.
As to why it's still better than a webmail solution, just one word: "Push".
Personally, I really hope that Camel, Thunderbird, et al all do get good IMAP support, and become Lemonade-capable. I'd quite happily ditch client development in an instant in favour of someone else's better alternative. Done that one with other projects quite happily. But right now, reading my email on the train with a laptop is simply not practical with these, and it's a lot of work to make it happen.
Posted by fejj at Wed Jan 24 01:24:11 2007:
You're certainly right in saying that there aren't many Lemonade capable servers right now - but what would be the point, when there's only one Lemonade capable client? And yet Cyrus IMAP is already full RFC4550, Oryx Mailsystems provides both IMAP and ESMTP, Isode provides both too. I know of other major developers working on Lemonade support too.
Now, offline access. There is absolutely no point whatsoever in downloading all your email to a mobile device. My email wouldn't fit on a 770, for starters. Making it available offline, on the other hand, is definitely useful - and in fact, if you shut down WiFi while using Telomer, you can look at messages it has cached. All I have left to do on that front is asynchronous mailbox synchronization and some mailbox listing caching. It really is just a feature. The forward without download took a couple of months to get right, on the other hand.
As to why it's still better than a webmail solution, just one word: "Push".
Personally, I really hope that Camel, Thunderbird, et al all do get good IMAP support, and become Lemonade-capable. I'd quite happily ditch client development in an instant in favour of someone else's better alternative. Done that one with other projects quite happily. But right now, reading my email on the train with a laptop is simply not practical with these, and it's a lot of work to make it happen.
I have to agree with Ross, here, (and, it turns out, Dave) in that offline support is just a feature and not even all that difficult to implement given the IMAP client code is nice enough to work with (which it probably is).
The Evolution IMAP code should be dropped like a dead horse and rewritten completely from scratch, it's just not fixable. There have been a couple of noble attempts at rewrites but none were ever completed, sadly.
The CHANGEDSINCE stuff looks exciting to me, it would cut down on a lot of IMAP traffic when selecting a folder which is where a big source of the slowness exists.
I must admit I've only skimmed the drafts, but I thought I saw something about being able to restore state at connect time (selection state, etc). I'm not sure how needed that really is (I presume this is more or less used when reconnecting during the same client session? I mean, if the client exits and reopens, it can save what folder had been selected itself and just return to that state). As far as reconnecting during the same session, while I think having state saved on the server is a nice convenience, I'm not sure it's really needed... I solved this problem client-side in one of the noble attempts at an IMAP rewrite mentioned above (imap4 module in camel).
Anyways, I think Push-IMAP stuff has some very good ideas, etc... hope to see someone implement it for Evolution.
Posted by Dave Cridland at Wed Jan 24 12:13:20 2007:
The Evolution IMAP code should be dropped like a dead horse and rewritten completely from scratch, it's just not fixable. There have been a couple of noble attempts at rewrites but none were ever completed, sadly.
The CHANGEDSINCE stuff looks exciting to me, it would cut down on a lot of IMAP traffic when selecting a folder which is where a big source of the slowness exists.
I must admit I've only skimmed the drafts, but I thought I saw something about being able to restore state at connect time (selection state, etc). I'm not sure how needed that really is (I presume this is more or less used when reconnecting during the same client session? I mean, if the client exits and reopens, it can save what folder had been selected itself and just return to that state). As far as reconnecting during the same session, while I think having state saved on the server is a nice convenience, I'm not sure it's really needed... I solved this problem client-side in one of the noble attempts at an IMAP rewrite mentioned above (imap4 module in camel).
Anyways, I think Push-IMAP stuff has some very good ideas, etc... hope to see someone implement it for Evolution.
fejj: The CHANGEDSINCE stuff is very good at reducing traffic, yes. It's also good for small devices because it means much less local filesystem access, which is also a slow thing to do.
The reconnection stuff turned out to be rather complex for servers to do, so we redesigned that, so it's now a reselect, and doesn't use any state on the server at all. It basically adds client state data to a SELECT, and turns selection and full resynchronization into a single round-trip for the common case. (And even most uncommon cases, actually). The server can provide some state in terms of keeping a record of which messages were expunged when - that does save a lot of effort for the client, and it's not a lot of state to keep.
But be wary of talking about "Push-IMAP" - there's an old proposal by Oracle called "Push-IMAP", or simply "P-IMAP", that's largely been abandoned, except by a couple of companies who lept onto that bandwagon a bit too early. Most of the good ideas from it were turned into Lemonade extensions, some copied more or less wholesale, some were redesigned from scratch.
Posted by fejj at Wed Jan 24 21:01:43 2007:
The reconnection stuff turned out to be rather complex for servers to do, so we redesigned that, so it's now a reselect, and doesn't use any state on the server at all. It basically adds client state data to a SELECT, and turns selection and full resynchronization into a single round-trip for the common case. (And even most uncommon cases, actually). The server can provide some state in terms of keeping a record of which messages were expunged when - that does save a lot of effort for the client, and it's not a lot of state to keep.
But be wary of talking about "Push-IMAP" - there's an old proposal by Oracle called "Push-IMAP", or simply "P-IMAP", that's largely been abandoned, except by a couple of companies who lept onto that bandwagon a bit too early. Most of the good ideas from it were turned into Lemonade extensions, some copied more or less wholesale, some were redesigned from scratch.
