Thursday, May 26, 2016

Akonadi for e-mail needs to die

So, I'm officially giving up on kmail2 (i.e., the Akonadi-based version of kmail) on the last one of my PCs now. I have tried hard and put in a lot of effort to get it working. However, it costs me a significant amount of time and effort just to be able to receive and read e-mail - meaning hanging IMAP resources every few minutes, the feared "Multiple merge candidates" bug popping up again and again, and other surprise events. That is plainly not acceptable in the workplace, where I need to rely on e-mail as means of communication. By leaving kmail2 I seem to be following many many other people... Even dedicated KDE enthusiasts that I know have by now migrated to Trojita or Thunderbird.

My conclusion after all these years, based on my personal experience, is that the usage of Akonadi for e-mail is a failed experiment. It was a nice idea in theory, and may work fine for some people. I am certain that a lot of effort has been put into improving it, I applaud the developers of both kmail and Akonadi for their tenaciousness and vision and definitely thank them for their work. Sadly, however, if something doesn't become robust and error-tolerant after over 5 (five) years of continuous development effort, the question pops up whether the initial architectural idea wasn't a bad one in the first place - in particular in terms of unhandleable complexity.

I am not sure why precisely in my case things turn out so badly. One possible candidate is the university mail server that I'm stuck with, running Novell Groupwise. I've seen rather odd behaviour in the IMAP replies in the past there. That said, there's the robustness principle for software to consider, and even if Groupwise were to do silly things, other IMAP clients seem to get along with it fine.

Recently I've heard some rumors about a new framework called Sink (or Akonadi-Next), which seems to be currently under development... I hope it'll be less fragile, and less overcomplexified. The choice of name is not really that convincing though (where did my e-mails go again)?

Now for the question and answer session...

Question: Why do you post such negative stuff? You are only discouraging our volunteers.
Answer: Because the motto of the drowned god doesn't apply to software. What is dead should better remain dead, and not suffer continuous revival efforts while users run away and the brand is damaged. Also, I'm a volunteer myself and invest a lot of time and effort into Linux. I've been seeing the resulting fallout. It likely scared off other prospective help.

Question: Have you tried restarting Akonadi? Have you tried clearing the Akonadi cache? Have you tried starting with a fresh database?
Answer: Yes. Yes. Yes. Many times. And yes to many more things. Did I mention that I spent a lot of time with that? I'll miss the akonadiconsole window. Or maybe not.

Question: Do you think kmail2 (the Akonadi-based kmail) can be saved somehow?
Answer: Maybe. One could suggest an additional agent as replacement to the usual IMAP module. Let's call it IMAP-stupid, and mandate that it uses only a bare minimum of server features and always runs in disconnected mode... Then again, I don't know the code, and don't know if that is feasible. Also, for some people kmail2 seems to work perfectly fine.

Question: So what e-mail program will you use now?
Answer: I will use kmail. I love kmail. Precisely, I will use Pali Rohar's noakonadi fork, which is based on kdepim 4.4. It is neither perfect nor bug-free, but accesses all my e-mail accounts reliably. This is what I've been using on my home desktop all the time (never upgraded) and what I downgraded my laptop to some time ago after losing many mails.

Question: So can you recommend running this ages-old kmail1 variant?
Answer: Yes and no. Yes, because (at least in my case) it seems to get the basic job done much more reliably. Yes, because it feels a lot snappier and produces far less random surprises. No, because it is essentially unmaintained, has some bugs, and is written for KDE 4, which is slowly going away. No, because Qt5-based kmail2 has more features and does look sexier. No, because you lose the useful Akonadi integration of addressbook and calendar.
That said, here are the two bugs of kmail1 that I find most annoying right now: 1) PGP/MIME cleartext signature is broken (at random some signatures are not verified correctly and/or bad signatures are produced), and 2), only in a Qt5 / Plasma environment, attachments don't open on click anymore, but can only be saved. (Which is odd since e.g. Okular as viewer is launched but never appears on screen, and the temporary file is written but immediately disappears... need to investigate.)

Question: I have bugfixes / patches for kmail1. What should I do?
Answer: Send them!!! I'll be happy to test and forward.

Question: What will you do when Qt4 / kdelibs goes away?
Answer: Dunno. Luckily I'm involved in packaging myself. :)


47 comments:

  1. +1! Finally someone has the balls to post this to Planet KDE!

    I have been pointing out this very issue for YEARS. Nobody wanted to listen. Akonadi is simply broken by design. The best thing the kdepim project could do is to go back to the 4.2 branch (IIRC, 4.3 already infected the address book with Akonadi) and port that to Qt 5 and KF5, replacing the current flawed code. (kdepimlibs must be reverted too, of course! The kdepim-noakonadi hack does not do that and so still drags in Akonadi and forces you to manually prevent it from running.) Everything since 4.3 has just been a pure waste of time on an experiment that cannot possibly work. Continuing work on it would just waste MORE time.

    ReplyDelete
    Replies
    1. The current version of kdepim-noakonadi does not need akonadi anymore at all. It's basically kdepim-4.4 with the addressbook from kdepim-4.3. I build it against recent kdepimlibs, which works fine. The only disadvantage is that kdepimlibs requires akonadi-server at build time, but nothing ever tries to start the server and it could probably just be deinstalled again.

      Delete
    2. This comment has been removed by the author.

      Delete
    3. It's not your time, isn't it? Akonadi/kdepim devs can do whatever they want in *their* free time. Why should they listen to you?

      You're free to fork a mantain a 4.2 branch of kdepim if you want to. Just don't assume that other people will do the work for you.

      Delete
    4. I'm totally support this article. Tried akonadi many times and finally give up kdepim in favor of other solutions just because of akonadi.

      Delete
  2. Akonadi has slowly driven me away from KDE. I have given up on kmail years ago when Akonadi came in, even though I had really liked it before. Now I'm stuck with Thunderbird - at least there is finally a good CardDAV extension (Cardbook) now, which integrates Thunderbird well with my Android phone via Davical. Calendaring also works decently with Lightning and syncs via Davical.
    Trojita is great, but still lacking many features. It's definitely a great IMAP client.
    I would love to see kdepim fixed. That would bring back the value of KDE for me.

    ReplyDelete
  3. I personally haven't had any issues on 16.04 and using Kmail, but would love to see more options progress, or even the existing/new options improved. My system is running so smoothly.

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Although I loved it, I switched to Thunderbird years ago for the same reasons. Time to move on.

    ReplyDelete
  6. I do use the "KDE4" version of KMail/Kontact personally and have few issues. Yet the Akonadi is what crashes it on and off. It was/is simply overambitious. I have yet to see another app/software actually accessing my email via Akonadi

    ReplyDelete
  7. I had very few issues with akonadi in the last years. There was only 1 issue in one of my 7 configured mailboxes under akonadi4. With akonadi5 I have no issues anymore. For me it works very fine and I am no KDE developer.
    It manages nearly a million mails here by the way. But indeed it is a very complex solution.

    ReplyDelete
  8. Akonadi has slowly driven me away from KDE too, as a user and as a developer... I was resetting my akonadi db every month, mail was ok for me but contacts and calendars...

    ReplyDelete
  9. I gave up on kmail2 years ago. Main problem was getting two copies of almost every mail and having to select which one to keep from a pop-up box. I used to toss a virtual coin at this point.

    I went to TB but got fed up with losing occasional long-filed mails through compacting folders whilst still online. Apparently MS mail systems have the same bug (feature?). I'm now with Claws so that I can avoid the mbox file format that was at least partly the cause of the problem. I find Claws a bit slow but at least it's safe. Still use TB for the calendar (Lightning) as it's the only one that does what I need.

    ReplyDelete
  10. My plan is to switch from kmail1 (yes, still using 1 because migration to kmail2 always failed for me) to claws-mail - http://www.claws-mail.org/

    ReplyDelete
  11. This comment has been removed by a blog administrator.

    ReplyDelete
    Replies
    1. I have deliberately refrained from any name-calling and would like to ask everyone else here to do so as well. Thank you.

      Delete
    2. You didn't want to call names, see what happened: http://famillemontel.org/index.php/2016/05/27/answer-about-akonadi-for-e-mail-needs-to-die-blog/

      Delete
  12. This comment has been removed by the author.

    ReplyDelete
  13. It can be worse. Currently they're porting KDE PIM to QtWebEngine which is not packaged and WILL NOT BE packaged for any mainstream distro anytime soon. It means that KDE PIM will not be packaged as well. But who cares?

    ReplyDelete
    Replies
    1. ?
      Fedora and Archlinux are already shipping it.

      Delete
    2. https://packages.gentoo.org/packages/dev-qt/qtwebengine
      ^ You mean this Gentoo package? :)

      Delete
  14. It's no rumor .. Sink is well along and Kube, a new application written in QML that uses the better parts of kdepim code for business logic, is using it. It is indeed simpler and should be quite a bit more robust.

    An IMAP resource is currently being worked on .. but that's where I have to chime in here and remind people things are not so simple.

    If it is indeed an issue with the IMAP protocol handling, then Kube may well continue to have issues, as the kdepim imap library will continue to be used. Christian is about to start on v2 of it, and this will bring a number of simplifications also, but it won't be a complete rewrite. So if there problems at the IMAP level, they may persist.

    That said, how Sink handles such problems can/should be rather more robust than how Akonadi currently manages. So if there are IMAP protocol issues, the resulting problems may not be as severe .. or they might be the same .. would depnd on the exact bug and how Sink responds to that exact issue.

    Debugging such issues with Sink should be a lot more straight-forward, though, as there is so much less indirection and circle-spinning that goes on relative to Akonadi.

    Finally, the applications (KMail, e.g.) also sometimes create their own issues .. or compound them with Akonadi ... so if it is indeed not a peculiarity with the IMAP protocol handling itself, it could be something that gets done with that data in the application(s) ...

    To that end, Sink provides a rather more robust mechanism for interacting with domain specific data (e.g. email). It makes handling incoming data and modifying it easier for the applications. .. and that in turn should mean fewer bugs. Not to mention having the UI in QML dramatically reduces the application size; internal complexity is held to where there is actual complexity (such as parsing out email bodies ... rather more complex than one might at first consider ...)

    tl;dr -> Sink and Kube represent a cleaner (and simpler) design without giving up a number of the nice things Akonadi attempted to (and in some cases even did) provide. That may or may not help you in and of of itself; so please engage and help with actionable feedback (which your blog certainly is not) when Kube is made available for general end-user testing. We are committed to making it the best email (and more!) client out there.

    ReplyDelete
    Replies
    1. Trouble is, after 6 *years* of "real soon right now", only to be abandoned for yet another "game changer" rewrite, the kmail team has pretty much zero credibility with the few users left.

      Delete
    2. Aaron, please help: We're using Kolab and also Kontact in our business on several Kubuntu-Desktops. It currently works but we're going to upgrade to 16.04 LTS soon and we fear that a KF5 based Kontact will not work reliably.

      We already thought about using roundcube only but "a real desktop application" is just so much more convenient.... So we're "stuck"?

      Delete
    3. Start the migration before it's too late.

      Delete
    4. Aaron - it seems I'm missing something here.

      Both https://github.com/KDE/sink and https://github.com/KDE/akonadi-next say:
      "It is currently entirely experimental and should not be considered a definite
      replacement for Akonadi 1 at this point in time."

      So are Kube and other software being built on top of Sink/akonadi-next part of validating it?

      Or are we "well along" building stuff on top still experimental base/foundation?

      Delete
  15. I have always had a problem where emails weren't marked as read in KMail. I would have to clear the filter in Akonadi control or browser (that's what it's called I think) and then redownload all emails for that filter. That really sucks. You can't expect us to only use KMail so it marks emails as read correctly.

    Two-three months ago KMail started notifying me that my imap connection is being denied. I was told that KMail is trying to establish more than 20 connections at a time. I am told the problem is resolved in latest version but not all of us are in position to get latest code.

    I am also thinking about what should I do swicth to Thunderbird?

    ReplyDelete
    Replies
    1. >I am also thinking about what should I do swicth to Thunderbird?

      Thunderbird is good. I was till recently running latest kmail on Opensuse Tumbleweed, dealing with the regularly glitches and needs for restarts.

      This posting motivated me to go back to Thunderbird, plugged in my email and password and it correctly set up all my accounts with rackspace servers, ssl etc. Sync'd very quickly and has had zero issues since.

      There were a few UI annoyances that I fixed with some plugins.

      Delete
  16. I liked the original idea of akonadi as a generic groupware data sink integrated with kde, it was very exciting. But it was always plagued with stability problems and fragile interfaces, notorious for bring plasma down with it.

    3rd party interfaces never happened, I suspect because of missing documentation and unstable ABI's.

    These days I wonder if the idea of a monster one size fits all message store has passed its use by date. I intrigued by mobile phone app concepts for the desktop - not the UI, but the idea of the server (gmail or whatever) being the primary message store and the desktop app acting as a caching interface to it.

    ReplyDelete
  17. Let's sum up. KDE lost it's browser and KHTML, KDE PIM is a failure, Plasma is a failure. Userbase is lost. All valuable KDE software is still Qt4 which is not supported anymore. That's it? The finish?
    Oh, I know! Let's port everything to QML!

    ReplyDelete
    Replies
    1. I don't miss one bit the KDE browser. Between Firefox and Chrome we have two great browsers.

      And I disagree, Plasma is not even remotely a failure. It's a really great desktop! And every application I need I either available in qt5 or being ported to it. While qt4 applications work just fine.

      Delete
    2. The only aspect where Plasma is great is crashing. And because of huge and bloated codebase nobody's able to patch and fix it. The only option is to wait til devs do something. That's not the Unix way.

      And don't tell me we don't need KHTML because there're some other browsers around. KHTML is lightweight and great to embed everywhere when you need a HTML viewer. There are simply no alternatives to KHTML.

      Delete
    3. This comment has been removed by the author.

      Delete
    4. I'm using plasma 5 here all day at work, no crashes. For years. Something's wrong with your setup or you are hitting a nasty code path that should be reported and fixed.

      Delete
    5. And from my experience, plasma 5 is way less annoying when crashing now than its early builds or even plasma 4: if the crash reporter window wasn't popping up, I wouldn't even notice.

      Delete
    6. Plasma is a different kettle of fish to akonadi, I think.

      Myself, currently using Plasma 5.6.4 (OS Tumbleweed), I find it stable, very slick looking and flexible. Dual Monitors works well for me and it the only DE I know apart from Windows and Unity that handles vertical taskbars well.

      I tried Gnome just this morning - ick. Rolled back my snapshot after 5 minutes.

      For me, it has several best of breed apps that I don't want to do with out - krunner, konsole and krdc. Every once in a while I even use baloo for file search.

      kmail used to be on the got list as well. But my email needs to be reliable. Its not, when i realised I always had my webmail open to cover for when akonadi barfed I switched back to thunderbird.

      Delete
    7. plasma always crashes/becomes problematic when lots of animations happen in the systray. i have a very problematic wifi-connection in my office, the rotating-circle-animation makes everything sluggish and in the end crashes plasma.

      as for akonadi: apart from the problem that i need to do quite a lot of actions in kmail at least twice (e.g.: delete mail, mail pops up again, ...), the search stopped working for me completely. but i stopped looking for the reason, using my phone-app now, when i really need to search a mail.

      Delete
    8. Plasma 5 works perfectly fine here; I haven't had a single crash on 3 PCs yet. Much better than my early Plasma 4 experiences years ago.

      Delete
  18. Failure? Have you even tried Plasma 5 and the ported Qt5 Apps? I don't think so. As of today it works fine in all of my installations, especially after Qt 5.6 came out. I think of of the negative commentators here should pay a bit more respect to the work of the KDE community. I mean you are using there software for free und yet you are insulting them? I don't get it.

    Cheers.

    ReplyDelete
  19. Thanks to this post, I went ahead and checked if my personal IMAP server now worked with KMail/Akonadi. I used to be unable to send mails.

    Well, turns out it does work now ! Goodbye Trojita, goodbye Google Mail, I'm back to managing all my mail accounts and aliases from a single place again, for the first time in years !

    There will always be bugs in every software... we just have to remember to check once in a while, if by chance some change in the code hasn't fixed our issues !

    ReplyDelete
  20. > Okular as viewer is launched but never appears on
    > screen, and the temporary file is written but
    > immediately disappears... need to investigate.)

    Maybe it has to do with the text "Since Okular hasn't been ported yet to Plasma, Krusader cannot load its part, like with Kate or Gwenview. When Okular will be released, this bug will be automatically fixed." found in https://bugs.kde.org/show_bug.cgi?id=356920

    ReplyDelete
  21. What do you mean "...needs to die"? When I run kmail, Akonodi already does die quite often, without letting me know. I only realize that it died when I notice I haven't gotten any email in an unusually long time. After having gotten tired of having to restart it, I went back to Thunderbird.

    ReplyDelete
    Replies
    1. You should try Kmail in ubuntu 16.04 with the backports ppa. It works perfectly for me and I haven't rebooted my system for over a week.

      Delete
    2. No, that's exactly the behaviour you can get with kdepim-16.04.1 as well. I can't use it without having a permanent GMail tab running.

      Delete
  22. For years I talked same thing.
    The Kmail was nice client at 4.1 and at 3.x series, but I just got annoyed about the designers control of the KDE project and making lots of terrible design choices by not understanding the context of each software, meaning one style, one theme and one logic doesn't always fit to different applications or so.

    KMail didn't receive a Ctrl+M for menubar hiding until in a year. Developers were against it. The same mistake was repeated with Kmail as was with Amarok 2, until developers finally bend and implemented it.

    Now while Kmail mainwindow can have a menubar hidden, the new email window doesn't!

    Then there is the common problems that is simply the legacy thing, no one wants to support HTML emails correctly, and then sniffing a email as pure text is as well more often miss than hit.

    And then comes the other things, like email should not be discussion tool, that is for newsgroups or chats. Email should be a personal (or notification for lists) where it is easy and simple to send, receive and manage emails like they would be your typical physical mail.

    Searching through email is just something that doesn't anymore work. Just give me a simple diff/find tool, not any context or such searching as I don't care. Let me select the sender, date or keyword etc and find stuff quickly.

    With Kmail I only liked about the filter bar, it worked well unlike with Thunderbird that couldn't find jack....

    ReplyDelete
  23. I switched to using fetchmail and procmail to create a maildir. Using kmail to read that maildir mostly works well, and it’s compatible with mu4e.

    And searching through my email works much better in kmail than in Thunderbird, since kmail can actually handle my 150k emails.

    So in essence, I adjusted my setup to (a) work well with KMail and (b) have a plan B.

    ReplyDelete
  24. From a user perspective, sadly I have to agree :'(

    I'm using kmail/kontact since a decade or so and it was always my favorite client by far. I liked its usability and design.

    However since upgrading from Kubuntu-14 to 16 I have serious trouble with kmail / akonadi. I send emails that never get delivered, mailboxes hang, constant fiddling with akonadiconsole, syncing takes ages, no notification when resources are offline (addressbook, calendar), ... in one word, it is really unstable.

    As longtime user I learned to expect that a certain amount of fiddling with the system is necessary after upgrades and I'm fully willing to invest this time, but this is just beyond me. Sometimes it works, sometimes it doesn't. When using email/addressbook/calendar in a production environment this is just not good enough. I'd rather have less features and more stability.

    I'll try a little longer, but as things look now I'll also have to switch to Thunderbird (who's interface I hate) and browser frontends.

    Anyway, a big thank you to the developers! For years I was happy to use kmail/akonadi (and will continue to do so on an old box which I won't upgrade)!

    ReplyDelete