So far, so good, this is the theory, now let's look at reality. In Gentoo, the main KDE distribution is split into 295 (!) packages. Many dont have any tests (oxygen-icons comes to my mind as a nice example). Of those that have, 37 now have all the test routines hard disabled, and I guess a few are missing that I still have to add to the list. Why? Well, first of all, you have to know that compilation usually runs in Gentoo as an unprivileged user detached from any graphical desktop session, and this user also runs the test routines. In addition, tests should also run fine "offline", i.e. without internet connection. One of our developers is running a great thing called a "tinderbox", where build and unit tests in various configurations are automated- in an isolated environment, and possibly even without network access. So, what can go wrong?
- The tests need X11. Well, actually we can get around this, with a nice hack called virtualx. A background framebuffer X server is started for the test phase only, and the tests are redirected to that display. Works even sometimes. :o)
- The tests need a dbus session bus. I think we could get around this too, but nobody bothered to do it so far and I'm not a dbus expert.
- The tests need an entire KDE desktop to interact with. Yes, that happens, and unfortunately, we have to disable them in this case, see above.
- The tests try to download a data set from, say, IMDB, CDDB, or similar. Bad, bad, bad, see above.
- Next, the tests pop up dialog boxes asking for user interaction. A window asking for a GnuPG passphrase is my personal favourite...
- And last, what I particularly like is a unit test that fails... and after digging into the test log, I see a fat comment "This does not work yet and needs to be fixed!"
PS. This blog post took slightly longer to write since a libreoffice-3.5 build in the background went berserk... load 65, >180 concurrent compilers... :D