Monday 8 December 2014

The Pillars of KDE 'Now'

Who here remembers the "Pillars of KDE 4"?

As you can see, I asked the Internet (and got little response). I was attending MeetingC++ as I wrote that tweet and spoke to some KDE folk who were attending. Between us (and online) we could remember:

Oxygen, Sonnet (remember that?), Solid, Plasma, Akonadi and Decibel... This list in one way or another might be wrong. Not so cool. But you know what is cool? People remembered the Pillars of KDE 4 existed and the effect they had.

What Were Those Pillars All About?

For me, the Pillars of KDE 4 were a battlecry: "KDE 4 is coming and these are its foundations! Look out!". Or something like that. When I first got involved with the KDE community KDE 4 development was in full swing and the first release was about 8 months away and these "pillars" were everywhere. We used them internally as points of coordination and we used them externally as points of marketing.

In my recent conversation I heard the pillars referred to as "just a piece of marketing". Humph. No, this was only oneof their functions and, for me, by far the least important.

The Pillars of KDE 4 were talismans. The community's good luck charms that we rallied around. The "things" that helped us to identify what was crucial to KDE 4. For me the keyword was "experience". All of these talismans were keys features of the KDE 4 experience; key technologies that were going to ensure our users (old and new) enjoyed "experiencing freedom".

I love "Experience Freedom" as KDE's tagline: what more prominent experience do computer users have, then their desktop?

Step Back In Time With Paul Adams

Once upon a time I was a teenager. It was back then that I discovered Free Software and first become involved. I was using Red Hat Linux or SunOS at school. GNOME was not to my taste (although not bad) and CDE was an affront. I build a preview of KDE 3 from a magazine cover CD and never looked back.

Why?

KDE has always been an extremely well-integrated experience. Sure, there are always things to be "polished" but, in general, everything worked and, more importantly, everything worked together. KDE was the one Free desktop where everything felt nicely integrated. It felt like a thoughtful product.

But KDE is no-longer a desktop. Remember? Wikipedia tells us:

KDE (/ˌkdˈ/) is an international free software community producing an integrated set of cross-platform applications designed to run on LinuxFreeBSDSolarisMicrosoft Windows, and OS X systems. It is known for its Plasma Desktop, a desktop environment provided as the default working environment on many Linux distributions, such as openSUSEMageia and Kubuntu and is default desktop environment on PC-BSD, a BSD operating system.

So What Is KDE?

The official party line (as supported by the Wikipedia entry) is that KDE is the community. KDE is a community of skilled individuals who all pull together to produce various ingegrated technologies, one of which happens to be a desktop (for various platforms). Many of the other projects directly relate to that desktop.

Hmmm... But wait a second. We have a separate definition, in the form of the KDE Manifesto:

We are a community of technologists, designers, writers and advocates who work to ensure freedom for all people through our software.

OK, so this is certainly compatible with the previous definition. Although perhaps a little less specific. Again, the clear point here is that KDE is a community. And this is something I feel increasingly dissatisfied with. What was wrong with KDE being a desktop with a well-integrated collection of awesome applications (which were also able to be run independently of that desktop on all major platforms)?

No really, what?

The Great KDE Disintegration

I first got involved with KDE because I was studying the management of Free Software communities for my PhD. In particular I was concerned about how communities remain productive. At the time I regularly made one specific warning: moving from centralised VCS to distributed will have enormous social implications way beyond the technical ramifications if the process is not monitored and managed properly.

Why? Because the VCS is a tool we communicate through. It is the most crucial tool we communicate through and it dictates how we can build our product(s). The way in which we design our software structure is constrained by how we communicate. Fundamentally change your communication structure and you will be forced to fundamentally change how your structure your code. This is called Conway's Law; I've mentioned it before in this blog and elsewhere. I dare you (nay, double dare you) to try and tell me that somehow it does not apply to Free Software or, more specifically, KDE. Go on. Give it a try.

With SVN KDE had a massive and unwieldy, but integrated codebase (heck, it was all in the same repo!). The community was also massive and unwieldy. But it was well-integrated and this was reflected in the product. KDE 3 was a phenomenon and so was KDE 4 early on.

Then we switched to GIT.

KDE became a collection of projects because each component of the community got its own repo. And we stopped working together as much. Our development became silo'd. There was even some weird behaviour within these silos. (My personal favourite... Adriaan de Groot being told to create his own repo for FreeBSD integration work for a particular application. Apparently creating a branch to work on in the existing repo was a bit too much to ask.)

Then it happened.

We became so disintegrated as a community that this became our branding. We, explicitly, stopped being a desktop project and became a collection of projects, one of which happened to be a desktop.

Let me be clear:

The VCS changed, this forced changes in the community structure and how we communicated, which encouraged us to rebrand in order to closer match our new communication patterns and how we view ourselves.

And, thanks to Melvin Conway, KDE became a less-integrated experience.

The Pillars of KDE "Now"

KDE 4 had its pillars. As the community disintegrated they fell to the wayside.

The KDE community continues to produce excellent technology. In particular, Plasma 5 is shaping up very nicely (although it is still not quite "there" enough for me to use it as my main desktop yet). But, inevitably, the fractured nature of the community will reflect in the user experience and, at that point, one of our killer features (being "the integrated experience") will be gone.

So I'm left wondering, what are the pillars of KDE now? What are the "things" we, as a community, can all get behind and say "these are the things that underpin the KDE experience".

Akonadi? Baloo? KF5? Plasma? Telepathy? KTeaTime? Heck... I'm gonna throw Akademy into the mix.

In Conclusion

Switching to GIT has brought many benefits to KDE (none of which are discussed here). But it has caused us social problems which will inevitably reflect in our products. It already reflects in our messaging, so it is really just a matter of (not much) time.

I think it is about time we took action to socially reintegrate before becoming a totally disperate collection of technologies. My 2c: KDE was strongest when we were all marching towards KDE 4, when the "pillars" were our talismans and we were one in community and vision.

So, tell me, what are the pillars of KDE "Now"?

Thursday 13 November 2014

KDEPIM: Any More Guesses?

Yesterday I posed a somewhat cryptic question: what happened in KDEPIM in late 2010 that might have caused such a dramatic spike in an unnamed (but real) metric? I want to leave the conversation running a little bit longer. So here is some further food for thought, to keep you guessing...

Some things to note:

  • The new metric is developer count... the number of unique contributors to KDEPIM each month;
  • Careful! These two plots are not on the same scale. Note the two different Y-axis scales. Mystery plot on the left, developer count on the right;
  • It may (or may not) be important that the huge spike in the mystery metric just so happens to be followed by a significant (for some definition of significant) drop in developer count. Again... I'm not going to tell you what the mystery metric is, but I will say that, in this case, causality is not out of the question. The huge spike in the mystery metric arguably, in some small way, might have contributed to the later drop in developers.

I'm now going to leave this "out there" for a day or two before getting back to it. This buys you guys some time to come up with some more suggestions as to what the mystery metric might be (and what caused the spike in it).

It also buys me some time to find out (for definite) what caused the spike... After all, I do know what the mystery metric is!

In Conclusion

Yes, I do know that the previously-red plot is now green and the new plot is red. One day I will master GNUPlot.

Tuesday 11 November 2014

The Great GNOME/Groupon Spat of 2014

If you have had your head in the sand today you might well have missed the story that the GNOME Foundation and Groupon are is a disagreement over the use of "GNOME".  I already shared my thoughts in Tweet form earlier... But now I'd like to talk a little more. Not about the event, but about how we (the Free Software community) reacted to it.

Where Do We Get Our News?

The first I heard about the GNOME/Group affair was on Facebook. A former colleague and friend of mine posted a link to the request for aid from the GNOME Foundation. Whilst it is no secret that I am a KDE user, community member and general doer-of-stuff, I am not tribal about it. Healthy competition breeds choice and innovation and am very happy that the GNOME project (along with all the other Free Software desktops) exist. What I saw here was something very simple: an attack by a wealthy corporation on a Free Software project.

Wait... what? "Attack"?

Like many people, my first reaction was one of "What. The. Hey? How dare this company attack my sisters and brother in Free Software?" Aaron Seigo kick-started a discussion on G+ about how we reacted to this situation and in particular, knowing that I would engage rather than enrage, picked up on my Tweet/G+ post. Sadly the conversation he started degraded, largely because some of the people engaging were failing to understand what he was saying... I digress...

I do not want to get into a discussion about the semantics of "attack". I stand by my assertion this is an attack. Why? The GNOME Foundation "call to arms" made it clear there has been an engagement with Groupon (they do have to defend the GNOME trademark after all) and that the engagement has broken down to the point they need to legally defend their trademark. For me, for right or for wrong, it is simple: allowing the situation to degrade to the point of lawyers needing to be deployed to actively stop trademark registrations is an attack of sorts. Groupon is large enough and smart enough to know that such a legal fight will cost GNOME dearly and allowing the discussion to degrade this far could easily be seen as an unpleasant tactic and an attack in itself.

Or perhaps not. More on this later.

What I want to focus on here is not on what we all wrote, but no how we wrote it.

With All Due Respect!

There is a frequently-used joke in my office about the passive aggression of the Brits. We are, after all, masters of deploying phrases such as "With all due respect..." when what we are really saying is "Look you tosser...".

My reaction to this news, as with many other people was one of anger. In my case this was fairly mild and so I think the accurate description would be "I was annoyed". It is not hard to see why so many had a similar reaction. The GNOME Foundation campaign page uses words like "defend" and "fight". They clearly feel threatened and that ain't cool.

I wrote a Tweet. Which automatically got posted to Facebook. I posted a modified (because G+ allows me to be more verbose) version on G+. Now what I was trying to do, of course, was raise some awareness of the GNOME Foundation's action...

Headless Chickens

So Aaron, perhaps a little harshly, likened the reaction of the Free Software world, in general, to headless chickens. Now Aaron's a pretty big fan of Free Software and the people who make it all happen. So he doesn't use such language lightly. Here is his own reasoning:

Yes, people are running around calling for boycotts, negative social media campaigns and flooding management inboxes .. none of which GNOME asked for .. and all within hours of hearing about the issues for the first time, and certainly having only the barest details. That's practically the definition of acting like a headless chicken.

The man has a point.

Social media allowed the Free Software community to spread the news pretty fast. Add to the pace of news the mildly aggressive language put forward by the GNOME Foundation and things escalated quickly.

By 10am I had the chance to speak face-to-face with my friend who posted the original news. Rather triumphantly he told me I should check out the shitstorm hitting Groupon's various social media outlets. Neither of us at this point had any idea about mailing campaigns.

Aaron's right: the GNOME Foundation did not call for any of this. Explicitly they asked for financial aid. Implicitly, arguably, they asked for people to spread the news. That was it. The uncoordinated "return fire" from the Free Software community on the GNOME Foundation's behalf does not paint a particularly professional picture of the Free Software community.

We let our collective anger show.

Anger Is a Gift

(Thanks, Zack de la Rocha).

I don't think there is anything particularly wrong about having gotten angry about this situation. But it really is all about how we deal with that anger.

In this case, many Free Software community members did not recognise the wishes of the GNOME Foundation: to fight this in the legal system, not in the court of public opinion. For this fight they need cash and that is what they asked for. Nothing else. Why? This is their fight (that is their word, remember) to fight the way they deem appropriate.

I am not going to comment on the GNOME Foundation's approach other than to say, if you contributed to the funding, thank you. Whether it is ultimately needed or not, your cash, I'm sure, will get put to good use in Free Software and for that we should all be grateful. Another good option would be to donate to the FSFE who, through their various campaigns are in a position to educate governments and businesses about how to deal effectively with Free Software communities (and their trademarks).

So, What Now?

Well, it looks like all the noise around this issue has had an effect. The Groupon engineering blog has this posting with the following key text:

And if we can’t come up with a resolution, we’ll be glad to look for another name.

For some definition of "win", this looks like a win to me. So long as Groupon stick to their word, keep engaging with the GNOME Foundation and, if needs be, rename their product.

So perhaps not an attack, after all. But certainly unfortunate management of the situation to allow things to get to the point that the GNOME Foundation feels a need to defend its trademarks legally. There's probably failings on both sides, if we are honest with ourselves.

But hey, a win. At what cost? Well, at the very least, Groupon (to some extent) may have an altered opinion on how it wishes to engage with Free Software. If nothing else, the eyes of the world (at least those paying attention) will have seen the collective forces of Free Software rally against Groupon and think "they probably could have done that a little more professionally".

In Conclusion

As I stated on G+:

I believe we have entered the world of social-constructivism and need to define "attack" :) How about this slightly less aggressive (and derived from +Aaron Seigo 's thoughts) language?... "This sucks for GNOME and, assuming they have acted reasonably towards Groupon, the passive aggresive attitude of Groupon towards the situation is certainly distinctly suboptimal."

Sunday 9 November 2014

Parsing GIT Logs With Files

So things have been a little quiet on the GIT log analysis front. Various out-of-bounds activities were in my way but, more importantly, so was a directly-related technical issue... Just how do you get file data out of GIT logs?

Not As Easy As It Sounds

The problem is this... In the good ol' days I was parsing SVN log files that were XML-formatted; svn log --xml. I also needed the file data to be included in that for many of my analysis tools; svn log -v --xml. Typical output looked something like this:

<?xml version="1.0" encoding="UTF-8"?>
<log>
  <logentry revision="58903">
    <author>padams</author>
    <date>2014-11-04T15:22:30.364435Z</date>
    <paths>
      <path action="A">/Some/Path/To/A/File</path>
    </paths>
    <msg>New WMD contract</msg>
  </logentry>
</log>

Nice and easy to parse. GIT does not provide any such XML output, however it does provide a --format option where you can specify the output format. For everything but the files. You see with GIT I can get all the information I want, just not in some handy format. And I certainly need to XMLify it myself if I want to make use of my existing parser; git log --name-only:

commit f2fe64b234c31c703c998cacdc1c3cff43f4e05f
Author: Lamarque V. Souza <lamarque@kde.org>
Date: Fri Nov 7 22:37:56 2014 -0200

    Doxygen configuration for http://api.kde.org/.

doc/api/Doxyfile.local

The Approach

I sat down with Ade at Academy and we came to the conclusion that actually the best way to get this file data out of GIT was to use libgit. Holy moly. No, just no. Academy was some time ago and that was the last time I had a chance to look into this. A little bit of Googling around led me to this post which did everything I needed apart from handle the file data. In fact,, I really could not find any information on extracting the file data out of a GIT log. Well, thankfully, the modification was easy. So many thanks to the author of that blog post and here is my recipe!

import pprint
import subprocess

GIT_COMMIT_FIELDS = ['id', 'author_name', 'author_email', 'date', 'message', 'files']
GIT_LOG_FORMAT = ['%H', '%an', '%ae', '%ad', '%s']
GIT_LOG_FORMAT = '%x1e' + '%x1f'.join(GIT_LOG_FORMAT) + '%x1f'

p = subprocess.Popen('git log --name-only --format="%s"' % GIT_LOG_FORMAT, shell=True, stdout=subprocess.PIPE)
(log, _) = p.communicate()
log = log.strip('\n\x1e').split("\x1e")
log = [row.strip().split("\x1f") for row in log]
log = [dict(zip(GIT_COMMIT_FIELDS, row)) for row in log]

for row in log:
    row['files'] = row['files'].strip('\n').split('\n')

pp = pprint.PrettyPrinter(indent=4)
pp.pprint(log)

At the end of that short script we are left with a Python dictionary representation of the log which can easily be processed within Python or outputted as XML for use by my other scripts. Hurrah!

Simple solution, but it took a while to get to it. Using the --format option to create sensible delimiters between log entries and their fields had already occurred to me and I had tried things like:

GIT_LOG_FORMAT = '<logentry revision="%H"><author>%an.......<paths>...'

The problem with this, of course... How do I insert the closing </paths> or the individual <path></path> tags? The files are not one of the items that can be formatted using git log --format!

Thanks to the other blog post, I realised that the XMLification should really be a separate step after extracting the required data.

Wednesday 8 October 2014

I Can Haz Internet?

The eagle-eyed of you might have noticed yesterday that I said my Bürgeramt ordeal is not over. As I said, the Bürgeramt was just a means to an end. They hand out a specific document and then I can get Internet (among other things, like paying taxes). Oh yes, the Internet. That concept has taken on a whole new meaning for me here in Berlin.

When Is BT Not BT?

The physical telecomms infrastructure in Germany is all controlled by Deutsche Telekom who also provide services on top. Other companies also have the ability to sell services on top of this physical infrastructure. Deutsche Telekom used to be state-owned/controlled.

Sounds familiar?

Both BT and Deutsche Telekom appear to have a very similar attitude and by that I mean that neither have learned properly that they have to compete on services and so they abuse their monopoly on the infrastructure to be a pain in the ass. Case in point... When you go to "the Internet" shop (it really does not matter which provider) and buy the Internet, you always end up dealing with Deutsche Telekom. In my case, I was told I would have to wait three weeks before DT we could around to my flat...

Interlude: On Speaking German

In Berlin it is fairly easy to survive on speaking a little German. I have heard stories of people having been in this city for years and basically making no effort with the language. Because, you really don't need to if you don't want to. Many Germans seem to be quite tolerant of this. It is a very nice trait of the local character. Before too long, I will make effort to learn German. For now, I am grateful to live in a town that is tolerant of my rudimentary German language skills.

Die Männer von Telekom: 1

So the three weeks pass and the first Telekom engineer comes around. He speaks absolutely no English whatsoever. No problem. I knew enough to grunt and point my way through the conversation. He took the main TAE Dose off the wall and inspected the wiring behind before hooking up the testing equipment. We then went all the way down into my cellar and he looked into the DT-owned demarcation point (a bad example fo which can be seen at the right hand side of he header image). He attached the other half of the testing equipment and grunted something at me. He waved some magic wand-like device around the demarcation point and grunted some more. Back up the 6 flights of stairs to my flat.

He knelt before the expose wiring again.

He looked up at me.

He shrugged.

He then offered me the only two words in English that he appeared to know...

"It's fucked!"

This appointment happened to be the day before I flew back to Scotland for the independence referendum. While I was away, my landlord sent around another engineer who told me what I suspected: that the wiring was all brand new and that there was nothing wrong at all. Specifically, he reported the problem was in fact that the line from my flat was not plugged into the demarcation point... As evinced by the large, loose cable with my address on it, in the basement. How had the first engineer (and me for that matter) missed this?

Die Männer von Telekom: 2

Earlier this week, after another long mutli-week wait, a second engineer visited. He came to my flat only to put the testing equipment in and then headed straight to the basement where he plugged my flat's cable into the demarcation point. He tested it (at least appeared to) and everything was OK. We climbed the 6 flights of stairs up to my flat, he double checks the line and off he ran! "Just give it 5 to 10 minutes", he said.

Nope. It's fucked.

Let The Finger Pointing Begin!

So now DT are blaming my ISP for the problems and my ISP are blaming DT. My next engineer visit is on Friday morning and my gin consumption is rising.

Tuesday 7 October 2014

Going Bürgeramtal: Part II (Going Buggeramtal)

Oh yes, the Bürgeramt. Remember that? I sure as hell do. By now I'd love to tell you that the story is over... It isn't. But, before we get to the end of the story, I need to bring you up to speed on the "middle" of the story. Let me start with the birds...

The Conditions Were Perfect... I think

OK OK... So first I need to admit something. Many, many people had warned me that I should simply make an appointment at the Bürgeramt, rather than just turning up. After my previous two failed attempted, this was starting to make sense. But, you see, my motivation for getting in there asap was simple... I needed the Anmeldebescheinigung because this is the gatekeeper document. Without it you cannot get: bank accounts, health insurance, Internet (important for someone who works form home!) and various other things. Those people, the ones who told me to just make an appointment? They were right.

So... A few weeks pass. I am now well past the theoretical deadline for registration. Oh well. At least I have my appointment.

The skies were blue that morning; not a cloud in sight. The sun was warm, the air was cool and the birds were chirping merrily as I boarded the M13 tram to take the short journey to my local Bürgeramt. When I arrived at the office, my number was already displayed on the screen and I had four minutes to spare before my appointment.

My number is called.

I go in and sit down.

I introduce myself and ask (in my best, polite German, of course) if we may speak English.

We can.

The dream of the Anmeldebescheinigung is slowly becoming reality...

Max Mustermann (The Stereotypical German)

The very nice lady in the Bürgeramt took a look at my documentation and read through it diligently. At least so I thought... Eventually she asks me if I was the home owner and I confirm that I was renting. She asks me, therefore, why did I not complete the name and address of my landlord. Well, the answer is simple, nowhere on the form is this information asked for. Apparently this information's to go into the one box I've left empty. The box which had no label telling me what information I was supposed to provide in there. How was I supposed to know this?

"Did you not read the supporting documentation that tells you how to fill in the form?"

At the time, possibly the most German thing I had ever heard.

She placed the documentation in my hand and asked me to provide the information. When I pointed out that I had no idea who my landlord was, nor where he lived, I was in trouble. She politely asked me to leave and make another appointment.

Thanks to the ingenuity of a friend of mine, I rolled the dice and pointed out that I was being sent out without the ability to get (legally required) health insurance. She paused for a while and then asked me to simply get her the name. If she were British I would swear by her intonation that she was happy for any name.

So... OK. Pretend that I am looking for a real name and just make up a name. Simple, right?

No.

Imagine my situation: I'm in a ever-so-slightly-higher-than-normal-pressure environment, in a foreign land, I do not know the information being required of me, the clock is ticking and all I have to do is come up with a name. At this point, it is very difficult to come up with something that does not sound very stereotypical and, thus, a piss-take...

Lars Schmidt.

Jens Schmidt.

Helmuth Schmidt (Oh sweet Lord, no).

Anything with "Schmidt".

Max Mustermann!

Let me tell you, when the name Max Mustermann gets stuck in your head when you are trying to make up a German name it is very hard to get it out and it is very hard not to laugh.

Thankfully, while I was pretending to search for my landlord's name, I actually found it. #winning

Epilogue: If We Must Be Formal, Call Me "Mr. Adams"

While I was searching for a (fake) name to provide for my landlord, the nice Bürgeramt lady was filling in all my details onto a computer using my passport. UK passports do not contain titles, only the name. So imagine my surprise when I leave the Bürgeramt, the proud owner of an Anmeldebescheinigung, to discover that I am no-longer "Herr Dr. Adams" but simply "Herr Adams".

Probably need another appointment to get that fixed.

Tuesday 9 September 2014

Presentation: David Faure Breaks The Law

  • Event: Akademy 2014
  • Location: Brno, Czech Republic
  • Date: 7.09.2104
  • References to sex acts: 1

The Short Version

David Faure is a pleasure to work with, both within KDAB and KDE. He maintains three important character traits that, as engineers, we should all aspire to:

  • He keeps a sense of fun about him at all times. This humour about his work makes him a joy to work with;
  • He is constantly learning and improving, even with the wealth of experience he has behind him;
  • He always has the actual end user in mind.

Despite these wonderful character traits, he has a dark side... He is a lawbreaker. So are many other KDE contributors.

In this presentation I talked about David Faure, Brooks' and Conway's laws and evaluated them in the context of KDE. Some rather fascinating, but alarming results are found!

Slides And Brief Notes

Slides 1 -- 6

David Faure: all-round legend and nice bloke of KDE. Fun to work with, thirst of knowledge and skill and, always thinks about the end user.

Slide 7

Positioning David Faure within the context of the wider KDE project and the people he works with.

Slides 11, 12

Introduce Brooks' and Conway's laws: it is incredible how many people identify as software engineers, but do not know what these laws are all about.

Slides 14 -- 16

How can we measure Brook's law in Free Software projects. We need to start by discussing how we measure anything at all! What are the sources for data in Free Software development.

Slide 17

How we model the community: nodes are contributors, edges show that the two contributors worked together. "Work" means they both edited the same resource in the observed timeframe. Edges get higher weights if the contributors work together in multiple resources.

From this we can find "cohesion", the average length of the shortest path between all pairs of nodes. The higher the cohesion, the more closely-knit the project is. If Brook's is correct then we should see cohesion drop with increased community size...

Slides 18 and 19

Yup, Brooks nailed it...

Slide 20

...or did he? KDE is clearly outperforming expectations here. Awesome. Well done KDE! Hint: This is where Conway's law comes into interest.

Slide 22

Wait, if we look at the data over time, something isn't right. Cohesion grows since 2004 until late 2007 (KDE 3.5, KDE 4.0?) but then cohesion declines...

Slide 23

Which just so happens to be the same pattern for funding of the eV, just offset by one year.

Slides 24 and 25

KDE has lost its way a little bit. Nothing disastrous yet, but the ship does need righting. As a start, being more like David and enjoying our work, continuing to learn and focusing on users, will get us headed the right way.