KDEPIM: Any More Guesses?

Yesterday I posed a somewhat cryptic question: what happened in KDEPIM in late 2010 that might have caused such a dramatic spike in an unnamed (but real) metric? I want to leave the conversation running a little bit longer. So here is some further food for thought, to keep you guessing...

Some things to note:

• The new metric is developer count... the number of unique contributors to KDEPIM each month;
• Careful! These two plots are not on the same scale. Note the two different Y-axis scales. Mystery plot on the left, developer count on the right;
• It may (or may not) be important that the huge spike in the mystery metric just so happens to be followed by a significant (for some definition of significant) drop in developer count. Again... I'm not going to tell you what the mystery metric is, but I will say that, in this case, causality is not out of the question. The huge spike in the mystery metric arguably, in some small way, might have contributed to the later drop in developers.

I'm now going to leave this "out there" for a day or two before getting back to it. This buys you guys some time to come up with some more suggestions as to what the mystery metric might be (and what caused the spike in it).

It also buys me some time to find out (for definite) what caused the spike... After all, I do know what the mystery metric is!

In Conclusion

Yes, I do know that the previously-red plot is now green and the new plot is red. One day I will master GNUPlot.

The Great GNOME/Groupon Spat of 2014

If you have had your head in the sand today you might well have missed the story that the GNOME Foundation and Groupon are is a disagreement over the use of "GNOME".  I already shared my thoughts in Tweet form earlier... But now I'd like to talk a little more. Not about the event, but about how we (the Free Software community) reacted to it.

Hey #KDE, our friends in the #GNOME project are being attacked by a corporation that should know better. http://t.co/xzfdvp7GMt #opensource — Paul Adams (@therealpadams) November 11, 2014

Where Do We Get Our News?

The first I heard about the GNOME/Group affair was on Facebook. A former colleague and friend of mine posted a link to the request for aid from the GNOME Foundation. Whilst it is no secret that I am a KDE user, community member and general doer-of-stuff, I am not tribal about it. Healthy competition breeds choice and innovation and am very happy that the GNOME project (along with all the other Free Software desktops) exist. What I saw here was something very simple: an attack by a wealthy corporation on a Free Software project.

Wait... what? "Attack"?

Like many people, my first reaction was one of "What. The. Hey? How dare this company attack my sisters and brother in Free Software?" Aaron Seigo kick-started a discussion on G+ about how we reacted to this situation and in particular, knowing that I would engage rather than enrage, picked up on my Tweet/G+ post. Sadly the conversation he started degraded, largely because some of the people engaging were failing to understand what he was saying... I digress...

I do not want to get into a discussion about the semantics of "attack". I stand by my assertion this is an attack. Why? The GNOME Foundation "call to arms" made it clear there has been an engagement with Groupon (they do have to defend the GNOME trademark after all) and that the engagement has broken down to the point they need to legally defend their trademark. For me, for right or for wrong, it is simple: allowing the situation to degrade to the point of lawyers needing to be deployed to actively stop trademark registrations is an attack of sorts. Groupon is large enough and smart enough to know that such a legal fight will cost GNOME dearly and allowing the discussion to degrade this far could easily be seen as an unpleasant tactic and an attack in itself.

Or perhaps not. More on this later.

What I want to focus on here is not on what we all wrote, but no how we wrote it.

With All Due Respect!

There is a frequently-used joke in my office about the passive aggression of the Brits. We are, after all, masters of deploying phrases such as "With all due respect..." when what we are really saying is "Look you tosser...".

My reaction to this news, as with many other people was one of anger. In my case this was fairly mild and so I think the accurate description would be "I was annoyed". It is not hard to see why so many had a similar reaction. The GNOME Foundation campaign page uses words like "defend" and "fight". They clearly feel threatened and that ain't cool.

I wrote a Tweet. Which automatically got posted to Facebook. I posted a modified (because G+ allows me to be more verbose) version on G+. Now what I was trying to do, of course, was raise some awareness of the GNOME Foundation's action...

Headless Chickens

So Aaron, perhaps a little harshly, likened the reaction of the Free Software world, in general, to headless chickens. Now Aaron's a pretty big fan of Free Software and the people who make it all happen. So he doesn't use such language lightly. Here is his own reasoning:

Yes, people are running around calling for boycotts, negative social media campaigns and flooding management inboxes .. none of which GNOME asked for .. and all within hours of hearing about the issues for the first time, and certainly having only the barest details. That's practically the definition of acting like a headless chicken.

The man has a point.

Social media allowed the Free Software community to spread the news pretty fast. Add to the pace of news the mildly aggressive language put forward by the GNOME Foundation and things escalated quickly.

By 10am I had the chance to speak face-to-face with my friend who posted the original news. Rather triumphantly he told me I should check out the shitstorm hitting Groupon's various social media outlets. Neither of us at this point had any idea about mailing campaigns.

Aaron's right: the GNOME Foundation did not call for any of this. Explicitly they asked for financial aid. Implicitly, arguably, they asked for people to spread the news. That was it. The uncoordinated "return fire" from the Free Software community on the GNOME Foundation's behalf does not paint a particularly professional picture of the Free Software community.

We let our collective anger show.

Anger Is a Gift

(Thanks, Zack de la Rocha).

I don't think there is anything particularly wrong about having gotten angry about this situation. But it really is all about how we deal with that anger.

In this case, many Free Software community members did not recognise the wishes of the GNOME Foundation: to fight this in the legal system, not in the court of public opinion. For this fight they need cash and that is what they asked for. Nothing else. Why? This is their fight (that is their word, remember) to fight the way they deem appropriate.

I am not going to comment on the GNOME Foundation's approach other than to say, if you contributed to the funding, thank you. Whether it is ultimately needed or not, your cash, I'm sure, will get put to good use in Free Software and for that we should all be grateful. Another good option would be to donate to the FSFE who, through their various campaigns are in a position to educate governments and businesses about how to deal effectively with Free Software communities (and their trademarks).

So, What Now?

Well, it looks like all the noise around this issue has had an effect. The Groupon engineering blog has this posting with the following key text:

And if we can’t come up with a resolution, we’ll be glad to look for another name.

For some definition of "win", this looks like a win to me. So long as Groupon stick to their word, keep engaging with the GNOME Foundation and, if needs be, rename their product.

So perhaps not an attack, after all. But certainly unfortunate management of the situation to allow things to get to the point that the GNOME Foundation feels a need to defend its trademarks legally. There's probably failings on both sides, if we are honest with ourselves.

But hey, a win. At what cost? Well, at the very least, Groupon (to some extent) may have an altered opinion on how it wishes to engage with Free Software. If nothing else, the eyes of the world (at least those paying attention) will have seen the collective forces of Free Software rally against Groupon and think "they probably could have done that a little more professionally".

In Conclusion

As I stated on G+:

I believe we have entered the world of social-constructivism and need to define "attack" :) How about this slightly less aggressive (and derived from +Aaron Seigo 's thoughts) language?... "This sucks for GNOME and, assuming they have acted reasonably towards Groupon, the passive aggresive attitude of Groupon towards the situation is certainly distinctly suboptimal."

Parsing GIT Logs With Files

So things have been a little quiet on the GIT log analysis front. Various out-of-bounds activities were in my way but, more importantly, so was a directly-related technical issue... Just how do you get file data out of GIT logs?

Not As Easy As It Sounds

The problem is this... In the good ol' days I was parsing SVN log files that were XML-formatted; svn log --xml. I also needed the file data to be included in that for many of my analysis tools; svn log -v --xml. Typical output looked something like this:

<?xml version="1.0" encoding="UTF-8"?>
<log>
  <logentry revision="58903">
    <author>padams</author>
    <date>2014-11-04T15:22:30.364435Z</date>
    <paths>
      <path action="A">/Some/Path/To/A/File</path>
    </paths>
    <msg>New WMD contract</msg>
  </logentry>
</log>

Nice and easy to parse. GIT does not provide any such XML output, however it does provide a --format option where you can specify the output format. For everything but the files. You see with GIT I can get all the information I want, just not in some handy format. And I certainly need to XMLify it myself if I want to make use of my existing parser; git log --name-only:

commit f2fe64b234c31c703c998cacdc1c3cff43f4e05f
Author: Lamarque V. Souza <lamarque@kde.org>
Date: Fri Nov 7 22:37:56 2014 -0200

    Doxygen configuration for http://api.kde.org/.

doc/api/Doxyfile.local

The Approach

I sat down with Ade at Academy and we came to the conclusion that actually the best way to get this file data out of GIT was to use libgit. Holy moly. No, just no. Academy was some time ago and that was the last time I had a chance to look into this. A little bit of Googling around led me to this post which did everything I needed apart from handle the file data. In fact,, I really could not find any information on extracting the file data out of a GIT log. Well, thankfully, the modification was easy. So many thanks to the author of that blog post and here is my recipe!

import pprint
import subprocess

GIT_COMMIT_FIELDS = ['id', 'author_name', 'author_email', 'date', 'message', 'files']
GIT_LOG_FORMAT = ['%H', '%an', '%ae', '%ad', '%s']
GIT_LOG_FORMAT = '%x1e' + '%x1f'.join(GIT_LOG_FORMAT) + '%x1f'

p = subprocess.Popen('git log --name-only --format="%s"' % GIT_LOG_FORMAT, shell=True, stdout=subprocess.PIPE)
(log, _) = p.communicate()
log = log.strip('\n\x1e').split("\x1e")
log = [row.strip().split("\x1f") for row in log]
log = [dict(zip(GIT_COMMIT_FIELDS, row)) for row in log]

for row in log:
    row['files'] = row['files'].strip('\n').split('\n')

pp = pprint.PrettyPrinter(indent=4)
pp.pprint(log)

At the end of that short script we are left with a Python dictionary representation of the log which can easily be processed within Python or outputted as XML for use by my other scripts. Hurrah!

Simple solution, but it took a while to get to it. Using the --format option to create sensible delimiters between log entries and their fields had already occurred to me and I had tried things like:

GIT_LOG_FORMAT = '<logentry revision="%H"><author>%an.......<paths>...'

The problem with this, of course... How do I insert the closing </paths> or the individual <path></path> tags? The files are not one of the items that can be formatted using git log --format!

Thanks to the other blog post, I realised that the XMLification should really be a separate step after extracting the required data.