Not so long ago, I blogged about KDEPIM and how contributions varied between 2014 and 2015. One of the comments on that blog post mentioned Akonadi. Now Akonadi has its own git repo; actually, it has about 14. So I decided to take a very quick glance into that, too.
Since I started working at Crate.IO I have, of course, had to spend some time getting to learn the technology. For me, overall Crate story is very compelling: distributed SQL at NoSQL-scale. How the technology clusters and distributes load is also very cool. In order to learn a little about the technology for myself I have been working on a specific "game"... What can I tweak in order to maximise the input rate? Here's what I found out so far...
It's been a while since I delved into KDE. With me no longer working at KDAB, my contributions are really restricted to helping out in KDE e.V. and, even then, normally only when someone specifically ask for me. KDE, of course, continues to be a community I care about very much. I wanted to do a very basic health check of my old stomping ground: KDEPIM.
If you have not seen it yet, take four minutes out of your life to watch the announcement that Bash is going to be natively supported on Windows 10. During the video you're probably going to notice some confused-looking folk in the audience. Heck, if I was at the Build Conference I'd be confused, too.
In fact, when I first heard this story yesterday, it was such a non-event in my life that I chortled a little and thought nothing more of it. I don't even remember who it was who told me. I just remember laughing and then going about my business... Because I assumed this was a joke. The I realise that this is real. Now I've had time to think on it this, clearly, is not a joke. But it is a little weird.
The eagle-eyed of you will have noticed that I have recently started a new career at Crate.IO. After 6 years in side the KDAB family (including Kolab Systems, KDAB UK and KDAB Germany) this came as a surprise to many people. KDAB gave me a lot of freedom to achieve some long-lasting successes as well as some failures for me to learn from. I will always be grateful. But it was time to move on. Here is how I ended up where I did.
1. Find an excellent team.
Having left KDAB I wanted to take the time to evaluate what I was trying to achieve with my career. I was never much of a programmer, but I like to think I have an excellent overview of software engineering as a whole. Whilst I did not want to be hacking (and no employer should want that, either) I did want to be kicking down doors. I want to kick down the doors that the other engineers did not even realise were in their way. This type of servant leadership is hugely enjoyable when you are working with the right team...
2. Product company, not consulting.
My career has been split between product and consulting work. Most recently, at KDAB, I was part of an incredible team on consultants. Those guys are solving problems with C++/Qt/OpenGL that most people would be outright scared of. The problem with consulting, though, is the lack of ownership. KDAB's engineers would achieve glorious results but not get any glory. Not only that, but often they would be working on multiple problems for multiple clients and not get the chance to feel a sense of real ownership. All software engineering is a team sport, both consulting and product work. But there is something awesome about being part of a team with a long-term product vision and ownership.
3. Free Software angle.
I have always worked in or around Free Software. This is something hugely important to me. Free Software was the subject of my PhD and has been at the heart of every job I've had. There is a certain sense of "doing it for a greater good" which really adds some good vibes to the work environment.
4. In Berlin.
I love Berlin. I might tongue-in-cheek write about the brokenness every now and then but, fundamentally, this is my kinda town. I've been here for almost two years, but I still feel like I'm new in town. In the grand scheme of things, I am. My time here is nowhere near done and so any new job had to keep me here. If this meant regular travel, so be it. Long commute? For the right job, perhaps. But with a large tech startup scene in Berlin this seemed unnecessary.
I write recently about my experience of attending the Crate.IO Snow Sprint. This gave me a great opportunity to meet the Crate team, learn about the technology and eat some phenomenal Leberkäse.
Needless to say, all of my boxes were ticked. I was impressed by the leadership team. The CEO and CTO were both very open about the company. This was not a recruitment sales pitch. Instead I was given the warts-and-all story of where the company and technology were at and where I might fit in to help. They are also generally cool guys with great humour under the pressure of leading a startup. Sadly, I did not get too much time to spend with the two team leaders during the sprint. What time I did spent with them gave me the impression that we all saw the world in very similar ways and that we were going to work well together. Everyone else was very welcoming of an outsider, had plenty of time to spare for me to help me understand the tech and generally made me feel like part of the team. Which I wasn't. As a nice touch, after deciding to evaluate how I might fit in, I even had the opportunity to speak with the newest member of the board, Ari Helgason. It was very nice to hear the investor's voice. And to hear that voice was genuinely excited about the technology...
...we definitely need to talk about the technology. If I was going to dedicate myself to a product company, the product had to be something epic. Something I cared about as least as much as Kolab. Crate is a departure from what I have done in the past but shares one crucial aspect that I care about most... solving a real problem. At this time I am not going into too much detail about Crate and what it does. What I will say is this: as a database, Crate helps to solve some seriously complex problems in the Big Data space, yet it is so easy to deploy that, at the Snow Sprint, I had a three-node cluster running on my laptop, with a simple schema, importing data, within minutes of me first sitting down to play with it. SQL with scaling, resilience, containerisation and such ease of use... it's a pretty compelling story.
In future posts I will write about what it is I am precisely doing at Crate.IO. For now there is a lot of hard work ahead of me and I am definitely going to enjoy the ride. I have already started to submit presentations on Crate to various conferences, so I hope to see you soon and we can talk about what I'm up to.
I recently moved this blog away from a self-hosted Wordpress (just too much hassle to keep maintained). The good news is that this new career change will present me the opportunity to go back to some of the metrics-related content I used to work on before. So, if you have been following my nonsense for some time, you (hopefully) will be pleased to know that more oversized data visualisations are on their way.
[If you haven't done so already, take a quick look at what I wrote yesterday about the Crate Snow Sprint].
So yesterday was the first "real" day of the Snow Sprint. I used the time to start implementing a very rudimentary metric processing tool. To recap: I want to build a service-oriented system with arbitrarily scalable components. In the cloud. (Bingo!). Whilst this is largely a demonstration of Crate and an opportunity for me to learn about that technology, it is a serious project that I would like to see grow into something meaningful.
So here I write about the nonsense code I wrote yesterday and what I have managed to achieve in prototyping my intended solution.
Many moons have passed since I was last paid to program in Python. The language has changed in some crucial ways since then, but not so much that it is hard to update your thinking from Python 2. I always find Python a real joy to work with, so updating my own thinking really was not a chore.
As well as defamiliarising with Python, my aim for yesterday was to get my head around the basics of certain technologies I was planning to use as part of this project, namely Flask and SQLAlchemy.
If you have yet to go back and read yesterday's blog post, now is a good time. Yesterday I envisaged a system with 4 distinct component services, each independently scalable:
Well... The Crate part of this is easy enough. Crate exists and works. And is extremely simple to deploy (read: "plays extremely well with Docker"). Yesterday my aim was to create a pipeline for getting metadata from Git into Crate.
Firstly, you can go and grab my code on Github: https://github.com/therealpadams/metre
What you will find in there:
Right now, you can run the urls script and fire in data from the create_commits script and it will fail. I will fix this all before the end of the day. If all goes well, but the end of the day there will be a simple metric (commits per day?) script in the repo, too.
This is far far far from great code. At the moment my aim is simply to learn the technologies I am working with and have a play with the overall pipeline for the data. After this playground edition is complete, I will stash it somewhere by itself in the repo and go about hacking "the real thing". Although this is not likely to happen after FOSDEM.
Another header photo taken at the Crate Snow Sprint. Again, thanks to Crate.io for sponsoring my trip to Austria.
Now for something a little different: Crate. Thanks to the kind sponsorship of Crate.io, I am attending the annual Snow Sprint. This is an event that has been in existence for many years (certainly over a decade); originally it was a get-together for Zope/Plone developers. Those of you with very long memories might remember that I used to be part of the Plone community and even worked for Zope Europe Association at one point. The Plone Snow Sprint is still a "thing". But, with former Zope/Plone developers involved in the company/community, the Crate Snow Sprint is also a "thing".
Crate is a high-performance, distributed database. Very easy to deploy (think: "the database storage for Docker") and designed to help manage subsets of very large datasets (Clusters of 100s of nodes? Why not!? Petabytes of data? Hells yeah! Hundreds of billions of rows? Come get some.)
From the marketing blurb:
Crate has been designed to be a highly distributed high performing database. Before Crate organizations had to compromise on performance if they wanted to keep the ease of use benefits of using SQL stores, or move to a No-SQL store and deal with the complexities of the query languages and rewriting their code. With Crate you get the best of both worlds: the No-SQL performance you require and the SQL syntax you want. Crate can be used for a variety of use cases, from a classic but scalable SQL database, to advanced usage incorporating full text search, geo shape and analytics support.
Big Data, of course, means we are not talking "classic" SQL. No foreign key support, for example; these just do not scale very well. Instead, all related data should just get fired into the same table. What you lose in space efficiency you gain in speed. Lots of speed.
The Big Data space is full of incredible technology and is evolving fast. With the growth in Big Data we have seen a growth in technologies to enable Big Data; containers are a huge part of this. If you are not clued-in on containers, go take a look at what Wikipedia has to say on it. Until now containerisation (that's a word, right?) has really been focused on application deployment. Crate is the primary contender as the storage for containers, allowing extremely easy deployments in order to arbitrarily grow the clusters. As I said, rolling a 100s of nodes in a Crate deployment is really not hard to do.
So why do I care? Who here remembers the SQO-OSS project? The premise of this project was simple: create a system to measure the quality of a piece of Free Software, where "quality" was defined by any arbitrary collection of metrics by the user. The system maintained clones of SVN repositories and regularly ensured they were up-to-date. Metric scripts would then be run against these repositories and an SQL store would keep the results which could then be viewed through a web client.
This project, like almost every EC-funded project I have ever worked on, was a successful failure. We built exactly what we intended to build and helped develop knowledge on software quality (metrics), for which we had an extraordinary number of publications to prove it. I say "failure", however, because the architecture of the system we built was heavily tied to the hardware purchased for the project (i.e. one seriously fat sparc server from Sun). Arbitrary scaling of the datastore, processing and front-end? Never taken into consideration. Ironically, the backbone to the tool we developed was Equinox which could have enabled such an architecture. We really did not make best use of that technology, however.
The... result.... was....... very........... slow.
Why do I care about Crate? Because I want to fix ^ that problem. In short: I want to create an arbitrary scaleable solution for metric processing, result caching and visualisation in a timely fashion.
I want to create a solution that is arbitrarily scaleable to the needs of those using it. To that end I envisage four discrete components that can be scaled according to need:
REST and web front-ends for data input, retrieval and visualisation. A typical deployment will not need many of these, I guess. However, a public deployment of the system for a popular Free Software project might well need extra oomph here. In the case of data retrieval, data will be grabbed from the results cache or, if the result is not available yet, a metric processing node will be triggered.
These would simply be responsible for gathering the metadata from (probably just git, at first) repositories and entering the data in the (potentially very large, but hardly "Big Data") table in the data storage. Python + SQLAlchemy here. These would need to be scaled up with the number of projects being analysed.
These will be scripts that process data from the metadata store and stash the result in the results cache. Mostly envisage Python for basic metrics or C/C++ libraries for anything heavyweight, dispy for process distribution.
Err... Crate.
Well, it is a long time since I did any "real" Python programming. Or any programming for that matter. So I'm starting from the very beginning and getting my head around the technology stack that I am envisioning.
As part of my work at the Snow Sprint, yesterday, I started work on a basic code for data entry using Python and SQLAlchemy; which works very nicely with Crate. The fruits of my labour can be found on Github (promise not to laugh!). By the end of today (Day 1) I hope to have a nice pipeline: git repo -> metadata -> data entry node -> crate. Then tomorrow I will implement a basic metric (daily commit count?) and my work for the Snow Sprint will be done. At the moment I have no documentation tucked away in there; I will sort this out sometime after the sprint.
For me the next step will be to really nail down the basic architecture, ensuring ease of deployment and scaling. Once I have my head around that, building of the actual v0.1 system will not be too hard, I think. At least not for very simple metrics.
At the moment the system I am building is nothing more than an interesting demo for Crate (it is really showcasing simplicity rather than, say, scalability, since this will never be a real BigData application). However, as someone who cares about software/community/developer metrics I would love to see this mini-project turn into something real. If you have an interested in helping me develop this into a real tool, something that developers could really benefit from, I would be very happy to hear from you and get this thing moving. Not picked a license yet, because I want to engage the whole project team in that discussion, if I manage to grow one! :) But, most definitely, Free Software.
If you have an interest in metrics and would like to talk to me about this project, feel free to reach me by any means sensible (all my social media/email links are at the top of the page). Alternatively, if you are there, feel free to grab me for a chat at either the FLOSS Community Metrics Meeting or FOSDEM later this week.
The header image on this page is a photo I took during the setup of the snow sprint. As you might imagine, it is not enough (or sensible) for us to make use of the router provided in the chalet. Lots of cabling everywhere for our own network, of course! I will post more photos later.
Yay @nigelbabu! Also, cool shoutout to @therealpadams just in time for @writethedocs Prague :-) #writethedocs https://t.co/rlxuTXJfsG
— Mikey Ariel (@thatdocslady) July 31, 2015
I spoke at the Write The Docs unconf in Berlin a year. It had been a long time since I had spoken at a non-engineering event. Even longer since I last spoke at an event without an explicit Free Software connection, too. I enjoyed the event and I obviously did not make a fool out of myself, because I am due to speak at the next Write The Docs European conference in Prague.
What are the rules documentarians play by?If you follow the link in the Tweet (above) or follow this link here you will find Nigel's report on what we concluded in that session. OK, so I learned a thing or two about how documents are prepared in the software product world. Cool. What's next?
My day-to-day contribution to Free Software has changed quite dramatically in the last couple of years. I'm no-longer as involved with any one project than I used to be. Somewhat correlated with my declining active contribution to Free Software projects is my decline in public speaking.
You wait forever for a bus and then two come along at the same time!
I shall be delivering two very different talks in the coming months: one in which I will remind the audience that PacMan is a real asshole and in the other I will talking about documentation communities within Free Software.
Thursday, July 30th at 19:00. IXDS studio, Paul-Lincke-Ufer 39/40, 10999 Berlin
"PacMan: The Original Troll. PacMan is one of the best-known, well-loved and often-copied games of all time. But PacMan is mocking you. You cannot beat PacMan; nobody can beat PacMan. He laughs in your face."I recently attended Enthusiasticon where one of the talks was about Pokémon and, more specifically, about how a programming error (almost certainly caused by an engineer wrestling with the GameBoy's very limited architecture) made the game bizarrely winnable. If you knew how. This inspired to get up and talk about PacMan; a game which has the opposite problem.
As an Ignite talk, I will only have 5 minutes. PacMan needs no introduction, so I can save some time there. So I will cut straight to the ghosts, the map and why you're doomed to never beat PacMan.
Spoiler alert: the "waka waka" noise that PacMan makes... that's him laughing in your face.
More of the event details can be found at the Ignite Berlin website.
Monday, August 31st and Tuesday, September 1st at 09:00. Club Lavka, Novotného lávka 201/1, 110 00 Prague
"Community Management in Free Software communities is still an emerging field and has produced a spectrum of practitioners: from master-manipulators of social media, to those more focused on metrics and data as a means to driving process. Either way, the state of the art is still largely driven by the needs of technical contributors to projects. Good documentation is a crucial component in a software product and yet often technical writers are overlooked as important stakeholders of the process. Within the community, there are undoubtedly common problems between engineers and technical writers. Software Engineering is full of laws; can we show that these laws apply to technical writers as a means to help bridge the chasm between developers and technical writers?? In this talk Paul walks us through a selection of his favourite laws of software engineering and explores how developers measure them and if technical writers must also obey them. Or not. And what that means for the success of documentation in a Free Software project."
This will be the second time that I have spoken at a Write The Docs event. These are very friendly events and very welcoming given my "deliberate outsider" position. As a software engineer I think it is fun and useful to engage with events like Write The Docs in order to reach and say "here is how I think document writers and engineers can work together better" from the engineering perspective.
In this particular talk I will be exploring some of the rules and laws of software engineering and showing how I think these laws equally apply to document creation. Do these common problems create a bridge between document creation and coding? A common "platform" to work from?
Spoiler alert: 90% of this talk will be crap.
More of the event details can be found at the Write The Docs Europe website.
The board of KDE eV has launched a new initiative to ensure that KDE remains awesome and relevant for the foreseeable future. Unlike previous approaches it is not a point-in-time solution, it is a continuous process of improvement. And it is a good thing.
Previously, I have written/spoken a lot about the role of Brooks' Law in the context of Free Software. Brooks' Law teaches us to be careful about the management of growth in our communities. Especially treated in consideration with the grossly under appreciated Conway's Law. There are, of course, other laws of Software Engineering that apply to Free Software development.
Last weekend I attended the KDE eV board meeting held in Berlin. I was part of a team of KDE community members who provided our thoughts on the board's desire to ensure the overall KDE community continues to work effectively and continues to deliver awesome software. A lot of the discussion involved the word "evolution". Now the context was the community... but our software continuously evolves, too. What are the implications on this for a Free Software Community? And are community evolution and software evolution somehow linked?
In the 1970s it was formulated (paywall, sorry) that any piece of software will fit into one of three categories:
(Source Wikipedia)
- An S-program: written according to an exact specification of what that program can do;
- A P-program: written to implement certain procedures that completely determine what the program can do (the example mentioned is a program to play chess);
- An E-program: written to perform some real-world activity; how it should behave is strongly linked to the environment in which it runs, and such a program needs to adapt to varying requirements and circumstances in that environment.
Now a Free Software project could, in theory, fall into any of these. I want to focus on KDE which, arguably, is almost entirely E-type software: software that must evolve to satisfy its shifting environment.
So let's take a look at Lehman's Laws and what they mean for Free Software. I'll couch them in terms of KDE as much as I can. Please note: this is not scientific, just my thoughts on what we can readily observe in the wild and what it means. This is also not exhaustive; I'm cherrypicking a selection of the laws.
an E-type system must be continually adapted or it becomes progressively less satisfactory
In this context "change" can mean various things: adding features, removing features, QA improvements... I'm not going to single out any particular parts of KDE, but I think it is fair to say that parts of our technology have failed to remain competitive because activity declined (and, thus, "change" with it). More positively, I think it is very easy to point out that much of KDE remained competitive because it did change; it's not very often that a part of KDE becomes completely moribund.
as an E-type system evolves, its complexity increases unless work is done to maintain or reduce it
Refactor, refactor, refactor. Once of the things I love about KDE is we are always marching forwards. We are very focused on staying competitive by adding new features or by reinvention. That's not to say we ignore Bugzilla, however. Time is always being spent in bug fixing. But what about other QA activities? The fine-polishing work is not consistent. Take, for example kdelibs. Large scale refactoring took place during the switch from Qt4 to Qt5 because, I assume, during there Qt4 lifespan of kdelibs, focus was on other things. KDElibs are now more competitive than ever and widely applicable to anyone doing Qt development.
the average effective global activity rate in an evolving E-type system is invariant over the product's lifetime
This one is a little tricky to explain. Perhaps a better presentation is this:
Beyond a certain upper limit, adding more resources or effort does not benefit the system in a meaningful way.
To be honest, I have no examples of this ever happening in Free Software. Ever heard of a project (legitimately) saying "You know what, we don't need more contributors." I would love to hear any examples people have.
Something to note: this is distinct from Brook's Law. They both relate to the growth of the team, but the context is very different between them.
as an E-type system evolves, all associated with it, developers, sales personnel and users, for example, must maintain mastery of its content and behaviour to achieve satisfactory evolution. Excessive growth diminishes that mastery. Hence the average incremental growth remains invariant as the system evolves.
The text of this law focuses on the effect of excessive growth: if you make too great a leap in your development you may end up lowering satisfaction. I think it is fair to say that is precisely what we experienced when KDE moved from KDE3 to KDE4. The Gnome community have seen something similar in the last few year's too, I think. This is not news to anyone... radical changes carry risks.
I like to think of this law from another angle, however. Too much growth will reduce satisfaction and, likewise, will a reduction in developers familiar with the codebase. Developer turnover can be even more dangerous than overreaching, in my opinion. Any of my previous green/blue blobs postings will give you an idea about developer turnover in various KDE projects.
I recently read this article on culture inside Open Source communities. There are plenty examples of projects/companies with that kind of messaging who genuinely mean it. There are, sadly, also examples of where Shanley is totally right. E-type software has an environment (most importantly, including users who interact with it). The environment is always shifting and the software must shift with it to remain relevant (the law of "continuous change"). Users generally do not like it when their software does not move with the times. But (the law of "conservation of familiarity") don't change too much! It's a fine balance and it is largely driven by continuously assessing the position of the software in the environment and understanding where to go next (the law of "feedback", not previously mentioned).
Projects and their communities evolve out of necessity. Engagement with the wider world drives this. I suspect most developers reading this will say "yup, that makes sense" for each of these laws. What they might not do, however, is realise the link between the laws and the overall culture of their project. It's a strange thought, but it is true...
Some projects want to ship a product. Some want to ship innovation. When you fire up Emacs and start writing that killer new feature rather than refactoring that spaghetti code you are, probably unconsciously, contributing to the overall culture of your project. Perhaps your community has a substantial focus on QA rather than innovation. Or maybe a balance of the two. Do you want to allow new contributors to join any part of the project they want and find their own way? Or do you want to focus on guiding them towards where effort is needed? There is no right or wrong.
But there are definitely users and you definitely need a mechanism for reacting to their needs and staying relevant. Why? Because wilfully choosing to become irrelevant ain't cool to the people/organisations who rely on that <thing> you wrote. If your users are crying out for more stability and you keep shipping new features instead, you're gonna run int trouble somewhere, sometime.
A software evolves and so must the community. Lehman's Laws could, arguably, apply to both. Why? Well, at the very least, Conway's Law implies symbiosis between community structure and product.
Another year, another opportunity to annoy Nuno Pinheiro with some terrible "engineer art" that I produced many moons ago (see header image). This, my friends in KDE, is why you should be going to Akademy and why you should talk there.
Akademy, in short, is the annual conference of the KDE community. If you are interested in KDE, you should go. If you contribute to KDE, you should definitely go. Why? Because Akademy is the event where we have the opportunity to evaluate our position and to look forward, in a face-to-face manner. It does not matter if you work on libs, or Plasma or that game. What is important is that this is the location for getting your voice heard.
Three major ways of doing this:
So there is a shiny shiny Call for Papers. On there you will see my name; I'm part of the committee evaluating talks. There are some specific topics we mention in that CfP that we would love to see people talk about. However, it is worth remembering, I'm a simple dude. Personally, I'm looking forward to seeing what people want to talk about. So submit your talk on anything, so long as it is relevant and interesting to the KDE community.
Akademy is very good at producing navel-gazing content. And I have no problem with that at all. This event is our one opportunity each year to evaluate where we are at. But it is always great to get more visionary content, too.
Where is Plasma going?
We have three (actually it is four) different types of content planned for this year's Akademy. Here is how I would think they are best used:
The deadline in March 31st. So get those talks in. And soon.
So, if there is something you want to talk about at Akademy, submit a talk. Submit many talks! This is your primary opportunity for the year to get your message across and make a difference to the future of our community!
Last year we had an entire track dedicated to oxidised brass instruments. Important stuff.
Strange things happen while at FOSDEM...
#Github: The social network for cat-hating flame warriors who code.
— Paul Adams (@therealpadams) February 1, 2015
...like losing that prized community visualisation code you've been tinkering with for years.
Short version:
Backing up and merging all the datasets between these machines without a hitch... apart from losing my log parser and visualisation code. So, I spent the last couple of hours rewriting them all. I used Amarok for test input data; not too much, not too little, just hmm-mmm good.
No comment, just sample output from my nearly-complete rewrite of my tools. Needless to say, I was listening to Spotify while visualising Amarok. (Click to enlarge).
You will notice that, at some point, the date format changes in the repo. I'm guessing this coincides with the switch from SVN to GIT. Speaking of coincidences... Look at the "performance" of Mark Kretschmann, the project founder. Project activity appears to be falling over recent years but, when Mark appears to stop work, the whole thing grinds a halt. Coincidence? Nope.
Who here remembers the "Pillars of KDE 4"?
Who here remembers the "Pillars of #KDE 4?" What are the pillars of KDE "now"? #freesoftware #opensource
— Paul Adams (@therealpadams) December 6, 2014
As you can see, I asked the Internet (and got little response). I was attending MeetingC++ as I wrote that tweet and spoke to some KDE folk who were attending. Between us (and online) we could remember:
Oxygen, Sonnet (remember that?), Solid, Plasma, Akonadi and Decibel... This list in one way or another might be wrong. Not so cool. But you know what is cool? People remembered the Pillars of KDE 4 existed and the effect they had.
For me, the Pillars of KDE 4 were a battlecry: "KDE 4 is coming and these are its foundations! Look out!". Or something like that. When I first got involved with the KDE community KDE 4 development was in full swing and the first release was about 8 months away and these "pillars" were everywhere. We used them internally as points of coordination and we used them externally as points of marketing.
In my recent conversation I heard the pillars referred to as "just a piece of marketing". Humph. No, this was only oneof their functions and, for me, by far the least important.
The Pillars of KDE 4 were talismans. The community's good luck charms that we rallied around. The "things" that helped us to identify what was crucial to KDE 4. For me the keyword was "experience". All of these talismans were keys features of the KDE 4 experience; key technologies that were going to ensure our users (old and new) enjoyed "experiencing freedom".
I love "Experience Freedom" as KDE's tagline: what more prominent experience do computer users have, then their desktop?
Once upon a time I was a teenager. It was back then that I discovered Free Software and first become involved. I was using Red Hat Linux or SunOS at school. GNOME was not to my taste (although not bad) and CDE was an affront. I build a preview of KDE 3 from a magazine cover CD and never looked back.
Why?
KDE has always been an extremely well-integrated experience. Sure, there are always things to be "polished" but, in general, everything worked and, more importantly, everything worked together. KDE was the one Free desktop where everything felt nicely integrated. It felt like a thoughtful product.
But KDE is no-longer a desktop. Remember? Wikipedia tells us:
KDE (/ˌkeɪdiːˈiː/) is an international free software community producing an integrated set of cross-platform applications designed to run on Linux, FreeBSD, Solaris, Microsoft Windows, and OS X systems. It is known for its Plasma Desktop, a desktop environment provided as the default working environment on many Linux distributions, such as openSUSE, Mageia and Kubuntu and is default desktop environment on PC-BSD, a BSD operating system.
The official party line (as supported by the Wikipedia entry) is that KDE is the community. KDE is a community of skilled individuals who all pull together to produce various ingegrated technologies, one of which happens to be a desktop (for various platforms). Many of the other projects directly relate to that desktop.
Hmmm... But wait a second. We have a separate definition, in the form of the KDE Manifesto:
We are a community of technologists, designers, writers and advocates who work to ensure freedom for all people through our software.
OK, so this is certainly compatible with the previous definition. Although perhaps a little less specific. Again, the clear point here is that KDE is a community. And this is something I feel increasingly dissatisfied with. What was wrong with KDE being a desktop with a well-integrated collection of awesome applications (which were also able to be run independently of that desktop on all major platforms)?
No really, what?
I first got involved with KDE because I was studying the management of Free Software communities for my PhD. In particular I was concerned about how communities remain productive. At the time I regularly made one specific warning: moving from centralised VCS to distributed will have enormous social implications way beyond the technical ramifications if the process is not monitored and managed properly.
Why? Because the VCS is a tool we communicate through. It is the most crucial tool we communicate through and it dictates how we can build our product(s). The way in which we design our software structure is constrained by how we communicate. Fundamentally change your communication structure and you will be forced to fundamentally change how your structure your code. This is called Conway's Law; I've mentioned it before in this blog and elsewhere. I dare you (nay, double dare you) to try and tell me that somehow it does not apply to Free Software or, more specifically, KDE. Go on. Give it a try.
With SVN KDE had a massive and unwieldy, but integrated codebase (heck, it was all in the same repo!). The community was also massive and unwieldy. But it was well-integrated and this was reflected in the product. KDE 3 was a phenomenon and so was KDE 4 early on.
Then we switched to GIT.
KDE became a collection of projects because each component of the community got its own repo. And we stopped working together as much. Our development became silo'd. There was even some weird behaviour within these silos. (My personal favourite... Adriaan de Groot being told to create his own repo for FreeBSD integration work for a particular application. Apparently creating a branch to work on in the existing repo was a bit too much to ask.)
Then it happened.
We became so disintegrated as a community that this became our branding. We, explicitly, stopped being a desktop project and became a collection of projects, one of which happened to be a desktop.
Let me be clear:
The VCS changed, this forced changes in the community structure and how we communicated, which encouraged us to rebrand in order to closer match our new communication patterns and how we view ourselves.
And, thanks to Melvin Conway, KDE became a less-integrated experience.
KDE 4 had its pillars. As the community disintegrated they fell to the wayside.
The KDE community continues to produce excellent technology. In particular, Plasma 5 is shaping up very nicely (although it is still not quite "there" enough for me to use it as my main desktop yet). But, inevitably, the fractured nature of the community will reflect in the user experience and, at that point, one of our killer features (being "the integrated experience") will be gone.
So I'm left wondering, what are the pillars of KDE now? What are the "things" we, as a community, can all get behind and say "these are the things that underpin the KDE experience".
Akonadi? Baloo? KF5? Plasma? Telepathy? KTeaTime? Heck... I'm gonna throw Akademy into the mix.
Switching to GIT has brought many benefits to KDE (none of which are discussed here). But it has caused us social problems which will inevitably reflect in our products. It already reflects in our messaging, so it is really just a matter of (not much) time.
I think it is about time we took action to socially reintegrate before becoming a totally disperate collection of technologies. My 2c: KDE was strongest when we were all marching towards KDE 4, when the "pillars" were our talismans and we were one in community and vision.
So, tell me, what are the pillars of KDE "Now"?
Yesterday I posed a somewhat cryptic question: what happened in KDEPIM in late 2010 that might have caused such a dramatic spike in an unnamed (but real) metric? I want to leave the conversation running a little bit longer. So here is some further food for thought, to keep you guessing...
Some things to note:
I'm now going to leave this "out there" for a day or two before getting back to it. This buys you guys some time to come up with some more suggestions as to what the mystery metric might be (and what caused the spike in it).
It also buys me some time to find out (for definite) what caused the spike... After all, I do know what the mystery metric is!
Yes, I do know that the previously-red plot is now green and the new plot is red. One day I will master GNUPlot.
If you have had your head in the sand today you might well have missed the story that the GNOME Foundation and Groupon are is a disagreement over the use of "GNOME". I already shared my thoughts in Tweet form earlier... But now I'd like to talk a little more. Not about the event, but about how we (the Free Software community) reacted to it.
Hey #KDE, our friends in the #GNOME project are being attacked by a corporation that should know better. http://t.co/xzfdvp7GMt #opensource
— Paul Adams (@therealpadams) November 11, 2014
The first I heard about the GNOME/Group affair was on Facebook. A former colleague and friend of mine posted a link to the request for aid from the GNOME Foundation. Whilst it is no secret that I am a KDE user, community member and general doer-of-stuff, I am not tribal about it. Healthy competition breeds choice and innovation and am very happy that the GNOME project (along with all the other Free Software desktops) exist. What I saw here was something very simple: an attack by a wealthy corporation on a Free Software project.
Wait... what? "Attack"?
Like many people, my first reaction was one of "What. The. Hey? How dare this company attack my sisters and brother in Free Software?" Aaron Seigo kick-started a discussion on G+ about how we reacted to this situation and in particular, knowing that I would engage rather than enrage, picked up on my Tweet/G+ post. Sadly the conversation he started degraded, largely because some of the people engaging were failing to understand what he was saying... I digress...
I do not want to get into a discussion about the semantics of "attack". I stand by my assertion this is an attack. Why? The GNOME Foundation "call to arms" made it clear there has been an engagement with Groupon (they do have to defend the GNOME trademark after all) and that the engagement has broken down to the point they need to legally defend their trademark. For me, for right or for wrong, it is simple: allowing the situation to degrade to the point of lawyers needing to be deployed to actively stop trademark registrations is an attack of sorts. Groupon is large enough and smart enough to know that such a legal fight will cost GNOME dearly and allowing the discussion to degrade this far could easily be seen as an unpleasant tactic and an attack in itself.
Or perhaps not. More on this later.
What I want to focus on here is not on what we all wrote, but no how we wrote it.
There is a frequently-used joke in my office about the passive aggression of the Brits. We are, after all, masters of deploying phrases such as "With all due respect..." when what we are really saying is "Look you tosser...".
My reaction to this news, as with many other people was one of anger. In my case this was fairly mild and so I think the accurate description would be "I was annoyed". It is not hard to see why so many had a similar reaction. The GNOME Foundation campaign page uses words like "defend" and "fight". They clearly feel threatened and that ain't cool.
I wrote a Tweet. Which automatically got posted to Facebook. I posted a modified (because G+ allows me to be more verbose) version on G+. Now what I was trying to do, of course, was raise some awareness of the GNOME Foundation's action...
So Aaron, perhaps a little harshly, likened the reaction of the Free Software world, in general, to headless chickens. Now Aaron's a pretty big fan of Free Software and the people who make it all happen. So he doesn't use such language lightly. Here is his own reasoning:
Yes, people are running around calling for boycotts, negative social media campaigns and flooding management inboxes .. none of which GNOME asked for .. and all within hours of hearing about the issues for the first time, and certainly having only the barest details. That's practically the definition of acting like a headless chicken.
The man has a point.
Social media allowed the Free Software community to spread the news pretty fast. Add to the pace of news the mildly aggressive language put forward by the GNOME Foundation and things escalated quickly.
By 10am I had the chance to speak face-to-face with my friend who posted the original news. Rather triumphantly he told me I should check out the shitstorm hitting Groupon's various social media outlets. Neither of us at this point had any idea about mailing campaigns.
Aaron's right: the GNOME Foundation did not call for any of this. Explicitly they asked for financial aid. Implicitly, arguably, they asked for people to spread the news. That was it. The uncoordinated "return fire" from the Free Software community on the GNOME Foundation's behalf does not paint a particularly professional picture of the Free Software community.
We let our collective anger show.
(Thanks, Zack de la Rocha).
I don't think there is anything particularly wrong about having gotten angry about this situation. But it really is all about how we deal with that anger.
In this case, many Free Software community members did not recognise the wishes of the GNOME Foundation: to fight this in the legal system, not in the court of public opinion. For this fight they need cash and that is what they asked for. Nothing else. Why? This is their fight (that is their word, remember) to fight the way they deem appropriate.
I am not going to comment on the GNOME Foundation's approach other than to say, if you contributed to the funding, thank you. Whether it is ultimately needed or not, your cash, I'm sure, will get put to good use in Free Software and for that we should all be grateful. Another good option would be to donate to the FSFE who, through their various campaigns are in a position to educate governments and businesses about how to deal effectively with Free Software communities (and their trademarks).
Well, it looks like all the noise around this issue has had an effect. The Groupon engineering blog has this posting with the following key text:
And if we can’t come up with a resolution, we’ll be glad to look for another name.
For some definition of "win", this looks like a win to me. So long as Groupon stick to their word, keep engaging with the GNOME Foundation and, if needs be, rename their product.
So perhaps not an attack, after all. But certainly unfortunate management of the situation to allow things to get to the point that the GNOME Foundation feels a need to defend its trademarks legally. There's probably failings on both sides, if we are honest with ourselves.
But hey, a win. At what cost? Well, at the very least, Groupon (to some extent) may have an altered opinion on how it wishes to engage with Free Software. If nothing else, the eyes of the world (at least those paying attention) will have seen the collective forces of Free Software rally against Groupon and think "they probably could have done that a little more professionally".
As I stated on G+:
I believe we have entered the world of social-constructivism and need to define "attack" :) How about this slightly less aggressive (and derived from +Aaron Seigo 's thoughts) language?... "This sucks for GNOME and, assuming they have acted reasonably towards Groupon, the passive aggresive attitude of Groupon towards the situation is certainly distinctly suboptimal."
So things have been a little quiet on the GIT log analysis front. Various out-of-bounds activities were in my way but, more importantly, so was a directly-related technical issue... Just how do you get file data out of GIT logs?
The problem is this... In the good ol' days I was parsing SVN log files that were XML-formatted; svn log --xml. I also needed the file data to be included in that for many of my analysis tools; svn log -v --xml. Typical output looked something like this:
<?xml version="1.0" encoding="UTF-8"?>
<log>
<logentry revision="58903">
<author>padams</author>
<date>2014-11-04T15:22:30.364435Z</date>
<paths>
<path action="A">/Some/Path/To/A/File</path>
</paths>
<msg>New WMD contract</msg>
</logentry>
</log>
Nice and easy to parse. GIT does not provide any such XML output, however it does provide a --format option where you can specify the output format. For everything but the files. You see with GIT I can get all the information I want, just not in some handy format. And I certainly need to XMLify it myself if I want to make use of my existing parser; git log --name-only:
commit f2fe64b234c31c703c998cacdc1c3cff43f4e05f
Author: Lamarque V. Souza <lamarque@kde.org>
Date: Fri Nov 7 22:37:56 2014 -0200
Doxygen configuration for http://api.kde.org/.
doc/api/Doxyfile.local
I sat down with Ade at Academy and we came to the conclusion that actually the best way to get this file data out of GIT was to use libgit. Holy moly. No, just no. Academy was some time ago and that was the last time I had a chance to look into this. A little bit of Googling around led me to this post which did everything I needed apart from handle the file data. In fact,, I really could not find any information on extracting the file data out of a GIT log. Well, thankfully, the modification was easy. So many thanks to the author of that blog post and here is my recipe!
import pprint
import subprocess
GIT_COMMIT_FIELDS = ['id', 'author_name', 'author_email', 'date', 'message', 'files']
GIT_LOG_FORMAT = ['%H', '%an', '%ae', '%ad', '%s']
GIT_LOG_FORMAT = '%x1e' + '%x1f'.join(GIT_LOG_FORMAT) + '%x1f'
p = subprocess.Popen('git log --name-only --format="%s"' % GIT_LOG_FORMAT, shell=True, stdout=subprocess.PIPE)
(log, _) = p.communicate()
log = log.strip('\n\x1e').split("\x1e")
log = [row.strip().split("\x1f") for row in log]
log = [dict(zip(GIT_COMMIT_FIELDS, row)) for row in log]
for row in log:
row['files'] = row['files'].strip('\n').split('\n')
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(log)
At the end of that short script we are left with a Python dictionary representation of the log which can easily be processed within Python or outputted as XML for use by my other scripts. Hurrah!
Simple solution, but it took a while to get to it. Using the --format option to create sensible delimiters between log entries and their fields had already occurred to me and I had tried things like:
GIT_LOG_FORMAT = '<logentry revision="%H"><author>%an.......<paths>...'
The problem with this, of course... How do I insert the closing </paths> or the individual <path></path> tags? The files are not one of the items that can be formatted using git log --format!
Thanks to the other blog post, I realised that the XMLification should really be a separate step after extracting the required data.
Say what? @valoriez is surprised to hear what I am going to say about #KDE development in my talk tomorrow. #Akademy pic.twitter.com/FLPeJQYNHb
— Paul Adams (@therealpadams) September 6, 2014
David Faure is a pleasure to work with, both within KDAB and KDE. He maintains three important character traits that, as engineers, we should all aspire to:
Despite these wonderful character traits, he has a dark side... He is a lawbreaker. So are many other KDE contributors.
In this presentation I talked about David Faure, Brooks' and Conway's laws and evaluated them in the context of KDE. Some rather fascinating, but alarming results are found!
David Faure: all-round legend and nice bloke of KDE. Fun to work with, thirst of knowledge and skill and, always thinks about the end user.
Positioning David Faure within the context of the wider KDE project and the people he works with.
Introduce Brooks' and Conway's laws: it is incredible how many people identify as software engineers, but do not know what these laws are all about.
How can we measure Brook's law in Free Software projects. We need to start by discussing how we measure anything at all! What are the sources for data in Free Software development.
How we model the community: nodes are contributors, edges show that the two contributors worked together. "Work" means they both edited the same resource in the observed timeframe. Edges get higher weights if the contributors work together in multiple resources.
From this we can find "cohesion", the average length of the shortest path between all pairs of nodes. The higher the cohesion, the more closely-knit the project is. If Brook's is correct then we should see cohesion drop with increased community size...
Yup, Brooks nailed it...
...or did he? KDE is clearly outperforming expectations here. Awesome. Well done KDE! Hint: This is where Conway's law comes into interest.
Wait, if we look at the data over time, something isn't right. Cohesion grows since 2004 until late 2007 (KDE 3.5, KDE 4.0?) but then cohesion declines...
Which just so happens to be the same pattern for funding of the eV, just offset by one year.
KDE has lost its way a little bit. Nothing disastrous yet, but the ship does need righting. As a start, being more like David and enjoying our work, continuing to learn and focusing on users, will get us headed the right way.