Sunday 19 April 2015

Evolving KDE: Lehman's Laws of Software Evolution In The Community

The board of KDE eV has launched a new initiative to ensure that KDE remains awesome and relevant for the foreseeable future. Unlike previous approaches it is not a point-in-time solution, it is a continuous process of improvement. And it is a good thing.

Previously, I have written/spoken a lot about the role of Brooks' Law in the context of Free Software. Brooks' Law teaches us to be careful about the management of growth in our communities. Especially treated in consideration with the grossly under appreciated Conway's Law. There are, of course, other laws of Software Engineering that apply to Free Software development.

Last weekend I attended the KDE eV board meeting held in Berlin. I was part of a team of KDE community members who provided our thoughts on the board's desire to ensure the overall KDE community continues to work effectively and continues to deliver awesome software. A lot of the discussion involved the word "evolution". Now the context was the community... but our software continuously evolves, too. What are the implications on this for a Free Software Community? And are community evolution and software evolution somehow linked?

What's Your Type?

In the 1970s it was formulated (paywall, sorry) that any piece of software will fit into one of three categories:

  • An S-program: written according to an exact specification of what that program can do;
  • P-program: written to implement certain procedures that completely determine what the program can do (the example mentioned is a program to play chess);
  • An E-program: written to perform some real-world activity; how it should behave is strongly linked to the environment in which it runs, and such a program needs to adapt to varying requirements and circumstances in that environment.
(Source Wikipedia)

Now a Free Software project could, in theory, fall into any of these. I want to focus on KDE which, arguably, is almost entirely E-type software: software that must evolve to satisfy its shifting environment.

The Laws

So let's take a look at Lehman's Laws and what they mean for Free Software. I'll couch them in terms of KDE as much as I can. Please note: this is not scientific, just my thoughts on what we can readily observe in the wild and what it means. This is also not exhaustive; I'm cherrypicking a selection of the laws.

Continuous Change

an E-type system must be continually adapted or it becomes progressively less satisfactory

In this context "change" can mean various things: adding features, removing features, QA improvements... I'm not going to single out any particular parts of KDE, but I think it is fair to say that parts of our technology have failed to remain competitive because activity declined (and, thus, "change" with it). More positively, I think it is very easy to point out that much of KDE remained competitive because it did change; it's not very often that a part of KDE becomes completely moribund.

Increasing Complexity

as an E-type system evolves, its complexity increases unless work is done to maintain or reduce it

Refactor, refactor, refactor. Once of the things I love about KDE is we are always marching forwards. We are very focused on staying competitive by adding new features or by reinvention. That's not to say we ignore Bugzilla, however. Time is always being spent in bug fixing. But what about other QA activities? The fine-polishing work is not consistent. Take, for example kdelibs. Large scale refactoring took place during the switch from Qt4 to Qt5 because, I assume, during there Qt4 lifespan of kdelibs, focus was on other things. KDElibs are now more competitive than ever and widely applicable to anyone doing Qt development.

Conservation of Organisational Stability

the average effective global activity rate in an evolving E-type system is invariant over the product's lifetime

This one is a little tricky to explain. Perhaps a better presentation is this:

Beyond a certain upper limit, adding more resources or effort does not benefit the system in a meaningful way.

To be honest, I have no examples of this ever happening in Free Software. Ever heard of a project (legitimately) saying "You know what, we don't need more contributors." I would love to hear any examples people have.

Something to note: this is distinct from Brook's Law. They both relate to the growth of the team, but the context is very different between them.

Conservation of Familiarity

as an E-type system evolves, all associated with it, developers, sales personnel and users, for example, must maintain mastery of its content and behaviour to achieve satisfactory evolution. Excessive growth diminishes that mastery. Hence the average incremental growth remains invariant as the system evolves.

The text of this law focuses on the effect of excessive growth: if you make too great a leap in your development you may end up lowering satisfaction. I think it is fair to say that is precisely what we experienced when KDE moved from KDE3 to KDE4. The Gnome community have seen something similar in the last few year's too, I think. This is not news to anyone... radical changes carry risks.

I like to think of this law from another angle, however. Too much growth will reduce satisfaction and, likewise, will a reduction in developers familiar with the codebase. Developer turnover can be even more dangerous than overreaching, in my opinion. Any of my previous green/blue blobs postings will give you an idea about developer turnover in various KDE projects.

Evolving Evolution (Or, "What's This All About, Paul?")

I recently read this article on culture inside Open Source communities. There are plenty examples of  projects/companies with that kind of messaging who genuinely mean it. There are, sadly, also examples of where Shanley is totally right. E-type software has an environment (most importantly, including users who interact with it). The environment is always shifting and the software must shift with it to remain relevant (the law of "continuous change"). Users generally do not like it when their software does not move with the times. But (the law of "conservation of familiarity") don't change too much! It's a fine balance and it is largely driven by continuously assessing the position of the software in the environment and understanding where to go next (the law of "feedback", not previously mentioned).

Projects and their communities evolve out of necessity. Engagement with the wider world drives this. I suspect most developers reading this will say "yup, that makes sense" for each of these laws. What they might not do, however, is realise the link between the laws and the overall culture of their project. It's a strange thought, but it is true...

Some projects want to ship a product. Some want to ship innovation. When you fire up Emacs and start writing that killer new feature rather than refactoring that spaghetti code you are, probably unconsciously, contributing to the overall culture of your project. Perhaps your community has a substantial focus on QA rather than innovation. Or maybe a balance of the two. Do you want to allow new contributors to join any part of the project they want and find their own way? Or do you want to focus on guiding them towards where effort is needed? There is no right or wrong.

But there are definitely users and you definitely need a mechanism for reacting to their needs and staying relevant. Why? Because wilfully choosing to become irrelevant ain't cool to the people/organisations who rely on that <thing> you wrote. If your users are crying out for more stability and you keep shipping new features instead, you're gonna run int trouble somewhere, sometime.

In Conclusion

A software evolves and so must the community. Lehman's Laws could, arguably, apply to both. Why? Well, at the very least, Conway's Law implies symbiosis between community structure and product.

  • Continuous Change: The KDE community must always evolve...
  • Increased Complexity: ... in a controlled manner. We must always adapt ("refactor") the community to suit the needs of the challenges we address
  • Conservation of Organisational Stability + Conservation of Familiarity: We must drive forward, we must grow and we must adapt, but we must also keep in mind that experience is precious and we should support our most experience contributors as they support the next generation of users.