Wednesday, April 07, 2010

one team, two teams! small team, big team!

From time to time I hear things to the effect of, "It's hard to get anything done in KDE. Where's the project-wide direction and management?" Indeed, it's not trivial. We could try to excuse the issue and say, "Well, KDE is huge now. 600+ developers contributing to the last release, even more translators, artists and others. That's a lot of people to move about!" While this is true, I don't think it is the whole picture.

In the beginning, some 14+ years ago, "KDE" was a software project to create the "Kool Desktop Environment" (quickly becoming the "K Desktop Environment"). There were a little over a dozen contributors and they all knew each other as a result. It was an ambitious project given how much software they planned on writing, and the results of the first go-around (KDE 1.x) were impressive though certainly small compared to what KDE (which now refers to a community rather than a specific project) puts out today.

Over the years, from that one team sprouted dozens of others. Each of these teams works very similarly to how that first group did: a generally meritocractic process of consensus based development held together by personal relationships. Each of these teams has a similarly strong send of purpose and tends to get things done. We call the sum total of all of them "KDE".

What didn't happen during the creation of all those teams, which was a purely organic and non-engineered process, was the creation of working bonds between the different teams. Sometimes they exist due to the people being involved in more than one (often over a period of time rather than concurrently), but many teams are fairly self-contained. This is not a bad thing, but it makes doing things "across all of KDE" more tricky.

In addition to this, the different teams have different interests and goals. In Plasma we are interested in things like mobile devices and media centers in addition to the desktop/netbook/tablet space. I'm not so sure the KDE Finance apps care much about media centers. I doubt the KDE Games people care that there's a use case difference between netbook and desktop: their stuff almost naturally works equally well in both cases, while the Plasma stuff required engineering efforts for that to happen!

A decade ago we had one set of goals and one group of people who knew each other marching towards them. Today we have a variety of (mutually supportive) goals and many groups of people who know each other marching towards different ones. This is the challenge to getting "global" vision.

What to do? In part, I think we need to drop the idea of getting "all of KDE" to all do the same thing all of the time. It is completely alright for just some of us to focus on mobile, for instance, while others focus on making great photo manipulation software for desktops and workstations. (To pick random examples out of the air. :)

We also need to realize that there are some things that we need "all of KDE" to coordinate on, otherwise they just don't work. The git migration (which is nearing the finish line at this point) and brand repositioning efforts are two obvious, if somewhat trivial, examples of this. They are not the only ones, however. To get those things done, we need to work on the bonds between the teams. We need to re-find our own KDE identity and put it into practice through personal relationships and work across team boundaries. We need to be able to go into shared efforts and not feel the need to henpeck each other just to prove that we're contributing in some way, or feel unease at other's (unknown) intentions because we don't really know those people very well personally.

In Plasma, we are trying to reach out more and more to other teams that are relevant to and interested in our efforts. This includes teams like KWin, Oxygen and Solid (we share goals) but also teams like Skrooge (we share needs). We also strive to tell our story to the rest of KDE so "you all" know what "we all" are up to. We have more to do, but it's a reasonable start. Each of us in each of the teams involved in this "are KDE", but through these cooperative efforts we help build personal relationships, create shared values, generate goals and spread expertise around.

In the coming days and years, if we can build more inter-team bonds then getting things done "in KDE" might be a lot easier and more successful. It may never be as easy as it was when Ettrich first started hacking on his Kool concept (koncept? ;), but that's a price of success. We can, however, make it not only possible to create more global vision more easily but actually do it, too.

11 comments:

Tom said...

Glad to hear you focus on mobile devices. Those are megaüberimportant IMO. I really hope QML will bring a lot of smoothness to Plasma/KDE (shiny & smooth & fast would be awesome)

PS. And it is Etterich, Mr. Sigo.

Tom said...

Sorry, Mr. Seigo. It isn't. You were right, it is Ettrich.

David said...

If I may, please let me argue the counter argument, that is for a smaller lighter, faster KDE. As it is now KDE wants to inform me of things I care little about, like special servers starting. When I want to place a window to the border of my screen it thinks I want to maximize it and when I move my mouse near the corner I get all windows. Granted each of these can be turned off, but it makes me why we have a desktop that seems so noisy.

When I need to help my wife out with her MacBook, it amazes me how things work so well and effectively with a small sense of modesty. Take the Mac startup, an animated 32x32 pixmap of a spinning wheel - just enough to show the user its working without wasting CPU cycles, or disk access loading up multiple pixmaps that basically convey the same message.

KDE is a great desktop, I just wish it were a bit more modest. Thanks for giving me the space to voice my opionion.

Tomasu said...

I suppose this argument could be similar to David's..

But I've been slowly noticing something thats been happening lately in the KDE world. Nothing seems to ever get finished. At least not to the point that was announced. So nothing gets polished it seems. Thinks /look/ ok on the surface, but look under all that and there are significant bugs.

I'm stuck with a mostly useless Desktop machine (Quad Core Q6600) because it doesn't have enough ram to reliably run KDE 4 anymore (2 out of 4 sticks died recently), and I really don't want to run anything else, so it sits unused most of the time. But 2GB ram should be MORE than enough to run KDE 4, but sadly it isn't any more. 2GB is the minimum these days to run Plasma, Kontact, Konversation, Kopete, and a web browser.

Possibly the two largest symptoms of the problem are probably Kmail and Konqueror. Kmail uses up 500MB on its own fairly often, and I've seen konqueror use GBs of ram. However it seems Konqueror is dead in the water, it only got buggier with each KDE 4 release till I had to switch back to Firefox which I can barely stand (it also has memory issues >:( ). Both have cpu use issues as well, where all of a sudden you're stuck with a cpu or two pegged and no way to fix it but kill them (or in kmail's case, wait for it to finish scanning the large folder, but that could take an hour or more, until then kontact's ui is mostly unusable). A possibly third symtom is the knotify4 process. It can and will consume GBs of memory when using the Xine Phonon backend. And until recently it was my only choice since the only other available backend, gstreamer, didn't output any sound what so ever.

I dearly love KDE. I even put on a KDE Release Party here in Edmonton on release day (I wish I could have done it later, but I had to fly out to vancouver a couple days later). I really just want KDE to succeed in a very large and visible way. And unfortunately the actual problem behind the symptoms I've listed will stop that from happening any time soon. People will continue to consider it slow, bloated, and incomplete till things change.

Now I'm aware that linux's mem use statistics is not completely accurate, but I'd like to show you what ps aux has to say right now about all of the kde/desktop related processes: http://pastebin.ca/1859742 Around 2GB of ram used on start up, once my session is restored. This machine starts to KDM with somewhere around 200-300MB used. After my KDE session is restored, its 2-3GB.

And its not just me thats having these problems. There are plenty of bug reports about leaks and mem use, as well as I know a few people that have had to switch to XFCE or Gnome to get a usable computer again.

And before some clever person chimes in to say "Why don't you help?" well, I do report bugs on a semi regular basis, most of which sit unconfirmed and forgoten, and the most egregious problems I face are pretty much being ignored by the entire KDE project as it seems KMail 1 is being dropped, and (seemingly) sits unsupported now in favor of KMail 2 (who knows when that'll see the light of day), and Konqueror hasn't worked adequately enough since KDE 4.0/4.1 (I might go so far as to say 3.5, or NEVER). So I don't see what I can do if the rest of the community is ignoring the problems.

So thats my rant. I really hope this improves near term, or KDE will start bleeding users left-right-and-center. And that can't be a good thing.

I do fancy myself a decent programmer. In C and C++/Qt. So I /can/ help if any one is interested in helping me get started in solving some of these problems.

Thank you for reading. I did not intend any of this to be a flame, or flame-bait. I would truly like to see this stuff improved, even if it means I get my hands dirty.

p.s. I don't really know of a better place to post this, so here it is.

Aaron J. Seigo said...

@David: "Take the Mac startup, an animated 32x32 pixmap of a spinning wheel - just enough to show the user its working without wasting CPU cycles, or disk access loading up multiple pixmaps that basically convey the same message."

i'm not a huge fan of the current splash screen in KDE 4, though it does look nice. i would be highly cautious of jumping to conclusions such as "wating CPU cycles, or disk access loading up multiple pixmaps" without having done some actual work measuring those things. you'd be very surprised how optimized the splash screen is, actually.

i do agree that optics are important and making it seem light and what not are important, since people are generally horrifically bad at accurately guaging such things except by the smoke-and-mirrors o "how things look to my eyes"

Aaron J. Seigo said...

@Tomasu: "2GB is the minimum these days to run Plasma, Kontact, Konversation, Kopete, and a web browser."

and yet people do it in less than that every day, all day. i have 2 GB in my machine here which i use for development and run full debug builds on. day in, day out.

can things be smaller? almost certainly. are they as bad as you try to make out? not even close.

it's really hard to keep a good conversation and movement in the right direction going when the topic is so distorted.

"Kmail uses up 500MB on its own fairly often, and I've seen konqueror use GBs of ram. However it seems Konqueror is dead in the water, it only got buggier with each KDE 4 release till I had to switch back to Firefox which I can barely stand (it also has memory issues >:( )"

KHTML (not so much Konqueror) is an issue, yes. we're working on that. but if you are using Firefox, the it really isn't the source of your issues.

"Both have cpu use issues as well, where all of a sudden you're stuck with a cpu or two pegged and no way to fix it but kill them (or in kmail's case, wait for it to finish scanning the large folder, but that could take an hour or more, until then kontact's ui is mostly unusable)"

hopefully you've reported exactly what triggers this (not just vague "it gets stuck scanning a large folder"). i have very large folders in kmail and have never seen this, but you could very well have a configuration that triggers something very severe.

then again, that's why they are moving to akonadi in the first place.

"Around 2GB of ram used on start up, once my session is restored."

the memory output you posted in completely meaningless. i'm not sure how many times it can be said, in how many ways, but: do not try and do your own memory measurements unless you really know what you are doing.

here, ps says plasma-desktop (running constantly for over a day now doing various things with it, full debug build) is using 332MB VSS and 37MB RSS.

ksysguard's more meaningful (though still fuzzy) numbers show it using 17MB of actual memory. it also shows the 13MB its using in x.org memory, which ps doesn't show at all.

the numbers you posted look even more inaccurate that the ones ps is giving me here.

it really comes down to the quality of the packages and the quality of tools used to measure the results.

"A possibly third symtom is the knotify4 process. It can and will consume GBs of memory when using the Xine Phonon backend"

afaik, this was fixed some time ago. thankfully there is work on Phonon ongoing.

"And its not just me thats having these problems."

agreed.

"There are plenty of bug reports about leaks and mem use, "

and plenty of such bugs have been fixed. some also exist completely outside the KDE stack, but we've done a rather good job of tracking down the ones inside our code to date.

"So I don't see what I can do if the rest of the community is ignoring the problems."

by working on akonadi to solve architectural issues in kmail (something you're evidently running into), the community is not ignoring the problems.

by spending endless hours profiling memory, tracking down leaks and fixign them in plama-*, the community is not ignoring the problems.

each release of KDE SC 4.x has gotten faster and more reliable.

so while there is certainly room for improvements yet, your doomsdaying doesn't match up with what's actually going on.

"So I /can/ help if any one is interested in helping me get started in solving some of these problems."

there's great documentation on techbase.kde.org to get you started, and you can find us in #kde-devel on irc.freenode.net and on the numerous KDE mailing list.

there's no end of community help available, but you need to take the step towards that help. hundreds do that every year, and so can you :)

Tomasu said...

(cont)
My main concern is that people don't seem to be following through. They start a project, go part way through, and look across at new project and go "ooh, shiny" and leave people up a creek with only partly usable programs.

Again, 2G+ is not reasonable. 1G is on the top end of reasonable imo. 2 and 3G is pushing it WAY too far. You should be able to start up your desktop, and a few favorite apps, and still have ram left for page/file cache on a 1G system.

My laptop is sitting at 2G ram used according to free -m. 1.5 or more of that is the desktop.

I'm just very frustrated by the lack of direction and follow through.

Do you see this improving? Given the pattern, I only see it getting worse.

"there's great documentation on techbase.kde.org to get you started, and you can find us in #kde-devel"

Get me started where? Obviously I'd like the memory issues fixed, but if the KDE experts can't fix them, how can I be expected to? And then theres Konqueror, it seems no one is interested in even working on it anymore. So I doubt there'd be anyone willing or possibly even able to help mentor a kde dev n00b.

I also don't want to turn into "the guy that fixes other people's messes".

I've actually been in #kde-devel (and #plasma) for ages. But I can't think of a single thing I can do, that doesn't trap me into working on something thats unsupported and deprecated.

Maybe you, or someone else can suggest something. Something that will help fix something real thats bugging me so I stop having this nagging doubt about KDE and its future.

Thank you for reading.

Tomasu said...

@aseigo

"and yet people do it in less than that every day"

Sure, but it runs like a dog. Right now, if I had only 2G physical ram, I'd be into swap for 500MB or more. I can't stand using a system thats paging every time I switch applications.

"it's really hard to keep a good conversation and movement in the right direction going when the topic is so distorted."

I'm sorry my experiences suck. Really I am. But my desktop ran fairly poorly with KDE 4.2, worse with 4.3, and with 4.4 it swaps quite a bit. With 4.4 my laptop, which has 4G ram occasionally swapps. That is just not acceptable to me.

"KHTML (not so much Konqueror) is an issue, yes. we're working on that. but if you are using Firefox, the it really isn't the source of your issues."

It is actually. The fact that KHTML and Konqueror itself are so buggy means I can't use them, and am stuck with the only other semi usable browser. I used to use konqueror fairly religiously. Even while it started getting buggier and buggier, I kept using it, because for a time, it was better with memory use than firefox. Last time I tried, it was far far worse. And flash got pretty buggy in it again too. For a long while, it worked better in konqueror than in firefox.

Rephrased, I'm using firefox because konqueror would be using MORE memory right now. So it clearly is the source of (some of) my issues. I'd much rather be using a kde based web browser right now.

"hopefully you've reported exactly what triggers this "

Yup and they know about it. The folder list is horribly broken. And they are punting to KMail 2 it seems. So no fix till they manage to get KMail 2 out. So every one is stuck with brokenness till then.

"the memory output you posted in completely meaningless."

No. It isn't. From KDM, to full KDE startup, theres a 2 to 3GB increase in memory use. Actual memory use. As in measuring free -m both times and comparing them (minus cache/buffers). I don't see how it could get any more meaningful.

As you mentioned sysguard, here's what it has to say: http://strangesoft.net/systemactivity-memuse1.png

Not very pleasant. Also, it (or krunner) crashed when I tried to ksnapshot it the first time. I don't seem to have debug packages installed, I'm not sure debian has any yet. (and before you ask, no I can't install from source, getting the python scripting working is way too time consuming and error prone for me, only once in the /many/ times I have installed KDE 4 from svn have I ever gotten the python support to work. once. so I'm stuck with debian packages)

"afaik, this was fixed some time ago. thankfully there is work on Phonon ongoing."

So I hear. But it still seems to be a problem.

"the community is not ignoring the problems."

So why with every KDE 4 release do I need to buy more memory? With the exception of 4.0 to 4.1. Mem use went down there due to lots of fixes in plasma and X.

"so while there is certainly room for improvements yet, your doomsdaying doesn't match up with what's actually going on."

Not quite doomsaying. Just stating my experience so far.

let me count the ways, plasma-desktop doesn't seem to be getting the attention it deserves, everyone is spending their time on the alternative versions (phone, netbook, mid, whatever). The long/oft promised "activity" support just isn't happening. Konqueror is completely broken as a web browser, and while kwebkitpart helps, it still doesn't integrate well with it. And the existing kmail application looks to me like it's been dropped in favor of a rewrite thats probably a year or two away from actual usability. Meanwhile I have a large number of akonadi processes using up ram that I just don't use. And every time I restart kde, it seems akonadi adds a new resource. I had to remove about 30 of them last week, that helped remove the memory pressure a tiny bit.
(to be continued)

Aaron J. Seigo said...

@Tomasu: "Sure, but it runs like a dog."

no, actually, it runs very nicely in far less ram than you have. i know because i do exactly that on some of the smaller machines around the house here.

now, i'm not doubting your personal experiences, but i do know that that experience is not indicative of what a KDE workspace is capable of right now.

"Right now, if I had only 2G physical ram, I'd be into swap for 500MB or more."

funny, i'm typing this on a machine with 2GB physical ram, running a web server and a database for a project i'm working on plus a full KDE Plasma workspace session with kontact, firefox, rekonq, konsole, open office and other apps open. i last shut it down yesterday, and we were watching movies on the t.v. via it yesterday... and it's using 550MB right now. no swap is being used.

either you don't know how to read the output of free, or your system is simply set up very, very poorly.

"I'd much rather be using a kde based web browser right now."

me too, and i'm easing into rekonq as it gets better and better over time. but i don't see that one issue as indicative of the entire sky falling, not when there's lots of other great stuff happening.

"And they are punting to KMail 2 it seems. So no fix till they manage to get KMail 2 out. So every one is stuck with brokenness till then."

can i point out the obvious ridiculousness of this statement? they are working on fixing it and you are still complaining about that, and trying to say that "My main concern is that people don't seem to be following through"

you aren't being consistent. you are, however, looking for every possible reason and way to complain. i don't have much time in my life for such things, and i don't think many other people who are actually want to get things done do, either.

Aaron J. Seigo said...

"plasma-desktop doesn't seem to be getting the attention it deserves, everyone is spending their time on the alternative versions (phone, netbook, mid, whatever)."

oh really? so what was i blogging about in "solving a small problem with a slightly bigger solution" then? have you seen the commits to things like the wallpaper config, the panel loading, etc? no, you just come here and say nobody is working on it. well, start calling me Nobody then.

"The long/oft promised "activity" support just isn't happening."

this is also untrue. Ivan's activity service work is in review, Chani's been working on the UI. there has been huge amounts of work on the design, an we had one whole track at tokamak 4 for that topic alone. we aren't just putting it into plasma-desktop but exposing it as a desktop-wide service (which has slowed immediate progress down, but is the Right Way to go long term)?

when we do deliver it, i do hope you step up and recognize.

"Get me started where?"

get you started on hacking. you said "So I /can/ help if any one is interested in helping me get started in solving some of these problems." and that would be great. but honestly, i don't believe you, because if you meant it you would get involved. there's nothing stopping you but yourself.

"but if the KDE experts can't fix them, how can I be expected to"

do some profiling, write some code that addresses the issues they turn up. it's not that "the KDE experts can't fix them" it's that there are a limited number of us with a hell of a lot to do.

"I also don't want to turn into "the guy that fixes other people's messes"."

that sounds like an excuse for not doing anything rather than a serious reason.

"I can't think of a single thing I can do, that doesn't trap me into working on something thats unsupported and deprecated."

another excuse, given that the overwhelming majority of code in KDE 4 is neither unsupported nor deprecated.

"Maybe you, or someone else can suggest something. Something that will help fix something real thats bugging me"

how about the activities support in plasma-desktop? come find me on #plasma this week sometime and i'll get you started.

Tomasu said...

I think we've gotten off track, and thats partly my fault. If I sounded overly accusatory, I apologize.

I do think you've hit my concerns on the head though when you said "it's that there are a limited number of us with a hell of a lot to do.". I totally agree. The point I wanted to get across, but failed miserably. That was that you've spread yourselves too thin. Take Plasma for example, instead of focusing on a single experience, polishing it up to a shine, and then moving onto the next. Instead theres four separate "experience" projects now. Not that the netbook or phone interfaces are bad, on the contrary, I think they will probably prove very /awesome/, but in the mean time its pulled the team's attention away from what they started with.

Basically what I've seen is things get started, then more things get started, and development on the previous things come to a crawl.

I'd appreciate it if we could keep things somewhat civil though. I don't think I once even implied you or KDE as a whole was /stupid/, so it would be nice if you could refrain from implying I am.

That said, I've been in #kde-devel and #plasma for some time. I just haven't found something I think I'd enjoy working on. Does the plasma activity support have a plan/design-spec yet? Or is that part of the assignment?