massif improvements

So a few nights ago Maksim-of-KHTML mentioned in passing that it might be interesting to run Plasma through Valgrind's Massif tool. He said he'd done so recently with Konqueror and it had shown some interesting things, such as the creation, parsing, writing, etc of a massive history configuration file (it was 1.5MB on his machine, 800k on mine .. not a small config file to be parsing or saving out constantly).

I tried to ignore it.

Really, i did.

But once the seed was planted it festered. I looked at the clock and it was late evening / early nighttime. "I should go to bed," I thought. But nooooooo... like the occasional *ahem* OCD sufferer that I am, I fired up Massif and pushed Plasma through it like a log through a chipper. The results were quite interesting.

My current plasma config has two desktop Containments/Activities and one Panel Containment/Activity. The panel is the standard layout for a laptop, and the one Activity has the standard folderview and wallpaper but also sports a few extra Plasmoids I use or are working on. The other desktop Activity is a FolderView pointed at a folder with 31 files in it and a different wallpaper. Both Activities are 1280x800 in size. This is the configuration I used for all the tests (except where I turned off wallpapers to see their impact on things) so as to emulate a real world scenario. Namely, mine. ;)

The first thing I noticed in the output was that Plasma was doing a lot of allocations at startup for parsing XML. What XML? SVGs. That was to be expected, really, but I didn't expect it to be the primary source of allocations. Eventually the wallpapers dwarf that memory usage, allocating some 9MB of memory for image data for my two desktop Activities. But still ..

For some time now I'd wanted to use a KPixmapCache to store the results of SVG parsing between runs of Plasma. We were already using a runtime memory cache via QPixmapCache, but KPixmapCache would take it one step further. So I started implementing a cache in Plasma::Theme that Plasma::SVG could use.

(Interesting side-note: KPixmapCache was one of last year's Google Summer of Code projects...)

I rapidly ran into issues because we use SVGs rather .. aggressively in Plasma. We don't just use them for static images, but also embed rendering hints in them, render specific elements of SVGs into larger composite images and various other tricks. This made the goal of "don't open a single SVG on startup" a bit harder to accomplish.

Over the last couple of days, Marco and I worked on various parts of what would be a complete solution, and today Marco nailed the final remaining blocker ... it is now possible to start Plasma up without touching a single SVG file. The cost is some disk space; how much disk space is something we'll work on nailing down. Right now it's a fixed sized cache, but I hope to make it an adaptive cache.

The wins are pretty signifcant. In my first massif runs Plasma would spike up to ~18-19MB of allocations and end up at around 7MB of allocations according to Massif. (Note that this is not the total memory usage, but counts of allocations ... slightly different things. =) Now it spikes up to about 1MB less than that, but more signficantly it ends up at ~2MB of allocations by the end of startup.

It also means we're reading already rendered images from an mmap'd cache file on disk, and this gains my Plasma config a ~10% start up improvement with warm caches on my system. That's not bad, of cousre .. but then again we're talking the difference between 1.8 to 1.9 and 1.6 to 1.65 seconds. So maybe not the most earth shattering of changes to the user. ;)

Update: I did some quick cold-cache timings: sync && echo 3 > /proc/sys/vm/drop_caches as root, then start plasma; did that four times both with and without disk caching, discarded the highest number in each sample set and averaged the remaining times; hopefully I got the methodology mildly right. ;) Assuming I did, the savings are also ~10% on cold start though that translates to 500ms instead of 200ms.

Now I'm seeing things like kDebug() usage and QLibraryPrivate::isPlugin as top allocators at start up. I've reported the latter to the Qt Software engineers, as improving that one dramatically shouldn't be too difficult as right now it's re-allocating (which means load and parse) the Trolltech.conf settings file with each plugin we load. You may have noticed that Plasma uses a "few" plugins "here and there". *cough*

The worst offender is still the wallpapers, of course. They are rendered in separate threads, and so only add ~150ms to start up with my above set-up, but they also cost about 5MB of peak allocation per 1280x800 Activity I have. I have some ideas for how to improve that situation somewhat, though.

Humorously (and I use that word as a replacement for "tragically"), the system tray takes nearly 7 times longer to load its icons into the tray than the rest of Plasma takes to start up. The more icons you have in your tray, the worse it gets. As we shave those 100ths of a second off of Plasma start up time, the system tray makes me cry a little more inside with each passing day.

Anyways, we now have the task ahead of us of profiling the benefits of different size caches and caching strategies in different usage scenarios. Caching can be turned on/off and the size controlled in the configuration file of the libplasma powered application, with something like this:

[CachePolicies]
CacheTheme=true
ThemeCacheKb=102400

A huge thanks to Marco for helping me get this rather detailed and perhaps less-than-fun (to say the least) set of work done. I applaud everyone who works on Free software, but doubly so those who take on the less glamorous work. Hats off to Marco, he is certainly this week's Plasma Hacker Extraordinaire! =)