Thursday, August 21, 2008

associating widgets with mimetypes

One of the neat features in Plasma is the ability to associated widgets with mimetypes. To associate a widget with a mimetype, you add a X-Plasma-DropMimeTypes key to the .desktop file for the widget.

For instance, the picture frame widget has this in its .desktop file:

X-Plasma-DropMimeTypes=image/jpeg,image/png


When I drop an image file onto the desktop, I get asked if I'd like to make a picture frame or an icon out of it. In 4.2, I can also drop raw image data onto the desktop and it does the same (except it doesn't give the option to create an icon for it as it isn't actually a file).

The way this works is that Plasma::Containment queries the widgets to see which, if any, supports the mimetype(s) in the drag object. It's smart enough to follow the mimetype priority if there is more than one type offered in the drag, so html content will be dropped as html content and not plain text. On my 4.2 system, that results in the mini-browser widget being created.

Inside the widget itself, the first argument passed in on the args list will be the URL to a file containing the data. In this way, the widget doesn't need to concern itself with caring about the difference between a "real" file and a blob of data dragged onto the Containment.

Inside the widget, the code can check for this in the constructor with:


KUrl url;
if (args.count() > 0) {
url = args.value(0).toString();
} else {
// do some sort of default set up instead
}


That's all there is to it: one line in the .desktop file and a couple lines in the widget itself.

For the user, it means they can drop "stuff" onto their desktop and the Right Thing(tm) happens depending on the content in the "stuff" without them having to do anything particularly special at all.

I'll be extending this to wallpapers as well so that wallpapers can also say, "oh yeah, i support that kind of data", which will result in the addition of a "Set as wallpaper" to the menu that pops up. Which means you will eventually be able to drop an image file onto a containment and choose between creating an icon shortcut for the image, creating an image frame widget for it or setting it as the background. And by "image file" I mean any file type that is supported by wallpapers or the image frame.

I think this could have some really cool applications if people use their imaginations a bit. To get you started: what could happen when a contact is dropped onto the desktop (e.g. a vcard or maybe even a contact listing from a social networking site)? what could happen when an email is dropped from kmail/thunderbird/whatever? what could happen when a movie is dropped? what could happen when a vnc connection is dropped? what could happen when a torrent is dropped?

What might be extra neat here is when combined with other widgets, like folderview and nepomuk. If the user is looking at a folderview showing all video files, once we have a working video player plasmoid the user could just drag one of these videos to the desktop and watch the video play without starting up another application. In fact, they could drag a couple of them onto the desktop and have multiple videos playing. And it would all Just Work(tm).

Now consider using something like this within an application ... say for instance Amarok2. You drag a movie into its plasma-driven context view, and it could quite easily pop up the movie widget with approximately 0 lines of code inside of Amarok2. I don't know if the Amarok2 peeps have such a thing planned, but it demonstrates the concept, one that could probably be extended to other applications as well.

I plan on playing with this auto-widget-generation-based-on-mimetype on small form factor devices as well as a way to perhaps limit the crazyness of opening entire apps for simple processes. Of course, on those devices it probably won't be triggered by physical drag 'n drops, but the idea and code paths will be the same. I'm not sure yet just how useful it will be in that context, but there's only one way to find out, right? =)

Note that in 4.1, drops are limited to applets or files. It's only in 4.2 that we have the non-file drop support (e.g. directly from a web page to the desktop, or from the image thumbnail in ksnapshot to the desktop). Props to Ivan for pointing out that the drop support should be extended to non-file drops as well, prompting me to flesh out this feature to its logical conclusion.

29 comments:

OTHMANE said...

With these features KDE will be simply wonderful!
Thank you Aseigo and plasma developers!

Tomasz Dudzik (sheytan) said...

I just cnat wait for 4.2 :D It's gonna be the best desktop ever! :D

Ravi Vagadia said...

KDE 4.2 would be rocking :D:D

thank you KDE devs!!!!!

[Knuckles] said...

I noticed the drop picture to create picture frame the other day and it was very cool -- one of those features that while unexpected (because you never saw it before), it's expected, because it makes sense, you drag a picture, you create a picture frame for it.

As for the other, new features I can just say: Dammit I want 4.2 today! :)

damiga said...

Might it be possible to highlight some text, drag it onto the desk top, and have it show up as a sticky note? Or even just a text file?

JT said...

"Might it be possible to highlight some text, drag it onto the desk top, and have it show up as a sticky note? Or even just a text file?"

That's a good idea. I use XP at work and I'm always using Notepad to dump draft text and pasted text in. If KDE 4.2 can open something automatically, it'd be quite convenient.

Aaron J. Seigo said...

@damiga: precisely.

Aaron J. Seigo said...

@damiga: or more verbosely, that already works in svn =)

morphado said...

cool, just one complain, before droping works on plasma ( desktop containement) applets must be unlocked witch is a little unconvenient,

yman said...

Isn't this very similar to the interface of MS Surface? If I remember from the demo videos, the main feature was that when a device was placed on it, it made it's features available in a way that's very similar to how such stuff is represented in the real world. If the device is the file, then the behavior seems to be very similar to what you just described.

It's just a thought that came out of nowhere. Anyway, this is really cool stuff!

Juha said...

Great improvements indeed.

Only thing that bothers is the ability to drag and drop things to containment and back while working with maximized windows - as I'm often doing. But I'm sure you've solved that one already as well - using drag'n'drop target zone in panel, perhaps?

Aaron J. Seigo said...

@yman: yes, there is overlap with what Microsoft (and others) are thinking with systems like surface.

except we have even a few more tricks up our sleeves which should be making their way up to the surface (excuse the pun ;) over the next year or so.

oh, that and we're not trying to lock people in, tie it to unrealistically expensive gadgets or introduce "solutions" which are really in search of problems (which is mostly what i perceive surface to be right now).

as an aside: have you ever noticed how nearly every multitouch demo ends up eventually devolving to "look, you can spin and resize photos that are scattered randomly!"? =) it's become the classic demo, but i have yet to figure out when i'd actually care about that use scenario in any significant way. i mean .. am i really supposed to be excited about working with documents as a pile of clutter using various magic hand gestures? (pinch is fairly intuitive, but most grouping and throwing exercises in these demos are not)

@juha: yes, you can also drag to the panel; i also would like to find a way to let a drag be "transported" to the dashboard view in a convenient manner, though that's apparently a bit trickier. but you are right that the "always maximized windows" work pattern does throw us some curveballs from time to time =)

jamboarder said...

@aaron:"also would like to find a way to let a drag be "transported" to the dashboard view in a convenient manner, though that's apparently a bit trickier."

Mouse guesture shortcut to activate/deactivate dashboard view maybe? just a random thought...

yman said...

@Aaron:
"oh, that and we're not trying to lock people in, tie it to unrealistically expensive gadgets or introduce "solutions" which are really in search of problems (which is mostly what i perceive surface to be right now)."

I really want to respond to that. Not to the content, but to the fact you even bothered wasting your time writing that mini-rant. But I won't, because I can't think of a right way to write it, so please just take note that I think it was the wrong thing to do.

"what could happen when a vnc connection is dropped?"
You'd have to reconnect, of course.

damiga said...

I can see opportunities with LinuxMCE too. When are we going to hear some new news on the LinuxMCE front?

[Knuckles] said...

Maybe if you dragged content on top of the "view dashboard widget" that you can have on the panel, the dashboard would activate, in a similar way as when you drag content to a window in the taskbar, that window activates, and you can drop whatever you're holding there.

Just a thought :)

Aaron J. Seigo said...

@yman: why do you think it was the "wrong thing to do"?

those things i mentioned are some of the biggest advantages *Free* (as in freedom) software has over technology like Microsoft Surface.

"You'd have to reconnect, of course."

lol! nice one ;)

@damiga: "When are we going to hear some new news on the LinuxMCE front?"

unfortunately, their project has hit some manpower issues; i met up with some of them at linux tag the other month and it was really productive, but they still are juggling more balls than they have hands for. unfortunately, i don't have time myself to help LinuxMCE find their feet again right now; hopefully within the next year i will, though, if they aren't back on track before then.

@[Knuckles]: good idea =)

damiga said...

@aaron: "they still are juggling more balls than they have hands for"

too bad, although to be honest it's more than I'm really looking for which is a 10' UI with basic music, movies, photos, etc. with KDE flavour.

yman said...

@Aaron:
Because it makes you seem to lack confidence in what you believe and what you are doing. I don't see anything that was said in the comment section to prompt an attack against MS, except for me mentioning it's name in a totally neutral context. So it seems you are over-sensitive and very defensive.

And the reason I see this in you is because I see this in myself, and it really annoys me. And as the previous sentence indicates, I'm also an apologist.

Aaron J. Seigo said...

@yman: it's not an attack on Microsoft, it's a statement of the benefits of what we're doing versus what they are doing in the areas where our efforts overlap.

i am not ashamed to stand up and say, "we do not lock people in, etc..", in fact i am proud to do so. it's one of the more compelling features of our software, in fact.

and the same can be said about the areas we overlap with MacOS, such as their dashboard widgets and, as we move towards the mobile space, their iPhone stack.

it's interesting that in the quest for treating Microsoft with respect (e.g. not referring to them as Micro$oft, or worse) we have tripped over to the other side where it's taboo to talk about our competitive benefits.

florian.hackenberger said...

The widget <-> mimetype association could be used to solve a problem I had in the past. Whenever new icons appear on the kde desktop, they were placed on the screen almost randomly. Say a new device is plugged in and the associated icon is placed on the desktop. With the mimetype <-> widget association we could configure an icon container (can't recall the plasma speak term for it) to accept only device icons, while another container will accept documents and another just applications. Result: some structure on your desktop...finally. Would that be a viable use case Aaron?

Michèle said...

be careful Aaron,, you start to blog too much :).
Go back to code ! Or evil comments like 'When will you FINALLY let us *hide* the annoying top right cashew?' might alter your karma.

yman said...

@Aaron:
It seems we both misunderstood each other. I was just bringing up a little anecdote. You took it as if I'm implying that Plasma doesn't have value since the feature appears elsewhere or something like that, so you pulled out all the guns in Plasma's defense.

Only problem is, I never meant to compare Plasma to Surface, so there was no need to defend it. I took the first two paragraphs of your reply to mean "here's the great stuff were planning", so to me the 3rd paragraph was attacking MS out of the blue for no reason.

I don't know if I properly understood you this time, but it seems this must be a misunderstanding, so we might as well forget it.

Aaron J. Seigo said...

@florian.hackenberger: that could certainly be done, yes. could be quite handy, as you note.

there are lots of possibilities here, really, and it will take us some time to explore them all and sort out the useful from the not-so-useful ones i'm sure =)

@yman: that's not how i took it at all. =)

yman said...

@Aaron:
I'm not going to waste your time asking for an explanation. In this case the truth doesn't matter: as long as I can explain you to myself I can let the subject drop. It was a useless subject anyway.

Ramon Antonio said...
This comment has been removed by the author.
Ramon Antonio said...

Hope all applets implement it. It would be interesting we could use this feature also dropping into a panel.

Also dropping into existing plasmoids (and applications) is interesting. I think such thing is already possible but not widely used. For example dropping an image into a icon applet would change icon image.

Michael said...

@how to drop content when there are maximized windows

i like the way mac os x or symphonyos for instance handle such issues by using the corners of your screen as entry points. you drag some content and move to one corner which activates expose to choose another window to drop it or you move to another corner which activates the dashboard or shows the desktop.

altough i have to admit that at least in osx the feature is hidden well and imho not activated by default. symphonyos has special "icons" in the corners and uses this concept throughout their mezzo desktop. in plasma this might be a use case for the cashew...

syler2000 said...

Great idea. Just an extension if I may; what about associating widgets with regular expressions (regex) in addition to mimetypes. Say have sth like X-Plasma-DropRegEx=http://* in the .desktop files. What im thinking is that if some text is selected (in an application/textbox etc) and copy-pasted/dragged to the desktop, plasma would be able to match the text to the regular expression and on finding a match in the .desktop files of the widgets, provides the options e.g.
1) regex: http://* , would offer opening up a webkit widget to show the url, or create a text file, or open the page in full browser (konqueror) etc.
2) regex: ftp://* , would offer the choice to display the contents of the ftp site in a folderview widget
3) regex: *.torrent , would offer to create a ktorrent widget and start downloading