Nepomuk – What Comes Next – Revised

[menotu]

Even though everything works fine on my systems I'm very pleased he decided to listen to peoples comments and feedback and revised his future plans - makes a lot of sense to me.

===============================================

http://trueg.wordpress.com/2011/09/19/nepomuk-what-comes-next-revised/

After reading your comments on my last blog about the next steps in Nepomuk and discussing with several people I decided to revise my plan for the next half year. Instead of directly diving into cool new features I will start with four very simple but very important topics:

    Fix all the crashes (and as many of the other bugs as possible) that have been reported on bugs.kde.org.

    Improve the Nepomuk startup time and lower the IO usage during Nepomuk startup.

    Fix any left-over issues in libstreamanalyzer (aka the strigi lib, used for file indexing in Nepomuk), meaning proper indexing of all files. This includes pdf problems, source code files, fonts, and others that might still be problematic.

    Properly show the search excerpts in Dolphin.

Four rather simple points that make a lot of difference for a lot of people. You should get exactly what you want from the fundraiser and nothing is more important for users and application developers alike than stability and performance. A fact that I sometimes loose sight of due to my “researchy” way of working.

Only once this is finished will I get back to creating cool new ways to expose the semantic desktop (in a stable system).

Hoping that this new focus in my work matches the needs of most of you let me spam you again with these two buttons:

============

His original plan

http://trueg.wordpress.com/2011/09/16/nepomuk-what-comes-next/

[menotu]

First Round of Bug Fixes

http://trueg.wordpress.com/2011/09/21/first-round-of-bug-fixes/

As promised I will not stop until I closed all Nepomuk crashes on bugs.kde.org. And I do not mean closing as in “WONTFIX/INVALID”. I mean really close as in “FIXED” or “DUPLICATE”.

This is the first status report on that effort: 106 bugs closed thus far. Sure, 96 of those are duplicates, but nonetheless it is a good start. In addition to that I posted patches for 6 bugs which still need testing.

More to come soon.

[menotu]

I Really Tried To Crash Nepomuk…

http://trueg.wordpress.com/2011/09/26/i-really-tried-to-crash-nepomuk/

… but it would not work. Not on my main system, not even in the VirtualBox which I only set up for that purpose. That is frustrating. There is a set of crash reports in bugs.kde.org for which I attached patches. But I cannot test them. Nepomuk simply does not want to crash for me!

Thus, I need your help. Anyone who manages to crash Nepomuk and is able to test a patch please have a look at those patches, bug  257176   even suggests that it might be prudent to apply more than one at the same time.

Anyway, I am now down to 111 open crash reports for which I did not request additional info yet (this is either: “does it still crash in 4.7.1″ or “please test this patch”). That means a total of 140 crash reports have been closed since 17.09.2011. Most of them are in fact duplicates but that is tedious work also.

[menotu]

The Hunt For Nepomuk Bugs Continues

Let me open with a few stats just to brag:

    Top bug killer on the last commit digest -  http://commit-digest.org/issues/2011-09-25/
    Number of Nepomuk crash reports now below 100
    Overall number of Nepomuk bugs down to 163 (this is actually not much, have a look at the related statistics) -  https://bugs.kde.org/weekly-bug-summary.cgi

    I closed some serious bugs this week (details below)

If you want to track the progress you can use the following links to check from time to time:

    Open Nepomuk crash reports
    Open Nepomuk bug reports excluding crashes
    Open Nepomuk wishes
    Closed Nepomuk bugs since this whole thing started

Finally I want to present two fixes I did this last week just to show what kind of work needs to be done in order to fix problems in Nepomuk:

http://trueg.wordpress.com/2011/10/05/the-hunt-for-nepomuk-bugs-continues/

Note - please see above url for more info and links to items mentioned above........

[GermanTux]

Great news, and concentrating on bugs instead of new features is definitely the way to go. 

[craesz]

Great news, and concentrating on bugs instead of new features is definitely the way to go. 

Absolutely.

[menotu]

Taking a Break From Crash Fixing For Usability

Fixing bugs is actually more fun than I thought. It is rewarding and seeing the bug count go down feels great. But after a few weeks of mostly hunting crashes I needed to do something different for a change.

Thus, I went after the file indexer for optimizations. As discussed in the comments of this very blog downloads and rapidly changing files in general have always been a problem – they are indexed way too often. This is a clear waste of resources. In a discussion the very good idea of introducing a delay after which to re-index a changed file was born. This is what I actually did. However, while doing that I found that it can be even more improved:

In addition to file modification events the file system can tell us when a file is closed after having been opened for writing. Thus, Nepomuk now uses that event instead. For downloads that means they will be indexed only a single time: when they are done.

So now Nepomuk will only re-index files that have actually been modified (modification event) and that have been closed (close after write event). And in addition the re-indexing is delayed for 5 seconds to ensure that we do not re-index rapidly changing files all the time.

All in all this is a great improvement for IO in Nepomuk. Thanks a lot for pointing it out and getting me on the right track. (I even backported this to KDE 4.7.3.)

.........................

http://trueg.wordpress.com/2011/10/13/taking-a-break-from-crash-fixing-for-usability/

[KernelKarter]

Great news, and concentrating on bugs instead of new features is definitely the way to go. 

Absolutely.

I find it extremely refreshing to see a developer concentrating on fixing the basics before hanging on the "bells 'n whistles" 

- Eddie

[menotu]

A Word (or Two) on Removable Storage Media Handling in Nepomuk

While fixing existing Nepomuk bugs and trying to close them as they come in I also look into other things. Last week it was the improved file indexer scheduling and file modification handling. This week it is about another improvement in the handling of queries which involve removable media. Ignacio Serantes already found one bug in the URL encoding before. This time he wanted to search through all mounted removable storage media and realized that he could not. I just fixed that. In order to understand how I did that we need to go into detail about how Nepomuk handles removable media.

Removable Storage Media in Nepomuk

Files on removable storage media are a problem when it comes to meta data stored in Nepomuk. As long as the medium is mounted we can simply identify the files through their local file path. But as soon as it is unmounted the paths are no longer valid. To make things worse we could mount the medium at another mount point the next time or mount another medium (which obviously does not contain the files in question) at the same mount point. So we need a way around that problem. Ever since 4.7 Nepomuk has a rather fancy way of doing that.

Internally Nepomuk uses a stack of Soprano::FilterModels which perform several operations on the data that passes through them. One of these models is the RemovableStorageModel. This model does one thing: it converts the local file URLs of files and folders on removable media into mount-path-independent URLs and vice versa. Currently it supports removable disks like USB keys or external hard disks (any storage that has a UUID), optical media, NFS and Samba mounts. The nice thing about it is that this conversion happens transparently to the client. Thus, a client simply uses the local file URLs according to the current mount path and does not care about anything else. It will always get the correct results.

To understand this better we should look at an example. Imagine we have a USB key inserted with UUID “xyz” which is mounted at /media/disk. Now if we add information about a file /media/disk/myfile.txt to Nepomuk the following happens: The RemovableStorageModel will convert the URL file:///media/disk/myfile.txt into filex://xyz/myfile.txt. This is a custom URL scheme which consists of the device UUID and the relative path. When querying the file the model does the conversion in the other direction. So far so simple.

Queries are where it gets a little more complicated. Imagine we want to query all files in a certain directory on the removable medium (ideally the SPARQL would be hidden by the Nepomuk query API). We would then perform a query like the following simplified one.

(please see link for the example(s) and the rest of the blog)

http://trueg.wordpress.com/2011/10/21/a-word-or-two-on-removable-storage-media-handling-in-nepomuk/

[menotu]

Memory Leaks in Nepomuk – Nah!

http://trueg.wordpress.com/

"OK – this is the fifth time I start the first sentence. So now I will just write like it comes to mind… backspace, backspace, mark/delete…. Oh, damn it. I just pushed a fix to master and 4.7 which fixes the memory leak in the filewatch service. The root of it was the same as in the file indexer service: no event loop in the work thread means DBus events piling up without ever being garbage collected.

Well, that is fixed now and the filewatch service will not steal all your memory anymore.

I have three days left until KDE 4.7.3 and I want to make them count!"

[menotu]

KDE 4.7.3 – The (First) Nepomuk Stability Release

Posted on November 2, 2011 by Sebastian Trüg

http://trueg.wordpress.com/

Now that KDE 4.7.3 has been released let me look back onto the work that I put into it over the last weeks.

    I fixed four actual crashes in 4.7.3. On first glance this might not sound like much but these four crash fixes entail 38 duplicates.

    I finally managed to close the memory leak in the file watcher service.

    I significantly improved the file indexer:

        Exclusion filters are now also correctly taken into account for folders.

        .xsession-errors is now always excluded from indexing.

        Rapidly changing files are only indexed once closed. This results in a lot less IO.

        The previous change also results in torrent downloads being indexed after finished.

        Files that are written over and over (like IRC logs for example) are only re-indexed once every five seconds.

        Nepomuk now always extracts the plain text from PDF files via pdftotext. This is a hack to make sure that we can at least search all PDFs by content. The next step will be to extract meta-data like title and author via poppler. (This is required since the PDF analyzer in libstreamanalyzer/Strigi is not powerful enough yet.)

        Symbolic links now have the correct mime type which means better search results.

        In case the indexer gets stuck (runs forever on one file) it is killed after a period of time.

        Jos van den Oever fixed a bunch of issues in libstreamanalyzer (Strigi) which results in less crashes and less endless PDF indexing. Stay tuned for Strigi 0.7.7.

    With Soprano 2.7.3 Nepomuk will now restart the storage if Virtuoso goes down due to a crash or a third-party kill.

    A running Virtuoso instance which was not shut down due to a crashed or killed Nepomuk will now gracefully be shut down before starting a new instance. This solves some startup issues.

    A small query performance improvement based on a pointless UNION.

    Smit Shah backported his patch which gets rid of the flickering Nepomuk indexer icon in the system tray. It now only becomes active if the indexer has been working for a certain period of time.

All in all 15 bugs are marked with “FIXED-IN: 4.7.3″. This does not include the fixes and improvements I made which did not have matching reports.

Today the next round of Nepomuk stability and performance begins. If all goes well KDE 4.7.4 should be rock-solid when it comes to Nepomuk. Thanks a lot for your continued support. I am still hopeful that I will find a more permanent solution soon:

[menotu]

Update on Bugs And Stuff

Posted on November 18, 2011 by Sebastian Trüg

There was not that much activity last week. That is simply because I took a few days off to spend time with my family on a farm – mostly so that my daughter could ride a pony every day.

Still, there are a few words I can say regarding Nepomuk bugs: 4 crashes have been reported for KDE 4.7.3. One of them I already fixed, one has a patch which is awaiting testing, one is very confusing and made me contact the mighty Dario for help, and the last one is the akonadi feeder which still has a memory leak. Apart from that blogging about my inotify usage had a very nice side-effect: Marcel Wiesweg from Digikam wants to use KInotfy and stumbled upon a serious bug which for some stupid reason I never caught. That bug means that files with umlauts and friends in their paths never keep their meta-data when moved. Urgh!

So KDE 4.7.4 will again contain a bunch of fixes. The hunt is not over yet. At least now all bugs are nicely categorized which makes triaging much easier.

And in other news: thank you so much for the amazing fundraiser. Only about 800€ to the goal which I feared was reaching for the stars. This is given me the energy to go on and try to find the required funding. No one stepped up so far but I am still hopeful as some are still “in discussion”. Keep your fingers crossed (and send ideas my way).

http://trueg.wordpress.com/2011/11/18/update-on-bugs-and-stuff/

[Archie]

I hope that we would be able to implement this on our next KDE upgrade. I am getting crashes (more and more lately) but I cannot report these on the forum (and seek help). I cannot even report it on the ML as it seems that no one else is getting crashes. Or perhaps I need to reinstall ... maybe I'll do that soon.

But this is definitely good news. Thanks for posting, menotu.

[menotu]

Posted on November 25, 2011 by Sebastian Trüg

The Different Places Something Can Go Wrong

This is just a little blog entry about the impact that the ontologies can have on functionality.

The ontologies are a set of vocabularies describing the types of resources stored in Nepomuk, the possible relations between these types, and the possible annotations. We have for example a type for local files, one for an address book entry, one for a person, one for music content and so on. We also have relations that describe that some person is the author or some piece of content and so on.

These ontologies are maintained in the Shared-Desktop-Ontologies project – to my knowledge the only real open-source project developing RDF ontologies.

Now to the actual topic. There once was a bug. Like so many other bugs it talked about file indexing in Nepomuk and like so many other bugs it said that some file could not be indexed. First it was Nepomuk’s fault, then it was the fault of libstreamanalyzer, but in the end I realized: there was a bug in the ontologies. More specificly in NMM – the Nepomuk MultiMedia ontology. (Granted this was not really the source of the hang the bug talks about but it was the reason the file could not be indexed.)

The problem was the domain of the nmm:setSize property. Each property has a domain and a range – the domain defines on which type of resource the property can be set, the range defines the type of the value. In other words they are defining the subject and object type of the triple. The domain is always a resource type (rdfs:Class), the range a resource or a literal type (typically one defined in the XML schema). In this case the domain of nmm:setSIze was set as nmm:MusicPiece whereas it should have been nmm:MusicAlbum. Thus, Nepomuk rejected the data generated by libstreamanalyzer as being invalid due to using an invalid domain.

The solution is shared-desktop-ontologies 0.8.1 with the fixed domain. Installing it will make Nepomuk re-parse the changed ontology and indexing the mp3 files in question will finally work.

Well, this was pretty verbose for a rather small issue. Still it gave a little introduction into how the ontologies are used in Nepomuk. One more thing to take care of in the “Nepomuk universe”.

Link

[menotu]

Posted on Dec 5, 2011 by Sebastian Trüg

Manually Forcing the (Re-)Indexing of Folders is Easy

Ever since the unicode bug in Virtuoso 6.1.3 many of us have broken unicode strings in our Nepomuk databases. Completely re-creating the database is IMHO not an option since that would mean loosing all manual annotations and things like download source URLs. One solution would be restoring a backup but I simply do not trust the Nepomuk backup until I had a deeper look into it. The perfect solution would be if Nepomuk could simply fix the data automatically. While that is of course my goal and I am looking into that it will take a while.

In the meantime I threw together a small desktop file which adds two new actions to the context menu of folders.

1    (Re-)index Folder contents will make the indexer update all the files in the folder indifferent of their state in Nepomuk. This includes fixed unicode strings.

2    (Re-)index Folder contents recursive does the same as the above except that it also recurses into sub folders.

Simply put the following into a file called “nepomuk-index-folder.desktop” and save it in “~/.kde/share/kde4/services/ServiceMenus”. At the next start of Dolphin or Konqueror the two new actions will be available.

Code and blog link