Snippets code from my daily experience

January 17, 2009

Rumbling about VisualDiffer

Filed under: komodo,xul — dafi @ 12:34 pm

After many months I’ve started again to work on VisualDiffer the most complex extension I ever realized.

Developing VisualDiffer is an exciting task at least for two reasons

  • Deeper level XUL usage to create complex user interfaces
  • Attention to code performances to make application more reactive

At this time the UI doesn’t present any special widget but I’ve some idea about line position sliders and comparison file results, they will be very fun to implement.

The implementation has a bottleneck in the Javascript sort function.

While VisualDiffer only runs on Komodo I’ve tried to run the directories sort routines on Firefox with TraceMonkey enabled and I noticed some speedup, nothing so amazing but with deep file trees  (~600 directories and ~2000 files) TraceMonkey completes in 10-20% less time (very good!!).

Due to the fact Komodo is based on Python, it should be easier to write core sort routines directly in Python, I will do it very soon.

VisualDiffer features list (in my mind)  is very long but the already implemented is a very short subset.

I think it is mandatory to adopt the Release Early, Release Often strategy otherwise I’ll release the next version in 2030.

A couple of new functionalities are complete

  • 2369618 quick comparing two open files selecting them from View Tab context menu (released)
  • 2515481 set two different base folders from Directory Panel (not released)

At this time I’ve abandoned the idea to add the three-file compare view I want to add many little time saver features like

  • copy/move folders from Folder sides
  • rename folders/files
  • reload changed files
  • syntax highlight (this should be easy within Komodo)
  • in place edit (no need to switch between Komodo and VisualDiffer)
About these ads

5 Comments

  1. I doubt switching to Python will give you any significant edge over Tracemonkey, and 2600 entries are not that many so if the performance sucks there should be definitely room for optimization.
    Where’s the sorting code, exactly?

    Comment by Giorgio Maone — January 17, 2009 @ 3:34 pm

  2. When all element present in same directory are read the sort function is called.
    If inside loop a directory element is found then it is scanned recursively (a very simple algorithm)
    The sort function uses a callback sorter but removing it the sort gains only few milliseconds, so the callback isn’t the guilty :P

    The code is present at

    http://dafizilla.svn.sourceforge.net/viewvc/dafizilla/komodo/visualdiffer/src/main/content/visualdiffer/commonDiff.js?revision=52&view=markup

    the readDirectory method contains the logic

    Comment by dafi — January 17, 2009 @ 3:46 pm

  3. Hrm, I run some micro-benchmarks on that code and I found something different than yours.
    I used a 16,000 files sample (with some directory recursion), and I found the preliminary file scanning (*before* sorting) accounts for about 50% of the processing time when the directory is already cached, and several times more on a cold cache.
    Also, the callback is all but neutral.
    On the same test, just eliminating the callback (by returning a counter which, on average, causes a sorting swap anyway and therefore shouldn’t neutralize the algorithm impact) reduced the processing time almost by 50%, therefore the built-in Array.sort() algorithm is *not* the culprit, but the comparator callback is almost as much heavy as the file scanning (non-cached I/O aside).

    I managed to shave more than 20% processing time from the (cached) file scanning, by caching nsIFile, using Array helpers and pre-branching conditionals:

    readDirectory: function(dir, recursive, level, parent, fileFilter) {
    var arr = [];

    var entries = dir.directoryEntries;

    const nsIFile = Components.interfaces.nsIFile;
    var entry;
    while (entries.hasMoreElements()) if ((entry = entries.getNext()) instanceof nsIFile) arr.push(entry);

    if (fileFilter) arr = arr.filter(filter.includes);
    arr = arr.map(recursive
    ? function(e) {
    var folderStatus = new FolderStatus(e, [], level, parent);
    if(e.isDirectory())
    folderStatus.subfolders = DiffCommon.readQuick(
    e, recursive, level + 1, folderStatus, fileFilter);
    return folderStatus;
    }
    : function(e) { return new FolderStatus(e, [], level, parent); }
    );

    arr.sort(DiffCommon.dirSorter);
    return arr;
    }

    The callback is yours to optimize :)

    Comment by Giorgio Maone — January 17, 2009 @ 5:14 pm

    • Thanks for your quick tests ;) I’ll repeat them ;)

      Comment by dafi — January 17, 2009 @ 5:22 pm

  4. oops, readQuick() should be readDirectory(), of course (copy and paste from my test code).

    Comment by Giorgio Maone — January 17, 2009 @ 5:57 pm


RSS feed for comments on this post.

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: