Snippets code from my daily experience

October 8, 2008

nsIProcess, Windows and Unicode

ViewSourceWith (VSW for friends) existence was impossible without nsIProcess, VSW must run programs and pass them at least one file name.

When arguments passed to nsIProcess are ASCII strings all works like a charm but you enter the hell if strings contain Unicode characters like those contained in Cyrillic or Japanese alphabets.

nsIProcess, especially under Microsoft Windows, doesn’t work very well with UTF-8 strings (see bugs 229379, 408923, 411511).

On VSW forum an user asked to me to fix the problem but it isn’t strictly related to VSW because the problem affects the XPCOM implementation.

After many attempts I surrendered and I decided to write my Win32 (Win64??) XPCOM component.

My IWinProcess has the same nsIProcess’s method signatures (Init, Run) but accepts wstrings and calls the unicode CreateProcessW Win32 API instead of its sibling CreateProcessA.

The Javascript caller (eg VSW) must pass UTF-8 strings but this is the only constraint.
Now finally I can open local files containing Unicode characters without tweaking the Windows ‘Regional Options’.

Linux seems to work fine (at least on my Gutsy box) simply converting Unicode strings with nsIScriptableUnicodeConverter.ConvertFromUnicode, no other special workarounds.

I don’t know if MacOSX works like Linux, I’m unable to test under Apple machines.

I don’t like code containing platform checks (if Windows … else if Linux … ) but to simplify VSW development the ‘if’ statement sounds reasonable.

The IWinProcess source and DLL are present on dafizilla SVN repository.

The next VSW release (0.4) will contain the IWinProcess component

21 Comments

  1. Is there a way to combine your efforst with the nsIProcess work being done?

    See: http://jamesboston.ca/cms/node/75

    Comment by Mark Finkle — October 8, 2008 @ 8:57 pm

  2. Nice work! I’ll be looking for an update to VSW. I’m unsure if this would have any side benefits in Komodo though because I believe we use Python for process creation under the hood. We *have* had our share of Unicode shenanigans in the past though.

    Comment by JeffG — October 9, 2008 @ 12:52 am

  3. @Mark Finkle Sure, I can try to find a way to combine all work consider I want to check if my Linux workaround fits on OSX, too

    Comment by dafi — October 9, 2008 @ 6:49 am

  4. @JeffG In the past I’ve “sniffed”πŸ˜‰ Komodo code but it is too much Python oriented to be used seamlessy. I wrote an XPCOM using the python wrapper but the extension file size grown too much so I decided to call Windows natively

    Comment by dafi — October 9, 2008 @ 6:52 am

  5. > The Javascript caller (eg VSW) must pass UTF-8 strings but this is the only constraint.

    This seems iffy to me. It seems like since you want a 16-bit filename on Windows, and nsIFile actually contains the information you want in the exact form you want, you should call GetPath/GetTarget on Windows rather than GetNativePath/GetNativeTarget.

    The type of mTargetPath could be nsCOMPtr, to save an #ifdef.

    Comment by Jason — October 9, 2008 @ 6:42 pm

  6. @Jason you are right but I want to pass 16-bit strings not only 16-bit file names (eg a specific command line argument in UTF-8 format)

    Comment by dafi — October 9, 2008 @ 6:45 pm

  7. I wrote a similar work around for this, but Windows only. I had to use
    char* file = ToNewUTF8String(filename); // (from nsStringAPI.h)
    before passing the file to CreateProcess.

    As Mark Finkle noted above, I’m working on a project to fix a bug in nsIProcess and also implement inter-process communication.
    Project page:
    http://zenit.senecac.on.ca/wiki/index.php/User:Jamesboston/nsIProcess
    Bug:
    https://bugzilla.mozilla.org/show_bug.cgi?id=442393

    I expect that the patch I’ve got at present will cause all kinds of regressions because I’m taking a new approach to starting processes that relies entirely on the Netscape Portable Runtime.

    Naturally, I don’t expect the patch to be accepted until I fix any regressions. I’m working on creating an xpcshell unit test. I’ll definitely add tests for international character sets. Would I be right in saying that if I can start a process with Japanese or Cyrillic characters in the filename then you are good to go?

    Comment by James Boston — October 10, 2008 @ 6:17 am

  8. @James If I can help you I’m hereπŸ˜‰ Consider I can drop my implementation and use yours.
    iWinProcess is Windows specific, too

    Comment by dafi — October 10, 2008 @ 2:38 pm

  9. I’ve downloaded your code. It’s very helpful. In my work around I only converted my args to UTF8. This is good enough for names that use only ASCII character. But I see that you need UTF16 to handle an international character set.

    I think that if I want to use the approach of only using the NSPR, I will need to merge your method into the mozilla code around here somewhere:
    http://mxr.mozilla.org/mozilla-central/source/nsprpub/pr/src/md/windows/ntmisc.c#299

    Comment by James Boston — October 10, 2008 @ 10:31 pm

  10. […] 0.4, this version mainly contains bug fixes but it is the first version using the IWinProcess the nsIProcess component replacement. I hope people using non latin charset can appreciate this little […]

    Pingback by Four years of ViewSourceWith: Version 0.4 released « Snippets code from my daily experience — November 8, 2008 @ 11:49 am

  11. I am a Firefox extension developer who suffers from the bad Unicode support of nIProcess. May I use your IWinProcess work in my GPL license extension NicoFox and report any problem if encountered?

    Comment by Littlebtc — November 9, 2008 @ 1:05 am

  12. @Littlebtc you feel free to use IWinProcess, actually works only in conjunction with ViewSourceWith so I think some problem can be present.
    Report problems you encounter and I will try to fix them

    Comment by dafi — November 9, 2008 @ 9:00 am

  13. @dafi
    Thank you, IWinProcess is really what I want, and after test, it works for me and solve the most serious problem in my extension!

    But I have found a bug: IWinProcess is unable to execute application with Unicode path. For example, D:\something.exe works but D:\γ‚γ„γ†γˆγŠ\something.exe doesn’t.

    I’m developing program in Traditional Chinese Windows, but application with only Traditional Chinese path still cannot be opened by IWinProcess.

    Comment by Littlebtc — November 14, 2008 @ 9:45 am

  14. @Littlebtc This bug isn’t strictly related to IWinProcess but to nsIFile. The application to run is passed to IWinProcess.init() in same manner of nsIProcess.init().

    I can try to fix it but for me it has a low priority

    Comment by dafi — November 14, 2008 @ 9:52 am

  15. @dafi Strangely, for me, with the same nsIFile with Unicode path, nsIProcess can executed the file sucessfully (though parameter has not Unicode support) but not IWinProcess. Why?

    Comment by Littlebtc — November 14, 2008 @ 10:02 am

  16. @Littlebtc Uh, very very strange I will take a look very soon, stay tunedπŸ˜‰

    Comment by dafi — November 14, 2008 @ 10:04 am

  17. @Littlebtc I’ve tried nsIFile and nsIProcess on my WinXP but they don’t work with path containing UTF-8 characters, maybe because I use an italian WinXP.

    I’ve patched IWinProcess to handle also UTF-8 executables, please may you help me to test it?

    The link to new DLL is shown below

    http://dafizilla.sourceforge.net/viewsourcewith/beta/winprocess.dll

    Comment by dafi — November 14, 2008 @ 1:44 pm

  18. @dafi After this patch, UTF-8 executable problem is fixed, but the Unicode support of parameters seems to be broken.

    Comment by Littlebtc — November 15, 2008 @ 4:45 am

  19. Wait, the patch works! I unknowingly changed back the code to re-use nsIProcess, so I have done a wrong test so the result is wrong!πŸ˜€

    Comment by Littlebtc — November 15, 2008 @ 4:58 am

  20. @Littlebtc Your first post killed meπŸ˜‰ Great! Do you think I can commit modifications to SVN?

    Comment by dafi — November 15, 2008 @ 8:21 am

  21. @dafi I think so, it is a helpful patch for Non-latin language computer users

    Comment by Littlebtc — November 20, 2008 @ 5:44 am


RSS feed for comments on this post.

Blog at WordPress.com.

%d bloggers like this: