Kodi Community Forum
Win Unicode Bug in most Addons - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=26)
+--- Thread: Win Unicode Bug in most Addons (/showthread.php?tid=125492)



Unicode Bug in most Addons - sphere - 2012-03-14

Hi,

while reading the forum I found this thread: http://forum.xbmc.org/showthread.php?tid=124921&pid=1039311#pid1039311

I did a few tests and can confirm that it is very easy to reproduce and effects most addons - in a not so unrealistic scenario:
- You just need an unicode letter in the xbmc path (which contains the windows-username on windows7). Not sure on the other OS's...

You can easy reproduce this (or test your addons) with starting xbmc in portable mode (add "-p" to you xbmc start shortcut) from a xbmc folder which contains an unicode letter, "ä" or "é" for example.
So just download xbmc again, install it to "C:\folder-with-unicode-ä-é\xbmc" and start "C:\folder-with-unicode-ä-é\xbmc\xbmc.exe -p".
You can also test it with starting xbmc with a local windows username which contans unicode-letters.

I cloned the eden plugins repo and got ~200 line matches without a real unicode decoding:
Code:
getAddonInfo('path')
- all usage of this path will fail without ".decode('utf-8')" added!
This effects all path's returned by getAddonInfo (because it returns string - not unicode) - you ALWAYS have to decode it properly.

In most cases the addons are using this path for icons, something like:
Code:
ICON = os.path.join(ADDON.getAddonInfo('path'), 'icon.png')
This will silently fail because xbmc skips images which aren't accessable - this won't be noticed by most users...

More bad is something like:
Code:
f = open(os.path.join( __settings__.getAddonInfo('profile'), "allmovies"),'wb')
This will fail with a script error: "WindowsError: [Error 3] The system cannot find the path specified."

But it can even avoid some addons from running with something like this:
Code:
sys.path.append(os.path.join(Addon(ADDON_ID).getAddonInfo("path"), "resources", "lib" ))
import module_in_the_above_path


Maybe a better solution would be that xbmc would return unicode-instances (and not str)...

Until that we have to do something like:
Code:
Addon.getAddonInfo('path').decode('utf-8')

Don't get me wrong - my Addons are also affected...

regards,
sphere


RE: Unicode Bug in most Addons - jmarshall - 2012-03-14

I don't believe it affects anything other than windows at this point, though that may possibly be due to most *nix users not having special-characters in their usernames.


RE: Unicode Bug in most Addons - Martijn - 2012-03-14

Hmm haven't had any issues with Artwork Downloader concerning this. Did have some issues with show text in the dialog that threw that error and the JSON requests.

But nothing in the cases you described


EDIT:

Ok i see your point. But this only happens in portable mode?

One add-on even made from this: XBMC_Edén
This: XBMC_Edén


Seems lilke my add-on doesn't even care about this without any code intervention Smile


RE: Unicode Bug in most Addons - sphere - 2012-03-14

(2012-03-14, 00:26)Martijn Wrote: Ok i see your point. But this only happens in portable mode?

One add-on even made from this: XBMC_Edén
This: XBMC_Edén


Seems lilke my add-on doesn't even care about this without any code intervention Smile

Sorry to say this - but you are wrong.

1. No, this is not only happening in portable mode. This happens at least in one of these scenarios:
a: You are on windows in portable mode with having xbmc installed in any path which contains a unicode letter
b: You are on windows in "normal" mode - but your windows username contains any unicode username (think about "Renè, André, ...")

2. Of course is your addon "Artwork downloader" affected! It is even crashing in some situations because of this bug:
Code:
07:44:19 T:3044   ERROR: Error Type: <type 'exceptions.WindowsError'>
07:44:19 T:3044   ERROR: Error Contents: (3, 'Das System kann den angegebenen Pfad nicht finden', 'D:\\Progs\\XBMC-Eden\xc3\xa4\\portable_data\\userdata\\addon_data\\script.artwork.downloader\\temp/*.*')
07:44:19 T:3044   ERROR: Traceback (most recent call last):
                                              File "D:\Progs\XBMC-Eden坰ortable_data\addons\script.artwork.downloader\default.py", line 840, in <module>
                                                Main()
                                              File "D:\Progs\XBMC-Eden坰ortable_data\addons\script.artwork.downloader\default.py", line 140, in __init__
                                                self.cleanup()
                                              File "D:\Progs\XBMC-Eden坰ortable_data\addons\script.artwork.downloader\default.py", line 218, in cleanup
                                                for x in os.listdir(self.fileops.tempdir):
                                            WindowsError: (3, 'Das System kann den angegebenen Pfad nicht finden', 'D:\\Progs\\XBMC-Eden\xc3\xa4\\portable_data\\userdata\\addon_data\\script.artwork.downloader\\temp/*.*')

regards,
sphere


RE: Unicode Bug in most Addons - Martijn - 2012-03-14

Well at least I had no one complaining so far Wink


I had to do this:
PHP Code:
__addonpath__   __addon__.getAddonInfo('path')
__addondir__    xbmc.translatePath__addon__.getAddonInfo('profile') ).decode('utf-8'

for it not to fail.

What about this one?
PHP Code:
THUMBS_CACHE_PATH xbmc.translatePath"special://profile/Thumbnails/Video" 

Should have the same problem I think.

Would be nice if this could be fixed in XBMC core instead of every addon is gonna need to fix this


RE: Unicode Bug in most Addons - amet - 2012-03-14

this is what I did in XBMC Subtitles(thx to chadoe), but it brought in more issues ... that chadoe fixed again Smile

testing it out now on OSX to see if it breaks anything Smile

EDIT: all good on OSX with above changes


RE: Unicode Bug in most Addons - Popeye - 2012-03-14

What I have found my addons is that xbmc always returns utf-8 from e.g. keyboard and dialogs. By always converting to unicode for internal code use and then convert back to utf-8 for output problems are solved...
If i were to use
PHP Code:
ICON os.path.join(ADDON.getAddonInfo('path'), 'icon.png'
I would do
PHP Code:
ICON os.path.join(unicode(ADDON.getAddonInfo('path'), 'utf-8'), 'icon.png'
ICON will then be e.g. u'/some/pathto/\xD6rn/icon.png'

Also urllib.qoute and urllib.quote_plus is tricky since they don't handle uincode.


RE: Unicode Bug in most Addons - jmarshall - 2012-03-14

Correct. ALL strings that XBMC gives python are currently utf8.

I suspect XBMC can make it much easier on plugins by returning unicode strings instead, but this is obviously something that will break API (i.e. post Eden).


RE: Unicode Bug in most Addons - Martijn - 2012-03-14

trac it so won't be forgotten and there can be some thought about this?


RE: Unicode Bug in most Addons - Popeye - 2012-03-14

(2012-03-14, 22:40)jmarshall Wrote: Correct. ALL strings that XBMC gives python are currently utf8.

I suspect XBMC can make it much easier on plugins by returning unicode strings instead, but this is obviously something that will break API (i.e. post Eden).

An idea, why not a xbmc return encoding tag in the addon settings.xml defaulting to utf-8 with unicode as the option?



RE: Unicode Bug in most Addons - jmarshall - 2012-03-15

It doesn't only affect addon settings though (and an add-on needn't have any settings) - basically every string we pass to python is utf8.


RE: Unicode Bug in most Addons - sphere - 2012-03-16

So what is the "official recommended way"?

- Should we (addon-authors) fix this? (80% of all addons are at least affected by not working picture-path-joins)
- Ignore and add a FAQ entry to refer to "do not use windows user-names with special characters on win7 or use portable-mode instead"?
- Wait for a core fix? (I guess only for post eden)

regards,
sphere


RE: Unicode Bug in most Addons - jmarshall - 2012-03-16

Core fix is only post-eden, yes. Remember that it only affects paths per-se, so if you're writing to or reading from the filesystem then you'll want those paths handled. Depending on your add-on this may be minimal.


RE: Unicode Bug in most Addons - sphere - 2012-03-17

Sure, for my 6-8 add-ons it is easy and I know what to do - I was just thinking about the other add-ons. I guess most devs still don't know about this.

thanks