Windows Unicode Bug in most Addons

sphere · 2012-03-14 00:13

Hi,

while reading the forum I found this thread: http://forum.xbmc.org/showthread.php?tid...pid1039311

I did a few tests and can confirm that it is very easy to reproduce and effects most addons - in a not so unrealistic scenario:
- You just need an unicode letter in the xbmc path (which contains the windows-username on windows7). Not sure on the other OS's...

You can easy reproduce this (or test your addons) with starting xbmc in portable mode (add "-p" to you xbmc start shortcut) from a xbmc folder which contains an unicode letter, "ä" or "é" for example.
So just download xbmc again, install it to "C:\folder-with-unicode-ä-é\xbmc" and start "C:\folder-with-unicode-ä-é\xbmc\xbmc.exe -p".
You can also test it with starting xbmc with a local windows username which contans unicode-letters.

I cloned the eden plugins repo and got ~200 line matches without a real unicode decoding:

Code:

getAddonInfo('path')

- all usage of this path will fail without ".decode('utf-8')" added!
This effects all path's returned by getAddonInfo (because it returns string - not unicode) - you ALWAYS have to decode it properly.

In most cases the addons are using this path for icons, something like:

Code:

ICON = os.path.join(ADDON.getAddonInfo('path'), 'icon.png')

This will silently fail because xbmc skips images which aren't accessable - this won't be noticed by most users...

More bad is something like:

Code:

f = open(os.path.join( __settings__.getAddonInfo('profile'), "allmovies"),'wb')

This will fail with a script error: "WindowsError: [Error 3] The system cannot find the path specified."

But it can even avoid some addons from running with something like this:

Code:

sys.path.append(os.path.join(Addon(ADDON_ID).getAddonInfo("path"), "resources", "lib" ))

import module_in_the_above_path

Maybe a better solution would be that xbmc would return unicode-instances (and not str)...

Until that we have to do something like:

Code:

Addon.getAddonInfo('path').decode('utf-8')

Don't get me wrong - my Addons are also affected...

regards,
sphere

**jmarshall** · 2012-03-14 00:25

I don't believe it affects anything other than windows at this point, though that may possibly be due to most *nix users not having special-characters in their usernames.

**Martijn** · 2012-03-14 00:26

Hmm haven't had any issues with Artwork Downloader concerning this. Did have some issues with show text in the dialog that threw that error and the JSON requests.

But nothing in the cases you described

EDIT:

Ok i see your point. But this only happens in portable mode?

One add-on even made from this: XBMC_Edén
This: XBMC_EdÃ©n

Seems lilke my add-on doesn't even care about this without any code intervention Smile

sphere · 2012-03-14 08:50

(2012-03-14 00:26)Martijn Wrote: Ok i see your point. But this only happens in portable mode?

One add-on even made from this: XBMC_Edén
This: XBMC_EdÃ©n

Seems lilke my add-on doesn't even care about this without any code intervention

Sorry to say this - but you are wrong.

1. No, this is not only happening in portable mode. This happens at least in one of these scenarios:
a: You are on windows in portable mode with having xbmc installed in any path which contains a unicode letter
b: You are on windows in "normal" mode - but your windows username contains any unicode username (think about "Renè, André, ...")

2. Of course is your addon "Artwork downloader" affected! It is even crashing in some situations because of this bug:

Code:

07:44:19 T:3044   ERROR: Error Type: <type 'exceptions.WindowsError'>

07:44:19 T:3044   ERROR: Error Contents: (3, 'Das System kann den angegebenen Pfad nicht finden', 'D:\\Progs\\XBMC-Eden\xc3\xa4\\portable_data\\userdata\\addon_data\\script.artwork.downloader\\te​mp/*.*')

07:44:19 T:3044   ERROR: Traceback (most recent call last):

                                              File "D:\Progs\XBMC-Eden坰ortable_data\addons\script.artwork.downloader\default.py", line 840, in <module>

                                                Main()

                                              File "D:\Progs\XBMC-Eden坰ortable_data\addons\script.artwork.downloader\default.py", line 140, in __init__

                                                self.cleanup()

                                              File "D:\Progs\XBMC-Eden坰ortable_data\addons\script.artwork.downloader\default.py", line 218, in cleanup

                                                for x in os.listdir(self.fileops.tempdir):

                                            WindowsError: (3, 'Das System kann den angegebenen Pfad nicht finden', 'D:\\Progs\\XBMC-Eden\xc3\xa4\\portable_data\\userdata\\addon_data\\script.artwork.downloader\\te​mp/*.*')

regards,
sphere

**Martijn** · 2012-03-14 09:24

Well at least I had no one complaining so far Wink

I had to do this:

PHP Code:

__addonpath__   = __addon__.getAddonInfo('path')
__addondir__    = xbmc.translatePath( __addon__.getAddonInfo('profile') ).decode('utf-8') 

for it not to fail.

What about this one?

PHP Code:

THUMBS_CACHE_PATH = xbmc.translatePath( "special://profile/Thumbnails/Video" ) 

Should have the same problem I think.

Would be nice if this could be fixed in XBMC core instead of every addon is gonna need to fix this

**amet** · 2012-03-14 09:27

this is what I did in XBMC Subtitles(thx to chadoe), but it brought in more issues ... that chadoe fixed again Smile

testing it out now on OSX to see if it breaks anything Smile

EDIT: all good on OSX with above changes

Popeye · 2012-03-14 14:12

What I have found my addons is that xbmc always returns utf-8 from e.g. keyboard and dialogs. By always converting to unicode for internal code use and then convert back to utf-8 for output problems are solved...
If i were to use

PHP Code:

ICON = os.path.join(ADDON.getAddonInfo('path'), 'icon.png') 

I would do

PHP Code:

ICON = os.path.join(unicode(ADDON.getAddonInfo('path'), 'utf-8'), 'icon.png') 

ICON will then be e.g. u'/some/pathto/\xD6rn/icon.png'

Also urllib.qoute and urllib.quote_plus is tricky since they don't handle uincode.

**jmarshall** · 2012-03-14 22:40

Correct. ALL strings that XBMC gives python are currently utf8.

I suspect XBMC can make it much easier on plugins by returning unicode strings instead, but this is obviously something that will break API (i.e. post Eden).

**Martijn** · 2012-03-14 23:04

trac it so won't be forgotten and there can be some thought about this?

Popeye · 2012-03-14 23:36

(2012-03-14 22:40)jmarshall Wrote: Correct. ALL strings that XBMC gives python are currently utf8.

I suspect XBMC can make it much easier on plugins by returning unicode strings instead, but this is obviously something that will break API (i.e. post Eden).

An idea, why not a xbmc return encoding tag in the addon settings.xml defaulting to utf-8 with unicode as the option?

Martijn Team-XBMC Posts: 7,715 Joined: Jul 2011 Reputation: 115 Location: Dawn of time	2012-03-14 23:04 Post: #9 trac it so won't be forgotten and there can be some thought about this? Always read the XBMC online-manual, FAQ and search the forums before posting. Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules. For troubleshooting and bug reporting, make sure you read this first For your mediacenter artwork go to
	find quote