Windows - Unicode Bug in most Addons

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
sphere Offline
Team-Kodi Member
Posts: 1,223
Joined: Jul 2009
Reputation: 50
Location: Germany
Post: #1
Hi,

while reading the forum I found this thread: http://forum.xbmc.org/showthread.php?tid...pid1039311

I did a few tests and can confirm that it is very easy to reproduce and effects most addons - in a not so unrealistic scenario:
- You just need an unicode letter in the xbmc path (which contains the windows-username on windows7). Not sure on the other OS's...

You can easy reproduce this (or test your addons) with starting xbmc in portable mode (add "-p" to you xbmc start shortcut) from a xbmc folder which contains an unicode letter, "ä" or "é" for example.
So just download xbmc again, install it to "C:\folder-with-unicode-ä-é\xbmc" and start "C:\folder-with-unicode-ä-é\xbmc\xbmc.exe -p".
You can also test it with starting xbmc with a local windows username which contans unicode-letters.

I cloned the eden plugins repo and got ~200 line matches without a real unicode decoding:
Code:
getAddonInfo('path')
- all usage of this path will fail without ".decode('utf-8')" added!
This effects all path's returned by getAddonInfo (because it returns string - not unicode) - you ALWAYS have to decode it properly.

In most cases the addons are using this path for icons, something like:
Code:
ICON = os.path.join(ADDON.getAddonInfo('path'), 'icon.png')
This will silently fail because xbmc skips images which aren't accessable - this won't be noticed by most users...

More bad is something like:
Code:
f = open(os.path.join( __settings__.getAddonInfo('profile'), "allmovies"),'wb')
This will fail with a script error: "WindowsError: [Error 3] The system cannot find the path specified."

But it can even avoid some addons from running with something like this:
Code:
sys.path.append(os.path.join(Addon(ADDON_ID).getAddonInfo("path"), "resources", "lib" ))
import module_in_the_above_path


Maybe a better solution would be that xbmc would return unicode-instances (and not str)...

Until that we have to do something like:
Code:
Addon.getAddonInfo('path').decode('utf-8')

Don't get me wrong - my Addons are also affected...

regards,
sphere

My GitHub. My Add-ons:
[Image: IOoywq0.jpg]
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,221
Joined: Oct 2003
Reputation: 178
Post: #2
I don't believe it affects anything other than windows at this point, though that may possibly be due to most *nix users not having special-characters in their usernames.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
Martijn Offline
Team Kodi
Posts: 11,614
Joined: Jul 2011
Reputation: 167
Location: Dawn of time
Post: #3
Hmm haven't had any issues with Artwork Downloader concerning this. Did have some issues with show text in the dialog that threw that error and the JSON requests.

But nothing in the cases you described


EDIT:

Ok i see your point. But this only happens in portable mode?

One add-on even made from this: XBMC_Edén
This: XBMC_EdÃn


Seems lilke my add-on doesn't even care about this without any code intervention Smile

Always read the XBMC online-manual, FAQ and search the forums before posting.
Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting, make sure you read this first

For your mediacenter artwork go to
[Image: fanarttv.png]
(This post was last modified: 2012-03-14 00:32 by Martijn.)
find quote
sphere Offline
Team-Kodi Member
Posts: 1,223
Joined: Jul 2009
Reputation: 50
Location: Germany
Post: #4
(2012-03-14 00:26)Martijn Wrote:  Ok i see your point. But this only happens in portable mode?

One add-on even made from this: XBMC_Edén
This: XBMC_EdÃn


Seems lilke my add-on doesn't even care about this without any code intervention Smile

Sorry to say this - but you are wrong.

1. No, this is not only happening in portable mode. This happens at least in one of these scenarios:
a: You are on windows in portable mode with having xbmc installed in any path which contains a unicode letter
b: You are on windows in "normal" mode - but your windows username contains any unicode username (think about "Renè, André, ...")

2. Of course is your addon "Artwork downloader" affected! It is even crashing in some situations because of this bug:
Code:
07:44:19 T:3044   ERROR: Error Type: <type 'exceptions.WindowsError'>
07:44:19 T:3044   ERROR: Error Contents: (3, 'Das System kann den angegebenen Pfad nicht finden', 'D:\\Progs\\XBMC-Eden\xc3\xa4\\portable_data\\userdata\\addon_data\\script.artwork.downloader\\te​mp/*.*')
07:44:19 T:3044   ERROR: Traceback (most recent call last):
                                              File "D:\Progs\XBMC-Eden坰ortable_data\addons\script.artwork.downloader\default.py", line 840, in <module>
                                                Main()
                                              File "D:\Progs\XBMC-Eden坰ortable_data\addons\script.artwork.downloader\default.py", line 140, in __init__
                                                self.cleanup()
                                              File "D:\Progs\XBMC-Eden坰ortable_data\addons\script.artwork.downloader\default.py", line 218, in cleanup
                                                for x in os.listdir(self.fileops.tempdir):
                                            WindowsError: (3, 'Das System kann den angegebenen Pfad nicht finden', 'D:\\Progs\\XBMC-Eden\xc3\xa4\\portable_data\\userdata\\addon_data\\script.artwork.downloader\\te​mp/*.*')

regards,
sphere

My GitHub. My Add-ons:
[Image: IOoywq0.jpg]
find quote
Martijn Offline
Team Kodi
Posts: 11,614
Joined: Jul 2011
Reputation: 167
Location: Dawn of time
Post: #5
Well at least I had no one complaining so far Wink


I had to do this:
PHP Code:
__addonpath__   __addon__.getAddonInfo('path')
__addondir__    xbmc.translatePath__addon__.getAddonInfo('profile') ).decode('utf-8'

for it not to fail.

What about this one?
PHP Code:
THUMBS_CACHE_PATH xbmc.translatePath"special://profile/Thumbnails/Video" 

Should have the same problem I think.

Would be nice if this could be fixed in XBMC core instead of every addon is gonna need to fix this

Always read the XBMC online-manual, FAQ and search the forums before posting.
Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting, make sure you read this first

For your mediacenter artwork go to
[Image: fanarttv.png]
(This post was last modified: 2012-03-14 09:25 by Martijn.)
find quote
amet Offline
Retired Team-Kodi Member
Posts: 4,090
Joined: Jul 2009
Reputation: 41
Location: Novi Sad / Dubai
Post: #6
this is what I did in XBMC Subtitles(thx to chadoe), but it brought in more issues ... that chadoe fixed again Smile

testing it out now on OSX to see if it breaks anything Smile

EDIT: all good on OSX with above changes
(This post was last modified: 2012-03-14 09:42 by amet.)
find quote
Popeye Offline
Posting Freak
Posts: 922
Joined: Aug 2009
Reputation: 26
Location: Sweden
Post: #7
What I have found my addons is that xbmc always returns utf-8 from e.g. keyboard and dialogs. By always converting to unicode for internal code use and then convert back to utf-8 for output problems are solved...
If i were to use
PHP Code:
ICON os.path.join(ADDON.getAddonInfo('path'), 'icon.png'
I would do
PHP Code:
ICON os.path.join(unicode(ADDON.getAddonInfo('path'), 'utf-8'), 'icon.png'
ICON will then be e.g. u'/some/pathto/\xD6rn/icon.png'

Also urllib.qoute and urllib.quote_plus is tricky since they don't handle uincode.

sverigesradio | Pneumatic | SABnzbd | XBMC that just works - openelec
(This post was last modified: 2012-03-14 14:13 by Popeye.)
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,221
Joined: Oct 2003
Reputation: 178
Post: #8
Correct. ALL strings that XBMC gives python are currently utf8.

I suspect XBMC can make it much easier on plugins by returning unicode strings instead, but this is obviously something that will break API (i.e. post Eden).

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
Martijn Offline
Team Kodi
Posts: 11,614
Joined: Jul 2011
Reputation: 167
Location: Dawn of time
Post: #9
trac it so won't be forgotten and there can be some thought about this?

Always read the XBMC online-manual, FAQ and search the forums before posting.
Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting, make sure you read this first

For your mediacenter artwork go to
[Image: fanarttv.png]
find quote
Popeye Offline
Posting Freak
Posts: 922
Joined: Aug 2009
Reputation: 26
Location: Sweden
Post: #10
(2012-03-14 22:40)jmarshall Wrote:  Correct. ALL strings that XBMC gives python are currently utf8.

I suspect XBMC can make it much easier on plugins by returning unicode strings instead, but this is obviously something that will break API (i.e. post Eden).

An idea, why not a xbmc return encoding tag in the addon settings.xml defaulting to utf-8 with unicode as the option?

sverigesradio | Pneumatic | SABnzbd | XBMC that just works - openelec
(This post was last modified: 2012-03-14 23:37 by Popeye.)
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,221
Joined: Oct 2003
Reputation: 178
Post: #11
It doesn't only affect addon settings though (and an add-on needn't have any settings) - basically every string we pass to python is utf8.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
sphere Offline
Team-Kodi Member
Posts: 1,223
Joined: Jul 2009
Reputation: 50
Location: Germany
Post: #12
So what is the "official recommended way"?

- Should we (addon-authors) fix this? (80% of all addons are at least affected by not working picture-path-joins)
- Ignore and add a FAQ entry to refer to "do not use windows user-names with special characters on win7 or use portable-mode instead"?
- Wait for a core fix? (I guess only for post eden)

regards,
sphere

My GitHub. My Add-ons:
[Image: IOoywq0.jpg]
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,221
Joined: Oct 2003
Reputation: 178
Post: #13
Core fix is only post-eden, yes. Remember that it only affects paths per-se, so if you're writing to or reading from the filesystem then you'll want those paths handled. Depending on your add-on this may be minimal.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
sphere Offline
Team-Kodi Member
Posts: 1,223
Joined: Jul 2009
Reputation: 50
Location: Germany
Post: #14
Sure, for my 6-8 add-ons it is easy and I know what to do - I was just thinking about the other add-ons. I guess most devs still don't know about this.

thanks

My GitHub. My Add-ons:
[Image: IOoywq0.jpg]
find quote