[WIP] Addon system refactor for multiple programming languages - Overview/Status
#31
Nice work - will take a look when I have some spare time (unfortunately I don't have a lot of that lately!)
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#32
Very cool - ditto on the time but I'm curious enough about how SWIG works with callbacks to take a look when I get a chance.

What's preventing making the link problem go away by modifying the build system? Just time to determine how to integrate the new lib, or some deeper problem?
Reply
#33
dbrobins Wrote:Very cool - ditto on the time but I'm curious enough about how SWIG works with callbacks to take a look when I get a chance.

SWIG calls it a "director" which equates to a two way proxy to shuffle calls between the navtive system and the scripting language. Look in the documentation under "Cross language polymorphism" (Section 21.5 in the 2.0 docs).

dbrobins Wrote:What's preventing making the link problem go away by modifying the build system? Just time to determine how to integrate the new lib, or some deeper problem?

I'm sure changing the build system wont be an issue but someone will need to determine the general rules for shared library linking including dependency requirements, etc. I'm not in a position to make those decisions so I just hacked my build for now.
Reply
#34
Just an FYI: The SWIG team took my patch:

http://sourceforge.net/tracker/?func=det...tid=101645

When version 2.0.1 comes out it will be a requirement to run the stuff I've been working on.
Reply
#35
I am hoping at some point to separate scrapers completely into their own library, with limited and well-defined interfaces between them and the rest of XBMC, possibly as a shared library. This will make it easier to do testing, including unit tests, (out of proc) background scraping, and maybe even have XBMC scrapers be used by other applications.

I would also like to support scrapers written in various languages - primarily Python, but also whatever else the addon system supports.

So, my question is, how general will the new addon system be, and how tied to the rest of XBMC? Do you have any plans for separation, i.e., making it into a library with loose ties to the rest of XBMC? Scrapers will need some way to call some external functions, but they won't need, for example, any GUI functionality, so ideally I could load the addon DLL, pass it a set of interfaces that loaded addons can call (bundled/pre-processed in the same way as you currently do for xbmc and xbmcgui), and then tell it to load an addon from a (C-Pluff) connection point and make various calls into the addon.

Is this along the lines of what you're building? If not currently, does it sound like something that would reasonably fit into the design?
Reply
#36
Well, the first thing I'm doing is attempting to provide the existing Python API to any number of other programming languages (maybe cleaning up some stuff along the way - I'm restructuring xbmcgui significantly but I'm making sure it's backward compatible). So initially the tie to XBMC will be tight as it depends on "core" functionality (and not the other way around).

If the core XBMC where componentized into a set of shared libraries plus a separate and minimal 'main' then I suppose what you're trying to do could be done. This would fit with my reading of the way cpluff is meant to be used so maybe something like this is the plan of the developers.

Hopefully this first step will provide a helpful grounds for a redesign of the Addon API (a version 2 perhaps) - if the task is deemed needed by the XBMC-team - though eventually (or maybe next) a step toward refactoring all of the xbmc functionality into cpluff modules making what is now "core" functionality into a series of "native addons" which could eventually be expanded in multiple programming languages.

If the later is the case, it may fit with your plans.
Reply
#37
Well. It looks like I have some decent progress to report. I basically have it working. It's still not packaged nicely but it works (there are some issues, see the bottom of the post)

If you want to try is you will need:

1) git repository:

git://github.com/jimfcarroll/xbmc-multi-language-addons.git

2) swig 2.0.1 installed.

3) configure with:

./configure --enable-newswiggyaddons --enable-external-python

4) The link still fails. Copy the link line and add the python library: -lpython2.6 (or whatever version you have installed).

5) You need to add an addon by hand. In ~/.xbmc/addons/script.module.newaddon/ use the following addon.xml:

Code:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<addon id="script.module.newaddon"
       name="newaddon"
       version="0.0.1"
       provider-name="jiminger.com">
  <requires>
    <import addon="xbmc.python" version="1.0"/>
  </requires>
  <extension point="xbmc.python.module"
             library="lib" />
  <extension point="xbmc.addon.metadata">
    <platform>all</platform>
  </extension>
</addon>

Then, in the lib directory of the addon, you need to symbolic link to the 4 SWIG generated python files:

xbmc.py -> [pathToSource]xbmc/lib/libPython/module/generatedSWIGBindings/xbmc.py
xbmcgui.py -> [pathToSource]xbmc/lib/libPython/module/generatedSWIGBindings/xbmcgui.py
xbmcplugin.py -> [pathToSource]xbmc/lib/libPython/module/generatedSWIGBindings/xbmcplugin.py
xbmcaddon.py -> [pathToSource]xbmc/lib/libPython/module/generatedSWIGBindings/xbmcaddon.py

... and include the dummy __init__.py

6) Issues:
a) SWIG doesn't support keywords. Any script you want to try you'll need to remove the keywords from all of the function/method calls.
b) SWIG error messages suck.
c) The ListItem.setInfo method takes a dictionary as the second parameter but it must be ALL STRINGS - or you will get an incomprehensible error message (see (b)).
d) plugin.video.revision3 works since that's what I did much of my testing on (and after the required mods were made (see a and c)) but I'm sure there'll be more issues.
e) Python thread yielding isn't incorporated in most of the places it used to be. I'm still considering the right way to do this.
f) I'm still working on the Window callbacks (Player callbacks work).
g) As mentioned earlier in this thread, the chdir functionality doesn't work correctly when USE_EXTERNAL_PYTHON is set. Any script that relies on "os.getcwd" to return the directory the running script is in will not work as expected (e.g.: Apple movie trailers). I'm looking for suggestions on fixing this one.
h) Instantiating a xbmcgui.Window using the id of an existing window is currently not supported. Working on putting that back.
i) I doubt the unicode stuff works right as SWIG out of the box has very little support. I might need help with this one.
j) since I've edited this post about 5 times to add more items to this list, let me just say ... a bunch more stuff.

Again, let me know what you think.
Jim
Reply
#38
6e) Handling the ability to allow the python threads to continue in a language neutral manner is now fixed.
Reply
#39
6h) You can now instantiate a Window using the windowid of an existing window.
Reply
#40
Very interesting stuff, jhcarrol.

I don't know if this has been considered from a design perspective, but it would be nice if the multi-language interface had built in process isolation. The main benefit being that a badly behaved addon could not bring XBMC down. This of course limits the addon's capabilities but it seems to be a popular design decision these days (Google Chrome with each tab/plugin/extension being a separate child process). I think a method invocation flow would look something like this:

XBMC native code -> c++ addon api -> bidirectional c++ json/xml marshalling -> process boundary -> bidirectional python json/xml unmarshalling -> addon client python api -> addon code

Thoughts?
MythBox for XBMC - http://mythbox.googlecode.com
Reply
#41
I dont really know what upu guys are talking about, but Im looking forward to and hoping for a java binding.
Use mythicalLibrarian to make a library out of your MythTV files. Leave the recording to MythTV and use XBMC as your library.
Installation and Instructions:http://wiki.xbmc.org/index.php?title=MythicalLibrarian
Technical Support:http://forum.xbmc.org/showthread.php?tid=65644
[url=http://forum.xda-developers.com/showthread.php?tid=1081892][/url]
Reply
#42
analogue Wrote:Very interesting stuff, jhcarrol.

I don't know if this has been considered from a design perspective, but it would be nice if the multi-language interface had built in process isolation. [...]

Thoughts?

Not a bad idea. The native API side of could eventually be implemented the way you suggest.

outleradam Wrote:I dont really know what upu guys are talking about, but Im looking forward to and hoping for a java binding.

I hope to be able to demonstrate that exact thing shortly (week or 2). I have to clean up some other things first (I want the apple movie trailer plugin to work so I need to address UTF-8 in SWIG first - then I need to work out how to bootstrap something other than python).
Reply
#43
analogue Wrote:Very interesting stuff, jhcarrol.

I don't know if this has been considered from a design perspective, but it would be nice if the multi-language interface had built in process isolation. The main benefit being that a badly behaved addon could not bring XBMC down. This of course limits the addon's capabilities but it seems to be a popular design decision these days (Google Chrome with each tab/plugin/extension being a separate child process). I think a method invocation flow would look something like this:

XBMC native code -> c++ addon api -> bidirectional c++ json/xml marshalling -> process boundary -> bidirectional python json/xml unmarshalling -> addon client python api -> addon code

Thoughts?

D-Bus is superior to JSON/XML marshaling - faster binary transport, better support for objects.

Marshaling to another process, using either transport, could conceivably be fit into jfcarroll's model as if it were just another language.
Reply
#44
6i) Unicode support should now work. I submitted a patch to SWIG. If they take it I can delete a SWIG typemap from the code base. In either case unicode should now be working.
Reply
#45
dbrobins Wrote:D-Bus is superior to JSON/XML marshaling - faster binary transport, better support for objects.

Marshaling to another process, using either transport, could conceivably be fit into jfcarroll's model as if it were just another language.

Dbus is still emerging on windows. It has not yet gained recognition. I agree with you though. It is a great protocol.
Use mythicalLibrarian to make a library out of your MythTV files. Leave the recording to MythTV and use XBMC as your library.
Installation and Instructions:http://wiki.xbmc.org/index.php?title=MythicalLibrarian
Technical Support:http://forum.xbmc.org/showthread.php?tid=65644
[url=http://forum.xda-developers.com/showthread.php?tid=1081892][/url]
Reply

Logout Mark Read Team Forum Stats Members Help
[WIP] Addon system refactor for multiple programming languages - Overview/Status1