[jfcarroll]

I know everyone's busy with the pending release but I wanted to give you an overview and status of my multi-language addon effort.

I) Overview.

The refactored Addon API is split into two parts, native functionality (C++) and language bindings. The Native API is a set of C++ classes and methods that has no dependency on a particular (potential) Addon language. This API mirrors the existing Python API (though I hope this will one day be deprecated) by module (namespace) and classes.

A "description" of this API is captured in an XML file. Originally I had looked at potentially automating this by using something like gcc2xml or an C++ variation of java's XDoclet or annotations, but it was too complicated to get something to show quickly and nothing prevents it from being done in the future.

This XML description drives a code generator (written in groovy) which applies the description to set of templates. This results in the generation of a set of C++ files that represent the binding to various languages (currently Python).

II) Some Details:

a) The Native API:
The Native functionality is a C++ set of modules and classes. These have no dependence on any particular Addon language. The current set is a transfer of the existing Python functionality into a Python-independent C++ implementation.

All native API classes and modules are in the namespace "XBMCAddon" and all of the code currently sits in the directory xbmc/lib/libAddon/module. All files that are part of the API are named Addon*.h/cpp.

i) Modules: In the native API Modules are represented as namespaces. The four existing (legacy) modules: xbmc, xbmcgui, xbmcaddon, xbmcplugin, are a set of global methods within their respective namespace, and nested within 'XBMCAddon'. For example, the 'sleep' method of the module "xbmc" can be accessed natively with "XBMCAddon::xbmc:leep()." Modules are in source files that are named AddonModule[Modulename].h/cpp.

ii) Classes: Classes are always within modules and are therefore named accordingly. For example, the Player class, which is in the "xbmc" module is "XBMCAddon::xbmc:layer."

There are several classes that contain objects that represent what are standard types (dictionaries, lists, strings) in some languages, but are not native to C++. These are also in files Addon*.h/cpp. So far these include (only):

AddonDictionary.h -> This is defined as template <class T> class Dictionary : public std::map<CStdString,T> and defines a "nullDictionary"
AddonString.h -> This is a CStdString but defines a "nullString" to use as a default value.

I'm not particularly thrilled with this solution but it works for now. Any suggestions on what might accomplish this better let me know.

b) The API specification
The specification contains the API definitions captured in a set of XML files and located at xbmc/lib/libAddon/module/specs. There is a separate XML file for each module. An example from the current xbmc.xml file is:

Code:

<?xml version="1.0" encoding="UTF-8"?>
<module name="xbmc">
<method name="log">
  <doc><![CDATA[... method documentation ...]]></doc>
  <parameter modifier="const" type="string">msg</parameter>
  ...
</method>
...
<class name="Player" >
   <doc><![CDATA[ ... class documentation ...]]></doc>
   <constructor>
     <parameter type="int" default="EPC_NONE">playerCore</parameter>
   </constructor>
   <method name="playStream" > ...

Another single file contains type metadata for parameters and return types. This file tells the generator how to handle the three different 'kinds' of 'types': 1) native C++ types (bool, int, long, etc). 2) Classes that are part of the API (String, Dictionary, etc), and 3) Classes that are defined in the API spec (xbmc.Player, xbmcgui.ListItem, etc). The file looks like:

Code:

<?xml version="1.0" encoding="UTF-8"?>
<meta>
  <parameter type="int" meta="native" />
  <parameter type="bool" meta="native" />
  ...
  <parameter type="string" meta="native" alias="char" modifier="*" />
  <parameter type="wstring" meta="object" alias="String" namespace="XBMCAddon" />
  <return type="string" meta="object" alias="String" namespace="XBMCAddon" />
  <parameter type="Dictionary&lt;CStdString&gt;" meta="object" namespace="XBMCAddon" />
</meta>

c) Code Generation.

The XML specification drives the execution of the generator over multiple templates. The entire process is driven by an XML file that contains meta data for the bindings. It currently looks like this:

Code:

<?xml version="1.0" encoding="UTF-8"?>
<bindings>
  <binding name="python">
    <param name="override" module="xbmc" class="Player" method="play" />
    <param name="override" module="xbmcgui" class="ListItem" method="setInfo" />
    <template in="PythonBindings.cpp.template"
              scope="permodule"
              out="${module}_PythonBindings.cpp"/>
    <template in="python.cpp.template"
              scope="allmodules"
              out="python.cpp"/>
  </binding>
  <template in="addons.cpp.template"
            scope="allbindings"
            out="addons.cpp"/>
</bindings>

The generator will apply the XML spec to the template file within the given scope and produce the given output file. For example, given the above, the generator will apply each XML module definition spec to the template contained in the file PythonBindings.cpp.template and will produce an output file for each resulting pass called {modulename}_PythonBindings.cpp (obviously the actual module name will be substituted). The templates and this metadata file are all found at xbmc/lib/libAddon/module/bindings

III) Current status.

I have the entire xbmc module now working, and some of the xbmcgui module. This included callbacks (The "Player" is the only class with callbacks in the xbmc module). I still need to:

1) Move the other modules into this system.
2) Remove the Python specifics from some of the code that bootstraps and calls into the Addon system
3) I'm pretty sure I can improve the callback system (it currently works the same way the offical Dharma one works) by having both synchronous calls and better asynchronous calling.
4) Add bindings (templates) for another language (groovy would be my first choice).

I hope this will lead to a redesign of the API itself (a version 2.0 perhaps) and perhaps a merging of some other portions of the system into a uniform API (c-pluff stuff and the new JSON-RPC functionality).

Let me know what you think. The git repository is:

http://github.com/jimfcarroll/xbmc-multi...addons.git

You'll need to switch to the 'Dharma' branch.

All of the code is in xbmc/lib/libAddon/module with the exception of some hooks in other places.

As always, feedback is welcome.

[jmarshall]

Nice work. One thing that I would suggest is you use std:tring rather than CStdString where you can - no need to include a dependency that you really don't require. Other than that, using a map for a dictionary seems reasonable to me. Having a strictly-typed interface is probably the way to go here, even if you lose a little bit in some languages.

I'm interested in how you see this fitting in with cpluff - particularly for binary addons. eg how do you see a visualisation written in Python fitting in with XBMC, given that we have some common C interface that we call into currently? Basically: Where does the support code go to interface with the visualisation (load python interpreter and pass information in and out)?

Cheers,
Jonathan

[jfcarroll]

Thanks.

I can change the CStdString easily enough. Will do.

As far as your other question - At this point, I don't know for sure. Based on the way it appears that you're setting things up, it should fit nicely ... eventually - and with some conceptual changes. For example (and please correct me if I'm wrong), in order for me to do a native visualization addon, I need to create a dynamic library that exports:

Code:

void (__cdecl* Start)(int iChannels, int iSamplesPerSec, int iBitsPerSample, const char* szSongName);
    void (__cdecl* AudioData)(const short* pAudioData, int iAudioDataLength, float *pFreqData, int iFreqDataLength);
    void (__cdecl* Render) ();
    void (__cdecl* GetInfo)(VIS_INFO *info);
    bool (__cdecl* OnAction)(long flags, const void *param);
    int (__cdecl* HasPresets)();
    unsigned int (__cdecl *GetPresets)(char ***presets);
    unsigned int (__cdecl *GetPreset)();
    unsigned int (__cdecl *GetSubModules)(char ***modules);
    bool (__cdecl* IsLocked)();

The current code generator simply generates what is effectively a bridge. I would think the same thing could be accomplished here. For each language binding you would generate the implementation of these calls which will be exported from a dynamic library. Inside the library this implementation would bridge to the language in question like it currently does with callbacks.

In the case of Python, the generated code would (perhaps) read in configuration that the visualization addon developer supplied. Perhaps this configuration would identify the Python code to bridge to by module or class, or maybe it would simply supply an entry point and the addon developer would be responsible for registering his implementation of the visualization through Python code.

In any case the bridge would be there. Code in the addon dynamic library would include generated bridge code as well as bootstrapping code specific to the language but not specific to the addon interface.

At this point, just speculation on my part. Does it seem like it might work?

Thanks again,
Jim

[dbrobins]

That is great... as part of my scraper abstraction I was thinking about how I could incorporate not just Python scrapers but scrapers in any language, and as you did I began thinking about a language agnostic interface (both in terms of scripts calling into XBMC and XBMC calling into scripts), and binding interfaces to languages. I was concerned about coming up with an interface description language, since I didn't want to reinvent the wheel and figured there must be some good IDLs out there already; so your XML interfaces fill in that missing piece for me.

Having said that, though: why invent your own interface definition language? There are some good existing ones - CORBA IDL, Google's Protocol Buffers, SWIG (which uses C), or Mozilla's XPIDL. There are probably some XML ones too, if you prefer that. There are likely a lot of corner cases that these existing IDLs have already solved.

How do you see it working from start to finish to do something like, say, writing a script in perl that pops up a modal window (like in the "HOW-TO build Python Scripts" wiki page)? Write a generator that transforms the XML API into a Perl (XS) module and some sort of framework to detect that an addon is in perl (file extension?) and have a language-specific class manage an embedded perl interpreter, with callbacks available through the aforementioned module ("use XBMC; XBMC::Callback()")?

I'm fairly new to git and am already pulling from the main XBMC repository. Can I apply your changes on top of that repository in a way that keeps them separate from the main master and my own local commits? Pointers/links as to how to do that appreciated - searching has been unfruitful, perhaps because I don't know the precise terms.

[spiff]

it's called a 'pull'. simply pull it into a separate branch in your own repo.

[gnif]

Please also note that when AudioEngine is ready, the visualizations will be using float data, not short, check the interface in the AudioEngine for the changes.

[jfcarroll]

dbrobins,

You need to pull the Dharma branch of my repository. I didn't know 'git' at all before doing this :-)

If you're checking out from scratch the following should work:

git clone git://github.com/jimfcarroll/xbmc-multi-language-addons.git xbmc
cd xbmc
git checkout Dharma

I keep it up to date with the Dharma branch of the main xbmc git repo.

On the XML, I wanted to process the actual C++ code so that data in two places wouldn't need to be kept in sync (that is, the C++ API and its XML description). If C++ had annotations like java I would have used it. As it is I hope to revisit that possibility at some point in the future.

Also, groovy is the best parsing language I've come across. It has built in XML parsing and template processing (the entire code generator is only on the order of 100 lines of code or so - not counting the templates and Helper class). It keeps it nice an simple. Maybe switching to one of these options will be a good phase 2 but this got me jump started.

Apart from jmarshall's question (the answer to which is not perfectly clear to me at this point) this should work the same way that Python scripts work today. Native objects and methods (the API itself) should be made available through a generated bridge to the addon language. I don't know perl well enough to say how that would be done there but JVM based languages would be available through generated JNI C++ code just like the Python is. If you can invoke perl from C++ and that perl script can call back on C then it should be straightforward.

[dbrobins]

dbrobins,
You need to pull the Dharma branch of my repository. I didn't know 'git' at all before doing this :-)

I did that - tried to pull it into my regular (non-Dharma) repo, got a lot of conflicts (with master, not my own changes), so reverted it. I probably got conflicts because I'm using master and not Dharma there. (Sorry if this is a silly question, but why sync to Dharma rather than master, since Dharma appears to be getting locked down in prep for the coming release?)

Anyway, I just cloned it into a new repo.

On the XML, I wanted to process the actual C++ code so that data in two places wouldn't need to be kept in sync (that is, the C++ API and its XML description). If C++ had annotations like java I would have used it. As it is I hope to revisit that possibility at some point in the future.

SWIG parses (a subset of?) C header files, so it's both done and doable, but perhaps not as precise as explicitly defining the API in XML.

Apart from jmarshall's question (the answer to which is not perfectly clear to me at this point) this should work the same way that Python scripts work today. Native objects and methods (the API itself) should be made available through a generated bridge to the addon language. I don't know perl well enough to say how that would be done there but JVM based languages would be available through generated JNI C++ code just like the Python is. If you can invoke perl from C++ and that perl script can call back on C then it should be straightforward.

What about multiple versions of languages? I've been writing a lot of utilities in Python 3, so I'd like to be able to use it for add-ons.

[jfcarroll]

I chose the Dharma branch because I assumed it had the most active development and that the changes there would be merged back into the main branch at the release. I'm not familiar with the source code lifecycle practices of the developers here so I'm not sure if that was a mistake. Maybe. Either way, so far my changes are mostly additions (I can switch between implementations (the original, or the new generated) with a #define flip right now) and I can lift them from where I have them, and incorporate them into a new branch (or the trunk) with about 15 minutes of work at most.

On the XML, take a look at what's there. If you can derive that information from the source code and want to help me out let me know.

Since Python 2.x and Python 3.x are almost different languages, it should be able to be done with another set of generated bindings easily enough.

[spiff]

we do it the other way around, trunk is the mainline and we backport the relevant stuff to dharma.

[jfcarroll]

we do it the other way around, trunk is the mainline and we backport the relevant stuff to dharma.

[STRIKE]Ah. Then I was right. Thanks.[/STRIKE]

EDIT: [STRIKE]Reading this again I'm not so sure. But[/STRIKE] like I said above, either way should be fine.

EDIT EDIT: So I guess I did get it wrong. Oh well. I'll move the changes to the main branch prior to my next checkin (hopefully this weekend. I should have xbmcgui complete then).

[alcoheca]

SWIG is ideal for this, and fits in perfectly with the existing C-Pluff approach.

Can't grab your tree right now, as on shitty mobile broadband with ridiculous download caps..

[jfcarroll]

OK. I'm spending some time looking at SWIG now. Maybe this is a better approach than what I'm doing.

[dbrobins]

On the XML, take a look at what's there. If you can derive that information from the source code and want to help me out let me know.

Do you have a DTD or spec for the full extent of your XML representation - and where are you getting the C++ functions that you're wrapping? If just the various Addon*.h files, I could write up a Python program fairly easily for going from .h to XML, taking a list of header files as input (is Python assumed to be available in the XBMC build environment?) (Minor quibble: it seems inconsistent for "string" to mean char * as a parameter but XBMCAddon::String as a return value - why not give char * a differentiated name ("charstring") and make "string" always mean XBMCAddon::String?)

Since Python 2.x and Python 3.x are almost different languages, it should be able to be done with another set of generated bindings easily enough.

I was thinking about languages a bit and took a step back and thought it might make more sense to go for language-agnostic - like you're trying to do, but not precisely in the same way. I'm talking about a more disconnected interface and after investigation I think D-Bus makes the most sense in that regard (it's used in Gnome and KDE and by HAL and is available in most distros). It's true there's possible worse performance than embedding a language directly, but I can't see any case where that would matter in practice - it's not like we're trying to do 3D rendering with scripts. I may work on a proof of concept for using D-Bus for scrapers.

D-Bus bindings should fit in from your design point of view as just another language binding. (XBMC already has a D-Bus server, but it's fairly limited in scope.) Any language could be hooked up to D-Bus bindings, making it unnecessary to figure out how to embed them all. Another good fit is that your <module> could just as easily define an interface on an object (which D-Bus is based around) as a set of functions in a namespace.

[jfcarroll]

Do you have a DTD or spec for the full extent of your XML representation

No, as the full extent is yet to be determined. I'm still expanding it.

Although, right now (literally) I'm looking at SWIG - so the generator may be superfluous.

- and where are you getting the C++ functions that you're wrapping? If just the various Addon*.h files, I could write up a Python program fairly easily for going from .h to XML, taking a list of header files as input

The Addon*.h files are the new refactored Addon API, so 'yes.' But again, this might be pointless if SWIG works out.

(is Python assumed to be available in the XBMC build environment?)

Currently I'm assuming groovy will be. :-) or if not a set of default pre-generated bindings could be checked in with the source code for those that fail that particular config check.

(Minor quibble: it seems inconsistent for "string" to mean char * as a parameter but XBMCAddon::String as a return value - why not give char * a differentiated name ("charstring") and make "string" always mean XBMCAddon::String?)

I went back and forth on this. The biggest issue was trying to understand what it means for a function to return a char* in terms of memory management, combined with being able to handle wstrings differently from strings, combined with limiting the proliferation (and therefore the handling) of different C++ types that do the same thing.

Though, like you say, it may be more consistent to do it your way and just allow the developer to do what he wants.

On D-Bus, I see what you're saying. A cross language RPC mechanism. Probably worth a look. If SWIG works we get the best of both worlds. Many languages out of the box and native access.

Thanks for the feedback and for taking a look.