Kodi Community Forum
XBMC.MyLibrary - Scan any source to the library (Hulu/Netflix/Plugins) - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+--- Thread: XBMC.MyLibrary - Scan any source to the library (Hulu/Netflix/Plugins) (/showthread.php?tid=98210)



- pkscout - 2011-12-12

TheCasa Wrote:Thanks for the quick response. For simplicity's sake, I changed the config to only pull in 2 movies. One of them is added to the library, the other is not. That mpg is not renamed though. Here's the log:
http://pastebin.com/pZD3nVrC
and here is my config:
http://pastebin.com/8T8vXdBJ

Thanks again in advance for the help. I'm a pretty quick learner, but I'm a virgin on this.

Two things I noticed:

1- You have Comskip enabled in the config file. It is possible that XBMC.MyLibrary is waiting for an EDL file to be present before processing the file. You might try turning disabling the Comskip functionality and re-run it.

2- I see you are trying to load movies from Hulu. I tried that a couple of times, and it seems like the Hulu parser doesn't do so well with the movies. If you do Hulu through PlayOn I think it works better. At least I remember that being true at one time.

And a request. If turning off Comskip doesn't help, bump the logging level to 4 and repost it.


- bradvido88 - 2011-12-12

TheCasa Wrote:Thanks for the quick response. For simplicity's sake, I changed the config to only pull in 2 movies. One of them is added to the library, the other is not. That mpg is not renamed though. Here's the log:
http://pastebin.com/pZD3nVrC
and here is my config:
http://pastebin.com/8T8vXdBJ

Thanks again in advance for the help. I'm a pretty quick learner, but I'm a virgin on this.
Yeah I can't tell what's happening from that log. Can you change the log level to INFO and try again?

I did test out your searchfilters and it sucessfully added the Hulu movies from the Hulu plugin for me.... so there shouldn't be any problem with the parsing.


- TheCasa - 2011-12-13

Sorry it took me so long to respond. I've been having issues. Just to make sure it wasn't something left over from problems I've been having, I just deleted the MyLibrary directory and made a fresh one. After cleaning the xbmc database, I set the config to be the example of Bluecop's from the users' guide. The only changes I made was limiting the max number of videos to import so it would run much faster. I'm having the same issue as before though. The videos are being added to the library and scraped, but they can't be viewed. In the dropbox directories, the files are still the mpg placeholders.

Here's the log with logging set to info.
http://pastebin.com/94w8qRfS

I don't have a PlayOn subscription, so I'm trying to do this with bluecop. Does anyone have a reference for how the subfolders are structured for that plugin? For example, I'm trying to import the Terra Nova tv show. I've tried /TV/T/Terra Nova/Full Episodes, I've tried /TV Shows/T/Terra Nova/Full Episodes, /TV Shows/Full Episodes/T/Terra Nova, etc. Lots of iterations with no luck.


- pkscout - 2011-12-13

TheCasa Wrote:Here's the log with logging set to info.
http://pastebin.com/94w8qRfS

From what I'm seeing, you have a straight up XBMC import problem. Did you make sure to add new sources to XBMC for the MyLibrary/TVShows and MyLibrary/Movies directories and set the proper scrappers?

That Amanda Knox movie probably isn't going to get scrapped at all. I couldn't find it in themoviedb.org. You could create a .nfo file for it and try and force the import that way.

TheCasa Wrote:Does anyone have a reference for how the subfolders are structured for that plugin? For example, I'm trying to import the Terra Nova tv show. I've tried /TV/T/Terra Nova/Full Episodes, I've tried /TV Shows/T/Terra Nova/Full Episodes, /TV Shows/Full Episodes/T/Terra Nova, etc. Lots of iterations with no luck.

The easiest thing to do is to go to the plugin and click through the options to get you where you want to go and write down each one as you go. It looks like Terra Nova can be gotten to by TV/Networks/FOX/All/Terra Nova/Episodes. The bad news with that path is that the Terra Nova episodes will go into a folder called Episodes. You'll have to use a .nfo file to get XBMC to understand that folder is Terra Nova, and you can't have any other TV Show end with Episodes.

Honestly, the easiest thing I found to do with Hulu was to add stuff to my queue and then just point XBMC.MyLibrary to my queue using:

Code:
<Hulu path="plugin.video.hulu">
  <subfolder name="Queue" type="episodes" />
</Hulu>

I only keep TV shows in my Hulu queue, so that makes it easy. The above will not properly load movies in my queue.


- TheCasa - 2011-12-13

pkscuot Wrote:From what I'm seeing, you have a straight up XBMC import problem. Did you make sure to add new sources to XBMC for the MyLibrary/TVShows and MyLibrary/Movies directories and set the proper scrappers?
Yes, the sources are setup and the content is set properly. XBMC is pulling the tv shows/movies into the library, but the mpgs aren't being renamed to strm and doing the database magic to make them play.

Quote:That Amanda Knox movie probably isn't going to get scrapped at all. I couldn't find it in themoviedb.org. You could create a .nfo file for it and try and force the import that way.
You're right, that movie didn't get scraped or imported, and honestly I don't care. I was just using the straight up example from the users manual to try to get it working in some way. But it's good to know how I'd have to go about getting movies or tv that can't be scraped.

Quote:The easiest thing to do is to go to the plugin and click through the options to get you where you want to go and write down each one as you go. It looks like Terra Nova can be gotten to by TV/Networks/FOX/All/Terra Nova/Episodes. The bad news with that path is that the Terra Nova episodes will go into a folder called Episodes. You'll have to use a .nfo file to get XBMC to understand that folder is Terra Nova, and you can't have any other TV Show end with Episodes.

Honestly, the easiest thing I found to do with Hulu was to add stuff to my queue and then just point XBMC.MyLibrary to my queue using:

Code:
<Hulu path="plugin.video.hulu">
  <subfolder name="Queue" type="episodes" />
</Hulu>

I only keep TV shows in my Hulu queue, so that makes it easy. The above will not properly load movies in my queue.

I'll just use that code and see if I can get it to pull the queue in. I'm just trying to get it working, then I'll try to get some stuff customized.

Thanks for your help. I'll run it with you code above and let you know how it goes......


- Ryoga79 - 2011-12-13

pkscuot Wrote:From what I'm seeing, you have a straight up XBMC import problem. Did you make sure to add new sources to XBMC for the MyLibrary/TVShows and MyLibrary/Movies directories and set the proper scrappers?

That Amanda Knox movie probably isn't going to get scrapped at all. I couldn't find it in themoviedb.org. You could create a .nfo file for it and try and force the import that way.



The easiest thing to do is to go to the plugin and click through the options to get you where you want to go and write down each one as you go. It looks like Terra Nova can be gotten to by TV/Networks/FOX/All/Terra Nova/Episodes. The bad news with that path is that the Terra Nova episodes will go into a folder called Episodes. You'll have to use a .nfo file to get XBMC to understand that folder is Terra Nova, and you can't have any other TV Show end with Episodes.

Honestly, the easiest thing I found to do with Hulu was to add stuff to my queue and then just point XBMC.MyLibrary to my queue using:

Code:
<Hulu path="plugin.video.hulu">
  <subfolder name="Queue" type="episodes" />
</Hulu>

I only keep TV shows in my Hulu queue, so that makes it easy. The above will not properly load movies in my queue.

I'm having a similar problem, and at least from what i can glean from the log, it looks like it may not be a databse problem, but a SQL problem. My log is very similar to his, in that i get repeated references to the files "not being in the database yet", but i also get repeated lines like this snippet below:

Code:
12/12/2011 08:10:08 PM ERROR   DBUpdate:TV Show Failed to check if file exists in XBMC's database using: SELECT idFile FROM files WHERE idPath IN(SELECT idPath FROM path WHERE LOWER(strPath) = 'y:\streaming\drop\tv shows\once.upon.a.time\season.1\') AND lower(strFileName) = 's01e06 - the shepherd.mpg'
12/12/2011 08:10:08 PM ERROR   DBUpdate:TV Show Could not execute query: SELECT idFile FROM files WHERE idPath IN(SELECT idPath FROM path WHERE LOWER(strPath) = 'y:\streaming\drop\tv shows\once.upon.a.time\season.1\') AND lower(strFileName) = 's01e03 - snow falls.mpg'
                                                    com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 's01e03 - snow falls.mpg'' at line 1
                                                        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                                                        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
                                                        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
                                                        at java.lang.reflect.Constructor.newInstance(Unknown Source)
                                                        at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
                                                        at com.mysql.jdbc.Util.getInstance(Util.java:381)
                                                        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1030)
                                                        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
                                                        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3536)
                                                        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3468)
                                                        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1957)
                                                        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2107)
                                                        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2642)
                                                        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2571)
                                                        at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1451)
                                                        at utilities.Database.getSingleInt(Database.java:205)
                                                        at utilities.XBMCInterface.updateFilePointers(XBMCInterface.java:555)
                                                        at mylibrary.importer.updateDatabase(importer.java:1561)
                                                        at mylibrary.importer.importVideos(importer.java:357)
                                                        at mylibrary.importer.main(importer.java:25)

I get a copy of that line from every video file that was imported in the previous step. I checked my sources, and it looks like the files are getting imported into the library just fine, it's just that the query is failing when it goes back to check for the files the second time, so the files never get renamed. I'm trying to do the same import as above, with 10 "popular" tv shows and movies from bluecop's hulu plugin (of which i'm also using the latest version).

Config.xml pastebin here: http://pastebin.com/KYPxRPCK

Full Log pastebin here: http://pastebin.com/C8Pzt3kE

Can anyone enlighten me on what i might have done wrong that the second database query is failing?

Thanks!


- bradvido88 - 2011-12-13

Ryoga79 Wrote:I'm having a similar problem, and at least from what i can glean from the log, it looks like it may not be a databse problem, but a SQL problem. My log is very similar to his, in that i get repeated references to the files "not being in the database yet", but i also get repeated lines like this snippet below:

Code:
12/12/2011 08:10:08 PM ERROR   DBUpdate:TV Show Failed to check if file exists in XBMC's database using: SELECT idFile FROM files WHERE idPath IN(SELECT idPath FROM path WHERE LOWER(strPath) = 'y:\streaming\drop\tv shows\once.upon.a.time\season.1\') AND lower(strFileName) = 's01e06 - the shepherd.mpg'
12/12/2011 08:10:08 PM ERROR   DBUpdate:TV Show Could not execute query: SELECT idFile FROM files WHERE idPath IN(SELECT idPath FROM path WHERE LOWER(strPath) = 'y:\streaming\drop\tv shows\once.upon.a.time\season.1\') AND lower(strFileName) = 's01e03 - snow falls.mpg'
                                                    com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 's01e03 - snow falls.mpg'' at line 1
                                                        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                                                        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
                                                        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
                                                        at java.lang.reflect.Constructor.newInstance(Unknown Source)
                                                        at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
                                                        at com.mysql.jdbc.Util.getInstance(Util.java:381)
                                                        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1030)
                                                        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
                                                        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3536)
                                                        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3468)
                                                        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1957)
                                                        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2107)
                                                        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2642)
                                                        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2571)
                                                        at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1451)
                                                        at utilities.Database.getSingleInt(Database.java:205)
                                                        at utilities.XBMCInterface.updateFilePointers(XBMCInterface.java:555)
                                                        at mylibrary.importer.updateDatabase(importer.java:1561)
                                                        at mylibrary.importer.importVideos(importer.java:357)
                                                        at mylibrary.importer.main(importer.java:25)
I get a copy of that line from every video file that was imported in the previous step. I checked my sources, and it looks like the files are getting imported into the library just fine, it's just that the query is failing when it goes back to check for the files the second time, so the files never get renamed. I'm trying to do the same import as above, with 10 "popular" tv shows and movies from bluecop's hulu plugin (of which i'm also using the latest version).

Config.xml pastebin here: http://pastebin.com/KYPxRPCK

Full Log pastebin here: http://pastebin.com/C8Pzt3kE

Can anyone enlighten me on what i might have done wrong that the second database query is failing?

Thanks!

It looks like MySQL is failing at executing that query. What version of MySQL are you using? Can you try and execute the query manually (if you know how)
Code:
SELECT idFile FROM files WHERE idPath IN(SELECT idPath FROM path WHERE  LOWER(strPath) = 'y:\streaming\drop\tv  shows\once.upon.a.time\season.1\') AND lower(strFileName) = 's01e06 -  the shepherd.mpg'

EDIT-Actually, after a closer look, this is a problem with my code. It looks like i didn't properly handle the '\' character in SQL queries. I thought i had it automatically escaped, but maybe it broke with some of the recent changes.

All my sources are smb://, and they only use forward slashes, so I don't get the same problem. In the meantime, you could try that, but I'll look into it.


- bradvido88 - 2011-12-13

TheCasa Wrote:Yes, the sources are setup and the content is set properly. XBMC is pulling the tv shows/movies into the library, but the mpgs aren't being renamed to strm and doing the database magic to make them play.


You're right, that movie didn't get scraped or imported, and honestly I don't care. I was just using the straight up example from the users manual to try to get it working in some way. But it's good to know how I'd have to go about getting movies or tv that can't be scraped.



I'll just use that code and see if I can get it to pull the queue in. I'm just trying to get it working, then I'll try to get some stuff customized.

Thanks for your help. I'll run it with you code above and let you know how it goes......

I exclusively use Hulu Bluecop plugin for most of my content and it works great.

Here's my exact code to get all full episodes from the top 25 most popular seasons:

Code:
<subfolder name="TV/Popular TV \([0-9]+\)/Popular Shows \([0-9]+\)"
                    max_series="25" type="episodes" >
                    <filter>
                            <regexp>/Episodes \([0-9]+\)/</regexp>                        
                    </filter>
                    <exclude>                                      
                            <regexp>/season[s]? \([0-9]+\)</regexp>  
                            <contains>/movie trailers</contains>                            
                    </exclude>
            </subfolder>
Try that out. Many of the Hulu movies are kind of obscure and won't be found by XBMC's scraper, so you'll have to wait until NFO's are written for those ones (this is configurable with <ManualArchiving enabled="true"> in Config.xml).

Also, adding your queue works, but make sure you only add one kind of content to the queue (either episodes OR movies) and set the proper type in the config file.


- Ryoga79 - 2011-12-13

bradvido88 Wrote:It looks like MySQL is failing at executing that query. What version of MySQL are you using? Can you try and execute the query manually (if you know how)
Code:
SELECT idFile FROM files WHERE idPath IN(SELECT idPath FROM path WHERE  LOWER(strPath) = 'y:\streaming\drop\tv  shows\once.upon.a.time\season.1\') AND lower(strFileName) = 's01e06 -  the shepherd.mpg'

Trying to do a crash course in mysql right now to find out the second part, but the version is:

Server version: 5.1.54-1ubuntu4 (Ubuntu)

Running on ubuntu server (natty)


- bradvido88 - 2011-12-13

Ryoga79 Wrote:Trying to do a crash course in mysql right now to find out the second part, but the version is:

Server version: 5.1.54-1ubuntu4 (Ubuntu)

Running on ubuntu server (natty)
See my edited post above, it's a problem with me escaping backslashes in the query. In the meantime, you could try a smb share as those paths shouldn't have backslashes.

I'll post a fix (probably tomorrow but no guarantees)


- Ryoga79 - 2011-12-13

bradvido88 Wrote:See my edited post above, it's a problem with me escaping backslashes in the query. In the meantime, you could try a smb share as those paths shouldn't have backslashes.

I'll post a fix (probably tomorrow but no guarantees)

Do you mind filling me in on the SMB syntax for the config.xml? I was using
Code:
    <Dropbox>
        <streaming>smb://FEEZFILESERV/Misc/Streaming/drop/Movies/</streaming>
        <downloaded>smb://FEEZFILESERV/Misc/Streaming/drop/TV Shows/</downloaded>
    </Dropbox>

and in the log it comes out as:

Code:
12/12/2011 10:12:55 PM INFO    Search:Hulu      Creating shortcut failed: smb:\FEEZFILESERV\Misc\Streaming\drop\Movies\Movies\The House Next Door.mpg (The specified path is invalid)

in the log - no files get written. Any suggestions? I'm new to this XML thing, clearly Smile


- TheCasa - 2011-12-13

After pulling only my queue in, again, the files were downloaded, added to the xbmc library, but not converted to strm for playing. Here's my searchfilter:

Code:
    <Hulu path="plugin://plugin.video.hulu">
        <subfolder name="Queue" type="episodes" />
                                
        <exclude>
            <contains>previous page</contains>                    
            <contains>next page</contains>                        
        </exclude>
    </Hulu>

Here's the dropbox:
Image
And the Library:
Image

Weird....


- bradvido88 - 2011-12-13

Ryoga79 Wrote:Do you mind filling me in on the SMB syntax for the config.xml?
...
in the log - no files get written. Any suggestions? I'm new to this XML thing, clearly Smile
Sorry, I meant set up the path for your source as smb in XBMC. Then in the config file, you use whatever syntax your OS uses to access samba shares. For windows its
Code:
\\server\path\to\share
TheCasa Wrote:After pulling only my queue in, again, the files were downloaded, added to the xbmc library, but not converted to strm for playing. Here's my searchfilter:

Code:
<Hulu path="plugin://plugin.video.hulu">
        <subfolder name="Queue" type="episodes" />
                                
        <exclude>
            <contains>previous page</contains>                    
            <contains>next page</contains>                        
        </exclude>
    </Hulu>
Weird....
Do you happen to have the log file for this?
You are really close. The reason it's not converted to a .strm is because this program never finds it in XBMC's database.
The way I see it is that one of three things is wrong:

  1. Your dropbox\Tv Shows directory has not been added as a source in XBMC and had it's content type set to "TV Shows" (this should be done after some .mpg's exists in the dropbox). From your screenshot, it looks like the .mpg is in the library, so this probably isn't the issue.
  2. (most likely) - The path to your TV Shows directory in XBMC is different than the path you specified in the Config file. For example, one uses a network share and the other uses a local path. They have to be the same. I personally use a network share "\\onyx\\dropbox\\tvshows" in the config file, and in xbmc I set the source up as a smb path to the same directory.
    To troubleshoot this, can you post the contents of your sources.xml file (in XBMC's userdata directory) as well as your dropbox config in the Config.xml so I can compare.
  3. The query to your SQL database to determine if the dummy .mpg files is in the library is failing. However we would see this in the logs as an ERROR and I don't think we've seen that.
As a disclaimer, my testing is limited to a Windows environment so things may be slightly different if your running a different OS


- bradvido88 - 2011-12-13

bradvido88 Wrote:See my edited post above, it's a problem with me escaping backslashes in the query. In the meantime, you could try a smb share as those paths shouldn't have backslashes.

I'll post a fix (probably tomorrow but no guarantees)

I fixed this bug, so now local paths should work for everyone.
The new version can be downloaded here. To upgrade, you only need to replace the XBMC.MyLibrary.jar in your current version with the one from the new version.

Also, this is compatible with Eden only (tested against Dec-10 build), if you're on Dharma, sorry I don't have time to fix that build.


- TheCasa - 2011-12-13

bradvido88 Wrote:
  1. Your dropbox\Tv Shows directory has not been added as a source in XBMC and had it's content type set to "TV Shows" (this should be done after some .mpg's exists in the dropbox). From your screenshot, it looks like the .mpg is in the library, so this probably isn't the issue.
I think this is the problem! They were setup as sources when the dropbox was empty! I'll try it when I get home and see if that resolves it.
Quote:
  • (most likely) - The path to your TV Shows directory in XBMC is different than the path you specified in the Config file. For example, one uses a network share and the other uses a local path. They have to be the same. I personally use a network share "\\onyx\\dropbox\\tvshows" in the config file, and in xbmc I set the source up as a smb path to the same directory.
    To troubleshoot this, can you post the contents of your sources.xml file (in XBMC's userdata directory) as well as your dropbox config in the Config.xml so I can compare.
  • I think I'm ok here, but I'll check that as well
    Quote:
  • The query to your SQL database to determine if the dummy .mpg files is in the library is failing. However we would see this in the logs as an ERROR and I don't think we've seen that.
  • You're right, no errors. I'll post the log when I get home if resetting the sources doesn't solve it.
    Quote:As a disclaimer, my testing is limited to a Windows environment so things may be slightly different if your running a different OS
    I'm running windows, so that's cool.