Kodi Community Forum
MySQL insert script (CLI scraper) - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: General Support (https://forum.kodi.tv/forumdisplay.php?fid=111)
+---- Forum: OS independent / Other (https://forum.kodi.tv/forumdisplay.php?fid=228)
+---- Thread: MySQL insert script (CLI scraper) (/showthread.php?tid=133820)

MySQL insert script (CLI scraper) - aTTilaz - 2012-06-12

Hi there,

Before i went off to write my own solution I thought i would see if something like this existed already. Now that I have a central XBMC MySQL database on my server I want to have a CLI script that I can pass a filename argument (like a tv episode) have it download all the relevant data and insert in the DB (same as what the XBMC client does basically). This way no xbmc client would have to scan and retrieve a new eps data, they would just exist in the DB already.

Essentially I guess it would be a CLI scraper. Anyone aware of anything like this? Sorry if this is the wrong place to post this question.

EDIT: Thought I might do it as CPAN perl module and utilize MySQL and TVDB modules to access the db and reference thetvdb respectively.

Thanks

RE: MySQL insert script (CLI scraper) - Ned Scott - 2012-06-12

Not that I know of, but let us know if you give this a shot, because a lot of people would be interested in it.

RE: MySQL insert script (CLI scraper) - aTTilaz - 2012-06-13

Thanks for your reply mate.

Yeah definately going to do it if theres nothing else like this. I already have automatic downloading of my shows via an rss feed, then have written scripts to rename and sort new content into there appropriate directories. These scripts already pull all the show data from the TVDB via its API. Would use this existing logic to get all the ep metadata and create the relevant entries in the XBMC MySQL DB. Everything done on the server completely independent of any XBMC client. Nice and neat. Just fire up one of my XBMC machines on my network and new show content already there and ready to watch.

RE: MySQL insert script (CLI scraper) - notyjoey - 2012-06-13

(2012-06-13, 00:11)aTTilaz Wrote: Thanks for your reply mate.

Yeah definately going to do it if theres nothing else like this. I already have automatic downloading of my shows via an rss feed, then have written scripts to rename and sort new content into there appropriate directories. These scripts already pull all the show data from the TVDB via its API. Would use this existing logic to get all the ep metadata and create the relevant entries in the XBMC MySQL DB. Everything done on the server completely independent of any XBMC client. Nice and neat. Just fire up one of my XBMC machines on my network and new show content already there and ready to watch.

Care to share the script which does your downloading? I've been using Sickbead + SABnzbd but wouldn't mind trying something a little more lightweight.

RE: MySQL insert script (CLI scraper) - aTTilaz - 2012-06-13

(2012-06-13, 00:15)notyjoey Wrote: Care to share the script which does your downloading? I've been using Sickbead + SABnzbd but wouldn't mind trying something a little more lightweight.

It is a torrent based solution so probably no good for you. If you are interested though, its a web frontend for Transmission i helped develop with a friend. Have a look at the following if you like:

http://sourceforge.net/projects/auto-for-trans/

Its monitors a user specified RSS feed for matches to a specified list (inclusions and exclusions), automatically downloads and sorts. Can automatically remove torrent/data at pre-determined days of seeding and days of inactivity. Sends user activty notifications via email.

RE: MySQL insert script (CLI scraper) - notyjoey - 2012-06-13

(2012-06-13, 00:46)aTTilaz Wrote:
(2012-06-13, 00:15)notyjoey Wrote: Care to share the script which does your downloading? I've been using Sickbead + SABnzbd but wouldn't mind trying something a little more lightweight.

It is a torrent based solution so probably no good for you. If you are interested though, its a web frontend for Transmission i helped develop with a friend. Have a look at the following if you like:

http://sourceforge.net/projects/auto-for-trans/

Its monitors a user specified RSS feed for matches to a specified list (inclusions and exclusions), automatically downloads and sorts. Can automatically remove torrent/data at pre-determined days of seeding and days of inactivity. Sends user activty notifications via email.

Hey now, it would save me using news groups AND I have transmission installed on my box too! I'll definitely check it out Wink


RE: MySQL insert script (CLI scraper) - vicbitter - 2012-06-13

With the latest code from head you now create a libxbmc.so shared library that can be used to create your own CLI (or server-based) XBMC applications... Smile

Here is a sample Video Library scanner application that uses libxbmc.so to update XBMCs Video Library...

http://pastebin.com/WpEvRTws

Enjoy!


RE: MySQL insert script (CLI scraper) - jhsrennie - 2012-06-13

Is this possible in Windows? I can't find any XBMC.dll in the build directories.

RE: MySQL insert script (CLI scraper) - vicbitter - 2012-06-13

(2012-06-13, 11:13)jhsrennie Wrote: Is this possible in Windows? I can't find any XBMC.dll in the build directories.

I am not sure as I use Linux but the libxbmc.so is not built by default... You need to run ./configure --enable-shared-lib

Maybe give it a try on Windows and see what gets built?

RE: MySQL insert script (CLI scraper) - aTTilaz - 2012-06-14

(2012-06-13, 10:43)vicbitter Wrote: With the latest code from head you now create a libxbmc.so shared library that can be used to create your own CLI (or server-based) XBMC applications... Smile

Here is a sample Video Library scanner application that uses libxbmc.so to update XBMCs Video Library...

http://pastebin.com/WpEvRTws

Enjoy!

Thanks vicbitter, is there any documentation that explains this process. Not to familar with C++

RE: MySQL insert script (CLI scraper) - vicbitter - 2012-06-14

(2012-06-14, 00:31)aTTilaz Wrote:
(2012-06-13, 10:43)vicbitter Wrote: With the latest code from head you now create a libxbmc.so shared library that can be used to create your own CLI (or server-based) XBMC applications... Smile

Here is a sample Video Library scanner application that uses libxbmc.so to update XBMCs Video Library...

http://pastebin.com/WpEvRTws

Enjoy!

Thanks vicbitter, is there any documentation that explains this process. Not to familar with C++

Unfortunately, it doesn't seem so at the moment, I had to trawl through the code and built the sample application based on my discoveries... I think the main code is generic enough to use and the only part(s) that would need changing is dependent on the internal APIs you want to use...

I hope that makes sense?



RE: MySQL insert script (CLI scraper) - Hansaplast - 2012-06-24

I'd love to see a script that could also run on a NAS (QNAP, etc) or the "server" that is running the MySQL database.
This way the NAS (or server) could do the scraping for XBMC ... Smile
I'm by no means experienced enough to build my own scraper though Sad