XBMC Library Updater Addon

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
robweber Offline
Posting Freak
Posts: 902
Joined: Sep 2009
Reputation: 27
Post: #1
One of XBMC's most powerful features are it's Video and Music libraries. With most people updating their libraries daily, having to restart XBMC or manually re-scan for content can be a chore. While cron jobs or batch scripts can take care of the update process for you, this add-on keeps the library management within XBMC without having to reply on another method.

I've never started a thread for the XBMC Library Updater addon before - mostly because the code behind the addon was so simple I didn't really think it was necessary. Every once in a while though I come across questions or suggestions to it that are hard to keep track of since there isn't a unified place to keep them.

With Eden coming the XBMC Library Updater has been re-written to use the new xbmc.service extension point. This allows more functionality than the original addon had and may allow for some modifications that would make it more feature-rich than the Dharma versions. Below is a link to the google code page for this addon and I encourage anyone with issues, comments, or suggestions to use this post as a place to put them.

[color=#0000CD][size=medium]Gotham Testing - https://github.com/robweber/xbmclibrarya...ze][/color]

https://github.com/robweber/xbmclibraryautoupdate

thanks to everyone using this addon, it has certainly helped keep my library up to date!

Version 0.7.3

added support for multipathed sources in verify sources

updated xbmc python version

Version 0.7.2

Fixed error with cleaning library

Version 0.7.1

Started using utils.py as common framework for logging/notifications

fixed unicode error in showing notifications - thanks koying!

added storage server (common plugin cache) as a dep

Version 0.7.0

increment version to keep Eden branch separate

Version 0.6.5

added setting to prompt user before doing scheduled clean. Defaults to False.

added ability to schedule cleaning separate

additional translations

Version 0.6.4

- added strings from notifications and logging into the strings.xml files

- strings reorganization (affected translations)

- updated to french translation file

Version 0.6.3

- added French translation file, thanks to foX aCe

Version 0.6.2

- needed a catch in case the last_run.txt file is blank, or has non-integer data. thanks to mmounirou for catching this

Version 0.6.1

- added 2 more custom library path options

Version 0.6.0

- added "Cleaning" category to schedule a clean operation of the music/video databases. This operation can happen immediately after a scan or once per day/week/month. Verifying source paths before a clean is also supported

Version 0.5.9

- added a custom video path option to only scan a specific video path instead of the entire library

Version 0.5.8

- added a 1 minute delay timer before running a scan if XBMC has just exited playback. This should help in scenerios where ending media viewing results in an immediate scan that the user didn't want.

Version 0.5.7

- merged 'standard' and 'advanced' usage to follow more of the same codebase. Now the standard timer uses a cron expression as well and will start at the top of every hour

Version 0.5.6

-updated the manual run interface to include information about when the updater will run again

Version 0.5.5

-changed cron expression library. Croniter will allow iterating through the cron expressions and show the next update time

-added methods to display a "countdown" when the next update will occur, and settings to display notifications

Version 0.5.4

-fixed issue with startup timer, thanks stevenD

Version 0.5.2

-fixed os import error

Version 0.5.1

-merged changes from pkscuot's branch.
-rounds last_run to top of the minute (timer executes at 00 not anywhere in minute)
-creates addon data directory if it doesn't exist

Version 0.5.0

-major changes to settings, split them by General and Timer category
-Advanced timer functions now add the ability to do cron-like scheduling of the update process, thanks to pkscuot for the timer ideas
-option to skip during media playback or run the update anyway

Version 0.4.1

-added extra setting for a "startup delay" timer. This will only affect the addon when xbmc starts.
-the last running time is now set to a variable so that manual updates will reset the timer, and system resets will start the service where it left off

Version 0.4.0

- Had a user suggestion to allow for a manual launch of the process as well as the service. Since the service point will ALWAYS launch on startup the manual option will kick off the library update process.

Version 0.3.9

- running video and music scans side by side never really worked. Now checks if scan is running and waits until complete before running the next scan.

Version 0.3.5

- now runs as a service instead of needing the autoexec.py file
- removed sample autoexec.py

Version 0.3.1

- fixed a really stupid indent error
- added cancelalarm call in case run more than once
(This post was last modified: 2014-01-20 21:51 by robweber.)
find quote
dynaflash Offline
Senior Member
Posts: 167
Joined: Nov 2010
Reputation: 0
Post: #2
Rob, as per http://forum.xbmc.org/showthread.php?p=9...post985289 I have ceased working on my implementation of your addon on my own as since then it was updated by yourself. I agree it is a very valuable addition to any decent xbmc implementation!

I mainly reworked it after dharma because at that time it wasn't working with pre - eden. One issue I did have with the new service addon as opposed to as a program is that there is no way I know of to invoke it manually. though maybe I am missing something. Anyway since that post I have gone back to using your new revision. Thanks and please realize I was not attempting to overtake your work but build on it as I was not sure after dharma at that time you would continue on with it.

Once again, Kudos!
find quote
TheStretchedElf Offline
Senior Member
Posts: 202
Joined: Jan 2011
Reputation: 0
Thumbs Up   
Post: #3
Thanks for this, just what I need for my daughters new ATV2.
find quote
vikjon0 Offline
---
Posts: 2,500
Joined: Apr 2009
Reputation: 8
Location: Sweden
Post: #4
Quote:there is no way I know of to invoke it manually

I have though about that. Is it not possible to have multiple extension points?
find quote
Martijn Offline
Team Kodi
Posts: 11,577
Joined: Jul 2011
Reputation: 167
Location: Dawn of time
Post: #5
vikjon0 Wrote:I have though about that. Is it not possible to have multiple extension points?

Yes it is

PHP Code:
<extension point="xbmc.python.script" library="default.py">
     <
provides>executable</provides>
</
extension>
<
extension point="xbmc.service" library="default.py" start="startup|login"/> 

Always read the XBMC online-manual, FAQ and search the forums before posting.
Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting, make sure you read this first

For your mediacenter artwork go to
[Image: fanarttv.png]
find quote
vikjon0 Offline
---
Posts: 2,500
Joined: Apr 2009
Reputation: 8
Location: Sweden
Post: #6
Quote:Yes it is
Thanks!
find quote
robweber Offline
Posting Freak
Posts: 902
Joined: Sep 2009
Reputation: 27
Post: #7
dynaflash Wrote:I mainly reworked it after dharma because at that time it wasn't working with pre - eden. One issue I did have with the new service addon as opposed to as a program is that there is no way I know of to invoke it manually. though maybe I am missing something. Anyway since that post I have gone back to using your new revision. Thanks and please realize I was not attempting to overtake your work but build on it as I was not sure after dharma at that time you would continue on with it.

Once again, Kudos!

No problem! I wasn't really focused on Eden until late in its development so I can understand wanting something that will work with the nightly builds and such. without a "proper" way for people to even ask for requests there was no way of knowing I would continue doing anything with it. Your idea with the scanning definitely got me thinking of how this could be incorporated into a more functional addon.

Martijn Wrote:Yes it is

Code:
<extension point="xbmc.python.script" library="default.py">
     <provides>executable</provides>
</extension>
<extension point="xbmc.service" library="default.py" start="startup|login"/>

I had no idea you could do multiple points like that. This is something that should be added to the wiki. I'll try to get this put into the Updater addon soon so it can be launched manually. Since the service will always launch the manual option will be more of a quick way to launch the normal "Scan for Updates" functionality; plus reset the Library Updater timer. Thanks for the info!
find quote
Martijn Offline
Team Kodi
Posts: 11,577
Joined: Jul 2011
Reputation: 167
Location: Dawn of time
Post: #8
robweber Wrote:I had no idea you could do multiple points like that. This is something that should be added to the wiki. I'll try to get this put into the Updater addon soon so it can be launched manually. Since the service will always launch the manual option will be more of a quick way to launch the normal "Scan for Updates" functionality; plus reset the Library Updater timer. Thanks for the info!

'Several other add-ons also use it and one even acts like it own repo Wink

The wiki is user driven so you can always add things yourself when you think they are missing Smile

Always read the XBMC online-manual, FAQ and search the forums before posting.
Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting, make sure you read this first

For your mediacenter artwork go to
[Image: fanarttv.png]
find quote
michaeldecharon Offline
Senior Member
Posts: 144
Joined: May 2009
Reputation: 0
Location: Netherlands
Post: #9
What about after updating the library execute clean library.

Also nice, update the library when content is added to a specific folder and clean library when content is deleted in a specific folder.
find quote
robweber Offline
Posting Freak
Posts: 902
Joined: Sep 2009
Reputation: 27
Post: #10
michaeldecharon Wrote:What about after updating the library execute clean library.

Also nice, update the library when content is added to a specific folder and clean library when content is deleted in a specific folder.

Regarding the clean library functionality please see this page on the wiki: http://wiki.xbmc.org/index.php?title=Adv...library.3E You can add a setting to your advancedsettings.xml file so that the library always cleans when it updates. This is the easiest way to make sure this happens without having to separately click the "Clean Library" setting under Video Settings.

As far as monitoring specific folders; the problem implementing that type of functionality is exactly why this addon uses the built in update functionality of XBMC and runs on a timer. Since files can be connected with many different types of protocols (NFS, Samba, local hard disks, etc) it is very hard to develop a cross platform way to monitor for files (event driven updates). The most common way to do this is with some type of folder polling (watchdog services); but those are inefficient. Writing something like that in python would basically duplicate the actions already happening via the XBMC Update command (look through each folder for new files). There are a lot of forum posts on this but the one I find the most interesting is this one: http://forum.xbmc.org/showthread.php?tid=66348 . Several XBMC developers explain the issue with having an event driven system very well.

Generally running the update processes every few hours works fairly well for most people. The scan process is usually pretty quick and most of the time happens when you aren't even using the system. I do agree though that a cross platform, event driven, system would be the best solution eventually.
find quote
Martijn Offline
Team Kodi
Posts: 11,577
Joined: Jul 2011
Reputation: 167
Location: Dawn of time
Post: #11
@robweber

I agree that running it every xx hours should suffice.
That aside could something like this work cross platform?:

PHP Code:
try:
                    
date os.path.getmtimebase_path )
                    print 
date
                except
:
                    
date 0
                
# we only refresh if it's been more than a day, 24hr * 60min * 60sec
                
refresh = ( ( time.time() - ( 24 60 60 ) ) >= date 

Found it in a script somewhere

Always read the XBMC online-manual, FAQ and search the forums before posting.
Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting, make sure you read this first

For your mediacenter artwork go to
[Image: fanarttv.png]
find quote
robweber Offline
Posting Freak
Posts: 902
Joined: Sep 2009
Reputation: 27
Post: #12
Martijn Wrote:@robweber

I agree that running it every xx hours should suffice.
That aside could something like this work cross platform?:

PHP Code:
try:
                    
date os.path.getmtimebase_path )
                    print 
date
                except
:
                    
date 0
                
# we only refresh if it's been more than a day, 24hr * 60min * 60sec
                
refresh = ( ( time.time() - ( 24 60 60 ) ) >= date 

Found it in a script somewhere

From what I can see that would flag a folder to be refreshed in the event that its modified date was greater than a day. I'm guessing you could then code the refresh value to trigger if the folder should be scanned or not?

I found the following related to how the "os.path" module in Python functions:

Quote:The os.path module is always the path module suitable for the operating system Python is running on, and therefore usable for local paths.

So while it would probably work for local paths, sources that are connected via samba, nfs, etc probably wouldn't return the right values. The docs I found even said that unc paths on windows systems wouldn't work.

Thinking about it I'm still not sure if doing separate polling of the individual paths and then specifying (via the updatelibrary() function) that path to update is worth the extra effort. The built in xbmc scanner seems to do a good job of detecting new files very quickly; maybe if you had lots of sources it would be worth only scanning folders that have changed.

I think ideally it would be nice if the built in XBMC scanner had a cross platform way of checking for changed files so that it could selectively know what folders to scan, this would put the burden on the core xbmc process rather than having something like an addon do the work. Again, I'm sure the problem is the cross platform piece of it.

Nice find with that piece of code. I'm by no means a Python expert so I'll have to remember that function in case I ever need something similar.
find quote
Martijn Offline
Team Kodi
Posts: 11,577
Joined: Jul 2011
Reputation: 167
Location: Dawn of time
Post: #13
That was my guess also that it only works for local path.
I don't find it useless to update on every 'change'. Once or two times a day would be more than sufficient in my case.
Once you made is a service/program they also run it theirselves if they want.
Same as we did with Artwork Downloader.

Maybe you can even trigger it when you finished the library update Wink
On the other hand we offer a one time a day run and they can start it manually
I already looked into it briefly once but couldn't find a way to detect if it finished the update.

Always read the XBMC online-manual, FAQ and search the forums before posting.
Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting, make sure you read this first

For your mediacenter artwork go to
[Image: fanarttv.png]
(This post was last modified: 2012-01-12 23:02 by Martijn.)
find quote
GJones Offline
Fan
Posts: 438
Joined: Nov 2009
Reputation: 0
Post: #14
I notice that the service is running an update on startup. Is there any way that we could delay the scan so that it does not run at startup? In particular, I am using the watchlist script and it hits the database heavily at startup. It would be nice to have an option to delay the first run of this script for 1,2 or 5 minutes after startup to let the user get settled into the interface before running the update.
find quote
GJones Offline
Fan
Posts: 438
Joined: Nov 2009
Reputation: 0
Post: #15
GJones Wrote:I notice that the service is running an update on startup. Is there any way that we could delay the scan so that it does not run at startup? In particular, I am using the watchlist script and it hits the database heavily at startup. It would be nice to have an option to delay the first run of this script for 1,2 or 5 minutes after startup to let the user get settled into the interface before running the update.

As a suggestion, the last_run could be written to file for comparison so that when the service started (like on an xbmc restart), it could determine if there was a need to update the library on startup.

Additionally, a sleep time after startup could be specified in service.py at the beginning of the first while loop. This would allow the user to specify a period after xbmc startup to wait before updating the library. This would allow less contention between the library update, watchlist and recentlyadded scripts. On large libraries, the impact of all 3 running simultaneously is somewhat disruptive.
find quote
Post Reply