[RELEASE - BETA] service.datasync

  Thread Rating:
  • 1 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
DecK Offline
Senior Member
Posts: 136
Joined: Nov 2008
Reputation: 4
Location: Round Rock, TX
Post: #16
I've done some work on the JSON interface, but have hit a roadblock in the Files.GetDirectory method. I am not able to see the last modified timestamp on a file, which is a key requirement for incremental syncing.

In the mean time I've been playing around with the ftplib as a possible alternative for gettting the file. That's working (not yet committed, will do so tonight) but I think I will to extentd it to use SFTP (SSH) before I go beta. I don't wnat users to have to be running an FTP server, unless I can't get anything else to work.
find quote
robweber Offline
Posting Freak
Posts: 882
Joined: Sep 2009
Reputation: 26
Post: #17
Awesome. Glad to know you're still working on this idea.

I agree that having users configure an outside service is kind of a pain - keeping everything within the addon (and xbmc) as much as possible will really make it useful. There would be countless ways (rsync, scripts, etc) to do this using just the OS, having it as a pure xbmc addon with no outside configuration would be ideal.
find quote
Bstrdsmkr Offline
Posting Freak
Posts: 803
Joined: Oct 2010
Reputation: 17
Post: #18
This is a W.A.G., but look into using the vfs interface. Something like http://192.168.1.2:8080/vfs/special://home/xbmc.log

I'm doing this from memory so that address might not be 100%, but I think that should give you the log file via http. Maybe you get lucky and can get the file details from one of the headers?

Alternatively, I think you could write a quick python script to be stored in the addon's folder. The script creates a file with a rolling list of files that have changed. Your client side addon calls the script (to make sure the list is up to date), downloads this file, determines which files it needs, then grabs them.

Hope that points you in the right direction!
find quote
DecK Offline
Senior Member
Posts: 136
Joined: Nov 2008
Reputation: 4
Location: Round Rock, TX
Post: #19
(2012-04-18 23:32)Bstrdsmkr Wrote:  Alternatively, I think you could write a quick python script to be stored in the addon's folder. The script creates a file with a rolling list of files that have changed. Your client side addon calls the script (to make sure the list is up to date), downloads this file, determines which files it needs, then grabs them.

Great idea! I already have a setting for Server (the box to pull from) versus Client (the box doing the pull). I can add this script and run it only on the server side and make it available to all the clients for the compare.
find quote
DecK Offline
Senior Member
Posts: 136
Joined: Nov 2008
Reputation: 4
Location: Round Rock, TX
Post: #20
All
I've updated the first post to reflect that fact that I am now in Beta. Testers would be greatly appreciated at this point.


Cheers
Declan
find quote
Bstrdsmkr Offline
Posting Freak
Posts: 803
Joined: Oct 2010
Reputation: 17
Post: #21
Awesome, now you can start work on storing a copy on Google Drive }=)
find quote
RockDawg Online
Posting Freak
Posts: 1,429
Joined: Jun 2005
Reputation: 1
Post: #22
I am not able to get this working. I installed the addon on two of my machines. My desktop is a Win7 machine using local Thumbnails with no symlink or pathsubstitution and I set that one up as the server sharing only Thumbnails. The second machine is running openelec with no shared thumbs (just an empty local Thumbnails fodler) and the addon is configured as a client with the IP set the same as my server's IP. The port is set to 8090 which is what the server computer has set in settings>network>services>allow control of XBMC via HTTP>port. I have the Data to Sync set to Thumbnaiils same as the server and the schedule is set to hourly and II set the starting hour and minute to the next closest time and nothing ever seems to happen. One thing that seems a bit odd, is the the Next Run is greyed and only ever reads "Friday, 01 Jan 1971 00:00:00". I'm not sure if that matter.

Nothing shows in a XBMC debug log. Should there be a progress/running indicator or a notification when it starts? Any ideas what's wrong?


I am really looking forward to this addon. Thanks for doing this.
find quote
DecK Offline
Senior Member
Posts: 136
Joined: Nov 2008
Reputation: 4
Location: Round Rock, TX
Post: #23
This is a service addon, which are not enabled when installed, you need to reboot both machines before the service will start. I should have put that in the first post, sorry.

Can you try rebooting both (server first) and see what happens.
The first thing that should happen is on the client as soon as you reboot, the next run should be updated to the correct time and date based on the setting you have entered. 1/1/71 is my default date, which will be updated as soon as the add-on runs. The fact that it is still 1/1/71 suggests that the add-on isn't yet running (at least on the client side).

If it still isn't working, please post a debug log from both machines.
find quote
RockDawg Online
Posting Freak
Posts: 1,429
Joined: Jun 2005
Reputation: 1
Post: #24
I rebooted both machines and now the client PC gives an error on start up saying the script failed. Here are the logs:

Server PC log: http://pastebin.com/Rgqzd9Np

Client PC log: http://pastebin.com/MsMv5HTV
find quote
bosel Offline
Junior Member
Posts: 49
Joined: Jun 2010
Reputation: 0
Post: #25
Just wanted to let you know that I think this is great, very useful indeed.

After a quick fix to the error referred to above, the whole thing seems to work without problem.
One thing I've noticed is that shutdown of xbmc seems to take quite a bit of time since I've started using this, but i've not yet tried to figure out why.

Anyway, thanks!
(This post was last modified: 2012-04-29 21:20 by bosel.)
find quote
RockDawg Online
Posting Freak
Posts: 1,429
Joined: Jun 2005
Reputation: 1
Post: #26
(2012-04-29 21:19)bosel Wrote:  After a quick fix to the error referred to above, the whole thing seems to work without problem.

Are you talking about the error I mentioned? If so, what was the quick fix?
find quote
Bstrdsmkr Offline
Posting Freak
Posts: 803
Joined: Oct 2010
Reputation: 17
Post: #27
(2012-04-29 21:19)bosel Wrote:  Just wanted to let you know that I think this is great, very useful indeed.

After a quick fix to the error referred to above, the whole thing seems to work without problem.
One thing I've noticed is that shutdown of xbmc seems to take quite a bit of time since I've started using this, but i've not yet tried to figure out why.

Anyway, thanks!

This is probably due to the service sleeping. For some reason, XBMC won't forcibly kill a sleeping thread when it exits. It will wait for the thread to wake, finish and then exit
find quote
bosel Offline
Junior Member
Posts: 49
Joined: Jun 2010
Reputation: 0
Post: #28
@RockDawg

I think it was, yes.

Just get the current version:
https://github.com/declankenny/XBMC_datasync

That should stop the script failing
find quote
RockDawg Online
Posting Freak
Posts: 1,429
Joined: Jun 2005
Reputation: 1
Post: #29
The newer version stops the error message, but now the log says:

Code:
18:32:29 T:3018127104  NOTICE: Samba is idle. Closing the remaining connections
18:35:49 T:2995223360  NOTICE: Data Sync Service:: Starting Sync
18:35:49 T:2995223360  NOTICE: Data Sync Service:: Cannot Connect to Server, please check IP Address and Port Number
18:35:49 T:2995223360  NOTICE: Data Sync Service:: No Files to Sync
18:35:49 T:2995223360  NOTICE: Data Sync Service:: Finished Sync
18:35:49 T:2995223360  NOTICE: Data Sync Service:: Next Run is Sun, 29 Apr 2012 19:35:00

I double-checked the config and the IP address is the correct address for my server PC. I have the port number the same as it's set in settings>network>services>allow control of XBMC via HTTP>port on the server PC. That's the correct port number, right?
find quote
robweber Offline
Posting Freak
Posts: 882
Joined: Sep 2009
Reputation: 26
Post: #30
(2012-04-29 23:28)Bstrdsmkr Wrote:  
(2012-04-29 21:19)bosel Wrote:  Just wanted to let you know that I think this is great, very useful indeed.

After a quick fix to the error referred to above, the whole thing seems to work without problem.
One thing I've noticed is that shutdown of xbmc seems to take quite a bit of time since I've started using this, but i've not yet tried to figure out why.

Anyway, thanks!

This is probably due to the service sleeping. For some reason, XBMC won't forcibly kill a sleeping thread when it exits. It will wait for the thread to wake, finish and then exit

Not sure of the inner workings here, but reducing the amount of a time a thread is sleeping will allow it to check for the xbmc.abortRequested command a lot faster. Many services have the thread sleep time set to 10 seconds or less to avoid hangs when trying to exit. it is especially troublesome when you want to close XBMC and then try to open it but the process hasn't exited yet.

There may be some tweaking that can be done with the sleep times as long as it doesn't disrupt the workings of the script. Based on my quick look it won't affect the clients since they check for a sync time inside of the loop. The server though may be trying to parse the file list each time so there may need to be an extra check in there (similar to the client) if the thread time is decreased to avoid a lot of file IO for no reason. Overall this addon is coming along really well, been working in the few tests I've done.
find quote