[Release] XBMC Backup

  Thread Rating:
  • 5 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
PatK Offline
Donor
Posts: 3,423
Joined: Mar 2010
Reputation: 75
Location: Canada
Post: #31
Love the concept.. I'm always backing up, yet never current.. weeks old, this would seem to take the drudgery outo of this task... but alas as others have noted.

here's a snip of the critical area... the pastebin was a bit too big to post.

11:26:53 T:3600 INFO: -->Python script returned the following error<--
Error Type: <type 'exceptions.UnicodeEncodeError'>
11:26:53 T:3600 ERROR: Error Contents: 'ascii' codec can't encode character u'\xf7' in position 50: ordinal not in range(128)
11:26:53 T:3600 ERROR: Traceback (most recent call last):
File "C:\Users\PatK\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 188, in <module>
11:26:53 T:3600 ERROR: backup.run()
File "C:\Users\PatK\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 119, in run
self.syncFiles()
File "C:\Users\PatK\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 132, in syncFiles
self.fileManager.createFileList(self.Addon)
File "C:\Users\PatK\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 26, in createFileList
self.walkTree(self.walk_path + "addons/")
File "C:\Users\PatK\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 65, in walkTree
self.addFile(filePath)
File "C:\Users\PatK\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 70, in addFile
self.fHandle.write(str(filename) + "\n")
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf7' in position 50: ordinal not in range(128)
11:26:53 T:3600 INFO: -->End of Python script error report<--

I look forward with baited breath fr an update Smile
find quote
loopez Offline
Junior Member
Posts: 19
Joined: Jan 2011
Reputation: 0
Post: #32
love idea - got mq3 and script error.
find quote
robweber Online
Posting Freak
Posts: 889
Joined: Sep 2009
Reputation: 27
Post: #33
quick update, i've looked over a lot of stuff tonight and think i have a solution. each of the errors i'm seeing are a result of ascii encoding errors when writing the "restore.txt" file that is used to write the files between destinations. I've removed this entirely and now pass the filenames around internally via a filename list instead. I'm going to test over the next day using different combinations (including MQ) but shoulld release to the repo soon.

Everyone please hang tight, hopefully this will fix things up.
find quote
matt77303 Offline
Fan
Posts: 359
Joined: Dec 2011
Reputation: 6
Post: #34
any updates?

Matt

CPU - AMD Athlon™ II X2 Dual-Core 3.4Ghz
MB - ASUS M5A78L-M/USB3, Nvidia 610 2GB
RAM - 4GB DDR3 Dual-Channel
OS - XBMC Frodo
TV - Panasonic TX-P50S21B
Home Cinema - Pioneer VSX-S300 5.1
Remote - Harmony one
find quote
schumi2004 Offline
Posting Freak
Posts: 1,376
Joined: Aug 2011
Reputation: 14
Post: #35
I pulled the latest git today and backup is running with MQ3 skin enabled, so far so good.
But then i looked at what exactly is being backuped and i still miss a few things form userdata folder.

Folders not backuped are:
- keymaps
- peripheral_data (not that important i guess)

EDIT:
I checked code and noticed that those 2 folders aren't defined so maybe that's why it's not backuped.

In OpenELEC there is also a config folder outside userdata folder , is there a way i can add this?

I also noticed it's doing a full update every time, maybe doing incremental updates is better?
(This post was last modified: 2012-05-23 10:51 by schumi2004.)
find quote
robweber Online
Posting Freak
Posts: 889
Joined: Sep 2009
Reputation: 27
Post: #36
(2012-05-23 10:37)schumi2004 Wrote:  I pulled the latest git today and backup is running with MQ3 skin enabled, so far so good.
But then i looked at what exactly is being backuped and i still miss a few things form userdata folder.

Folders not backuped are:
- keymaps
- peripheral_data (not that important i guess)

EDIT:
I checked code and noticed that those 2 folders aren't defined so maybe that's why it's not backuped.

In OpenELEC there is also a config folder outside userdata folder , is there a way i can add this?

I also noticed it's doing a full update every time, maybe doing incremental updates is better?

Glad that it is working. I should probably add the keymaps folder at the least as some people may be making custom keymap entries there. As far as OpenELEC support I could add an extra toggle for OpenELEC specific files in the settings. Are there any other folders under the user profile that OpenELEC creates that may be important?

As far as the incremental updates - I totally agree that would be the best. The caveat is that there is no reliable way to get the timestamps for files from remote systems. The XBMC JSON RPC does not return them, and the built in python modules only do local files/folders very well. If I was to restrict the addon to something like only SMB shares I may be able to find a smb python library that would get the timestamps, but I'd like to use the built in xbmc protocols where possible and the functionality for that just isn't there yet. As such it can still take quite a while if you have multiple skins or lots of thumbnails/fanart to copy everything over.

I'm holding off submitting this to the addons repo just yet, I have found an issue with the way the JSON Files.GetDirectory command returns filenames for zip files that garbles them. On github I have added a catch to filter these files out, but am waiting back from some info on if this can be fixed before submitting.
find quote
schumi2004 Offline
Posting Freak
Posts: 1,376
Joined: Aug 2011
Reputation: 14
Post: #37
I'll check OpenELEC for you.

If incremental is hard to accomplish maybe add time stamps to the backup folders?
Just a thought Wink
find quote
robweber Online
Posting Freak
Posts: 889
Joined: Sep 2009
Reputation: 27
Post: #38
(2012-05-23 22:59)schumi2004 Wrote:  I'll check OpenELEC for you.

If incremental is hard to accomplish maybe add time stamps to the backup folders?
Just a thought Wink

Thanks, I'll add the OpenELEC stuff before submitting to the repo, shouldn't be too hard. I got an answer on the zip file paths so that is sorted out.

What do you mean by add time stamps to the backup folders? I'd be happy to consider something but the different ways I've looked at it all require that I can gather some type of timestamp information from the backup location once the files have been sent, which I can't do. Once the files are in the backup local all I can get from them are name and full file path.

The one thing I can think of that might work would be to create a list of files and timestamps that is created locally when the backup job runs, and then saved to the addon_data folder (as a txt or xml file). Upon the next run the timestamps in the file could be used to compare the local file against what it was the last time the backup ran. The restore side would just restore everything, as I think it should. Is that more like what you were thinking? It would require the creation of a map data structure to compare against (which would eat up some processing time) but the trade off would be not transferring large files that may not have changed, which would be worth the extra overhead I think.
(This post was last modified: 2012-05-23 23:10 by robweber.)
find quote
schumi2004 Offline
Posting Freak
Posts: 1,376
Joined: Aug 2011
Reputation: 14
Post: #39
I tried latest git on my OpenELEC system and it fails with the following error.


23:49:20 T:140627724441344 NOTICE: Thread Background Loader start, auto delete: false
23:49:20 T:140627724441344 NOTICE: Previous line repeats 1 times.
23:49:20 T:140627724441344 WARNING: CreateLoader - Unsupported protocol(addons) in addons://more/executable.tbn
23:49:25 T:140627724441344 NOTICE: Thread XBPyThread start, auto delete: false
23:49:25 T:140627724441344 NOTICE: -->Python Interpreter Initialized<--
23:49:25 T:140627724441344 ERROR: Error Type: <type 'exceptions.IndentationError'>
23:49:25 T:140627724441344 ERROR: Error Contents: ('unindent does not match any outer indentation level', ('/storage/.xbmc/addons/script.xbmcbackup/default.py', 36, 64, ' self.walkTree(self.walk_path + "userdata/Database")\n'))
23:49:25 T:140627724441344 ERROR: IndentationError: ('unindent does not match any outer indentation level', ('/storage/.xbmc/addons/script.xbmcbackup/default.py', 36, 64, ' self.walkTree(self.walk_path + "userdata/Database")\n'))

Don't understand what is going wrong.
Going to check it out.

EDIT:
My bad, mixed up spaces and tabs.

EDIT2:
root / # find / -name autostart.sh
/storage/.config/autostart.sh
root / # find / -name Lircmap.xml
/storage/.xbmc/userdata/Lircmap.xml
/storage/backup/backup/xbmc_backup/userdata/Lircmap.xml
/usr/share/xbmc/system/Lircmap.xml
root / #

This is the location of OE specific files.
/storage/.config
*.xml and *.sh

For some reason i keep getting errors, will check out tomorrow.


(2012-05-23 23:10)robweber Wrote:  
(2012-05-23 22:59)schumi2004 Wrote:  I'll check OpenELEC for you.

If incremental is hard to accomplish maybe add time stamps to the backup folders?
Just a thought Wink

Thanks, I'll add the OpenELEC stuff before submitting to the repo, shouldn't be too hard. I got an answer on the zip file paths so that is sorted out.

What do you mean by add time stamps to the backup folders? I'd be happy to consider something but the different ways I've looked at it all require that I can gather some type of timestamp information from the backup location once the files have been sent, which I can't do. Once the files are in the backup local all I can get from them are name and full file path.

The one thing I can think of that might work would be to create a list of files and timestamps that is created locally when the backup job runs, and then saved to the addon_data folder (as a txt or xml file). Upon the next run the timestamps in the file could be used to compare the local file against what it was the last time the backup ran. The restore side would just restore everything, as I think it should. Is that more like what you were thinking? It would require the creation of a map data structure to compare against (which would eat up some processing time) but the trade off would be not transferring large files that may not have changed, which would be worth the extra overhead I think.
And what if you checked file-size and date before overwriting? No need to keep track of files in a separate file.
Again just a thought.
(This post was last modified: 2012-05-24 09:52 by schumi2004.)
find quote
PatK Offline
Donor
Posts: 3,423
Joined: Mar 2010
Reputation: 75
Location: Canada
Post: #40
Ver .8 auto-update report some mixed results. Default settings script dies...(same kind of errors reported previously) exclude user addons and add-on data, and the script seems to work but with some extras... which don't seem to be part of the back-up, but did appear. http://pastebin.com/f9qc4YYA keep in mind these are folders... not files and within the folders are empty... with the exception of the 'userdata' folder which contains... the usual xml files and folders, the thumbnails are populated, and the database, all the rest are empty.. playlists included.

In my case, I'd just prefer a global back-up of the roaming directory, (at least for now) without the selective functions.

Eden 11, T! win7/64
find quote
dallasnights Offline
Fan
Posts: 487
Joined: Jan 2011
Reputation: 0
Post: #41
not working looking for files to backup at about 10 min in it causes XBMC to crash and returns to apple screen
Is there a FIX please advise

Thanks

Here is the log
11:36:19 T:165150720 NOTICE: script.xbmcbackup: Starting
11:36:19 T:165150720 NOTICE: script.xbmcbackup: Local Dir: /var/mobile/Library/Preferences/XBMC/
11:36:19 T:165150720 NOTICE: script.xbmcbackup: Remote Dir: smb://MEDIACENTER-HP/Users/Media Center/Documents/xbmc/
11:36:19 T:165150720 NOTICE: script.xbmcbackup: Remote Path exists - may have old files in it!
11:36:46 T:149860352 NOTICE: weather.wunderground - satellite image downloaded
11:36:47 T:149860352 NOTICE: weather.wunderground - available locations: 1
11:36:47 T:149860352 NOTICE: weather.wunderground - finished
11:37:53 T:110497792 NOTICE: Samba is idle. Closing the remaining connections
11:46:57 T:165150720 NOTICE: userdata/addon_data/
11:46:57 T:165150720 NOTICE: userdata/Database/
11:46:57 T:165150720 NOTICE: userdata/keymaps/
11:46:57 T:165150720 NOTICE: userdata/peripheral_data/
11:46:57 T:165150720 NOTICE: userdata/playlists/
11:46:57 T:165150720 NOTICE: userdata/plugin_data/
11:46:57 T:165150720 NOTICE: userdata/Thumbnails/
11:46:57 T:165150720 NOTICE: userdata/advancedsettings.xml
11:46:57 T:165150720 NOTICE: userdata/favourites.xml
11:46:57 T:165150720 NOTICE: userdata/guisettings.xml
11:46:57 T:165150720 NOTICE: userdata/LCD.xml
11:46:57 T:165150720 NOTICE: userdata/passwords.xml
11:46:57 T:165150720 NOTICE: userdata/profiles.xml
11:46:57 T:165150720 NOTICE: userdata/RssFeeds.xml
11:46:57 T:165150720 NOTICE: userdata/sources.xml
11:46:57 T:165150720 NOTICE: userdata/upnpserver.xml
(This post was last modified: 2012-05-27 18:54 by dallasnights.)
find quote
matt77303 Offline
Fan
Posts: 359
Joined: Dec 2011
Reputation: 6
Post: #42
hi mate

dont know what youve done, but just updated via git hub ands it working perfect.

Thanks

Matt

CPU - AMD Athlon™ II X2 Dual-Core 3.4Ghz
MB - ASUS M5A78L-M/USB3, Nvidia 610 2GB
RAM - 4GB DDR3 Dual-Channel
OS - XBMC Frodo
TV - Panasonic TX-P50S21B
Home Cinema - Pioneer VSX-S300 5.1
Remote - Harmony one
find quote
digitaltomj Offline
Fan
Posts: 343
Joined: May 2011
Reputation: 1
Location: Miami
Post: #43
yay its totally working now Smile
find quote
matt77303 Offline
Fan
Posts: 359
Joined: Dec 2011
Reputation: 6
Post: #44
is there a way to automate this?

ie, if i leave my computer on on a sunday night at 2am it will back it up to my server? say weekly?

Matt

CPU - AMD Athlon™ II X2 Dual-Core 3.4Ghz
MB - ASUS M5A78L-M/USB3, Nvidia 610 2GB
RAM - 4GB DDR3 Dual-Channel
OS - XBMC Frodo
TV - Panasonic TX-P50S21B
Home Cinema - Pioneer VSX-S300 5.1
Remote - Harmony one
find quote
robweber Online
Posting Freak
Posts: 889
Joined: Sep 2009
Reputation: 27
Post: #45
Thanks for the feedback everyone, nice to know the most current version (0.0.8) is working better for most people. Anyone still having problems please make sure you are using the latest version (check XBMC repo if not set to auto update). Also, it may be worth deleting your old backup directory and starting over to avoid any issues.

@PatK - the debug log you posted is not anything this addon has produced. Not sure if you meant to post this. XBMC logs where there is actually some kind of python error are most useful, I don't see any errors here.

@dallasnights - I don't have an appltTV so I can't say for certain that this addon works on it, although it uses default XBMC libraries so there is no reason it shouldn't (anyone else have any comments here?). The section of the log you posted does not have any errors in it, it looks like it built the file list just fine, could there be a network connectivity problem writing to your backup directory? Without a log that actually includes an error of some type I don't really know what to advise.

@matt77303 - If you want to automate stuff check out this addon: https://github.com/robweber/cronxbmc As you'll see in the disclaimer it is not 100% finished, but it has instructions on how to edit the cron.xml needed to schedule XBMC scripts to run as needed.
find quote
Post Reply