XBMC Community Forum
[Release] XBMC Backup - Printable Version

+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Help and Support (/forumdisplay.php?fid=33)
+--- Forum: Add-ons Help and Support (/forumdisplay.php?fid=27)
+---- Forum: Program Add-ons (/forumdisplay.php?fid=151)
+---- Thread: [Release] XBMC Backup (/showthread.php?tid=129499)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42


[Release] XBMC Backup - robweber - 2012-04-20 21:21

For a lot of different reasons (testing, new xbmc system, computer crashes) it is a good idea to backup your XBMC data. I've spent enough time copying my userdata folder around that I decided that being able to just click a button and send it on it's way was worth the time coding an addon for it.

Gotham Testing Branch: https://github.com/robweber/xbmcbackup/tree/gotham-dev

The XBMC Backup Addon

This is a real quick way to export your XBMC data to another folder or network share for backup purposes. Scheduling backups and having a max number of archived backups are also configurable options. This allows you to use the addon as a completely hands off backup solution. The types of files you can backup from XBMC include:
  • User Addons - Addon's you've downloaded
  • Addon Data - Settings associated with your addons
  • Database - The SQLite Databases
  • Playlists - Your playlist folder
  • Thumbnails/Fanart - downloaded images
  • Config Files - sources.xml, guisettings.xml, advancedsettings.xml, etc
  • Up to 2 non-xbmc directories on your local system

Since this addon is using the xbmcvfs python module to actually do the file copy you should be able to backup to any location read/writeable by XBMC. You can also write directly to a Dropbox target without needing the Dropbox client on your HTPC. Right now you can download the addon from the main XBMC Add-on repo or the github page for this project. Please post any problems and suggestions here.

Download
In XBMC: Settings->Add-ons->Get Add-ons->Programs
or
Github: https://github.com/robweber/xbmcbackup

FAQ

I can't see any restore points when choosing "Restore", what is the problem?

If you've created restore points with an older version of the addon (pre 0.3.6) you may see this issue. New versions of the addon look for a file called xbmcbackup.val to validate that a folder is a valid restore archive. Your older restore folders may not have this file. All you need to do is create a blank text file and rename it to xbmcbackup.val. Then put this file inside the archive directory. Your restore points should show up after selecting "Restore" in the addon again.

Several settings aren't being restored, this includes views, weather, etc. How do I get these back?

GUISETTINGS.xml is a configuration file used heavily by XBMC for remembering GUI specific settings. Due to the fact that XBMC reads this file on startup, and writes from memory to this file on shutdown; it is not possible to restore this file while XBMC is running. You must manually move this file from your backup archives if you wish to restore it. User SouthMark has posted the following steps for restoring in the OpenELEC system where this is more difficult:

1. Run the restore of your backup
2. SSH using putty to the IP Address of your media centre username: root Password openelec
3. Type touch /var/lock/xbmc.disabled and then press enter
4. Type kill all -9 xbmc.bin and then press enter - Your media center machine should now go blank
5. Connect to your machine using WinSCP and copy the guisettings.xml file to the userdata folder (this is the guisettings.xml file from your backup)
6. go back to your putty window and type rm /var/lock/xbmc.disabled

Why is the Addon prompting me to restart XBMC to continue?

If you have an advancedsettings file in your restore folder the addon will ask you if you want to restore this file and restart xbmc to continue. This is because the advancedsettings file may contain path substitution information that you want to be loaded when doing the rest of your restore. By restoring this file and restarting xbmc it will be loaded and the rest of your files will go where they are supposed to. If you know your file does not contain any path substitutions you can select "no" and continue as normal.


RE: [Release] XBMC Backup - avp387 - 2012-04-22 15:27

Good effort. This is definitely needed. At this point though, on a linux system I find the following method to work well for me:

http://www.htpcbeginner.com/automatic-backup-of-xbmc-settings-databases-on-ubuntu-linux/

It is automatic and came in handy for me at least once when I had to restore all my xbmc settings.


RE: [Release] XBMC Backup - robweber - 2012-04-22 16:55

Thanks for the link. As with a lot of things there are numerous ways to get the job done. I guess with this addon I'm aiming at cross-platform, or people on "out of the box" systems like XBMCbuntu that may not want to muck around with the command line. As far as automation goes, I don't think that is something I'll put specifically into this addon, but you could schedule it with something like Cron XBMC (again, keep it all within the media center). Once the restore mode is finished I think this will really bring things full circle. I think it would be great to install XBMC, download this one addon, then pull a restore to get back everything I had previously.


RE: [Release] XBMC Backup - paddycarey - 2012-04-24 15:10

you can walk non-local paths (or local ones for that matter) using the json api instead of os.walk. It's pretty nice and has a few added benefits like being able to walk inside compressed directories (the compressed backup file for instance). I use it in my mmabrowser plugin to walk a defined share, whether local or remote like so:

Code:
def getDirList(path):
    dirList = []
    currentLevelDirList = [path]
    while True:
        prevLevelDirList = []
        if len(currentLevelDirList) > 0:
            for dirName in currentLevelDirList:
                prevLevelDirList.append(dirName)
                dirList.append(dirName)
            currentLevelDirList = []
        else:
            break
        for dirName in prevLevelDirList:
            log('Checking for directories in: %s' % dirName)
            json_response = xbmc.executeJSONRPC('{ "jsonrpc" : "2.0" , "method" : "Files.GetDirectory" , "params" : { "directory" : "%s" , "sort" : { "method" : "file" } } , "id" : 1 }' % dirName.encode('utf-8').replace('\\', '\\\\'))
            jsonobject = simplejson.loads(json_response)
            if jsonobject['result']['files']:
                for item in jsonobject['result']['files']:
                    if item['filetype'] == 'directory':
                        currentLevelDirList.append(item['file'])
    return dirList

def getFileList(path):
    fileList = []
    dirList = getDirList(path)
    for dirName in dirList:
        log('Checking for files in: %s' % dirName)
        json_response = xbmc.executeJSONRPC('{ "jsonrpc" : "2.0" , "method" : "Files.GetDirectory" , "params" : { "directory" : "%s" , "sort" : { "method" : "file" } , "media" : "video" } , "id" : 1 }' % dirName.encode('utf-8').replace('\\', '\\\\'))
        jsonobject = simplejson.loads(json_response)
        if jsonobject['result']['files']:
            for item in jsonobject['result']['files']:
                if item['filetype'] == 'file':
                    fileList.append(item['file'])
                    log('Found video: %s' % item['file'])
    return fileList

Might help with some of the caveats above.

Paddy


RE: [Release] XBMC Backup - robweber - 2012-04-24 15:52

That is awesome, thanks for the code. I had seen the Files.GetDirectory method in the JSON API but never thought of using it that way. My solution for the time being was to write a restore.txt file during the backup that included all files and directories, and then read that file back in later for the restore. I didn't really like this as a permanent solution though, I like your cross platform way of walking through the directory tree much better.


RE: [Release] XBMC Backup - dallasnights - 2012-04-24 18:42

Great work I am using it on ATV2 to my media share on a Windows 7 pc
Would love to see if you can add only backup updates since last backup

Thanks


RE: [Release] XBMC Backup - Bobby Blixberg - 2012-04-24 21:21

Great job.


RE: [Release] XBMC Backup - paddycarey - 2012-04-26 03:36

I took a quick stab at writing something to solve the walk problem and i came up with this https://github.com/paddycarey/script.module.devtoolkit/blob/master/lib/DevToolkit/vfs.py

it's still rough round the edges but i'll fix it up as best i can

if you just drop it in your addon for now (until i get the module shipshape), do a:

Code:
import vfs as xbmcvfs

and you should now have all the functionality of xbmcvfs plus a walk and listdir method which should work on local or remote shares


RE: [Release] XBMC Backup - robweber - 2012-04-26 04:00

(2012-04-26 03:36)paddycarey Wrote:  I took a quick stab at writing something to solve the walk problem and i came up with this https://github.com/paddycarey/script.module.devtoolkit/blob/master/lib/DevToolkit/vfs.py

it's still rough round the edges but i'll fix it up as best i can

if you just drop it in your addon for now (until i get the module shipshape), do a:

Code:
import vfs as xbmcvfs

and you should now have all the functionality of xbmcvfs plus a walk and listdir method which should work on local or remote shares

That is really great - thank you for taking this on! Am I understanding the comparepathlists() function correctly? The way it looks to me is that by giving it two paths, say C:\xbmc\userdata\ and nfs:\\192.168.1.102\foo\xbmc\userdata it will return the common directory of "xbmc\userdata". Is that right?

I was working on the code snippet you gave me but this is much better than the one or two functions I would have done. I like the idea of making it a comprehensive module for extending xbmcvfs. I should get around to integrating this with the backup addon within a day or two. I'm watching your repo so as you improve it I can keep up.


RE: [Release] XBMC Backup - paddycarey - 2012-04-26 04:08

comparepathlists isn't quite finished yet, it's intended to be a function to use to compare the contents of two directories. so one list of files that exist in both lists, one list of files that exists only in the first list but not the second list and vice versa.

The results could then be used to perform a sort of one way sync, copying any new files, skipping any common files and removing any deleted files.

Not sure of any way to get a timestamp from a remote file though so you can't be sure which file is newer.


RE: [Release] XBMC Backup - robweber - 2012-04-28 00:33

(2012-04-26 04:08)paddycarey Wrote:  comparepathlists isn't quite finished yet, it's intended to be a function to use to compare the contents of two directories. so one list of files that exist in both lists, one list of files that exists only in the first list but not the second list and vice versa.

The results could then be used to perform a sort of one way sync, copying any new files, skipping any common files and removing any deleted files.

Not sure of any way to get a timestamp from a remote file though so you can't be sure which file is newer.

I think you're right, I saw in another thread once that someone was using the JSON-RPC functions for file operations and they were having the same issue. Might be something to add to XBMC's JSON functionality so it can be exposed for stuff like this.


RE: [Release] XBMC Backup - e2zippo - 2012-05-03 11:19

This looks great, thanks a lot!


RE: [Release] XBMC Backup - klunde - 2012-05-03 12:07

Just downloaded and testet it - it didn't work

Code:
12:02:56 T:10624  NOTICE: -->Python Interpreter Initialized<--
12:02:56 T:10624  NOTICE: script.xbmcbackup: Starting
12:02:56 T:10624  NOTICE: script.xbmcbackup: Local Dir: C:\Users\moluo\AppData\Roaming\XBMC\
12:02:56 T:10624  NOTICE: script.xbmcbackup: Remote Dir: C:\Data\Backup\xbmc_backup/
12:02:56 T:10624  NOTICE: script.xbmcbackup: Remote Path exists - may have old files in it!
12:02:56 T:10624   ERROR: Error Type: <type 'exceptions.UnicodeEncodeError'>
12:02:56 T:10624   ERROR: Error Contents: 'ascii' codec can't encode character u'\xc2' in position 59: ordinal not in range(128)
12:02:56 T:10624   ERROR: Traceback (most recent call last):
                                              File "C:\Users\moluo\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 188, in <module>
                                                backup.run()
                                              File "C:\Users\moluo\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 119, in run
                                                self.syncFiles()
                                              File "C:\Users\moluo\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 132, in syncFiles
                                                self.fileManager.createFileList(self.Addon)
                                              File "C:\Users\moluo\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 26, in createFileList
                                                self.walkTree(self.walk_path + "addons/")
                                              File "C:\Users\moluo\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 65, in walkTree
                                                self.addFile(filePath)
                                              File "C:\Users\moluo\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 70, in addFile
                                                self.fHandle.write(str(filename) + "\n")
                                            UnicodeEncodeError: 'ascii' codec can't encode character u'\xc2' in position 59: ordinal not in range(128)



RE: [Release] XBMC Backup - robweber - 2012-05-03 21:42

From the error it looks like there was a character in one of you filenames that couldn't be encoded to ASCII (for writing the copy list). I'm curious as to what the filename was but it doesn't look like we're going to get that info from this log file. If you could add the following line after line 69 in the default.py file it should print out the filename to the log right before it calls the function that seems to cause the error. Make sure it is on the same indent line as the next line and it should run. Please post your results back here.

Code:
xbmc.log(str(filename))



RE: [Release] XBMC Backup - Bond13579 - 2012-05-05 21:00

I'll post the logs with that line of code since I'm having the same issue. Looks like it doesn't like the dash, '-'.


14:58:42 T:3808 NOTICE: addons\skin.aeonmq.3\media\flags\music\recordlabel\Republic.png
14:58:42 T:3808 NOTICE: addons\skin.aeonmq.3\media\flags\music\recordlabel\Rhino.png
14:58:42 T:3808 NOTICE: addons\skin.aeonmq.3\media\flags\music\recordlabel\Roadrunner.png
14:58:42 T:3808 NOTICE: addons\skin.aeonmq.3\media\flags\music\recordlabel\Roc-A-Fella.png
14:58:42 T:3808 ERROR: Error Type: <type 'exceptions.UnicodeEncodeError'>
14:58:42 T:3808 ERROR: Error Contents: 'ascii' codec can't encode character u'\u201d' in position 58: ordinal not in range(128)
14:58:42 T:3808 ERROR: Traceback (most recent call last):
File "C:\Users\Parnell\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 189, in <module>
backup.run()
File "C:\Users\Parnell\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 120, in run
self.syncFiles()
File "C:\Users\Parnell\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 133, in syncFiles
self.fileManager.createFileList(self.Addon)
File "C:\Users\Parnell\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 26, in createFileList
self.walkTree(self.walk_path + "addons/")
File "C:\Users\Parnell\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 65, in walkTree
self.addFile(filePath)
File "C:\Users\Parnell\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 68, in addFile
xbmc.log(str(filename))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201d' in position 58: ordinal not in range(128)