[Release] XBMC Backup

Love the concept.. I'm always backing up, yet never current.. weeks old, this would seem to take the drudgery outo of this task... but alas as others have noted.

here's a snip of the critical area... the pastebin was a bit too big to post.

11:26:53 T:3600 INFO: -->Python script returned the following error<--
Error Type: <type 'exceptions.UnicodeEncodeError'>
11:26:53 T:3600 ERROR: Error Contents: 'ascii' codec can't encode character u'\xf7' in position 50: ordinal not in range(128)
11:26:53 T:3600 ERROR: Traceback (most recent call last):
File "C:\Users\PatK\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 188, in <module>
11:26:53 T:3600 ERROR: backup.run()
File "C:\Users\PatK\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 119, in run
self.syncFiles()
File "C:\Users\PatK\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 132, in syncFiles
self.fileManager.createFileList(self.Addon)
File "C:\Users\PatK\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 26, in createFileList
self.walkTree(self.walk_path + "addons/")
File "C:\Users\PatK\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 65, in walkTree
self.addFile(filePath)
File "C:\Users\PatK\AppData\Roaming\XBMC\addons\script.xbmcbackup\default.py", line 70, in addFile
self.fHandle.write(str(filename) + "\n")
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf7' in position 50: ordinal not in range(128)
11:26:53 T:3600 INFO: -->End of Python script error report<--

I look forward with baited breath fr an update

love idea - got mq3 and script error.

quick update, i've looked over a lot of stuff tonight and think i have a solution. each of the errors i'm seeing are a result of ascii encoding errors when writing the "restore.txt" file that is used to write the files between destinations. I've removed this entirely and now pass the filenames around internally via a filename list instead. I'm going to test over the next day using different combinations (including MQ) but shoulld release to the repo soon.

Everyone please hang tight, hopefully this will fix things up.

any updates?

Matt

I pulled the latest git today and backup is running with MQ3 skin enabled, so far so good.
But then i looked at what exactly is being backuped and i still miss a few things form userdata folder.

Folders not backuped are:
- keymaps
- peripheral_data (not that important i guess)

EDIT:
I checked code and noticed that those 2 folders aren't defined so maybe that's why it's not backuped.

In OpenELEC there is also a config folder outside userdata folder , is there a way i can add this?

I also noticed it's doing a full update every time, maybe doing incremental updates is better?

(2012-05-23 10:37)schumi2004 Wrote:  I pulled the latest git today and backup is running with MQ3 skin enabled, so far so good.
But then i looked at what exactly is being backuped and i still miss a few things form userdata folder.

Folders not backuped are:
- keymaps
- peripheral_data (not that important i guess)

EDIT:
I checked code and noticed that those 2 folders aren't defined so maybe that's why it's not backuped.

In OpenELEC there is also a config folder outside userdata folder , is there a way i can add this?

I also noticed it's doing a full update every time, maybe doing incremental updates is better?

Glad that it is working. I should probably add the keymaps folder at the least as some people may be making custom keymap entries there. As far as OpenELEC support I could add an extra toggle for OpenELEC specific files in the settings. Are there any other folders under the user profile that OpenELEC creates that may be important?

As far as the incremental updates - I totally agree that would be the best. The caveat is that there is no reliable way to get the timestamps for files from remote systems. The XBMC JSON RPC does not return them, and the built in python modules only do local files/folders very well. If I was to restrict the addon to something like only SMB shares I may be able to find a smb python library that would get the timestamps, but I'd like to use the built in xbmc protocols where possible and the functionality for that just isn't there yet. As such it can still take quite a while if you have multiple skins or lots of thumbnails/fanart to copy everything over.

I'm holding off submitting this to the addons repo just yet, I have found an issue with the way the JSON Files.GetDirectory command returns filenames for zip files that garbles them. On github I have added a catch to filter these files out, but am waiting back from some info on if this can be fixed before submitting.

I'll check OpenELEC for you.

If incremental is hard to accomplish maybe add time stamps to the backup folders?
Just a thought

(2012-05-23 22:59)schumi2004 Wrote:  I'll check OpenELEC for you.

If incremental is hard to accomplish maybe add time stamps to the backup folders?
Just a thought

Thanks, I'll add the OpenELEC stuff before submitting to the repo, shouldn't be too hard. I got an answer on the zip file paths so that is sorted out.

What do you mean by add time stamps to the backup folders? I'd be happy to consider something but the different ways I've looked at it all require that I can gather some type of timestamp information from the backup location once the files have been sent, which I can't do. Once the files are in the backup local all I can get from them are name and full file path.

The one thing I can think of that might work would be to create a list of files and timestamps that is created locally when the backup job runs, and then saved to the addon_data folder (as a txt or xml file). Upon the next run the timestamps in the file could be used to compare the local file against what it was the last time the backup ran. The restore side would just restore everything, as I think it should. Is that more like what you were thinking? It would require the creation of a map data structure to compare against (which would eat up some processing time) but the trade off would be not transferring large files that may not have changed, which would be worth the extra overhead I think.

I tried latest git on my OpenELEC system and it fails with the following error.


23:49:20 T:140627724441344 NOTICE: Thread Background Loader start, auto delete: false
23:49:20 T:140627724441344 NOTICE: Previous line repeats 1 times.
23:49:20 T:140627724441344 WARNING: CreateLoader - Unsupported protocol(addons) in addons://more/executable.tbn
23:49:25 T:140627724441344 NOTICE: Thread XBPyThread start, auto delete: false
23:49:25 T:140627724441344 NOTICE: -->Python Interpreter Initialized<--
23:49:25 T:140627724441344 ERROR: Error Type: <type 'exceptions.IndentationError'>
23:49:25 T:140627724441344 ERROR: Error Contents: ('unindent does not match any outer indentation level', ('/storage/.xbmc/addons/script.xbmcbackup/default.py', 36, 64, ' self.walkTree(self.walk_path + "userdata/Database")\n'))
23:49:25 T:140627724441344 ERROR: IndentationError: ('unindent does not match any outer indentation level', ('/storage/.xbmc/addons/script.xbmcbackup/default.py', 36, 64, ' self.walkTree(self.walk_path + "userdata/Database")\n'))

Don't understand what is going wrong.
Going to check it out.

EDIT:
My bad, mixed up spaces and tabs.

EDIT2:
root / # find / -name autostart.sh
/storage/.config/autostart.sh
root / # find / -name Lircmap.xml
/storage/.xbmc/userdata/Lircmap.xml
/storage/backup/backup/xbmc_backup/userdata/Lircmap.xml
/usr/share/xbmc/system/Lircmap.xml
root / #

This is the location of OE specific files.
/storage/.config
*.xml and *.sh

For some reason i keep getting errors, will check out tomorrow.

(2012-05-23 23:10)robweber Wrote:  

(2012-05-23 22:59)schumi2004 Wrote:  I'll check OpenELEC for you.

If incremental is hard to accomplish maybe add time stamps to the backup folders?
Just a thought

Thanks, I'll add the OpenELEC stuff before submitting to the repo, shouldn't be too hard. I got an answer on the zip file paths so that is sorted out.

What do you mean by add time stamps to the backup folders? I'd be happy to consider something but the different ways I've looked at it all require that I can gather some type of timestamp information from the backup location once the files have been sent, which I can't do. Once the files are in the backup local all I can get from them are name and full file path.

The one thing I can think of that might work would be to create a list of files and timestamps that is created locally when the backup job runs, and then saved to the addon_data folder (as a txt or xml file). Upon the next run the timestamps in the file could be used to compare the local file against what it was the last time the backup ran. The restore side would just restore everything, as I think it should. Is that more like what you were thinking? It would require the creation of a map data structure to compare against (which would eat up some processing time) but the trade off would be not transferring large files that may not have changed, which would be worth the extra overhead I think.

And what if you checked file-size and date before overwriting? No need to keep track of files in a separate file.
Again just a thought.

Ver .8 auto-update report some mixed results. Default settings script dies...(same kind of errors reported previously) exclude user addons and add-on data, and the script seems to work but with some extras... which don't seem to be part of the back-up, but did appear. http://pastebin.com/f9qc4YYA keep in mind these are folders... not files and within the folders are empty... with the exception of the 'userdata' folder which contains... the usual xml files and folders, the thumbnails are populated, and the database, all the rest are empty.. playlists included.

In my case, I'd just prefer a global back-up of the roaming directory, (at least for now) without the selective functions.

Eden 11, T! win7/64