[Release] Simple downloader

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
milli Offline
Senior Member
Posts: 211
Joined: Feb 2011
Reputation: 2
Post: #46
Is network location supported for the download folder? For example using an SMB share. This is on Apple TV 2 where local storage is limited and I'm trying to use an SMB share location to download videos. Is there anyway to do it?

Thanks!
find quote
shawonmq Offline
Senior Member
Posts: 212
Joined: Jun 2012
Reputation: 8
Post: #47
Hi
I am trying to add this script in my plugin...
I am able to start downloading to designated file..
But I am trying to clear the download queue before starting..
Other words if user close the xbmc the queue still stays..so want to clear that

StreamOn Plugin
TV Streams, Shows, Sports
Install Repository
Forum Link
find quote
milli Offline
Senior Member
Posts: 211
Joined: Feb 2011
Reputation: 2
Post: #48
(2012-12-09 07:33)milli Wrote:  Is network location supported for the download folder? For example using an SMB share. This is on Apple TV 2 where local storage is limited and I'm trying to use an SMB share location to download videos. Is there anyway to do it?

Thanks!

Actually found a way to do it. It was using os method to move file from temporary location to original destination after downloading it. Changed to use XBMC API to copy the file and delete the temporary file. It works with SMB share path.
find quote
ulion Offline
Team-Kodi Member
Posts: 84
Joined: Dec 2012
Reputation: 0
Post: #49
I modified the _downloadURL method to let it support xbmc formated url (with header options in the url after '|').
now addons can set cookies, referer, necessary headers to the download request, same with xbmc core does.
also, urlopen could raise exceptions, so I move these lines also into the try...except inner block.

Code:
def _downloadURL(self, filename, item):
        self.common.log(filename)
        url = item['url']
        url = url.split('|', 2)
        if len(url) == 2:
            url, opts = url
            opts = opts.split('&')
            opts = dict(opt.split('=', 2) for opt in opts)
            opts = dict([unquote_plus(key), unquote_plus(val)] for key, val in opts.iteritems())
        else:      
            url = url[0]
            opts = {}
        req = urllib2.Request(url)
        for key, val in opts.iteritems():
            req.add_header(key, val)
        req.add_header('User-Agent', item.get('useragent', opts.get('User-Agent', self.common.USERAGENT)))
                    
        try:
            file = self.common.openFile(item["path_incomplete"], "wb")
            con = urllib2.urlopen(req)
            
            item["total_size"] = 0
            chunk_size = 1024 * 8
            
            if con.info().getheader("Content-Length").strip():
                item["total_size"] = int(con.info().getheader("Content-Length").strip())
            
            params = {"bytes_so_far": 0, "mark": 0.0, "queue_mark": 0.0, "obytes_so_far": 0}
            item["percent"] = 0.1
            item["old_percent"] = -1

            while "quit" not in params:
find quote
ulion Offline
Team-Kodi Member
Posts: 84
Joined: Dec 2012
Reputation: 0
Post: #50
another problem, I need a way to stop current downloading, but seems _downloadURL method does not supply such possible, could that be done?
find quote
tknoll Offline
Junior Member
Posts: 6
Joined: Jan 2013
Reputation: 0
Post: #51
Hey there,

I'm working on a seminar paper for recording internet radio streams in xbmc, so the functionality to the simple
downloader is quite similar and I get a lot of good ideas from the source code :-). However at the moment I'm thinking
of having a possibility to abort an active download (I do it with streamripper). For the simple downloader I realized that
a "quit" parameter is evaluated, however I'm just wondering whether the youtube plugin gives the user the chance
to actually set this parameter directly in XBMC? At least I didn't find any cancel-button or the like...

Hope someone can help as I'm pretty new to plugin development in XBMC.

Regards

Tobias
find quote
michel551813 Offline
Junior Member
Posts: 2
Joined: May 2013
Reputation: 0
Post: #52
(2012-12-19 04:06)milli Wrote:  
(2012-12-09 07:33)milli Wrote:  Is network location supported for the download folder? For example using an SMB share. This is on Apple TV 2 where local storage is limited and I'm trying to use an SMB share location to download videos. Is there anyway to do it?

Thanks!

Actually found a way to do it. It was using os method to move file from temporary location to original destination after downloading it. Changed to use XBMC API to copy the file and delete the temporary file. It works with SMB share path.

As I'm running into the same issue and am quite new to this, would you mind sharing what exactly you changed to fix the downloading to a SMB share issue?
find quote
milli Offline
Senior Member
Posts: 211
Joined: Feb 2011
Reputation: 2
Post: #53
(2013-05-12 01:34)michel551813 Wrote:  
(2012-12-19 04:06)milli Wrote:  
(2012-12-09 07:33)milli Wrote:  Is network location supported for the download folder? For example using an SMB share. This is on Apple TV 2 where local storage is limited and I'm trying to use an SMB share location to download videos. Is there anyway to do it?

Thanks!

Actually found a way to do it. It was using os method to move file from temporary location to original destination after downloading it. Changed to use XBMC API to copy the file and delete the temporary file. It works with SMB share path.

As I'm running into the same issue and am quite new to this, would you mind sharing what exactly you changed to fix the downloading to a SMB share issue?

I did it quite a while ago and since then I'd upgraded/downgraded many times. So I don't have those changes any more. For me I realized it wouldn't help much where I have to remember to keep my NAS up while it is downloading. So never really used it.
(This post was last modified: 2013-05-18 02:12 by milli.)
find quote
Nwardezir Offline
Junior Member
Posts: 5
Joined: May 2013
Reputation: 0
Post: #54
(2013-05-18 02:12)milli Wrote:  
(2013-05-12 01:34)michel551813 Wrote:  
(2012-12-19 04:06)milli Wrote:  Actually found a way to do it. It was using os method to move file from temporary location to original destination after downloading it. Changed to use XBMC API to copy the file and delete the temporary file. It works with SMB share path.

As I'm running into the same issue and am quite new to this, would you mind sharing what exactly you changed to fix the downloading to a SMB share issue?

I did it quite a while ago and since then I'd upgraded/downgraded many times. So I don't have those changes any more. For me I realized it wouldn't help much where I have to remember to keep my NAS up while it is downloading. So never really used it.

I'm very interested in having some sort of fix, or at least some hints and pointers to try fixing it by myself ...
I'm struggling for days with the Youtube Addon, precisely the downloading feature. Indeed, Simple Downloader service seems not being able to download files to an smb:// or nfs:// location. Sad
I'd like to save Youtube videos and other downloaded stuff on my NAS, with all my other media stuff (movies, , tv shows, clips, music etc).

I posted an issue in the gigantic YT addon dedicated thread here : http://forum.xbmc.org/showthread.php?tid...pid1424390 ... but I guess it's more a Simple Downloader issue Frown

Any help would be appreciated.
find quote
milli Offline
Senior Member
Posts: 211
Joined: Feb 2011
Reputation: 2
Post: #55
(2013-05-19 06:31)Nwardezir Wrote:  
(2013-05-18 02:12)milli Wrote:  
(2013-05-12 01:34)michel551813 Wrote:  As I'm running into the same issue and am quite new to this, would you mind sharing what exactly you changed to fix the downloading to a SMB share issue?

I did it quite a while ago and since then I'd upgraded/downgraded many times. So I don't have those changes any more. For me I realized it wouldn't help much where I have to remember to keep my NAS up while it is downloading. So never really used it.

I'm very interested in having some sort of fix, or at least some hints and pointers to try fixing it by myself ...
I'm struggling for days with the Youtube Addon, precisely the downloading feature. Indeed, Simple Downloader service seems not being able to download files to an smb:// or nfs:// location. Sad
I'd like to save Youtube videos and other downloaded stuff on my NAS, with all my other media stuff (movies, , tv shows, clips, music etc).

I posted an issue in the gigantic YT addon dedicated thread here : http://forum.xbmc.org/showthread.php?tid...pid1424390 ... but I guess it's more a Simple Downloader issue Frown

Any help would be appreciated.

You need to drill down the code of simple downloader which is usually under user preferences folder and find where it is renaming the file using os.rename or os.move. You would need to replace it with xbmcvfs.copy and xbmcvfs.delete.
find quote
Nwardezir Offline
Junior Member
Posts: 5
Joined: May 2013
Reputation: 0
Sad  RE: [Release] Simple downloader
Post: #56
(2013-05-26 22:48)milli Wrote:  You need to drill down the code of simple downloader which is usually under user preferences folder and find where it is renaming the file using os.rename or os.move. You would need to replace it with xbmcvfs.copy and xbmcvfs.delete.
Plugin is already using xbmcvfs, from what I found in SimpleDownloader.py
Code:
if status == 200:
                   if self.xbmcvfs.exists(item["path_incomplete"]):
                       self.common.log("Moving %s to %s" % (repr(item["path_incomplete"]), repr(item["path_complete"])))
                       self.xbmcvfs.rename(item["path_incomplete"], item["path_complete"])
                       self._showMessage(self.language(203), filename)

It's in the _processQueue method, and I guess the 200 status code is OK, thus file downloaded and need to be moved.

Only other thing I found is in the xbmcvfsdummy.py file where they defined :
Code:
def rename(origin, target):
    return os.rename(origin, target)
Which I replaced with :
Code:
def rename(origin, target):
    return xbmcvfs.rename(origin, target)
And I also added : import xbmcvfs on top of it.

I did several test and ... it still not working. As shown on log :
Code:
00:22:50 T:147722240  NOTICE: [YouTube-4.4.4] _downloadURL : 'Loop done'
00:22:50 T:147722240  NOTICE: [YouTube-4.4.4] _downloadURL : 'Done'
00:22:50 T:147722240  NOTICE: [YouTube-4.4.4] _processQueue : 'Moving u'/var/mobile/Library/Preferences/XBMC/userdata/addon_data/script.module.simple.downloader/Skrillex vs Daft Punk vs Queen vs House Of Pain - Skrillex Rock (Loo & Placido vs DJ Hero Bootleg)-[0vle6n5fGZg].mp4' to u'smb://SLAUGHTERLOGY/Videos/Vrac/FromTheNet/Skrillex vs Daft Punk vs Queen vs House Of Pain - Skrillex Rock (Loo & Placido vs DJ Hero Bootleg)-[0vle6n5fGZg].mp4''

00:22:50 T:147722240   ERROR: MoveFile - cant move file </var/mobile/Library/Preferences/XBMC/userdata/addon_data/script.module.simple.downloader/Skrillex vs Daft Punk vs Queen vs House Of Pain - Skrillex Rock (Loo & Placido vs DJ Hero Bootleg)-[0vle6n5fGZg].mp4>. trying lower case </var/mobile/library/preferences/xbmc/userdata/addon_data/script.module.simple.downloader/skrillex vs daft punk vs queen vs house of pain - skrillex rock (loo & placido vs dj hero bootleg)-[0vle6n5fgzg].mp4>

00:22:50 T:147722240   ERROR: Rename - Error renaming file /var/mobile/Library/Preferences/XBMC/userdata/addon_data/script.module.simple.downloader/Skrillex vs Daft Punk vs Queen vs House Of Pain - Skrillex Rock (Loo & Placido vs DJ Hero Bootleg)-[0vle6n5fGZg].mp4

00:22:51 T:147722240  NOTICE: [StorageClient-2.5.0] 'lock' : 'SimpleDownloaderQueueLock'

I grep-ed the whole SimpleDownloader code and found no match for any other rename, move, copy or delete command.
Any ideas ? Sad
(This post was last modified: 2013-05-27 00:38 by Nwardezir.)
find quote
Lewpy Offline
Junior Member
Posts: 7
Joined: Jul 2013
Reputation: 1
Location: UK
Post: #57
(2013-05-27 00:37)Nwardezir Wrote:  Plugin is already using xbmcvfs, from what I found in SimpleDownloader.py
Code:
if status == 200:
                   if self.xbmcvfs.exists(item["path_incomplete"]):
                       self.common.log("Moving %s to %s" % (repr(item["path_incomplete"]), repr(item["path_complete"])))
                       self.xbmcvfs.rename(item["path_incomplete"], item["path_complete"])
                       self._showMessage(self.language(203), filename)

It's in the _processQueue method, and I guess the 200 status code is OK, thus file downloaded and need to be moved.
[...]
Any ideas ? Sad
That is the issue right there: you can only rename a file if it is on the same drive in Windows. You cannot rename a file between drives: you must perform a copy and delete (a move, in effect).

So change the lines
Code:
if status == 200:
                   if self.xbmcvfs.exists(item["path_incomplete"]):
                       self.common.log("Moving %s to %s" % (repr(item["path_incomplete"]), repr(item["path_complete"])))
                       self.xbmcvfs.rename(item["path_incomplete"], item["path_complete"])
                       self._showMessage(self.language(203), filename)
to
Code:
if status == 200:
                   if self.xbmcvfs.exists(item["path_incomplete"]):
                       self.common.log("Copying %s to %s" % (repr(item["path_incomplete"]), repr(item["path_complete"])))
                       self.xbmcvfs.copy(item["path_incomplete"], item["path_complete"])
                       self.common.log("Deleting %s" % (repr(item["path_incomplete"])))
                       self.xbmcvfs.delete(item["path_incomplete"])
                       self._showMessage(self.language(203), filename)
and it should now work: it does for me Big Grin

Of course, this is not "optimal" if the temp location ("path_incomplete") and the final location ("path_complete") are on the same drive.
A "perfect" solution would check the drive locations and perform "rename" if the same, or copy/delete if different.

Update: okay, I've just noticed it is not deleting the files from the temporary location, so a little more work needs done. But it does copy it to the correct NAS location: 50% success Big Grin

Update 2: a further check shows that it only occasionally fails to delete the temp copy of the downloaded file - 1 in 10 of the Apple iTunes Trailers I tested last night, and not always the first on to be queued up either. Guessing there is a timing issue going on with the file still being open? I really don't know Python at all, so fully debugging the script is beyond me at the moment :-(
(This post was last modified: 2013-07-08 12:52 by Lewpy.)
find quote
michel551813 Offline
Junior Member
Posts: 2
Joined: May 2013
Reputation: 0
Smile  RE: [Release] Simple downloader
Post: #58
I have tried the adjustments you suggested and it works properly for me, even removing the files for the temp folder.

As for the file in the temp folder not always being deleted:
The lock on the file may not be released when the script tries to remove the file. Maybe a small timer might solve the issue, but a check would be nicer...
I'm no expert in scripting, but I now know were to look Smile
find quote
KingsCross Offline
Junior Member
Posts: 33
Joined: Aug 2011
Reputation: 1
Post: #59
I don't know whether this is the right place to ask but I'm also having problems with downloading videos. It gets stuck on 'preparing download' and freezes. Clearing commoncache used to work but it freezes so often on 'preparing download' that it isn't really an ideal solution. The other issue is that a commoncache file no longer appears to be being generated, so I can't clear it anyway. I've tried deleting and reinstalling the cache plugging and downloader plugging but it hasn't changed anything. I also did a fresh install of openelec and that didn't fix the issue of getting stuck on 'preparing download' either.

Any help would be greatly appreciated.
find quote
AddonScriptorDE Offline
Skilled Python Coder
Posts: 707
Joined: May 2012
Reputation: 56
Location: Germany
Post: #60
(2013-07-16 20:35)KingsCross Wrote:  I don't know whether this is the right place to ask but I'm also having problems with downloading videos. It gets stuck on 'preparing download' and freezes. Clearing commoncache used to work but it freezes so often on 'preparing download' that it isn't really an ideal solution. The other issue is that a commoncache file no longer appears to be being generated, so I can't clear it anyway. I've tried deleting and reinstalling the cache plugging and downloader plugging but it hasn't changed anything. I also did a fresh install of openelec and that didn't fix the issue of getting stuck on 'preparing download' either.

Any help would be greatly appreciated.
Same here. Tested on 2 systems - after a few downloads it always freezes on "prepraring download".
Would be great if this bug will be fixed.

Great addon btw! (when its working)

https://github.com/AddonScriptorDE
Categories | Chrome Launcher | Commands | SimplePlaylists | SubCentral | Video Screensavers | XBMC WebBrowser
BestOfYoutube | CBSnews | DailyMotion | Discovery Networks | EuroNews | IGN | LoveFilm | NetfliXBMC | RedBull | Reddit | TheOnion | TIME | VEVO TV | VICE | WIRED | YouTube Billboard/Channels
German Add-ons
(This post was last modified: 2013-07-29 01:18 by AddonScriptorDE.)
find quote