[Release] Simple downloader

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
cpw83 Offline
Junior Member
Posts: 3
Joined: Aug 2013
Reputation: 0
Location: Germany
Post: #61
KingsCross Wrote:The other issue is that a commoncache file no longer appears to be being generated, so I can't clear it anyway. I've tried deleting and reinstalling the cache plugging and downloader plugging but it hasn't changed anything.

Confirmed - exactly the same problem here. It seems Simple Downloader doesn't use the Commoncache (the SQLite-database in /storage/.xbmc/userdata/addon_data/script.common.plugin.cache) anymore. Nevertheless it does cache data, especially a download queue:

If I delete every temporary, unfinished download in /storage/.xbmc/userdata/addon_data/script.module.simple.downloader and try to download a video from the YouTube-addon, an empty file with a name from a download that seems to have failed before (so not the actual file I'm trying to download now) appears in that directory. So it seems something crashed during the last download(s), something is corrupted and one or more files are stuck in the download queue.

I assume this queue was stored in the Commoncache before, but is now stored elsewhere. I looked through all the Simple Downloader-directories and files and tried to grep the name of the temporary file from all files in the file system, but I can't find anything.

So deleting/resetting Simple Downloader's download queue might help temporarily - but where is it?
find quote
divingmule Offline
Skilled Python Coder
Posts: 1,297
Joined: Oct 2008
Reputation: 61
Post: #62
I think you'll find the commoncache_db in userdata/cache.
find quote
cpw83 Offline
Junior Member
Posts: 3
Joined: Aug 2013
Reputation: 0
Location: Germany
Post: #63
OK, found it - Common Plugin Cache's commoncache.db seems to be no longer in /storage/.xbmc/userdata/addon_data/script.common.plugin.cache, but in /storage/.xbmc/temp

I copied it via SCP, opened it in SQLite Database Browser, ran

Code:
UPDATE Downloader SET data = '' WHERE name = 'SimpleDownloaderQueue';

and copied it back. That resulted in the downloader not responding at all, not even showing "Preparing Download", probably because Simple Downloader expects to be at least something with a defined syntax there; an empty String won't work.

So I deleted /storage/.xbmc/temp/commoncache.db and rebooted - downloads are now working again.

Here's an excerpt from my XMBC log - the YouTube-URLs contain a lot of GET-variables that I don't know what they're for exactly, so I substituted anything looking like sensitive information (e.g. potentially password hashes etc.) with [x], because I don't want to post that on an open forum.

Code:
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'SimpleDownloaderLock'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'Downloader'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'Done : true'
12:25:06 T:2706754368  NOTICE: [YouTube-4.4.6] _startDownload : 'Downloader not active, initializing downloader.'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'SimpleDownloaderQueueLock'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'Downloader'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'Done : true'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'get' : 'SimpleDownloaderQueue'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:06 T:2706754368  NOTICE: [YouTube-4.4.6] _addItemToQueue : 'Moved Ashens and the Quest for the GameChild - Full Movie HD  Geek Week-[VS2Cnx_eL6s].mp4 to front of queue. - 8'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'set' : 'SimpleDownloaderQueue'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'unlock' : 'SimpleDownloaderQueueLock'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'unlock' : 'Done: true'
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] _addItemToQueue : 'Done'
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] _processQueue : ''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'SimpleDownloaderQueueLock'
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'Downloader'
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'Done : true'
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'get' : 'SimpleDownloaderQueue'
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] _getNextItemFromQueue : 'queue loaded : u'[(u"The Stable of Solitude ft. Chris O\'Dowd  KnightHood and Decoy  YouTube Geek Week-[UWmDye4QxAA].mp4", {\'Title\': u"The Stable of Solitude ft. Chris O\'Dowd | KnightHood and Decoy | YouTube Geek Week", \'url\': u\'http://r17---sn-4g57kn7e.c.youtube.com/videoplayback?ip=[x.x.x.x]&ipbits=8&fexp=[x]&mt=[x]&sver=3&expire=1375893945&upn=[x]&cp=[x]&sparams=cp,id,ip,ipbits,itag,ratebypass,source,upn,expire&id=[x]&key=yt1&source=youtube&ratebypass=yes&mv=m&itag=37&ms=au&signature=[x]\', \'download_path\': \'/storage/downloads/\', \'videoid\': \'UWmDye4QxAA\', \'action\': \'download\', \'path\': \'/root/subscriptions/new\'}), (u\'Ashens and the Quest for the GameChild - Full Movie HD  Geek Week-[VS2Cnx_eL6s].mp4\', {\'download_path\': \'/storage/downloads/\', \'url\': u\'http://r5---sn-4g57lnel.c.youtube.com/videoplayback?sver=3&cp=[x]&source=youtube&upn[x]&itag=37&mv=m&ms=au&ratebypass=yes&ipbits=8&fexp=[x]&sparams=cp,id,ip,ipbits,itag,ratebypass,source,upn,expire&expire=1376154160&mt=[x]&key=yt1&ip=[x.x.x.x]&signature=[x]\', \'Title\': u\'Ashens and the Quest for the GameChild - Full Movie HD | Geek Week\', \'videoid\': \'VS2Cnx_eL6s\', \'action\': \'download\', \'path\': \'/root/subscriptions/new\'}), (u\'Nostalgia Critic Sailor Moon-[6625851].mp4\', {\'download_path\': \'/storage/downloads/\', \'url\': \'http://j69.video2.blip.tv/14080011970964/NostalgiaCritic-NostalgiaCriticSailorMoon932.m4v?ir=181450&sr=4024\', \'videoid\': \'6625851\', \'useragent\': \'curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3\', \'action\': \'download\', \'path\': \'/root/my_favorites\'}), (u\'Top 40 Shitty Shark Movies PROMO-[VKQKlzudcyo].mp4\', {\'download_path\': \'/storage/downloads/\', \'url\': u\'http://r4---sn-4g57lnez.c.youtube.com/videoplayback?key=yt1&ip=[x.x.x.x]&upn=[x]&fexp=[x]&itag=37&mt=1375872492&mv=m&ratebypass=yes&id=[x]&ms=au&source=youtube&expire=1375893496&sparams=cp,id,ip,ipbits,itag,ratebypass,source,upn,expire&sver=3&ipbits=8&signature=[x]\', \'Title\': u\'Top 40 Shitty Shark Movies PROMO\', \'videoid\': \'VKQKlzudcyo\', \'action\': \'download\', \'path\': \'/root/subscriptions/new\'}), (u\'How to watch Ashens and the Quest for the GameChild-[l-a_LxkonsE].mp4\', {\'Title\': u\'How to watch Ashens and the Quest for the GameChild\', \'url\': u\'http://r4---sn-4g57lnel.c.youtube.com/videoplayback?source=youtube&fexp=[x]&ipbits=8&ratebypass=yes&ip=[x.x.x.x]&key=yt1&expire=1375894272&sver=3&itag=22&ms=au&cp=[x]&upn=[x]&id=[x]&mv=m&mt=1375872492&sparams=cp,id,ip,ipbits,itag,ratebypass,source,upn,expire&signature=[x]\', \'download_path\': \'/storage/downloads/\', \'videoid\': \'l-a_LxkonsE\', \'action\': \'download\', \'path\': \'/root/subscriptions/new\'}), (u\'Pogo Pins in Action-[XfxvxwSIjrU].mp4\', {\'download_path\': \'/storage/downloads/\', \'url\': u\'http://r2---sn-4g57knz7.c.youtube.com/videoplayback?mv=m&id=[x]&ms=au&cp=[x]&source=youtube&upn=[x]&expire=1375893852&mt=[x]&key=yt1&ip=[x.x.x.x]&ipbits=8&sparams=cp,id,ip,ipbits,itag,ratebypass,source,upn,expire&itag=22&fexp=[x]&sver=3&ratebypass=yes&signature=[x]\', \'Title\': u\'Pogo Pins in Action\', \'videoid\': \'XfxvxwSIjrU\', \'action\': \'download\', \'path\': \'/root/subscriptions/new\'}), (u\'The Matrix Sweded ft. Bart Baker and Edward Vilderman  Green Swede-[wkJx_SObhAY].mp4\', {\'Title\': u\'The Matrix Sweded ft. Bart Baker and Edward Vilderman | Green Swede\', \'url\': u\'http://r6---sn-4g57knes.c.youtube.com/videoplayback?expire=1375895915&mv=m&source=youtube&ratebypass=yes&key=yt1&cp=[x]&mt=[x]&id=[x]&sver=3&fexp=[x]&sparams=cp,id,ip,ipbits,itag,ratebypass,source,upn,expire&upn=EIda2fDmopI&itag=22&ipbits=8&ip=[x.x.x.x]&ms=au&signature=[x]\', \'download_path\': \'/storage/downloads/\', \'videoid\': \'wkJx_SObhAY\', \'action\': \'download\', \'path\': \'/root/subscriptions/new\'}), (u\'Asylum Seeker Almighty Thor-[6626036].mp4\', {\'download_path\': \'/storage/downloads/\', \'url\': \'http://j69.video2.blip.tv/14190011971385/MikeyJevs-AsylumSeekerAlmightyThor779.m4v?ir=12721&sr=1897\', \'videoid\': \'6626036\', \'useragent\': \'curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3\', \'action\': \'download\', \'path\': \'/root/my_favorites\'})]''
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] _getNextItemFromQueue : 'returning : The Stable of Solitude ft. Chris O'Dowd  KnightHood and Decoy  YouTube Geek Week-[UWmDye4QxAA].mp4'
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'unlock' : 'SimpleDownloaderQueueLock'
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'unlock' : 'Done: true'
12:25:07 T:3018868480   ERROR: ClearBlock: Trying to clear non existent block /storage/.xbmc/addons/script.module.simple.downloader
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] _setPaths : 'Removing incomplete u"/storage/.xbmc/userdata/addon_data/script.module.simple.downloader/The Stable of Solitude ft. Chris O'Dowd  KnightHood and Decoy  YouTube Geek Week-[UWmDye4QxAA].mp4"'
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] _downloadURL : 'The Stable of Solitude ft. Chris O'Dowd  KnightHood and Decoy  YouTube Geek Week-[UWmDye4QxAA].mp4'
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] openFile : 'u"/storage/.xbmc/userdata/addon_data/script.module.simple.downloader/The Stable of Solitude ft. Chris O'Dowd  KnightHood and Decoy  YouTube Geek Week-[UWmDye4QxAA].mp4" - 'wb''
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] openFile : 'Trying normal: wb'
12:25:07 T:2706754368   ERROR: Exception in thread Thread-1:
                                            Traceback (most recent call last):
                                              File "./Lib/threading.py", line 551, in __bootstrap_inner
                                              File "./Lib/threading.py", line 504, in run
                                              File "/storage/.xbmc/addons/script.module.simple.downloader/lib/SimpleDownloader.py", line 127, in _startDownload
                                                self._processQueue()
                                              File "/storage/.xbmc/addons/script.module.simple.downloader/lib/SimpleDownloader.py", line 170, in _processQueue
                                                status = self._downloadURL(filename, item)
                                              File "/storage/.xbmc/addons/script.module.simple.downloader/lib/SimpleDownloader.py", line 548, in _downloadURL
                                                con = urllib2.urlopen(url)
                                              File "./Lib/urllib2.py", line 126, in urlopen
                                              File "./Lib/urllib2.py", line 406, in open
                                              File "./Lib/urllib2.py", line 519, in http_response
                                              File "./Lib/urllib2.py", line 444, in error
                                              File "./Lib/urllib2.py", line 378, in _call_chain
                                              File "./Lib/urllib2.py", line 527, in http_error_default
                                            HTTPError: HTTP Error 403: Forbidden

To me it looks like Simple Downloader stumbles over an 403 by YouTube and gets stuck.

BTW: Before Simple Downloader stops working (while it is still working/downloading), it shows some script-errors, but I can't find them anywhere in the log.

I noticed there's a GET-Variable "ip" in the YouTube-URL (in the log above substituted with [x.x.x.x]). Here in Germany it's quite common that you get a new IP-address for your DSL-connection every 24 hours - in fact none of the IP-addresses in the YouTube-URLs are identical with the one I have now. So maybe the downloads break if the router reconnects and gets a new IP while downloading?


BTW 2: I'm using OpenELEC 3.1.5 testing.

/EDIT:

divingmule Wrote:I think you'll find the commoncache_db in userdata/cache.
There is no such directory - maybe OpenELEC handles this differently.
(This post was last modified: 2013-08-10 15:19 by cpw83.)
find quote
spyder Offline
Fan
Posts: 333
Joined: Feb 2012
Reputation: 4
Post: #64
(2013-07-29 01:04)AddonScriptorDE Wrote:  
(2013-07-16 20:35)KingsCross Wrote:  I don't know whether this is the right place to ask but I'm also having problems with downloading videos. It gets stuck on 'preparing download' and freezes. Clearing commoncache used to work but it freezes so often on 'preparing download' that it isn't really an ideal solution. The other issue is that a commoncache file no longer appears to be being generated, so I can't clear it anyway. I've tried deleting and reinstalling the cache plugging and downloader plugging but it hasn't changed anything. I also did a fresh install of openelec and that didn't fix the issue of getting stuck on 'preparing download' either.

Any help would be greatly appreciated.
Same here. Tested on 2 systems - after a few downloads it always freezes on "prepraring download".
Would be great if this bug will be fixed.

Great addon btw! (when its working)
I would like to add to this, as I just had this problem as well. Deleting the commoncache.db file did get things working again(after trying other pointless fixesTongue). I've downloaded plenty of videos with no problem, and didn't know this was even an issue.
find quote
aksarge Offline
Junior Member
Posts: 25
Joined: Apr 2012
Reputation: 0
Post: #65
I got stuck in queue hell too. User inflicted.

I have had ZERO problems with this addon until I caused them.

It took me longer than I want to admit to get to this thread to figure out what to do to escape the queue hell!

Once here and finding out to delete the commoncache.db did the trick in a mere 4 seconds.
find quote
bpuk Offline
Junior Member
Posts: 1
Joined: Jan 2014
Reputation: 0
Post: #66
I also got stuck in the 403 queue. A quick workaround is to catch HTTPError from urllib2.urlopen.

SimpleDownloader.py:
Change
Code:
file = self.common.openFile(item["path_incomplete"], "wb")
con = urllib2.urlopen(url)

to
Code:
try:
    con = urllib2.urlopen(url)  
except urllib2.HTTPError, e:
    self.common.log("Probable 403 Error. Download Failed")
    self._showMessage(self.language(204), "ERROR")
    return 500

file = self.common.openFile(item["path_incomplete"], "wb")

Not the neatest solution, but it stops 403 errors from jamming the queue.

Is there an active git repo for this addon that I can pull from?

Ta
find quote
kit500 Online
Junior Member
Posts: 11
Joined: Mar 2014
Reputation: 0
Post: #67
I have tested downloading from youtube plugin on win7. Appears a problem with cpu load increase to 90%, and slow-downing xbmc. Have tested both on frodo and gotham. (i can provide a log on which debug mode is needed)
find quote
casperklein Offline
Junior Member
Posts: 2
Joined: Apr 2014
Reputation: 0
Post: #68
Hello,

i am very new to python. However I hacked together some snippets and got a very well working video addon Big Grin

It basically downloads a text file, containing a list auf stream urls and additional infos and put them into a xbmc video list.

Here is the code: http://pastebin.com/wxbiD6BM

I'd like to add a "Download movie" option via the context menu. Can anybody please be so kind to give me a suggestion how to integrate simple downloader?

Thanks in advance!
find quote
Post Reply