[joe.dev.1981]

I have been working on an experiemental version of videodevil that allows viewing of the websites simultaneously.

I have decided to upload it to my own repository so users can test it out. Viewing individual websites should work as normal with the usual breaks that occur when regex need to be updated.

Many things don't work when viewing all sites but it is functional. You can browse the webpage videos and play them, view categories and search. Under categories not all categories are listed only common categories. To view all categories offered by a certain site you most browse to that site specifically. Sorting and other folders appear but either don't work or don't work well.

I am learning python using this project and welcome all improvements/suggestions.

Let me know what you think. (sorry the folder in folder in the zip file, I am also new to github. Just place the folder named "plugin.video.videodevil-joedev1981" [without quotes] in your addons folder)

https://github.com/joedev1981/joedev1981...mental.zip

Issues:

1. clean_safe inefficient....reported by anonymous(until I get permission to post his/her name), fixed by anonymous and joe.dev.1981
   
   • Need to create a pull request to videodevil in github
     
2. cookie handling.....reported by anonymous(until I get permission to post his/her name), fixed by anonymous
   
   • Need to create a pull request to videodevil in github
     
3. spaces in URLs scraped handled improperly........reported by hstegeman, should be fixed in my version
   
   • consider opening an issue in github
     
   • wait for someone to step up and fix in videodevil
     
   • wait until my version is stable so that I may take a look at videodevil(no promises I will find a solution)
     

Features wish list for videodevil:

1. view all sites - mostly complete in my version
   
2. caching "category" only in view all sites - complete in my version
   
3. downloading queues - completed in my version using simple downloader
   
   • Need to create a pull request to videodevil in github
     
4. pagination - to be worked on in the future("This future should be limited to type "next" and enabled with item_skill=pagination)
   
   • Next page - on click works as usual
     
   • context menu item "view pagination"?
     
5. enable/disable sites in settings - done in my version(videodevil uses skill=add/skill=remove in site configs but to my knowledge it is disabled)
   

to-do for stable unofficial version:

1. release v0.0.4 for testing
   
2. fix directory filename(videodevil creates a random file name and renames, I am thinking about just appending the plugin handle to the name) --> v0.0.5
   
3. create a class (listitems) to handle all items to be added to xbmc or saved to a list. (should fix the progress bar) --> v0.0.6
   
4. add better debugging --> v0.0.7
   
5. update sites, debug --> v0.0.8
   
6. cleanup code --> v0.0.9
   
7. debug --> v0.1.0
   

potential to-do for stable unofficial version (v0.0.5-v0.0.6):

1. transfer load catcher to localparser
   
2. add a mechanism to item rules that allows scraping grouped infos
   
   • this is useful for websites that use javascript to create html tables ie:
     <script>
     video_ids = [1,2,3]
     video_titles = [t1,t2,t3]
     video_icons = [1.jpg, 2.jpg, 3.jpg]
     </script>
     

to-do for next stable version:

1. improved caching based on type for both single site view and all site view(ie:videos cached 1hr, category cached 2 days, sort cache cached 1wk)
   
   • this should be made user configurable
     
2. enable/disable individual sites separately for single site view and all site view in settings
   
   • consider automatically generating these settings from sites.list
     

Let's work together to improve videodevil!!!

[joe.dev.1981]

Not sure if I should be posting this here or the video addon section. I have an updated experimental version. I have changed the version number to 0.0.2. Code is messy but most things work. I have improved the multithreading code. Previously it would open a thread for each webpage and download/parse simultaneously. This worked without problems but having so many threads parsing at the same time was detrimental to the overall efficiency. It now opens a new thread for each webpage while current thread waits for a response from a queue to start parsing. The overall time was reduced to half of the previous version. This only affects viewing all sites. The real world times I measure were 15-20s for the previous version 9-13s after these changes. Once parsing was complete ~2s was required to process and pass everything to xbmc. My goal was to reduced this time as much as possible by improving the efficiency of the function calls while parsing to do as much work as possible while waiting for webpage responses. I was able to get it down to about ~hundredth of a second. I discovered several significant inefficiency in my previous version. I am not sure if they exist in the official release of videodevil but the result was night and day. The problem was from a combination of many things. First clean_safe had the biggest effect and was responsible for roughly 1-2s. I changed loadRemote from one major "for" loop with lots of "if" statements to "if" statements with small "for" loops and did the same with infoFormatter. I gained another 0.5-1.5s. There were many other changes but they had much smaller gains time time. Overall this version seems to be limited by urllib2.urlopen and the lack of multiprocessing in xbmc(to my knowledge). I am not complaining, parsing +30 sites 3-7s! is good enough for me. The variability in times seems to be related to website response. While timing site response time I noticed certain sites took 3-7s to respond. If I disabled these sites(1-2 sites overall times dropped to 3s). Timing the sites without doing anything else ranged from 2.4 - 4.5s(gold standard). So I don't think there is much room for improvement. But it bugs me that there is 2.5s time difference in real world and gold standard. I plan on including a timeout that can be configured in the setting in future versions. This version is less stable than the previous version related to urllib.urlopen error while multithreading. Proper handling of errors/killing threads is required as xbmc will crash or start improperly. I suspect this is partly due to my failing computer and partly due to the code. Anyone up for helping test this? Lastly, I am unsure if this was introduced in the previous version or this(it's too late for me to check the previous code) but I have incorporated Simple Downloader to queue downloads. I don't think Simple Downloader will be able to start downloads that are old but it worked perfectly for downloads that were queued with the hour. I would be nice if Simple Downloader offered a mechanism to call a plugin to resolve a url at the time it ready to download.

Well here it is:

https://github.com/joedev1981/joedev1981...-0.0.2.zip

[joe.dev.1981]

Bug fix: fetchHTML returned a unicode format if an error occurred and remoteDataMiner checked for a str format.

In addition, I went ahead and added a 10s timeout for urlopen but haven't added an option in the setting to select a desired value. I added another type of rule (item_type=video2, link_type=video2) to be able to handle sites that provide a list of embedded links in the video page instead of a direct playable link. Previously I treated the list of embedded links as catcher targets but these are processed in the quality priority specified in the settings/catcher. Users would have to set quality priority to "ask" in order to choose which site they prefer to stream. "video2" is processed partly as type "video" and partly as 'rss'.

BTW, rule types are treated differently. Types:
"video" is only displayed in non-directory views and is playable/downloadable. It never saved to a list.
"video2" is only displayed in non-directory views but is not playable/downloadable. It provides a link to another non-directory view
"next" is only displayed in non-directory views while viewing all sites and any views in single site view mode
"search" self-explanatory, provides a search, also used in all sites to search sites that don't have categories or don't have the selected category
"category" is the last type relevant to all sites, in all sites it is cached and similar categories are grouped together while less common categories are dropped. (this may change as I implemented automatic search for sites that do not have the selected category)
"once" displayed in single site views and only on the first page.(may need some debugging haven't check if this is working or not but eventually)



custom types only work in single site views and are save to a list if in non-directory views or displayed if in a directory view that type(example: item_type=list in directory views displays items of type "list" but not "category" or "sort"...Note: "rss", "sort", "tags", "subcategory" are not considered custom types but function the same).

As mentioned, categories are now cached in all sites to reduced disk writes and improve display speed. Currently, expiration time is hard coded to 1hr. I will try to add a setting option for this as well.

I am considering caching all urlopen responses configurable per type by the user(for certain types and a default for remaining types)

It say release in the zip but this only mean these are version I release to user for testing. They are experimental versions of VideoDevil and you should install the official version of VideoDevil if you want a stable version. XBMC Adult Addons does not support this version. If you have any questions or suggests please post them here and not VideoDevil/XBMC Adult Addons official threads. I wish I could change the title to something like "Unofficial experimental videodevil development" but not sure if its possible.

https://github.com/joedev1981/joedev1981...-0.0.3.zip

[Memphiz]

title changed.

[hstegeman]

The addon videodevil is giving some problems when I press next page. The coding of videodevil is used by all sub addons, so I see the problem on allmost all addon parts.

Addon redtube, searching for "bobbi starr"

bobbi starr, search page 1
"http://www.redtube.com/pornstar/bobbi+starr"
10:11:58 T:2288 DEBUG: Saving fileitems [plugin://plugin.video.videodevil-joedev1981/?cfg=redtube.com.cfg&director=VideoDevil&genre=RedTube&icon=C%3a%5cUsers%5cHarry%5cAppData%5cRoaming%5cXBMC%5caddons%5cplugin.video.videodevil-joedev1981%5cresources%5cimages%5csearch.png&mode=11&title=%20%20Search%20%20&type=search&url=http%3a%2f%2fredtube.com%2f%3fsearch%3d%25s]

I would expect to see the url part as "http://www.redtube.com/pornstar/bobbi+starr"

bobbi starr, search page 2
"http://www.redtube.com/pornstar/bobbi+starr?page=2"
10:14:45 T:2288 DEBUG: Saving fileitems [plugin://plugin.video.videodevil-joedev1981/?cfg=redtube.com.cfg&director=VideoDevil&genre=RedTube&icon=C%3a%2fUsers%2fHarry%2fAppData%2fRoaming%2fXBMC%2faddons%2fplugin.video.videodevil-joedev1981%2fresources%2fimages%2fnext.png&mode=10&title=Next%20Page&type=next&url=http%3a%2f%2fwww.redtube.com%2f]

The problem is that the second page is showing the content from the website homepage, new.
I would expect to see the url part as "http://www.redtube.com/pornstar/bobbi+starr?page=2"

top rated, page 1
"http://www.redtube.com/top"
10:36:07 T:2288 DEBUG: Saving fileitems [plugin://plugin.video.videodevil-joedev1981/?cfg=redtube.com.cfg&director=VideoDevil&genre=RedTube&icon=C%3a%2fUsers%2fHarry%2fAppData%2fRoaming%2fXBMC%2faddons%2fplugin.video.videodevil-joedev1981%2fresources%2fimages%2fface_devil_grin.png&mode=12&title=%20Top%20Rated%20&type=links&url=http%3a%2f%2fredtube.com%2ftop]

top rated, page 2
"http://www.redtube.com/top?page=2"
10:37:07 T:2288 DEBUG: Saving fileitems [plugin://plugin.video.videodevil-joedev1981/?cfg=redtube.com.cfg&director=VideoDevil&genre=RedTube&icon=C%3a%2fUsers%2fHarry%2fAppData%2fRoaming%2fXBMC%2faddons%2fplugin.video.videodevil-joedev1981%2fresources%2fimages%2fnext.png&mode=10&title=Next%20Page&type=next&url=http%3a%2f%2fwww.redtube.com%2f]

The problem is that the second page is showing the content from the website homepage, new.
I would expect to see the url part as "http://www.redtube.com/top?page=2"

The folder top rated is not on top of the menu, it's between the video items.

The addon ah-me does have simular problems, by example the categorie section.

anal, page 1
"http://www.ah-me.com/channels/43/anal/most-recent/page1.html"
10:19:07 T:2288 DEBUG: Saving fileitems [plugin://plugin.video.videodevil-joedev1981/?cfg=ahme.com.cfg&director=VideoDevil&genre=Ah-Me&icon=http%3a%2f%2fahmestatic.fuckandcdn.com%2fah-me%2fahmethumbs%2fmisc%2fcat43.jpg&mode=10&title=%20Anal%20&type=category&url=http%3a%2f%2fwww.ah-me.com%2fchannels%2f43%2fanal%2fmost-recent%2fpage1.html]

anal, page 2
"http://www.ah-me.com/channels/43/anal/most-recent/page2.html?sort=most-recent"
10:20:38 T:2288 DEBUG: Saving fileitems [plugin://plugin.video.videodevil-joedev1981/?cfg=ahme.com.cfg&director=VideoDevil&genre=Ah-Me&icon=C%3a%2fUsers%2fHarry%2fAppData%2fRoaming%2fXBMC%2faddons%2fplugin.video.videodevil-joedev1981%2fresources%2fimages%2fnext.png&mode=10&title=Next%20Page&type=next&url=http%3a%2f%2fwww.ah-me.com%2fp]

The problem is that the second page is empty, I don't see content.
I would expect to see the url part as "http://www.ah-me.com/channels/43/anal/most-recent/page2.html?sort=most-recent"

The original videodevil has simular problems, see for my post on that thread. Post: #119
http://forum.xbmc.org/showthread.php?tid...pid1719088

[joe.dev.1981]

If it is in the official version of videodevil as well as this experimental version I suspect the issue resides in the site configs. Either the site configs have to be updated or if there isn't a mechanism in videodevil to handle/build the parsed url one may need to be incorporated. The "Next page" in my latest version posted here has an even bigger problem. It builds the "Next page" urls incorrectly most of the time. This bug was introduced when I reverted loadRemote back to a "for" loop. I have a fix for it already as well as some patches provided by another member(would like to give him credit but given the nature of this program I won't mention names here unless they have given me permission or post here). I have eliminated unnecessary items rules from being sent to loadRemote. Overall this will be the fastest version I will post here(when its ready and all so I am moving to start a new job tuesday, may take some time but hopefully not more than a couple of weekends).

With all that said, I have to iron out one more major bug in this next version then I start updating the site configs. This is where most of the bugs pop up. I will start by targeting your findings. Hopefully, I can find a solution but I am a real amature when it comes to python and most things for me to accomplish require much research.

Wish me luck.

[joe.dev.1981]

Got to thinking more about your bug report. btw thanks for test these out! I realized this is one of the reasons I began experimenting with videodevil to begin with. in earlier versions I would perform an action on rules that were affected in this way. The problem is when we scrape "http://www.......bobbie star....." some sites expect "http://www.......bobbie+star....." while others expect "http://www.......bobbie%20star.....". my strategy was to hard code videodevil to return "http://www.......bobbie%20star....." and add an action to be performed in the rule actions in the site configs to return "http://www.......bobbie+star....." if necessary. this worked well and I kept it in the code for a long while. Recently I stripped videodevil down to only perform the minimum required to display items and play them. Slowly I have been adding back all the processing stuff I removed while trying to optimize them. This one slipped through the cracks. I implemented the fix but keep in mind it may break other sites, those sites will need some kind of action place in the item rule to get them working


00:14:10 T:2228 DEBUG: Saving fileitems [plugin://plugin.video.videodevil-joedev1981/?cfg=redtube.com.cfg&director=VideoDevil&genre=RedTube&icon=C%3a%5cUsers%5chtpc%5cAppData%5cRoaming%5cXBMC%5caddons%5cplugin.video.videodevil-joedev1981%5cresources%5cimages%5cnext.png&mode=10&title=Next%20Page&type=next&url=http%3a%2f%2fwww.redtube.com%2f%3fsearch%3dbobbi%20star%26page%3d2]


BTW, even though you have debug enabled in xbmc, videodevil has its own debug switch gives us an easier to read log of what params were passed. For example the above DEBUG line looks like this:

00:14:58 T:4704 NOTICE: currentView:
title=Next Page
url=http://www.redtube.com/?search=bobbi%20star&page=4
cfg=redtube.com.cfg
director=VideoDevil
genre=RedTube
type=next
icon=C:\Users\htpc\AppData\Roaming\XBMC\addons\plugin.video.videodevil-joedev1981\resources\images\next.png

Currently my debugging isn't all that great but I am trying to improve it slowly, would like to eventually make it function with xbmc's debug log with levels. Good news is what I thought would take several days to debug may have been resolved while I was looking into a fix for this. Will test some before I commit another version. Will check out xham... later.

EDIT
I haven't tested this lately with these sites but you should be able to add an action to the rules of the affected sites to replace space with %20 or space with +.
redtube in the "next" rule(note, the order of the items is important and may endup giving you runtime errors if the correct order is not specified). "item_infos_actions" goes in the line before "item_url_build="
item_infos_actions=replace('url', ' ', '%20')

[hstegeman]

The debug option is very useful, it is so much easier to read the xbmc log file.
Load remote just shows the loaded url, without the strange url recoding. You can just copy past an url for testing.

Edit:
I saw load remote in the original videodevil, in your version the xbmc log is less readable.

[hstegeman]

@joe.dev.1981,
I know that you want to make an alternative version from videodevil, but would it be possible that you could look at the original version, make a fork version from videodevil to solve problems that have been there for a long time?

[joe.dev.1981]

The problem is when we scrape "http://www.......bobbie star....." some sites expect "http://www.......bobbie+star....." while others expect "http://www.......bobbie%20star.....".

Unless videodevil itself is patched with "item_infos_actions" I don't think there is an easy efficient way to do it.

It may be using "info_name" but this would cause video devil to parse the entire webpage over again for each item it finds. In addition you would have to create many "info_name=" to scrape each part of the url and then join them by naming the info_name with ".append" for 30 items you may end up parsing an entire website +60 times. I am not an expert in manipulating the site config files if anyone know an easy/efficient way to accomplish this let me know as it would make things a lot easier.

I can fork it but I can't merge "item_infos_actions" into the fork. Videodevil uses paired list for its items and I use a dictionary. This would take time away from this project. If anyone wants to help, I can guide them and point them to the relevant code that needs to ported back into a fork.

I am working on getting better debugging in my version but the code changes so frequently it takes to much time to code relevant debug info. This is going to change soon I am reaching a point when no new features will be added and I can devote time to stuff like relevant debug info.

If some want to take responsibility to patch videodevil let me know, you may also want to ask XBMC Adult Addons if they are interested in such a patch.

EDIT:

Eventually I will submit pull requests for portions of this version that will benefit videodevil. I would love to see this project merged into videodevil. But only small uncomplicated patch are accepted. Rightly so, videodevil is expected to work and merging my project as is now would be a nightmare for XBMC Adult Addons devs. I post these here to document my progress and in hopes of getting the XBMC community involved in testing and development. The end goal is to improve videodevil without burdening XBMC Adult Addons devs. Contributing anyway you would be helpful. Thank you hstegeman for bringing this to my attention.

EDIT:
With the introduction of more item types and handling them differently having compatible site configs is not currently possible.