html source retrived...differ by simple wget
#1
I hope can explain what make me scraper... Rolleyes

if i get this page http://www.mymovies.it/film/2009/avatar/trailer/ with <url> tag, the html source returned is different by a simple linux wget.

Linux wget: http://paste2.org/p/918238 see line 725
Xbmc scraper: http://paste2.org/p/918257 see line 727

wget:
Code:
so.addVariable('plugins','gapro-1,adtvideo');
so.addVariable('adtvideo.config','http://www.mymovies.it/v9/adv/preroll/config_nestea.asp')
  
so.addVariable('file','[b]http://pad.mymovies.it/filmclub/2008/03/090/trailer.flv[/b]');
  
so.addVariable('gapro.accountid', 'UA-259522-1');
so.write('mediaspace');

xbmc:
Code:
so.addVariable('plugins','gapro-1');
                                              
so.addVariable('file','[b]http://www.mymovies.it/v9/include/player/playlist/editoriale.asp?codicefilm=2008/03/090[/b]');
so.addVariable('repeat','list');
                                              
so.addVariable('gapro.accountid', 'UA-259522-1');
so.write('mediaspace');

There is other little change, a javacript <script> block from 745 to 764 line, not present in wget version.

I use latest xbmc pre-10.5 r31718

With 'old' xbmc this didn't happen...and also with xbmc4xbox now.
What is browser identify of xbmc?!

any suggestion?!

thanks!

p.s. another question: for submit an addons i must send a mail to: [email protected] whit subject "[Svn pull] xxxx scraper" and in body url to svn for example!?!?
Reply
#2
can i submit as bug?! Sad

...someone for question in ps?!
Reply
#3
I suspect your guess is accurate and that the website is returning different things based on useragent. I believe you can spoof the referer at least - not sure about the user-agent.

And yes - ensure you take note of the addons guidelines (eg icon size etc.) then just send an SVN pull request to the mailing list.

Cheers,
Jonathan
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#4
thank you!!!

I'll try with referrer, but it's strange that work good with xbmc 9.11 and xbmc4xbox Eek

Quote:And yes - ensure you take note of the addons guidelines (eg icon size etc.) then just send an SVN pull request to the mailing list.

ok!
Reply

Logout Mark Read Team Forum Stats Members Help
html source retrived...differ by simple wget0