+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=26)
+--- Thread: How to scrape an "infinite" scrolling website like this? (/showthread.php?tid=337527)
How to scrape an "infinite" scrolling website like this? - A2020 - 2018-11-15
So I saw some methods about how to scrape "infinite" scrolling websites, but they don't apply to the site I'm trying to scrape.
Maybe someone with more experience with scrappers can take a look and help me.
PS: It's an adult site.
json:https://www.myfreecams.com/#Homepage
I'm not sure on how to scrape a website like that in "pages"
Once again thanks for any help.
RE: How to scrape an "infinite" scrolling website like this? - Roman_V_M - 2018-11-15
Self-quote from another topic:
(2018-11-05, 14:37)Roman_V_M Wrote: with modern JavaScript-based websites the situation gets more complicated. You need to use either a headless browser engine, which is not a trivial task for a Kodi addon, or to reverse-engineer the front-end JavaScript application to find REST endpoints it gets its data from and use those endpoints in your addon to fetch data.
RE: How to scrape an "infinite" scrolling website like this? - A2020 - 2018-11-16
(2018-11-15, 22:40)Roman_V_M Wrote: Self-quote from another topic:Thanks, will take a look into things like that I see what I can find about it
(2018-11-05, 14:37)Roman_V_M Wrote: with modern JavaScript-based websites the situation gets more complicated. You need to use either a headless browser engine, which is not a trivial task for a Kodi addon, or to reverse-engineer the front-end JavaScript application to find REST endpoints it gets its data from and use those endpoints in your addon to fetch data.
