Local 'Scraping'

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
Ned Scott Offline
Team-Kodi Wiki Guy
Posts: 20,394
Joined: Jan 2011
Reputation: 258
Location: Arizona, USA
Post: #16
Anything wihtout an nfo file wouldn't be added at all, because there wouldn't be a real database of info to get any info from, thus no entry.

This actually sounds like a really easy and clever solution to the request, which is a somewhat common request too.

You can make easy links to the XBMC wiki using double brackets around common XBMC words: [[debug log]] = debug log, [[Video library]] = Video library, [[SMB]] = SMB , [[userdata]] = userdata, etc
find quote
Koying Online
Team-Kodi Member
Posts: 1,975
Joined: Sep 2008
Reputation: 43
Location: Brussels, Belgium
Post: #17
Mmm... Are you sure of that?

When my current movie scrapper doesn't find an entry in its database, a blank entry is still added to the XBMC library, with no other info than the filename, which is actually desirable in this instance, as it reminds me to update the entry.

But it is maybe scrapper dependent?
BTW, there is a ticket (4 yrs old !) for that which also proposed the "null" scrapper solution: http://trac.xbmc.org/ticket/5382
(This post was last modified: 2012-07-21 11:41 by Koying.)
find quote
scudlee Offline
Team-Kodi Member
Posts: 839
Joined: Jul 2011
Reputation: 51
Post: #18
(2012-07-21 10:46)vdrfan Wrote:  How would it be dirtier code-wise? From what i had in mind the user chooses a "local importer" scraper. As long as there's a full nfo, the scraper code should not try to fetch something from the web. Note, this is only theory Wink Will have to try it later today.

I tried to do this a while back just out of curiosity. One issue I found was that you still need to have a valid URL in CreateSearchUrl to pass to GetSearchResults. I managed to get around it by using the file:/// protocol to point to a non-empty local file - it worked, but there was no way I could see to make it "just work" for everyone without them having to set the path manually. (I had hoped the internal special:// protocol would work, so the dummy file could be stashed with the scraper, but alas no.)
find quote
Koying Online
Team-Kodi Member
Posts: 1,975
Joined: Sep 2008
Reputation: 43
Location: Brussels, Belgium
Post: #19
Oups... I'm not familiar with XBMC ui but it seems going the "add an option" way would break all the skins, right?

Better to go the pseudo-scraper way, then...
find quote
Koying Online
Team-Kodi Member
Posts: 1,975
Joined: Sep 2008
Reputation: 43
Location: Brussels, Belgium
Post: #20
And here is the pull request: https://github.com/xbmc/xbmc/pull/1192
find quote
topfs2 Offline
Team-Kodi Developer
Posts: 4,049
Joined: Dec 2007
Reputation: 10
Post: #21
Just wanted to say that heimdall (my GSoC, where I try to design a new scraper engine for xbmc) will most likely handle NFO the same way as imdb, themoviedb etc. The user of the engine (xbmc) would still need to decide how it should handle the returned data but essentially what you guys want here I think should be possible.

From what I understand you want to be able to:
a) Use NFO data if existant and skip internet lookup (as we do now)
b) Use NFO data in conjunction with internet lookup (as we do now)
c) Skip any NFO existant and just do folder based lookup (not as we do now)
d) Skip internet lookup, only use what, if any, data NFO gives

Have I understood the requests right? Just so I know what I need for heimdall!

If you have problems please read this before posting

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.

[Image: badge.gif]

"Well Im gonna download the code and look at it a bit but I'm certainly not a really good C/C++ programer but I'd help as much as I can, I mostly write in C#."
find quote
Koying Online
Team-Kodi Member
Posts: 1,975
Joined: Sep 2008
Reputation: 43
Location: Brussels, Belgium
Post: #22
(2012-07-21 19:12)topfs2 Wrote:  d) Skip internet lookup, only use what, if any, data NFO gives

That is indeed what I'm wanting to achieve...
find quote
radix Offline
Junior Member
Posts: 6
Joined: Nov 2012
Reputation: 0
Post: #23
I add myself for this request.

(sorry for my english, i speak spanish)

My problem is that i made a menu item for concerts, and i have a lot of them, but i many of them are non listed in any online database, so the scrapers cant find them and never add them to my library. So my only way to watch them is from the files browser,

We need some way to add local info in each rare movie, concert or musicvideo to alow them to exist into our library.
find quote
DiMag Offline
Senior Member
Posts: 209
Joined: Nov 2012
Reputation: 0
Post: #24

I support the local scraper as herein specified, indeed I have suggested something similar in http://forum.xbmc.org/showthread.php?tid...pid1297694 (not sure whether I should consider this post closed).

What I would like to add, and please all post participants take notice, is that the same issue keeps popping up (1) under extremely different headings and (2) spread over the entire forum rather than concentrated in one section. We are far more numerous than we suppose.

find quote
henricos Offline
Junior Member
Posts: 14
Joined: Jan 2013
Reputation: 0
Post: #25
+1 for a Local "Scraping" (only) feature

@DiMag

I believe your suggestion differs from this one

Local "Scraping" here means:
1) We already have a full NFO (and artwork) that was carefully edited - manually or with an external tool
2) We want to skip internet lookup to prevent any mess-up in that information or to avoid unnecessary bandwidth consumption

Your suggestion is something like:
1) Automatically generate NFO from filenames and media info
2) Automatically generate thumbs
3) Allow metadata to be merged into NFO from generic XML supplied by the user

See the difference?
find quote
stepir Offline
Junior Member
Posts: 27
Joined: May 2006
Reputation: 0
Post: #26
+1 here

I have a main Minimac xbmc, + 3 tablet XBMC, + 1 android dongle XBMC on my RV. It's a pain had to correct all the time the errors of TMDB scraping. Much better have a first pass on the main XBMC, have a third party tool to populate the nfo files and use only local info on all the other XBMC!
find quote
AdolfoDias Offline
Junior Member
Posts: 11
Joined: Jan 2011
Reputation: 0
Post: #27
+1 here also.

Finally i've found a thread regarding specifically this important issue.
There are many bits and pieces about/similar to this subject scattered all over this forum, but i believe this topic has not received the attention it deserves.
Cudos for AnalogKid for starting this thread.

For whatever different reasons people want local scraping, it is important that XBMC respects and presents the metadata that we previously prepared and in the way we wanted/intended to be.
I have all my media in a NAS that is prepared and organised just the way i want. Every player in the house access the same NAS and it should just be necessary to point out the locations for music/videos,etc and it should behave the same in all of them. In the way i wanted and spent many time preparing.
That doesn't happen, and the scraping produces erratic and unpredictable results.

XBMC has grown so much and so well in all these years. It has been added with so many features that now it does almost everything except making coffee Wink
Surely online scraping is a great feature, but this produces many errors (more, i believe, for languages other than English).
While this is usefull and acceptable for many people, forcing users with organised and structured collection of media to use this, doesn't seem an improvement.

It seems that we spend all the time looking for errors in the library, analising, finding solutions, correcting and tweaking things rather than actually enjoying our media.
The geak part of me likes tweaking things, but at some point i'll just want to be able to enjoy it and have something to show for.

Please think a little more in terms of usefullness for the dedicated and organised users, and not so much in "swiss army knife" development and "bring a bare media file, we do the rest" ideology.
find quote
DiMag Offline
Senior Member
Posts: 209
Joined: Nov 2012
Reputation: 0
Post: #28
@henricos (belatedly...):
You are right, my quest includes the preparatory step of producing the NFO file with the correct, correctly structured, and maybe customizable, metadata. I have found that it is a far more difficult task than the subsequent one of trying to keep this NFO uncontaminated from unwanted internet lookups (in itself no simple task either, thank you XBMC scrapers).

@all thread participants:
Please join me in requesting (suggesting?) that a new section be opened in the forum to discuss everything library-related, in particular:
  • SQL engine optimization,
  • database schema design (add/remove/link tables),
  • NFO schema design (adjust the XML schema to the SQL schema),
  • creating, per SQL/XML schema tweaking, new media filetypes (in particular recognition of documentaries as a distinct filetype not a genre) and adapting old ones to new tasks (movie.nfo needs for instance only 4 or 6 new tags to be able to serve as metadata container for opera ballet, and theater performances),
  • XSP tweaking,
  • unifying unitary (currently movies) and episodic (currently tvshows) video content under a common video node, a feature currently badly needed in the case of documentaries and anime (you can mix music and musicvideos in the same XSP but you cannot link, per XSP or otherwise, movies to tvshows),
  • customization of video nodes,
  • and skin integration of all the above.
In a perfect world scrapers should also have to be included to library-related issues, but they are presently well organized under add-ons so this is not an issue to be raised.

find quote
Paradise Offline
Member
Posts: 75
Joined: Jun 2009
Reputation: 0
Post: #29
Hi,

is this now somewhere? I see a scrolling text in Gotham in set content: "local information only".
But where is the setting for that?

This online stuff is the thing that is holding me back since years from xbmc.
From time to time i check back if i now can get rid off online sraping.

For me a perfect world would be to put the imdb url ad the end of my nfo and just
get the rating from time to time.

Now there is also this artwork downloader. Nothing shows up until you let it run and
every time i reload my movie information i have to do it again.
If i turn of for example clearart and there is a clearart.png than it does not show up.
If i turn it on and there is no clearart.png it gets downloaded even if i don't want to download stuff...

Damn i only want my local stuff and really want to use xbmc - but since years its only messing
up my handmade information Sad
find quote
Memphiz Offline
Team-Kodi Developer
Posts: 10,855
Joined: Feb 2011
Reputation: 113
Location: germany
Post: #30
get proper nfo files in place next to your movies along with thumbnail and fanart and use "local information only" - this will then only use the local found stuff. I do this all the time.

AppleTV2/iPhone/iPod: HowTo find debug logs and everything else which the devs like so much: click here
HowTo setup NFS for XBMC: Wiki NFS
HowTo configure avahi (zeroconf): Wiki Avahi
READ THE IOS FAQ!: iOS FAQ
find quote