Make library mode work without scrapers as well - import all files into the database?

  Thread Rating:
  • 2 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
Gamester17 Offline
Team-XBMC Forum Moderator
Posts: 10,523
Joined: Sep 2003
Reputation: 9
Location: Sweden
Lightbulb   
Post: #16
spiff Wrote:see, what you actually want is a file node in library. not to scan all vids into the db
Well, not exactly, if all files where scanned into the database first on file/folder name alone, (before going online to look for meta data for it), then the user could still use the library search function and live search to find files. Without a database that stores all the file/folder names searching is not exactly usable if no metadata for the item was found online.

I touched this discussion with the MeediOS developers when I suggested that they implement XBMC's scraper parser into MeediOS so that we could share that code, and there I learned that MeediOS process flow is slightly different than XBMC.
http://www.meedios.com/forum/viewtopic.php?t=2238

Currently the process flow is like this in XBMC:
File/Folder-name => Regular Expression Importer Filter => Media Meta Data from XBMC Scrapers => New database entry

While with using MeediOS concept the process flow in XBMC would be a little more like this:
File/Folder-name => Regular Expression Importer Filter => New database entry => Extra Media Meta Data from XBMC Scrapers

I non XBMC developers do not know; what the "Regular Expression Importer Filter" does is try to cleanup the name before preforming the lookup, like remove the file-extension, replace dots with spaces, assume that four digests in a parentheses is the year it was produced, remove some known 'scene ripper' tags such as "xvid", "x264", "dvdrip", "dvdscr" and such.

Again, please read thoroughly through this related discussion if you are truly interested in what was said previously about this idea as it really depends on how easy we can first make it for new users to add sources to XBMC: http://forum.xbmc.org/showthread.php?tid=14186

If implemented then this should IMHO be the future default way to browse view mode in XBMC Nod
...sure we would still keep file-mode for those who insists on using but it would no longer be the default.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
(This post was last modified: 2008-09-16 13:57 by Gamester17.)
find quote
markiz Offline
Senior Member
Posts: 270
Joined: Feb 2008
Reputation: 0
Location: Sweden
Post: #17
Gamester17 Wrote:While with using MeediOS concept the process flow in XBMC would be a little more like this:
File/Folder-name => Regular Expression Importer Filter => New database entry => Extra Media Meta Data from XBMC Scrapers

Does your suggestion imply that XBMC scrapers are mandatory. I feel that XBMC scraping should be optional. I don't want to scrape all my movies since I already know that nothing good will come from it.

May I also suggest that you add the name to the .nfo-file, so that it will be easy to add new information manually (with XBMC Media Companion/notepad/...). AFAIK there is no software available that modifys the movie/tv-series information in the XBMC database directly.
find quote
Gamester17 Offline
Team-XBMC Forum Moderator
Posts: 10,523
Joined: Sep 2003
Reputation: 9
Location: Sweden
Lightbulb  Prepare yourself for a long read, step-by-step details on this process flow concept.
Post: #18
markiz Wrote:
Gamester17 Wrote:While with using MeediOS concept the process flow in XBMC would be a little more like this:
File/Folder-name => Regular Expression Importer Filter => New database entry => Extra Media Meta Data from XBMC Scrapers
Does your suggestion imply that XBMC scrapers are mandatory.
Not exactly, for it to be totally automatic with no user intervention and as user-friendly as possible for the majority of the XBMC userbase; yes all files would be scraped, but not like you seem to think.

File/Folder-name => Regular Expression Importer Filter => New database entry => Extra Media Meta Data from XBMC Scrapers

How this concept would work in practice is that first all files are scanned and each file generate an new entry item in the SQL databese, (and it should be noted that all that will be done in the background so that the user can still use XBMC while all the scanning is happening), this entry item would always contain the location of the file as well as the full file name (which is part of the location of course), it would in this scenario also contain the file-extension (so that it can be filtered/sorted out separately by the end-user if they which) and a name generated by the Regular Expression Importer Filter parser, and all this is done before XBMC goes online with the scrapers to look for additional meta data for this item.

For example:

The user set a source as movies and choose whichever movie scraper for it (as they still need to pick one, in this example we say IMDb), then XBMC starts scanning all files in that source into the database under the category "Movies".

The scanner finds a file named "Pulp.Fiction.(1994).XviD.DVDrip.avi" in that source, it enters the location path including the full file-name into database but it also runs the Regular Expression Importer Filter which cleans up the name it enters for than database entry as only "Pulp Fiction", this because the Regular Expression Importer Filter would parse the name to clean it up, entering the year for that item into the database (as it assume that four digests in a parentheses is the year it was produced), also replacing the dots with spaces and remove the known 'scene ripper' tags which in this example is "xvid" and "dvdrip", also the extension is stored so that the user can still sort by extension. Now if the end-user had an XML formatted NFO for that file then all the meta data from the XML formatted NFO would also have been entered into the database with the Regular Expression Importer Filter parser.

So far so good, our file named "Pulp.Fiction.(1994).XviD.DVDrip.avi" have been entered into the database, (and all that before going online yet), so if the user did not have the internet they would now still be able to search and browse it in the library more.

Now comes the next step which is done in a separate process thread; again, all done in the background without the user really noticing it as they will still be able to use XBMC as normal when all this is going on, being the scene so to speak; XBMC automatically beings to take those parsed names from the database that was generated with the help of the Regular Expression Importer Filter and send them to the IMDb movie scraper and it goes online with the name, and year if available, (note that XBMC now take those names and years from the SQL database entry, not from the file-name), the scraper tries to find a match online, if it does not find a match online then nothing more happens and you are still left with the name generated by Regular Expression Importer Filter to use in the library view mode, but if the scraper does find a match online which is would in our example with "Pulp Fiction" then it will append the meta data it collects to the database entry for that item, and it will also overwrite the initial name generated by the Regular Expression Importer Filter, (that is unless that name was picked up from an XML formatted NFO file, in which case it would use that name instead).


markiz Wrote:May I also suggest that you add the name to the .nfo-file, so that it will be easy to add new information manually (with XBMC Media Companion/notepad/...)
The name and location of the .NFO files does not need to be in the database as the location and full name of the file is always also already in the database, and the NFO file always have the same name as the file-name but with a .NFO as the extension instead. If you make any changes inside the NFO file then XBMC will pick that up on the scan refresh, ...as far as I know XBMC does this already.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find quote
markiz Offline
Senior Member
Posts: 270
Joined: Feb 2008
Reputation: 0
Location: Sweden
Post: #19
Thank you for your explanation. Very interesting! It seems as if this is a small modification to how XBMC works today!

Gamester17 Wrote:if it does not find a match online then nothing more happens and you are still left with the name generated by Regular Expression Importer Filter to use in the library view mode

What if there is a match, but the match is not the correct movie. Is there a way to remove or edit the incorrect data?

And what happens if there are several matches?

Gamester17 Wrote:The name and location of the .NFO files does not need to be in the database as the location and full name of the file is always also already in the database, and the NFO file always have the same name as the file-name but with a .NFO as the extension instead. If you make any changes inside the NFO file then XBMC will pick that up on the scan refresh, ...as far as I know XBMC does this already.

I was referring to the name of the movie. Not the name of the .NFO file. What I mean is that it should be easy to edit the information about the movie, and that the .NFO should contain the information (name of the movie) returned by the Regular Expression Importer Filter.
find quote
Gamester17 Offline
Team-XBMC Forum Moderator
Posts: 10,523
Joined: Sep 2003
Reputation: 9
Location: Sweden
Post: #20
Note that this is all a concept, not something that are in the works or even planned yet. We are still just brainstorming openly here, no one has gotten any closer to the dawning table.
markiz Wrote:
Gamester17 Wrote:if it does not find a match online then nothing more happens and you are still left with the name generated by Regular Expression Importer Filter to use in the library view mode
What if there is a match, but the match is not the correct movie. Is there a way to remove or edit the incorrect data?
Yes functions would have to be implemented so that you could choose only to use the Regular Expression Importer Filter generated name, and to edit that name directly from the XBMC GUI. Wrong matches is always going to happen and is unavoidable to get rid of completely. What will make it easier is to have more genres/subgenres in scrapers, video database, and library filters, some of which (like for example "Home Movies") will not be connected to a scraper (and only use NFO files if available).
http://forum.xbmc.org/showthread.php?tid=33710

markiz Wrote:what happens if there are several matches?
Same as today, (when you do a scan today on IMDB movie names get multiple hits), when scanning an entire directory XBMC will pick the top hit, it would be annoying if XBMC stopped to ask on every multiple hit. Again, mismatches will happen, the user should be able to go in after and current them, manually or by doing a new individual online lookup when you get the option to pick the corrent one on multiple hits just like today.

Gamester17 Wrote:The name and location of the .NFO files does not need to be in the database as the location and full name of the file is always also already in the database, and the NFO file always have the same name as the file-name but with a .NFO as the extension instead. If you make any changes inside the NFO file then XBMC will pick that up on the scan refresh, ...as far as I know XBMC does this already.
I was referring to the name of the movie. Not the name of the .NFO file. What I mean is that it should be easy to edit the information about the movie, and that the .NFO should contain the information (name of the movie) returned by the Regular Expression Importer Filter.[/quote]That is already how it works today in XBMC so that would not change, any strings that exist in a XML formatted NFO is prioritized over any meta data collected online with a scraper for that specific string, see: http://wiki.xbmc.org/?title=NFO

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find quote
sho Offline
Team-XBMC Member
Posts: 4,128
Joined: May 2004
Reputation: 3
Post: #21
markiz Wrote:Maby the system doesn't need to be altered at all. Maby the only thing that is needed is a new scraper that "scrapes" info from folders/file names. It could be called "Folder scraper" or "Filename scraper". Or "Default scraper" that looks at the setting "use foldernames when scraping".

I might be wrong here since I'm not a developer, and don't know anything about the inner workings of scrapers.
Did you read my post?

There is something very close to such a thing, it's called Music video NFO scraper.
If a file does not have an associated NFO file it will use the filename. (xxx - yyy may be required).
If you don't have a problem with them being called Music Videos, this could be exactly what you are looking for?

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #22
the way to proceed to add this EVIL (imo) "functionality" would indeed be to extend the filename scraping to be supported for movie content
find quote
sho Offline
Team-XBMC Member
Posts: 4,128
Joined: May 2004
Reputation: 3
Post: #23
I for one would love to have a "home movies" content type with a filename/nfo based scraper.
I had just decided on waiting for post 8.04 to start badgering the devs Wink

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find quote
TeknoJnky Offline
Fan
Posts: 424
Joined: Aug 2008
Reputation: 0
Post: #24
I really like Gamester17's idea and hope it is something that would get implemented in the future.

If it would be easier/quicker to implement, allowing the 'music video' scraper to be utilzed as a fall back for when IMDB or other selected scraper fails, might be a usable alternative.

spiff Wrote:the way to proceed to add this EVIL (imo) "functionality" would indeed be to extend the filename scraping to be supported for movie content

I am not sure why you think this is evil. I think this would be a good step into getting people to who use file mode, into trying/using library mode... especially in content situations as likened by markiz.
(This post was last modified: 2012-04-15 03:32 by Ned Scott.)
find quote
rwparris2 Offline
Team-XBMC Python Developer
Posts: 1,333
Joined: Jan 2008
Reputation: 2
Location: US
Post: #25
I tried searching but is there a request in trac for this yet?

If not I'll make one

Always read the XBMC online-manual, FAQ and search and search the forum before posting.
For troubleshooting and bug reporting please read how to submit a proper bug report.

If you're interested in writing addons for xbmc, read docs and how-to for plugins and scripts ||| http://code.google.com/p/xbmc-addons/
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #26
why i think it's evil? because it isnt a solution at all - it is just cluttering your library.
the solution is to make the library SUPPORT the content type you want in there.
find quote
TeknoJnky Offline
Fan
Posts: 424
Joined: Aug 2008
Reputation: 0
Post: #27
spiff Wrote:why i think it's evil? because it isnt a solution at all - it is just cluttering your library.
the solution is to make the library SUPPORT the content type you want in there.

Yes, but the problem appears to be, there can never be any (external) scrapers for custom/home/private content... yet this same content, the user may desire to have within the same library context as other publicly accessible media. IE home movies mixed with hollywood movies.

Besides, clutter is in the eye of the user. While some may consider home movies (or other content with no real metadata) mixed in with regular movies to be clutter, others may not.
(This post was last modified: 2012-04-15 03:32 by Ned Scott.)
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #28
sure. you asked me for my view. i gave you that.
find quote
rwparris2 Offline
Team-XBMC Python Developer
Posts: 1,333
Joined: Jan 2008
Reputation: 2
Location: US
Post: #29
TeknoJnky Wrote:Besides, clutter is in the eye of the user. While some may consider home movies (or other content with no real metadata) mixed in with regular movies to be clutter, others may not.

A better solution IMO is custom categories. Having Home Movies mixed with 'Real' Movies yet having TV Shows & Music Videos separated is inconsistent, even if you don't consider it clutter.

I created a ticket asking for custom categories here: http://trac.xbmc.org/ticket/4969

Always read the XBMC online-manual, FAQ and search and search the forum before posting.
For troubleshooting and bug reporting please read how to submit a proper bug report.

If you're interested in writing addons for xbmc, read docs and how-to for plugins and scripts ||| http://code.google.com/p/xbmc-addons/
(This post was last modified: 2008-09-23 19:20 by rwparris2.)
find quote
TeknoJnky Offline
Fan
Posts: 424
Joined: Aug 2008
Reputation: 0
Post: #30
rwparris2 Wrote:A better solution IMO is custom categories. Having Home Movies mixed with 'Real' Movies yet having TV Shows & Music Videos separated is inconsistent, even if you don't consider it clutter.

I created a ticket asking for custom categories here: http://trac.xbmc.org/ticket/4969

Good idea, for my own use, I would rather have all videos in one node and use filters when I want to show only 1 category.

Further, it would be sweet you could color code the categories so that they appeared differently while in an 'all videos' view.
(This post was last modified: 2012-04-15 03:43 by Ned Scott.)
find quote