Make library mode work without scrapers as well - import all files into the database?

  Thread Rating:
  • 2 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
rwparris2 Offline
Team-XBMC Python Developer
Posts: 1,333
Joined: Jan 2008
Reputation: 2
Location: US
Post: #31
TeknoJnky Wrote:Good idea, for my own use, I would rather have all videos in one node and use filters when I want to show only 1 category.

That could be done with my proposed feature, however at this point I have to wonder why not just use file mode?

Always read the XBMC online-manual, FAQ and search and search the forum before posting.
For troubleshooting and bug reporting please read how to submit a proper bug report.

If you're interested in writing addons for xbmc, read docs and how-to for plugins and scripts ||| http://code.google.com/p/xbmc-addons/
find quote
Gamester17 Offline
Team-XBMC Forum Moderator
Posts: 10,523
Joined: Sep 2003
Reputation: 10
Location: Sweden
Post: #32
rwparris2 Wrote:A better solution IMO is custom categories. Having Home Movies mixed with 'Real' Movies yet having TV Shows & Music Videos separated is inconsistent, even if you don't consider it clutter.

I created a ticket asking for custom categories here: http://trac.xbmc.org/ticket/4969
http://forum.xbmc.org/showthread.php?tid=33710

Wink

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find quote
rwparris2 Offline
Team-XBMC Python Developer
Posts: 1,333
Joined: Jan 2008
Reputation: 2
Location: US
Post: #33
Gamester17 Wrote:http://forum.xbmc.org/showthread.php?tid=33710

Wink

I forgot about that! Should have done a search... anyways, thanks for adding a link to the trac ticket for me.

Always read the XBMC online-manual, FAQ and search and search the forum before posting.
For troubleshooting and bug reporting please read how to submit a proper bug report.

If you're interested in writing addons for xbmc, read docs and how-to for plugins and scripts ||| http://code.google.com/p/xbmc-addons/
find quote
SwissElite Offline
Senior Member
Posts: 128
Joined: Jan 2011
Reputation: 0
Post: #34
jmarshall Wrote:In particular, I need to hear from those of you who make use of the Genre, Actors, Years, Directors, Studios "nodes" in the movie or tvshow library.

2. Could this be done better in an alternate way (eg by enabling a "filter" on one or more genres)?

I don't use the categories genre, actors, etc. for movies, im only interested in the title of the movies. However, in the videolibrary showing the titles of the movies things could be improved.

Movies that cannot be matched with the scrapper are not added to the library, they have to be added manually or with an .nfo file. This incurs additional effort a) identifying the files that have not been added, b) browsing to them and c) adding them manually to the library. This process could be improved by I.) adding all movies that could be matched with the scrapper as it is done today and II) adding all other movies that could not found during the scrapping but are located in the video folders unformatted (e.g. if the movie "averyniceday_CD1.avi" could not be matched, it should show up as "averyniceday_CD1" in the videolibrary. This way, the "unscrapped" movies could be identified quickly and renamed easily.
(This post was last modified: 2011-01-26 13:13 by Gamester17.)
find quote
SwissElite Offline
Senior Member
Posts: 128
Joined: Jan 2011
Reputation: 0
Post: #35
I read through the whole thread again in order to understand better the process how movies are added to the library. What I am looking for is a way to have all my movies displayed in the library, no matter whether they could be scrapped or not.

The following I read in your insightful post @Gamester17:


Gamester17 Wrote:For example:

The user set a source as movies and choose whichever movie scraper for it (as they still need to pick one, in this example we say IMDb), then XBMC starts scanning all files in that source into the database under the category "Movies".

The scanner finds a file named "Pulp.Fiction.(1994).XviD.DVDrip.avi" in that source, it enters the location path including the full file-name into database but it also runs the Regular Expression Importer Filter which cleans up the name it enters for than database entry as only "Pulp Fiction", this because the Regular Expression Importer Filter would parse the name to clean it up, entering the year for that item into the database (as it assume that four digests in a parentheses is the year it was produced), also replacing the dots with spaces and remove the known 'scene ripper' tags which in this example is "xvid" and "dvdrip", also the extension is stored so that the user can still sort by extension. Now if the end-user had an XML formatted NFO for that file then all the meta data from the XML formatted NFO would also have been entered into the database with the Regular Expression Importer Filter parser.

So far so good, our file named "Pulp.Fiction.(1994).XviD.DVDrip.avi" have been entered into the database, (and all that before going online yet), so if the user did not have the internet they would now still be able to search and browse it in the library more.

Now, if some movies are not displayed in the list of movies, altough there are located in one of the locations that have been selected as the sources of movies, it is not because they could not be scrapped, but because there is some problem with the importing process using the "Regular Expression Importer Filter parser"? How do i get to that conclusion? Following the logic in your post, the movies get added to the database, no matter wether there is an internet connection to the scraper database.

If my understanding is correct (correct me if not), then the process as it stands is as follows:

1. Importing of movies to database using "Regular Expression Importer Filter parser"

Outcome: If Moviename could be formatted correctly, it is added, otherwise it is not.

2. Adding additional information to the movie name by downloading it fromthe scrapper database.

Outcome: Moviename (those which were added) + Additional information.


My suggestion would be to change this process slightly.

1. Importing of movies to database using "Regular Expression Importer Filter parser"

Outcome: If Moviename could be formatted correctly, it is added as it is done today. Otherwise it is added with the filename and the annotation [format error]

2. Those movies in the database not containing a format error are looked up as it is done today in scrapper.

Outcome: Moviename (those which could be formatted) + additioanal information; Filename (those which could not be formatted) + annotation (format error)


With this process, the user sees all the movies that could not be formatted automatically, and can change them manually, but there is no need anymore to check for each movie wether it has been added and if not, to go and look for it, and add it manually.


Happy to receive your feedback, Gamester Smile (and rest Tongue)
find quote
SwissElite Offline
Senior Member
Posts: 128
Joined: Jan 2011
Reputation: 0
Post: #36
i'd still be interested in a feedback in a better working scraper Wink
find quote
psikohunter Offline
Junior Member
Posts: 4
Joined: Apr 2011
Reputation: 0
Post: #37
I agree with this. Unlike the scrapers/library options for Movies and TV show. the options for Music videos could be much better. The following are just a few I would personally like to see fixed/changed.

I have an extensive music video collection from all around the world. Including both new and older videos. And because of this only a small portion are found with any given scraper (e.g MTV Yahoo). Also the way music videos are listed in the library is confusing. no matter if its list or media info, By artist, album or title. it is always listed by title first making it difficult to find the video you want.

What I would like to see is an improvement in the way they are listed. for example if you chose to list by artist it should display the name as "artist - title." And truly list alphabetically by artist first, not song title. (My main issue)
Second I would like the option to "scrape" the folder and add them by file name. I have my videos named in the standard convention as such: "artist - title." This way all the videos will be added, and not just a portion. and they will all be correct. No misinformation or wrong names from scraping the internet.

Just my opinion.
find quote
mcosta1973 Offline
Junior Member
Posts: 1
Joined: Apr 2011
Reputation: 0
Post: #38
I am currently using mythtv and considering switching to XBMC, the only thing preventing me is the lack of this feature. The Video Library interface much slicker and would be my preferred browsing option - if I have to use file mode to view all my shows, I might as well stay with what I use currently.

I have around 9,000 movies/tv episodes and do not want to go through the process of renaming them to match the scraper format (although 80% of them already conform to it) and will definitely not be creating hundreds of .nfo files.

I'm confused as to why I can't add a file to the library and manually set the tvdb info through the gui. It seems to me that you should be able to edit all metadata information through the interface. Mythtv scans all new files into the library and associates them with the folder they are located in, then you have the opportunity to Edit Details - changing the tmdb/tvdb number, season and episode number, and everything else. Then you run the scraper on that entered info and it grabs the artwork/etc. from the site.

Let me give you an example of what I tried to do to add a file with metadata info to the library.

1. I entered File Browse mode and went to a folder I have called "Blue Planet".
2. Right-clicked on that folder to Set Content to TV Show. When to TV show information, it found Blue Planet and loaded in all the info for the show.
3. Inside the Blue Planet folder, I have many files in this format "BBC - Blue Planet 02 - The Deep.avi". If I right click on that file and select Episode Information, I would have expected to be able to edit the details manually for that episode (setting season and episode number) and then run the scraper with that info. Instead it runs the scraper automatically and returns no info found.
4. If I add the file manually to the library, it doesn't appear under the Blue Planet show in the Video Library.
5. If I manually edit the database, I can get the episode to appear under the Blue Planet show, but then clicking Episode Info on it just resets the data and it disappears from the library.

I know the answer now to make it work would be to rename the files, but there are potentially thousands of them and it is not something I am willing to do. What would be ideal would be the ability to individually scrape episodes inside the gui after you have entered the appropriate data.
find quote
sokolum Offline
Junior Member
Posts: 15
Joined: Apr 2011
Reputation: 0
Post: #39
spiff Wrote:where are those movies from? reason i ask if so we can make a scraper to cover them if a suitable backend is available.

Like homemade movies maybe?
I have a lot of them.

It sounds the option we dont have, adding subcategories/genres:
http://forum.xbmc.org/showthread.php?tid=33710

Edit: added link
(This post was last modified: 2011-05-01 08:57 by sokolum.)
find quote
stensgaard Offline
Junior Member
Posts: 41
Joined: Oct 2011
Reputation: 0
Post: #40
I would like to add my vote to the original posters idea!

but I think we could make this work so everyone would be happy, simply by changing the order things are done Smile

1: add movie to database using just the folder name and a default (file missing) cover!
2: check if there is a folder.jpg in the folder - and use that if there is one
(now everyone with home movies are happy)
3: run the scraper like we do today, and if movie is found update the info
and if the movie is not found, it's still visible on the wall Smile

so just by making proper folder names and adding a folder.jpg we can add any home movie to our movie wall!

hope this makes sense! Smile
find quote
Woolly Offline
Junior Member
Posts: 5
Joined: Dec 2011
Reputation: 0
Post: #41
Allow me to give my view of how this might be implemented:
The only important difference between file view and library view is that the file view displays all video files in correct directory structure, while the library view displays only the files that have scraped information. That difference is too small and confusing for new users.

So they have to be merged. Then the following may be needed in addition:
  • For each source an extra option should be available like 'add media files to library even if scrape is unsuccessful'.
  • Ideally, multiple scrapers can be set for each source, with a priority for each scraper; if one fails it tries the next one.
  • Maybe add an option to choose custom categories besides TV shows and Movies, selectable through filters or buttons. This allows for home videos, or anything else the user doesn't consider a regular movie or TV show
Then the functionality of the file view is almost completely encased into the library view. For a real file view we already have the 'file manager' under 'system'. Adding a link from the library ('Open in file manager') is also useful.
(This post was last modified: 2011-12-22 00:03 by Woolly.)
find quote
dlblank Offline
Junior Member
Posts: 30
Joined: Jun 2009
Reputation: 0
Post: #42
Gamester17 Wrote:Not exactly, for it to be totally automatic with no user intervention and as user-friendly as possible for the majority of the XBMC userbase; yes all files would be scraped, but not like you seem to think.

File/Folder-name => Regular Expression Importer Filter => New database entry => Extra Media Meta Data from XBMC Scrapers

How this concept would work in practice is that first all files are scanned and each file generate an new entry item in the SQL databese, (and it should be noted that all that will be done in the background so that the user can still use XBMC while all the scanning is happening), this entry item would always contain the location of the file as well as the full file name (which is part of the location of course), it would in this scenario also contain the file-extension (so that it can be filtered/sorted out separately by the end-user if they which) and a name generated by the Regular Expression Importer Filter parser, and all this is done before XBMC goes online with the scrapers to look for additional meta data for this item.

For example:

The user set a source as movies and choose whichever movie scraper for it (as they still need to pick one, in this example we say IMDb), then XBMC starts scanning all files in that source into the database under the category "Movies".

The scanner finds a file named "Pulp.Fiction.(1994).XviD.DVDrip.avi" in that source, it enters the location path including the full file-name into database but it also runs the Regular Expression Importer Filter which cleans up the name it enters for than database entry as only "Pulp Fiction", this because the Regular Expression Importer Filter would parse the name to clean it up, entering the year for that item into the database (as it assume that four digests in a parentheses is the year it was produced), also replacing the dots with spaces and remove the known 'scene ripper' tags which in this example is "xvid" and "dvdrip", also the extension is stored so that the user can still sort by extension. Now if the end-user had an XML formatted NFO for that file then all the meta data from the XML formatted NFO would also have been entered into the database with the Regular Expression Importer Filter parser.

So far so good, our file named "Pulp.Fiction.(1994).XviD.DVDrip.avi" have been entered into the database, (and all that before going online yet), so if the user did not have the internet they would now still be able to search and browse it in the library more.

Now comes the next step which is done in a separate process thread; again, all done in the background without the user really noticing it as they will still be able to use XBMC as normal when all this is going on, being the scene so to speak; XBMC automatically beings to take those parsed names from the database that was generated with the help of the Regular Expression Importer Filter and send them to the IMDb movie scraper and it goes online with the name, and year if available, (note that XBMC now take those names and years from the SQL database entry, not from the file-name), the scraper tries to find a match online, if it does not find a match online then nothing more happens and you are still left with the name generated by Regular Expression Importer Filter to use in the library view mode, but if the scraper does find a match online which is would in our example with "Pulp Fiction" then it will append the meta data it collects to the database entry for that item, and it will also overwrite the initial name generated by the Regular Expression Importer Filter, (that is unless that name was picked up from an XML formatted NFO file, in which case it would use that name instead).


The name and location of the .NFO files does not need to be in the database as the location and full name of the file is always also already in the database, and the NFO file always have the same name as the file-name but with a .NFO as the extension instead. If you make any changes inside the NFO file then XBMC will pick that up on the scan refresh, ...as far as I know XBMC does this already.

I would love this sort of library to be implemented.
find quote
wujj123456 Offline
Junior Member
Posts: 1
Joined: Dec 2011
Reputation: 0
Post: #43
Glad to see that some users have similar demand as I do, though mine might sound really odd to you. Anyway, let me add my scenario to the thread. Hopefully one day, you can implement such shortcuts for adding new, unorganized videos to library.

I setup XBMC on Ubuntu specifically for streaming my videos to XBOX. They are online clips, not movies, not TV shows, not anything can be organized logically. I stream them to XBOX so that I can enjoy them on a big screen with my friends. Unfortunately, only videos in the library are provided to XBOX. That's fine, and WMP does the same. However, WMP can brainlessly scan everything into library, which makes my job much easier. For XBMC, I have to manually add each of them into some genre, instead of a scan click.

Since the files are by nature unorganized, the best one can imagine is either a flat list or folder hierarchy. Either gets the job done. It can be in "stupid" genre, "unknown" artists, "non-sense" titles, "alien" contries, etc. I don't care, since they are just a temporary list for streaming. The best solution I can think of is what WMP did, an "unknown" category for everything can't be organized.
find quote
bgrau2000 Offline
Junior Member
Posts: 1
Joined: Feb 2012
Reputation: 0
Post: #44
it forked out off XBMC, but seems to have more common sense than some pig-headed devs of XBMC...
since they have a file only folder organised option...
haven't we forgotten that the user is King, and not an inflated ego dev?
I guess when you sell your product (like Plex do), you soon realise that indeed the Client (User) is King...
Let us hope XBMC keeps its user base instead of dying the slow death of those who think they know better than their users...
after all who the heck are you coding for?
find quote
AnalogKid Offline
Fan
Posts: 648
Joined: Feb 2009
Reputation: 141
Post: #45
Surely his point IS valid.

A scraper attempts to resolve information about media through whatever technique it sees fit (typically an online database, but it need not be the case).

Ergo, a scraper that makes an attempt to complete SOME metadata from 'deduction' (filename, foldername etc) is perfectly valid.

File view makes no such attempt to 'deduce' anything. It's raw - pure and simple. Whereas this theoretical scraper adds some semblance of intelligent deduction.

Admittedly it's not much of a scraper by comparison with TMDB et al, but if your media can't resolved online, it's not so easy to get it into the library.

This (in part) is precisely why Music Video's is a mess - folks are having to use creative workarounds to classify concert videos as a movie, or tv series or hand craft an NFO. Music Video is really geared to one video = one song.


There's another reason why it's a good idea too.... and you'll probably scream at this:
It would allow alternative metadata (fanart, thumbs, even nfo alternatives) to be used. It's just a 'scaper' that works locally instead of via the net. Perhaps that's already possible with the current scraper architecture?
find quote