Make library mode work without scrapers as well - import all files into the database?

Allow me to give my view of how this might be implemented:
The only important difference between file view and library view is that the file view displays all video files in correct directory structure, while the library view displays only the files that have scraped information. That difference is too small and confusing for new users.

So they have to be merged. Then the following may be needed in addition:

• For each source an extra option should be available like 'add media files to library even if scrape is unsuccessful'.
• Ideally, multiple scrapers can be set for each source, with a priority for each scraper; if one fails it tries the next one.
• Maybe add an option to choose custom categories besides TV shows and Movies, selectable through filters or buttons. This allows for home videos, or anything else the user doesn't consider a regular movie or TV show
  

Then the functionality of the file view is almost completely encased into the library view. For a real file view we already have the 'file manager' under 'system'. Adding a link from the library ('Open in file manager') is also useful.

Gamester17 Wrote:Not exactly, for it to be totally automatic with no user intervention and as user-friendly as possible for the majority of the XBMC userbase; yes all files would be scraped, but not like you seem to think.

File/Folder-name => Regular Expression Importer Filter => New database entry => Extra Media Meta Data from XBMC Scrapers

How this concept would work in practice is that first all files are scanned and each file generate an new entry item in the SQL databese, (and it should be noted that all that will be done in the background so that the user can still use XBMC while all the scanning is happening), this entry item would always contain the location of the file as well as the full file name (which is part of the location of course), it would in this scenario also contain the file-extension (so that it can be filtered/sorted out separately by the end-user if they which) and a name generated by the Regular Expression Importer Filter parser, and all this is done before XBMC goes online with the scrapers to look for additional meta data for this item.

For example:

The user set a source as movies and choose whichever movie scraper for it (as they still need to pick one, in this example we say IMDb), then XBMC starts scanning all files in that source into the database under the category "Movies".

The scanner finds a file named "Pulp.Fiction.(1994).XviD.DVDrip.avi" in that source, it enters the location path including the full file-name into database but it also runs the Regular Expression Importer Filter which cleans up the name it enters for than database entry as only "Pulp Fiction", this because the Regular Expression Importer Filter would parse the name to clean it up, entering the year for that item into the database (as it assume that four digests in a parentheses is the year it was produced), also replacing the dots with spaces and remove the known 'scene ripper' tags which in this example is "xvid" and "dvdrip", also the extension is stored so that the user can still sort by extension. Now if the end-user had an XML formatted NFO for that file then all the meta data from the XML formatted NFO would also have been entered into the database with the Regular Expression Importer Filter parser.

So far so good, our file named "Pulp.Fiction.(1994).XviD.DVDrip.avi" have been entered into the database, (and all that before going online yet), so if the user did not have the internet they would now still be able to search and browse it in the library more.

Now comes the next step which is done in a separate process thread; again, all done in the background without the user really noticing it as they will still be able to use XBMC as normal when all this is going on, being the scene so to speak; XBMC automatically beings to take those parsed names from the database that was generated with the help of the Regular Expression Importer Filter and send them to the IMDb movie scraper and it goes online with the name, and year if available, (note that XBMC now take those names and years from the SQL database entry, not from the file-name), the scraper tries to find a match online, if it does not find a match online then nothing more happens and you are still left with the name generated by Regular Expression Importer Filter to use in the library view mode, but if the scraper does find a match online which is would in our example with "Pulp Fiction" then it will append the meta data it collects to the database entry for that item, and it will also overwrite the initial name generated by the Regular Expression Importer Filter, (that is unless that name was picked up from an XML formatted NFO file, in which case it would use that name instead).


The name and location of the .NFO files does not need to be in the database as the location and full name of the file is always also already in the database, and the NFO file always have the same name as the file-name but with a .NFO as the extension instead. If you make any changes inside the NFO file then XBMC will pick that up on the scan refresh, ...as far as I know XBMC does this already.

I would love this sort of library to be implemented.

Glad to see that some users have similar demand as I do, though mine might sound really odd to you. Anyway, let me add my scenario to the thread. Hopefully one day, you can implement such shortcuts for adding new, unorganized videos to library.

I setup XBMC on Ubuntu specifically for streaming my videos to XBOX. They are online clips, not movies, not TV shows, not anything can be organized logically. I stream them to XBOX so that I can enjoy them on a big screen with my friends. Unfortunately, only videos in the library are provided to XBOX. That's fine, and WMP does the same. However, WMP can brainlessly scan everything into library, which makes my job much easier. For XBMC, I have to manually add each of them into some genre, instead of a scan click.

Since the files are by nature unorganized, the best one can imagine is either a flat list or folder hierarchy. Either gets the job done. It can be in "stupid" genre, "unknown" artists, "non-sense" titles, "alien" contries, etc. I don't care, since they are just a temporary list for streaming. The best solution I can think of is what WMP did, an "unknown" category for everything can't be organized.

it forked out off XBMC, but seems to have more common sense than some pig-headed devs of XBMC...
since they have a file only folder organised option...
haven't we forgotten that the user is King, and not an inflated ego dev?
I guess when you sell your product (like Plex do), you soon realise that indeed the Client (User) is King...
Let us hope XBMC keeps its user base instead of dying the slow death of those who think they know better than their users...
after all who the heck are you coding for?

Surely his point IS valid.

A scraper attempts to resolve information about media through whatever technique it sees fit (typically an online database, but it need not be the case).

Ergo, a scraper that makes an attempt to complete SOME metadata from 'deduction' (filename, foldername etc) is perfectly valid.

File view makes no such attempt to 'deduce' anything. It's raw - pure and simple. Whereas this theoretical scraper adds some semblance of intelligent deduction.

Admittedly it's not much of a scraper by comparison with TMDB et al, but if your media can't resolved online, it's not so easy to get it into the library.

This (in part) is precisely why Music Video's is a mess - folks are having to use creative workarounds to classify concert videos as a movie, or tv series or hand craft an NFO. Music Video is really geared to one video = one song.


There's another reason why it's a good idea too.... and you'll probably scream at this:
It would allow alternative metadata (fanart, thumbs, even nfo alternatives) to be used. It's just a 'scaper' that works locally instead of via the net. Perhaps that's already possible with the current scraper architecture?

(2012-02-23 09:02)bgrau2000 Wrote:  it forked out off XBMC, but seems to have more common sense than some pig-headed devs of XBMC...
since they have a file only folder organised option...
haven't we forgotten that the user is King, and not an inflated ego dev?
I guess when you sell your product (like Plex do), you soon realise that indeed the Client (User) is King...
Let us hope XBMC keeps its user base instead of dying the slow death of those who think they know better than their users...
after all who the heck are you coding for?

I think that's a little harsh, but it's not entirely without merit.
I don't think it's pig headed developers though... I think it's any combination of the following:

1) It's been considered before and discounted (and once that's happened, people forget that sometimes an old idea takes on new meaning / life), so it's wise to revisit it once in a while.

2) Despite folks THINKING they are open minded, we all get tunnel vision to some extent. You're a Windows fan, or you're a Linux or Mac fan etc and there's a Right and Wrong way to do it. The 'current' way is right. The more fundamental your idea, the harder it will be for devs to be motivated. I'm a developer and I know in my heart of hearts when something is flawed, but I STILL carry on and bury my head in the sand, because at any given moment in time, the workaround is a better option than the fundamental fix... except in the end, you've put it off for so long, it's just a nightmare to contemplate.

3) It's a total pain in the ass to code something that goes against every principle in your body, but the client wants. Esp when you're convinced it's illogical or downright wrong. That's because I think I'm smarter than the client, and it's my job to advise them on the best design for them. I don't often think I'm wrong.
If I were my own client, I think I'd have fired me.

I'll say this though... and it hurts like hell for 99% of developers of a product...

Put your product in the hands of someone who's never used it before, and watch your 'usability' theories go right out of the window. It beggars belief what users (many intelligent) will do or assume.

I would like it if (as mentioned above) scrapers and .nfo files were optional or at very least if you could tell the scraper easily (GUI would be great) that a video file was a certain movie/tv show. I copy a lot of my dvd's and tv series onto my computer and it doesn't pick most of them up.

If you went with a GUI that allowed you to select media files as movies/tv episodes it would need to allow you to pick from the season list downloaded from the TVDB site or just type in the details manually.

Thank you.

After spending this morning exporting path/filename/imdbID from the database and joining this information to CSV files containing the contents of all my Moviefolders, looking for files that do not have an IMDB ID and thus are not scraped into my database.
Having themoviedb on the left screen, changing these unknown filenames on the filesystem.

So after finding 18 unscraped movies in 700+ files i did not want to do this excercise for my TV show's.

I would really like an option to scan unknown files in my folders into a special category. not cluttering up my library view.

It's not the editing of the fileinformation that takes time, it's finding the unscraped files inbetween large numbers of files.

Konijntje: you do know about the Missing Movie Scanner addon, don't you? http://wiki.xbmc.org/index.php?title=Add...ie_Scanner

(2011-04-23 04:05)mcosta1973 Wrote:  Let me give you an example of what I tried to do to add a file with metadata info to the library.

1. I entered File Browse mode and went to a folder I have called "Blue Planet".
2. Right-clicked on that folder to Set Content to TV Show. When to TV show information, it found Blue Planet and loaded in all the info for the show.
3. Inside the Blue Planet folder, I have many files in this format "BBC - Blue Planet 02 - The Deep.avi". If I right click on that file and select Episode Information, I would have expected to be able to edit the details manually for that episode (setting season and episode number) and then run the scraper with that info. Instead it runs the scraper automatically and returns no info found.
4. If I add the file manually to the library, it doesn't appear under the Blue Planet show in the Video Library.
5. If I manually edit the database, I can get the episode to appear under the Blue Planet show, but then clicking Episode Info on it just resets the data and it disappears from the library.

I know the answer now to make it work would be to rename the files, but there are potentially thousands of them and it is not something I am willing to do. What would be ideal would be the ability to individually scrape episodes inside the gui after you have entered the appropriate data.

This.
+ 1 for add to library by name.