[DiMag]

There is a persisting request, its expression dispersed among several sections of this forum, for a built-in video scraper which takes as argument the filename (appropriately formatted, see below), writes an NFO file, adds content to its tags, adds artwork, and scans the whole to the library. The naming is confusing ---"Custom Video", "Generic Video", "Home Video", "Home Movie", are among the identifications you may encounter---, but once you actually read the posts, you discover a broad consensus as to function and features.

Let us feature specify this scraper as follows:

1. Take the video filename.
   
2. Format it appropriately (in accordance with user-provided settings): say capitalize words,
   
3. Format it appropriately, II: Convert _ or ~ to :, make it possible to use other special characters (such as /, |, ") which would be illegal as part of the filename.
   
4. Write an NFO file with the resulting library title.
   
5. Add mediainfo data (codec, resolution, runtime, etc).
   
6. Unless it already exists, add a screenshot to be user as poster. (The precise runtime point at which to take the screenshot can be customized in settings.)
   
7. Fill the NFO with all content (plot, outline) and filing (sets, tags, genres) related data the user has provided in a same-name xml file. Ideally one (or many) user customizable templates should be made available.
   
8. The above assumes no scrapable metadata exist for the video file in question. But what if some do exist? Ideally there should be a possibility to chain third party scrapers.
   
9. Equally useful would be the possibility to add URL references then let Artwork Downloader get them on disk.
   
10. The scraper must come in two versions, one for single file videos (what XBMC calls "Movies"), the other for episodic content videos (what XBMC calls "TV shows"). Say "Generic Movies" and "Generic TV shows".
    

The necessity for this scraper results from XBMC's limitations to three basic video file types --- movies, tvshows, musicvideos ---, whereas in the real world there are many video files which fall outside this tripartite schema. Thus it can be argued that a generic video scraper, and moreover a scraper for which no metadata sites need exist, should have been in XBMC from the start. Indirect evidence thereof is provided by the fact that as aforesaid, demand for this kind of scraper has been persistent

So has, however, been rejection of it by developers. Their arguments fall into three categories.

Firstly, they are of the sort familiar to viewers of the British TV series "Yes Minister/Yes Prime Minister", where an Oxford educated Permanent Undersecretary routinely blocks change with eloquent, and linguistically beautiful, speeches such as "careful consideration shall reveal that it is already being taken care of by existing institutional arrangements", or "it is an idea worth considering, but surely its time has not yet come". No comment to that.

Secondly, they point to the existence of a third party scraper called JustTheFilename. This is indeed an excellent tool, but as a basis for developing the scraper I am asking for. Once you put it to use, its limitations become obvious. As Tacitus said about Galba, I say about JustTheFilename "omnium consensu, capax imperii nisi imperasset".

Thirdly, I could make the argument (on behalf of the developers, who apparently missed it) that the job as specified above is better done by an external library management tool. And to this (hypothesized) reply, I have the following rejoinders:

Firstly, it is not sure this tool currently exists. The only one which to my knowledge does partly the job as specified above is Media Companion (MC), which has included a so-called "Home Movies" feature since its version (I believe) 3.5.1.5 (at this writing, it is at v3.5.2.0). I have only good things to say about MC's "Home Movies" feature ---I use it practically every day---, and yet it is IMO incomplete in two important regards. (A) It does not allow for user customization of the NFO file it produces. For a tool geared toward generic and custom videso, this omission is rather crucial. (B) It produces single file (movie) NFOs, not episodic (tvshow) ones. Given that many, perhaps most, of the generic/custom videos you wish to include in your library are likely to be of the episodic type ---think of the many multi-part documentaries or online lectures, or the videos you made of your girlfriend turned wife turned mother of your children (Eva, Our First Holiday, Our Marriage, Eva Pregnant)--- this omission too is rather crucial.

Secondly, and on the supposition that MC shall (quite probably, as I might say) quickly redress these omissions, shouldn't the fact that MC has demonstrated that there is need for a generic scraper alert XBMC's team to the possibility that there is something missing from XBMC?

ALSO POSTED in Add-Ons Help and Support/Metadata Scrapers. Moderators, feel free to allocate post where it should be filed.

[Ned Scott]

If you have an nfo file then XBMC already does this...

[Memphiz]

Well afaik scrapers are python addons. So what you propose is completly doable as an addon too. Its just finding someone who does this for you/the others. Its unlikly that core developers will spend time on it though. But we have a dozens of good addon devs in the community (i guess). So if the preparation of an nfo file is not cutting it, then you will need to find someone who is

1. Able to hackup some python
2. Maybe familiar with scraping in general
3. Not an ass like all core devs. :p

No comment on 3.

just my 2 cents.

[Martijn]

3. Damn I'm out then

[DiMag]

@ned Scott:
Yes - if. But have you realized how much time it takes? (1) Use a rudimentary scraper like JustTheFilename to just produce an editable NFO. (2) Retouche/edit this NFO. (3) Call again the scraper to scan it into the library. A fine exercise in wasting time, IMO. Firing the scraper I suggested on the video file and a same-name XML file containing extra data would cut this waste considerably.

[Ned Scott]

You don't need the scraper to make the nfo file. It's just a text file.

I would skip the nfo step completely. Have a scraper or option to add to library with zero info (similar to justthefilename) then use a database editor like XWMM or MediaElch to add the data.

[DiMag]

@ned Scott:
I think I have made what I meant only 50% clear, so let me clarify. You say

I would skip the nfo step completely.

You are right, in principle. You don't need a scraper to produce an NFO file; a text (XML) creator/editor would be far more efficient in that regard. Problem is, how do you create the NFO and populate it with the requisite data in one step? You need include three types of data. Firstly, media information data. To get them into the NFO, you need a utility like mediainfo (exe or dll) to read the information from the file then export it in xml. A text editor doesn't do the job; if you know otherwise, please comment. Secondly, pointers to artwork files, preferably to local locations (preferably on your NAS). To do that, you need a scraper. A text editor, even one paired with add-ons like Artwork Downloader, doesn't do the job; if you know otherwise, please comment. Thirdly, content-related metadata. It is here that, by definition, a text editor doesn't do the job and a scraper is needed. But suppose thet the job description isn't to scrape the metadata from some internet site but to write them into the NFO using a custom-made input. You may have maby reasons to prefer this to internet scraping, especially so if the media file in querstion is a video file falling outside XBMC's dichotomy movie/tvshow (musicvideos is, as we all know, not really videos at all).

You then say:---

Have a scraper or option to add to library with zero info (similar to justthefilename)...

Here you hit on a raw nerve. This ability should ---IMO--- have been in XBMC from the very beginning.

JustTheFileName isn't so good as to be considered a solution to this gap in XBMC's functionality specifications (in the same manner, say, as MyPicturesDB has been a solution to the lack of a pictures database). It is too slow (is it so because it resides on a distant server?). It outputs not the filename as found, but the filename in all-lowercase; for many users too ugly, and inviting needless post-scraping retouches. (JustTheFileName isn't customizable per a settings.xml.) And, in truth, once you have discovered the benefits on a just-the-filename "scraper", isn't it reasonable to ask for an option to input content provided in a same-name xml file into the NFO, thus obviating the need for post-scraping retouches and, for that matter, needless Internet lookups?

...then use a database editor like XWMM or MediaElch to add the data.

Oh boy, once again you hit on a raw nerve.

Let me first explain to the unsuspecting why an XBMC-savvy database editor is needed, and a simple SQL(ite) editor---and even more so an XML editor---won't do. XBMC's database is intentionally not fully normalized. This means you cannot expect to edit something in one central place and have this edit propagated by the SQL engine across all tables. (For example, you cannot edit genres that way.)

Then let me chide you for not telling to the uninitiated that this editor does not currently exist---and if it does, is not guaranteed to do the job under all circumstances.

XWMM, which you mention, is indeed the nonplusultra in web based library, and in part also system, administration. However, presumably exactly because of its trying to be so good at so many things, it is currently not available for Frodo. In other words, it currently exists only as a promise.

MediaElch, which you also mention, has currently dropped its ability to SQL-edit the database. I presume this was because it never did so in a satisfactory way, because my own experience with MediaElch was a disaster: it stripped all my by that time information-enriched NFOs of all content save title and media info, meaning all my hard work gone down the drain. When I posted this, the answer was I could have avoided this if I had implemented this and that at the time undocumented tweak; or more precisely, I could have perhaps avoided this, as noone knows for sure. Thank you very much MediaElch; I have great understanding for tools which do not do the job at first, but in my vocabulary a beta tseter isn't someone who doesn't mind losing data.

XBMC Nfo Editeur (XBNE), which you do not mention, is an even more promising library (though not system administration) tool than XWMM, in the sense that it not only includes the ability to directly edit the database (even an MySQL database), but also to use a multitude of scrapers available as XBMC add-ons. However, its behavior is unexpected in the sense that when it comes to putting these extra abilities to the test, it complains that "the .NET libraries available on this system are version 4.0, whereas the version expected by XBNE is 3.5". This is strange, as I, and undoubtedly most users, have both versions on my system as I need both, and they are both designed to coexist in part, and be a subset of one another in other part. So the best I can say o XBNE is that it, too, is currently but a promise.

FOR ALL THESE REASONS, I persist in my opinion that a scraper such as described in this post is needed. I do not ask for it to be coded here and now, nor do I ask for developers to take some extra time to consider how best to solve my particular needs. But I ask them to ponder the proposition that in not offering this feature, XBMC is an unfinished job in the same manner its lack of a pictures library was before the advent of MyPicturesDB.

[Ned Scott]

First, I don't think you realize how you really come off as offensive here. XBMC is what it is. It is not the perfect newbie friendly solution that the average joe can set up. We know this, and we aspire to fix it, but it is what it is. We do not need you to rub our noses in it as if we were dogs. I don't think you mean it to come out that way, but it does.

Second, XBMC is pretty damn good for what it is. While we aspire to be more user-friendly, we also know that our strength is in the power user. They're the ones that are demanding a media center like XBMC, and they are the ones who get our attention the most. After well, we're power users too.

Third, what on earth are you going on about? I'm basically agreeing with you that we need a better method of inputting entries that don't have scrapers (or will never have scrapers, like personal videos). XBMC needs a method to just add to the library. We actually had this ability before, but it was dropped in v11. I'm not fully sure why, and I do not doubt there were good reasons, but nothing has replaced it yet. I also think XWMM, or something like it, should come with XBMC as a default add-on. I'd even propose to the group that Slash, the author of XWMM, be invited to join the team, but I don't think that would make any difference since he probably just doesn't have a lot of time to work on it. Though I do hear XWMM is being updated for Frodo.

It's like you're trying to fight this philosophical battle that no one else is fighting against.

[DiMag]

I understand you to sum up The XBMC Team's position on the subject matter of this post as follows:

1. Generic (aka Custom aka Home) Video Scraper:

History
Implemented in v10 (Dharma), but met this and that obstacle in v11 (Eden). Please consult the github.

Current state:
Need thereof as a third scraper alongside movies and tvshows is universally acknowledged. Currently Under Development, so keep coming back here for updates.

2. Until then,---?

1. Use JustTheFileName as it is reported to work reasonably well.
   
2. Use an XML editor to custom edit your NFOs on a file-by-file basis.
   
3. Use an XML editor to custom edit (after export) your videodb.xml.
   
4. Use an SQL editor to directly edit the database, provided you understand that many tables are not normalized and you know how to deal with it.
   
5. Wait for XWMM for Frodo which will enable directly adding any video file to the database, then direct SQL-editing of the database---a far superior approach by all means.
   

[DiMag]

On the relative merits of JustUseTheFilename, see my post in http://forum.xbmc.org/showthread.php?tid...pid1297870.