[Kinematics]

I'm trying to get XBMC to handle my music videos, and having an awful time of it.

The videos I have are AMVs (anime music videos), and I recognize that they will -never- be available on any scraper, since the music video scrapers have information that's only partially related to the videos in question, even assuming the files were named such that it could locate the original song being used (and no, they are not being renamed). As such, I accept that I will only ever be able to work with my AMVs in file mode.

However, I'm trying to create some smart playlists to at least get a vague bit of control over managing the listings, and it seems that it flat out won't work. The source directory is marked as containing music videos, and each individual video shows up as a music video type, however when I create a playlist that sources on music videos, nothing shows up. The playlist filter is based on just the file name and/or path, but it seems to fail to find anything at all.

My best guess was the lack of info music video information since none of them could be scraped (I had similar issues when trying to get anime working properly). I tried adding a .nfo file for one video, updated, and now the playlist filters work -- except that that one file is the only one they'll ever find.

Now, I know XBMC recognizes video files separately from other types (eg: going in through file mode, I'll never see any random text files or whatever), and I know that it recognizes each individual file in the AMV directory as a music video. So why is it that it can't recognize them when creating a playlist? At the very least I should be able to get a file mode listing.

At the moment it looks like, in order to get XBMC to recognize the AMVs, I'd have to go in and manually create several thousand .nfo files by hand. That's just not happening. Is there any other means of making this workable?



Aside: Part of the purpose of this was to create a home menu item that I could use to access the AMV list. The default skin has no means of accessing music videos that I've been able to find; Aeon Nox requires a playlist that won't work; Metropolis at least provides a top-level entry for music videos, which is nice, but is still trapped by the need to manually define every single one if I define any at all (though it will default to a file view if I have no .nfo files defined; that works for the top-level menu, but means I can't do anything with regular playlists).

Edit: My mistake on the default skin. All my searching was before I added the .nfo file; now that it's there, I actually have a menu item for music videos.

[strikervi]

do you mean your adding the OP and ED of some of the anime shows ?
there was apost a while back about an AniDB scraper.... not sure where it went to

or something like
7!! - Sayonara Memory
FictionJunction - stone cold
ClariS - nexus (Music Video (full ver.))
Eri Kitamura - Happy Girl

those " Music videos" are found using the last.fm plug in under music.

granted some might not work if they arent close.. some need English names vs Japanese letters..
but works for me

[Kinematics]

No, not that.

For clarification, here's an AMV on YouTube: http://www.youtube.com/watch?v=kHieEhINI7g

Creator: Nostromo
Video Title: Auriga
Music: "Watch Them Fall Down" by DJ Spoke
Video Sources: dozens of anime, too many to list here; they're in the details section on the YouTube page


There's tons of variations in how they're constructed, but the basic premise is creating a music video using some chosen song or songs (or in some cases, things like the audio from a movie trailer), accompanied by video from one or more anime (or other animated) series. You could perhaps think of it as a mashup that uses audio+video instead of audio+audio.

I have a few thousand, and there's well over a hundred thousand cataloged as available, and many more that aren't readily cataloged. Unfortunately the information describing them doesn't fall neatly into the categories scrapers provide, and there's no uniform naming convention for easy lookup anyway, so it's unlikely (though, I suppose, possible) there will ever be an automated service that XBMC could use.

That leads to the issue of my original question -- how to deal with filtering based only on hard information (eg: paths, file names, etc) without trying to manually build thousands of .nfo files (though a better system would be some way to manually enter/edit details as convenient). The .nfo files don't handle all the information I'd want tracked, anyway. For example, 'album' isn't a relevant field, but 'video source' and 'contests' would be. The AMV creator would, I guess, be the 'director', but the song artist would be a secondary, rather than primary, field (and absolutely do not want there to be any mix-up as to whether these are 'official' videos for said artists).

[strikervi]

whoosh, over my head,
too many
and just a lot of variations.
gl in your quest.

[Ned Scott]

Currently, there is no easy way to do this. Smart playlists require that videos be in the library for them to work. There was a scraper for movies called "Just use the file name" that would add videos as movies with no metadata, but still technically set as "movies". It might be possible to modify this for music videos as well, but I'm not sure: http://wiki.xbmc.org/index.php?title=Vid...heFileName

[Kinematics]

Ok, got the justUseFilename scraper extension. Aside from deleting the random junk in the archive (a bunch of .swp files and an OSX directory), I needed to modify the addon.xml file. Changes made:

1) Renamed from metadata.justUseFilename.org to metadata.musicvideos.justUseFilename.org. Also renamed the directory when adding it to XBMC's addons direectory.
2) Changed the first extention point attribute from xbmc.metadata.scraper.movies to xbmc.metadata.scraper.musicvideos

That was sufficient to make it available as a scraper engine for the selected directory, when scanning for music videos. I changed that and rescanned the directory (a small one for testing), and it added all the files to the library with just the file name as the title (as advertised). All the added AMVs now show up under Videos/Music Videos, and I can filter them with smartlists based on title.

Thank you Ned, it's pretty much exactly what I needed.


I'm currently going through the plugin, along with the wiki information about how to create a scraper, to try to refine it some, since it appears to have some non-working code included. Maybe I can tweak it a little to get a tiny bit more info out of the title.

[Kinematics]

Haven't quite worked out how to rebuild the scraper code, but definitely going to need it. Somewhere along the way there's a fair bit of file name mangling going on (all dashes removed, a few bracketed values, some closing parentheses, some missing almost the entire file name, etc). Unfortunately I'm not sure how to determine where in the code path it's happening. From what I can tell, the scraper itself should be outputting exactly what it gets in, so it may be XBMC itself that's doing it.

[scudlee]

From what I can tell, the scraper itself should be outputting exactly what it gets in, so it may be XBMC itself that's doing it.

It is. cleandatetime and cleanstrings get run before the scraper sees the file name.

[Ned Scott]

Awesome, glad to hear it worked!

[Kinematics]

Ok. Found the source of some of the naming problems.

1) CleanString

Oasis - Falling Down (East of the Eden version)[x264 AC3].mkv

Cleanstrings matches: A close paren, followed by a bracketed label, followed by end of string. Since it drops everything it matches and following, I end up with the following name:

Oasis - Falling Down (East of the Eden version

Note the missing closing paren.

From what I can tell of the regex, the close paren should *not* be part of the first group (open paren is fine; close paren being the first matched character doesn't make any sense). This is potentially a bug in the default XBMC implementation (though I don't know why it was written that way, so there may be a legitimate use for it). I can override this in advanced settings:

Code:

<video>
        <!-- If a date is found, save the year (first year if year1-year2 format) and strip the date and anything following from the file name. -->
        <cleandatetime>(.+?)(?:[-_ ,.\(\[\{]*?)(?:_|\b)((?:19|20)\d{2})(?:\s*-\s*(?:19|20)\d{2})?\b(?!\s*\w+)[^\\/]*?$</cleandatetime>

        <cleanstrings>
            <regexp>[ _\,\.\(\[\-](ac3|dts|custom|dc|divx|divx5|dsr|dsrip|dutch|dvd|dvdrip|dvdscr|dvdscreener|screener|dvdivx|cam|fragment|fs|hdtv|hdrip|hdtvrip|internal|limited|multisubs|ntsc|ogg|ogm|pal|pdtv|proper|repack|rerip|retail|r3|r5|bd5|se|svcd|swedish|german|read.nfo|nfofix|unrated|ws|telesync|ts|telecine|tc|brrip|bdrip|480p|480i|576p|576i|720|720p|720i|1080p|1080i|hrhd|hrhdtv|hddvd|bluray|x264|h264|xvid|xvidvd|xxx|www.www|cd[1-9]|\[.*\])([ _\,\.\(\)\[\]\-]|$)</regexp>
            <regexp>(\[.*\])</regexp>
        </cleanstrings>
    </video>

2) From ADDONS::CScraper::FindMovie:

It explicitly removes dashes from the file names of music videos, replacing them with spaces (though that may be redundant, since the debug message that comes before that code already has the dashes removed).

It forces the filename to lowercase.

It does not pass the original filename as one of the parameters, so I can't do my own manipulations.

Overall, there's nothing I can do about this short of modifying the code and making my own build. Just have to live with it.


For anyone who may have to deal with this issue later, the following is the code I finally ended up with for justUseFilename.xml (the core of the scraper code):

Code:

<?xml version="1.0" encoding="UTF-8"?>
<scraper framework="1.1" date="2012-02-18">
    <!-- For a tutorial on scrapers, see: http://wiki.xbmc.org/index.php?title=HOW-TO:Write_media_scrapers  -->
    
    <!-- Default NFO loading -->
    <NfoUrl dest="3">
        <RegExp input="$$1" output="\1" dest="3">
            <expression noclean="1"></expression>
        </RegExp>
    </NfoUrl>
    <!-- Create a reference to the dummy web page.  Append any year that XBMC extracted from the filename, via $$2. -->
    <CreateSearchUrl dest="3">
        <RegExp input="$$1" output="&lt;url&gt;http://www.elbsolutions.com/xbmc/scraper/justUseFilename.php?task=getsearchresults&amp;string=\1::$$2&lt;/url&gt;" dest="3">
            <expression noclean="1">(.*)</expression>
        </RegExp>
    </CreateSearchUrl>
    <!-- Pull the info the page generated and form it into our own search results -->
    <GetSearchResults dest="8">
        <RegExp input="$$5" output="&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot; standalone=&quot;yes&quot;?&gt;&lt;results&gt;\1&lt;/results&gt;" dest="8">
            <RegExp input="$$1" output="&lt;entity&gt;&lt;title&gt;\1&lt;/title&gt;&lt;url&gt;http://www.elbsolutions.com/xbmc/scraper/justUseFilename.php?task=getdetails&amp;string=\1&lt;/url&gt;&lt;/entity&gt;" dest="5">
                <expression>&lt;title&gt;([^&lt;]+)&lt;/title&gt;</expression>
            </RegExp>
            <expression noclean="1"></expression>
        </RegExp>
    </GetSearchResults>
    <!-- Parse the results in order to break out info from the file name.
         Order them from less specific matches to more specific, since each match that succeeds overrides any previous ones.
         Modify these regexes to suit your needs. -->
    <GetDetails dest="3">
        <RegExp input="$$5" output="&lt;details&gt;\1&lt;/details&gt;" dest="3">
            <!-- Form: title::year? -->
            <RegExp input="$$1" output="&lt;title&gt;\1&lt;/title&gt;&lt;year&gt;\2&lt;/year&gt;" dest="5">
                <expression>&lt;title&gt;([^:&lt;]+)::([0-9]{4})?&lt;/title&gt;</expression>
            </RegExp>

            <!-- Form: title (series)::year? -->
            <RegExp input="$$1" output="&lt;title&gt;\1&lt;/title&gt;&lt;year&gt;\3&lt;/year&gt;&lt;plot&gt;\2&lt;/plot&gt;" dest="5">
                <expression>&lt;title&gt;([^:&lt;]+(?:\(([^)]{3,})\)))::([0-9]{4})?&lt;/title&gt;</expression>
            </RegExp>

            <!-- Form: creator - title::year? -->
            <RegExp input="$$1" output="&lt;title&gt;\2&lt;/title&gt;&lt;year&gt;\3&lt;/year&gt;&lt;director&gt;\1&lt;/director&gt;" dest="5">
                <expression>&lt;title&gt;(.+)   ([^:&lt;]+)::([0-9]{4})?&lt;/title&gt;</expression>
            </RegExp>

            <!-- Form: creator - title (series)::year? -->
            <RegExp input="$$1" output="&lt;title&gt;\2&lt;/title&gt;&lt;year&gt;\4&lt;/year&gt;&lt;director&gt;\1&lt;/director&gt;&lt;plot&gt;\3&lt;/plot&gt;" dest="5">
                <expression>&lt;title&gt;(.+)   ([^:&lt;]+(?:\(([^)]{3,})\)))::([0-9]{4})?&lt;/title&gt;</expression>
            </RegExp>

            <!-- Form: [creator] title::year? -->
            <RegExp input="$$1" output="&lt;title&gt;\2&lt;/title&gt;&lt;year&gt;\3&lt;/year&gt;&lt;director&gt;\1&lt;/director&gt;" dest="5">
                <expression>&lt;title&gt;(?:\[([^]]+)\])([^:&lt;]+)::([0-9]{4})?&lt;/title&gt;</expression>
            </RegExp>

            <!-- Form: [creator] title (series)::year? -->
            <RegExp input="$$1" output="&lt;title&gt;\2&lt;/title&gt;&lt;year&gt;\4&lt;/year&gt;&lt;director&gt;\1&lt;/director&gt;&lt;plot&gt;\3&lt;/plot&gt;" dest="5">
                <expression>&lt;title&gt;(?:\[([^]]+)\])([^:&lt;]+(?:\(([^)]{3,})\)))::([0-9]{4})?&lt;/title&gt;</expression>
            </RegExp>

            <!-- Form: [label] creator - title::year? -->
            <RegExp input="$$1" output="&lt;title&gt;\3&lt;/title&gt;&lt;year&gt;\4&lt;/year&gt;&lt;director&gt;\2&lt;/director&gt;&lt;plot&gt;\1&lt;/plot&gt;" dest="5">
                <expression>&lt;title&gt;(?:\[([^]]+)\])(.+)   ([^:&lt;]+)::([0-9]{4})?&lt;/title&gt;</expression>
            </RegExp>

            <!-- Form: [label] creator - title (series)::year? -->
            <RegExp input="$$1" output="&lt;title&gt;\3&lt;/title&gt;&lt;year&gt;\5&lt;/year&gt;&lt;director&gt;\2&lt;/director&gt;&lt;plot&gt;\1, \4&lt;/plot&gt;" dest="5">
                <expression>&lt;title&gt;(?:\[([^]]+)\])(.+)   ([^:&lt;]+(?:\(([^)]{3,})\)))::([0-9]{4})?&lt;/title&gt;</expression>
            </RegExp>

            <!-- Form: [studio][creator] title::year? -->
            <RegExp input="$$1" output="&lt;title&gt;\3&lt;/title&gt;&lt;year&gt;\4&lt;/year&gt;&lt;studio&gt;\1&lt;/studio&gt;&lt;director&gt;\2&lt;/director&gt;" dest="5">
                <expression>&lt;title&gt;(?:\[([^]]+)\])(?:\[([^]]+)\])([^:&lt;]+)::([0-9]{4})?&lt;/title&gt;</expression>
            </RegExp>

            <!-- Form: [studio][creator] title (series)::year? -->
            <RegExp input="$$1" output="&lt;title&gt;\3&lt;/title&gt;&lt;year&gt;\5&lt;/year&gt;&lt;studio&gt;\1&lt;/studio&gt;&lt;director&gt;\2&lt;/director&gt;&lt;plot&gt;\4&lt;/plot&gt;" dest="5">
                <expression>&lt;title&gt;(?:\[([^]]+)\])(?:\[([^]]+)\])([^:&lt;]+(?:\(([^)]{3,})\)))::([0-9]{4})?&lt;/title&gt;</expression>
            </RegExp>

            <!-- For testing --
            <RegExp input="$$1" output="&lt;title&gt;test&lt;/title&gt;" dest="5">
                <expression>fail</expression>
            </RegExp> -->

            <expression noclean="1"></expression>
        </RegExp>
    </GetDetails>
</scraper>

You can tweak the regexes in the GetDetails section to suit your own needs. The other sections shouldn't need to be modified, as long as the website that elbsolutions set up doesn't change.

I haven't figured out why it always seems to load each file twice (you can see the normal name, then the HTML-ified name, in the little notification window, for every file). Also, I don't know why, when you do a refresh from the media info page, it always asks you to confirm the file name (in ugly HTML formatting) even though there's only one entry. A means of tidying those issues up would be nice, though it's only a minor annoyance.

Edit: Had to make the same tweaks to <cleandatetime> as <cleanstrings>.

Edit: Recenly reviewing <cleandatetime> to give an answer to someone here, I've updated the above suggested value to better handle some additional edge cases. The new version provides more correct output than the default version or my original modified version.

[bornagainpenguin]

Hi, I know it's been awhile but did you ever get this finished and released for music videos? Not only do I also use AMVs, I'm also into Mashups (aka Bastard Pop) and I am well aware that the majority of those videos (assuming one gets made) will likely never see official scraping due to many of the same issues you mentioned in your first post. So I'm very interested in this and hope you've released your changes somewhere!

--borngainpenguin

[Kinematics]

While cleaning this up after penguin's note, I found a few more issues. Notably, the cleandatetime regex needs to be updated to fix a few edge cases, and the cleanstrings regex needs to be overhauled to avoid a lots of little issues. A number of problems are detailed in this thread.

Code:

<cleandatetime>(.+?)(?:\s*(?:(?:[[({])(?:[^])}]*)(?:_|\b))|[_.,-]\s*)((?:19|20)\d{2})(?:(?:_|\s)*-(?:_|\s)*(?:19|20)\d{2})?\b(?!(?:\s*\w)+)[^\\/]*?$</cleandatetime>

For example, the word "Tsundere" in the title will strip everything from that word forwards, due to the presence of "ts" in the cleanstrings regex. It also strips "Setsuna" because of the presence of "se". There are many other bad matches there, due to the fact that those trip terms are allowed to occur after a space, without any other punctuation separating it from the rest of the file name.

As a preliminary fix, I've broken the cleanstrings regex into two parts: A full list for after punctuation, or a partial list if it's only after a space. I also added "10-bit" to the list of terms.

Code:

<cleanstrings>
            <regexp>[[(_,.-]\s*(ac3|dts|custom|dc|divx|divx5|dsr|dsrip|dutch|dvd|dvdrip|dvdscr|dvdscreener|screener|dvdivx|cam|fragment|fs|hdtv|hdrip|hdtvrip|internal|limited|multisubs|ntsc|ogg|ogm|pal|pdtv|proper|repack|rerip|retail|r3|r5|bd5|svcd|swedish|german|read\.nfo|nfofix|unrated|ws|telesync|telecine|tc|brrip|bdrip|480p|480i|576p|576i|720|720p|720i|1080p|1080i|10-?bit|hrhd|hrhdtv|hddvd|bluray|x264|h264|xvid|xvidvd|xxx|www\.|cd[1-9])</regexp>

            <!-- Anything following a space is suspect, since it can be real words for the file title.  Remove anything that might be part of a real word or title. -->
            <regexp>[ ](ac3|dts|custom|divx|divx5|dsr|dsrip|dutch|dvd|dvdrip|dvdscr|dvdscreener|screener|dvdivx|fragment|hdtv|hdrip|hdtvrip|multisubs|ntsc|ogg|ogm|pdtv|repack|rerip|r3|r5|bd5|svcd|read\.nfo|nfofix|telesync|telecine|tc|brrip|bdrip|480p|480i|576p|576i|720|720p|720i|1080p|1080i|10-?bit|hrhd|hrhdtv|hddvd|bluray|x264|h264|xvid|xvidvd|xxx|www\.|cd[1-9])</regexp>
</cleanstrings>

As for the addon itself, I went through and did a full clean version of it, but since it's only slightly different from the original justUseFilename addon, I was going to see if I could get them to update their version instead (mainly since it depends on using their server for the URL query, since that's far faster and less bandwidth-intensive than using a Google search (even the stripped down version) or similar). For the very basic usage, you can just download the original addon and copy the following lines into the addon.xml file in the addon's directory:

Code:

<extension point="xbmc.metadata.scraper.tvshows" language="en" library="justUseFilename.xml" />
  <extension point="xbmc.metadata.scraper.albums" language="en" library="justUseFilename.xml" />
  <extension point="xbmc.metadata.scraper.musicvideos" language="en" library="justUseFilename.xml" />

This will allow the addon to be used on other file types than just movies.

For my own version, I also stripped out all the file name data extraction, since that's a very specialized usage with non-standard patterns, and it's only marginally useful.

Edit: Fixed the cleandatetime regex slightly.

[arolust]

I know this thread is almost a year old, but info from this thread should be added to the Kodi Wiki anime section http://kodi.wiki/view/Anime

Many anime watchers also watch AMVs/MEPs and being able to add them to the library is a big thing and needs more love.