[Release] TheAudioDb.com Music Video Scraper

  Thread Rating:
  • 3 Votes - 3 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
olympia Offline
Team-Kodi Member
Posts: 2,499
Joined: May 2008
Reputation: 32
Post: #16
In case Zag can implement heuristic search in theaudiodb.com API, then it would also cure the situation.
...but as he requested previously, he would need to see what crappy file naming conventions are available/possible to test with.
find quote
padfoot Offline
Junior Member
Posts: 19
Joined: May 2010
Reputation: 0
Post: #17
I must say, I am disappointed I was dismissed so quickly:

I did raise a bug regarding this issue: http://trac.xbmc.org/ticket/14222 which was simply closed as invalid.

1 Search the forums - yes I did search the forums. Simply didn't expect problem discussion for the plugin in the [RELEASE] announcement. Any reasonable person would expect a separate thread for any issues.
2. Attach a degub output - hmmmm, what was there to debug? Nothing crashed, it simply was not connecting to the server, and was reporting the fact (as it was programmed to behave).....or so I thought.

I do not think this is an invalid bug report, as, after reading here, I find the plugin IS connecting to the server, but reporting it isn't when it fails in a search. This is not the same behaviour of the other scrapers. And by throwing that error, well, most sane people would not bother continuing to scan their library as they think the server is down, thus the failure to locate 1 item is preventing people from scannng their entire library.

This is a valid bug report as the scraper is throwing an incorrect error, and for the simple fact that you MUST have the EXACT spelling and [space]-[space] to get a match Confused? This is not good behaviour. This scraper should not have been released with these issues. It should be able to take into account different naming conventions (as do the scrapers for music and videos), and should display the same behaviour on failing to locate an item (ie, bring up the name of the item for you to modify and search again) instead of throwing a completely invalid error message which will unnecessarily cause users to stop scraping as the "SERVER IS DOWN".

Cheers.
find quote
Martijn Offline
Team Kodi
Posts: 11,371
Joined: Jul 2011
Reputation: 165
Location: Dawn of time
Post: #18
and users should provide a proper Debug Log whenever they find any issues and almost every other user here always post their issue in the dedicated thread of the scraper/addons/skin

Always read the XBMC online-manual, FAQ and search the forums before posting.
Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting, make sure you read this first

For your mediacenter artwork go to
[Image: fanarttv.png]
find quote
olympia Offline
Team-Kodi Member
Posts: 2,499
Joined: May 2008
Reputation: 32
Post: #19
(2013-03-27 23:07)padfoot Wrote:  I must say, I am disappointed I was dismissed so quickly:
I must say I will cry all the night today.

(2013-03-27 23:07)padfoot Wrote:  I do not think this is an invalid bug report, as, after reading here, I find the plugin IS connecting to the server, but reporting it isn't when it fails in a search. This is not the same behaviour of the other scrapers. And by throwing that error, well, most sane people would not bother continuing to scan their library as they think the server is down, thus the failure to locate 1 item is preventing people from scannng their entire library.
The "plugin" which we call add-on by the way has not throwing any error, XBMC does.

(2013-03-27 23:07)padfoot Wrote:  ...and for the simple fact that you MUST have the EXACT spelling and [space]-[space] to get a match Confused? This is not good behaviour.
OK, can you tell me how this could be different? Please, please, I will fix up the scraper.

(2013-03-27 23:07)padfoot Wrote:  This scraper should not have been released with these issues.
You have no clue what you are talking about. This scraper is the only music video scraper. It has a limitation in it's naming convention for known and understandable reasons, but there is a music video scraper. If you searched and read the initial post, then it is clearly stated in it. It this scraper wasn't existing, then there was no music video scraper at all.

Patches are welcome.

Edit:
...and as you could've read in the last couple of posts, we have some ideas and trying to improve the situation.
...but instead of providing naming conventions to zag, you are opening invalid bug reports and start ranting...
Nice self-introduction on our forums.
(This post was last modified: 2013-03-28 00:25 by olympia.)
find quote
padfoot Offline
Junior Member
Posts: 19
Joined: May 2010
Reputation: 0
Post: #20
(2013-03-28 00:19)olympia Wrote:  
(2013-03-27 23:07)padfoot Wrote:  I must say, I am disappointed I was dismissed so quickly:
I must say I will cry all the night today.

Really, is this comment necessary? Being a bitch to a user with a valid concern is not going to help the XBMC project in any way.

(2013-03-28 00:19)olympia Wrote:  
(2013-03-27 23:07)padfoot Wrote:  I do not think this is an invalid bug report, as, after reading here, I find the plugin IS connecting to the server, but reporting it isn't when it fails in a search. This is not the same behaviour of the other scrapers. And by throwing that error, well, most sane people would not bother continuing to scan their library as they think the server is down, thus the failure to locate 1 item is preventing people from scannng their entire library.
The "plugin" which we call add-on by the way has not throwing any error, XBMC does.

The error thrown is still incorrect, so it is still a bug.

(2013-03-28 00:19)olympia Wrote:  
(2013-03-27 23:07)padfoot Wrote:  ...and for the simple fact that you MUST have the EXACT spelling and [space]-[space] to get a match Confused? This is not good behaviour.
OK, can you tell me how this could be different? Please, please, I will fix up the scraper.

Again, is it really necessary to respond in this way? Perhaps you should have a look at how the video and music scrapers take file names in varying formats and convert them to a format suitable for the scraper.

(2013-03-28 00:19)olympia Wrote:  
(2013-03-27 23:07)padfoot Wrote:  This scraper should not have been released with these issues.
You have no clue what you are talking about. This scraper is the only music video scraper. It has a limitation in it's naming convention for known and understandable reasons, but there is a music video scraper. If you searched and read the initial post, then it is clearly stated in it. It this scraper wasn't existing, then there was no music video scraper at all.

Now you are making assumptions that I have no idea what I am talking about. You know nothing about me, my history, knowledge or abilities. Again, a completely unnecessary comment.

(2013-03-28 00:19)olympia Wrote:  Patches are welcome.

Edit: ...and as you could've read in the last couple of posts, we have some ideas and trying to improve the situation.
...but instead of providing naming conventions to zag, you are opening invalid bug reports and start ranting...

As I stated in my original post, I did not expect to find bugs being discussed in the release announcement post. I searched on the Linux support forum for any issues people may have been having with the scraper, as that is the platform I use.

Please tell me where I was ranting? My original post above simply states that I was disappointed with the bug report being dismissed as invalid, when it is not, and stated my case as to why.

No where in that post did I rant or make snide remarks about you for closing it.

I am sorry, but replying to people in that manner will only hurt xbmc in the long run.

Cheers.
find quote
Martijn Offline
Team Kodi
Posts: 11,371
Joined: Jul 2011
Reputation: 165
Location: Dawn of time
Post: #21
Ungrateful posts without any helpful information will give you such responses from people who spend many hours trying to provide a great media center experience.

Reporting that post will get you nowhere

Always read the XBMC online-manual, FAQ and search the forums before posting.
Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting, make sure you read this first

For your mediacenter artwork go to
[Image: fanarttv.png]
(This post was last modified: 2013-03-28 00:43 by Martijn.)
find quote
padfoot Offline
Junior Member
Posts: 19
Joined: May 2010
Reputation: 0
Post: #22
(2013-03-28 00:40)Martijn Wrote:  Ungrateful posts without any helpful information will give you such responses from people who spend many hours trying to provide a great media center experience.

Reporting that post will get you nowhere

How was my post ungrateful? I was simply stating a case regarding the closure of a bug I believe was unnecessary.

Nowhere have I made any snide remarks about the xbmc project or it's developers.

Obviously not going to get anywhere here. It's a pity the project is represented on these forums by developers who think it is ok to demean their users with unnecessary comments.

Great way to shore up your user base.
find quote
n1md4 Offline
Fan
Posts: 408
Joined: Nov 2012
Reputation: 8
Post: #23
(2013-03-28 00:59)padfoot Wrote:  Obviously not going to get anywhere here.

Enable debugging, run the update again, and post the xbmclog appropriately.

Submit your problem in this way, and you will likely get a good reply :-)

HTPC XBMC Gotham OpenELEC, NYXBoard, OCZ SSD
Storage 4x 2TB Green HDD BTRFS RAID1
Display Sony Bravia 32"
find quote
mojganin Offline
Junior Member
Posts: 1
Joined: Jul 2012
Reputation: 0
Post: #24
(2013-03-28 00:59)padfoot Wrote:  Nowhere have I made any snide remarks about the xbmc project or it's developers.
Except everything in each one of your comments in this thread.

(2013-03-28 00:59)padfoot Wrote:  It's a pity the project is represented on these forums by developers who think it is ok to demean their users with unnecessary comments.

Great way to shore up your user base.

For the record, I was very happy to see padfoot's posts responded to in this way, and it really did make me more appreciative of these forums. Thanks!

As for example naming conventions, I actually use the style that the scraper looks for (the artist names-with-hyphen problem continues to be an issue for me, however). But, I've got a couple of videos where the artist is... unclear for various reasons, so for those I just use the title, with nothing before it. Potentially somebody could do Artist - Album - Track, particularly if the song is remix/rerelease/new version. I also have a couple (for instance, Ok Go's This Too Shall Pass) where there are multiple videos for the same song. In that case, it's "Artist - Title - Version" (Ok Go - This Too Shall Pass - Rube Goldberg Machine Version). In this case, the song itself is on the DB, but it doesn't account for the two different versions: perhaps additional material like that could be ignored in the search, but re-appended for its entry in XBMC?

EDIT: Oh, also, as I work my way through my videos, I've also got some that are Artist ft. Another Artist (or and or with) - Title. Or Artist - Title ft Another Artist. Be great if it could parse out those additions, and search for the primary artist, but (ideally) again, without losing that information for the final XBMC entry.
(This post was last modified: 2013-03-28 04:58 by mojganin.)
find quote
zag Offline
Team-Kodi Member
Posts: 1,704
Joined: Oct 2007
Reputation: 20
Location: UK
Post: #25
So, back on topic.....

Artist-Track
Artist - Track
Artist - Title - Version
Artist ft. Another Artist - Title

And some scene stuff like this (just few examples)

lostprophets-bring_em_down-convert-x264-2012-srp.mkv
b.o.b._ft_taylor_swift-both_of_us-dvdrip-x264-2012-fray.mkv
usher_ft_rick_ross-lemme_see-ddc-x264-2012-fray.mkv
usher-climax-dvdrip-x264-2012-fray.mkv
rjd2-might_as_well_step_forward-x264-2011-srpx.mkv

HTPC - XBMC Gotham, OpenELEC, Harmony Smart Remote, Intel Haswell NUC, 40gb intel SSD, Core i3, 4gb RAM
Storage - 2 x qnap 8tb 419p+ NAS
Display LG 46" LCD + Casio Bulbless projector [PICS]
[Image: widget]
(This post was last modified: 2013-03-28 11:39 by zag.)
find quote
zag Offline
Team-Kodi Member
Posts: 1,704
Joined: Oct 2007
Reputation: 20
Location: UK
Post: #26
And just some logic notes for me:

- Explode first 2 strings (artist, title) with separator -
- Remove anything after feat. or ft.
- Remove anything after 2nd hyphen (this may cause problems with Jay-Z ect)

Or 2nd choice is to take the entire string and do a FULLTEXT search on artist + track field

- Remove "convert", "year", "ddc", "x264", "xvid", "ft.%-", last trailing -groupname

Anyone with a more logical brain than me got any better ideas? Smile

HTPC - XBMC Gotham, OpenELEC, Harmony Smart Remote, Intel Haswell NUC, 40gb intel SSD, Core i3, 4gb RAM
Storage - 2 x qnap 8tb 419p+ NAS
Display LG 46" LCD + Casio Bulbless projector [PICS]
[Image: widget]
(This post was last modified: 2013-03-28 12:08 by zag.)
find quote
olympia Offline
Team-Kodi Member
Posts: 2,499
Joined: May 2008
Reputation: 32
Post: #27
(2013-03-28 00:36)padfoot Wrote:  Really, is this comment necessary? Being a bitch to a user with a valid concern is not going to help the XBMC project in any way.
As much necessary as your comment about your feelings was. It is not me who started to bitch, but you.
...and you know, it's not wise to bite the hand which feeds you...

(2013-03-28 00:36)padfoot Wrote:  Again, is it really necessary to respond in this way? Perhaps you should have a look at how the video and music scrapers take file names in varying formats and convert them to a format suitable for the scraper.
I don't need to look at the other scrapers because I know them by heart since I was writing them all! Don't you think that if it was possible, then the music video scraper wouldn't be better? I was not asking for a code here, just simply please explain me the logic who it could work.

(2013-03-28 00:36)padfoot Wrote:  Now you are making assumptions that I have no idea what I am talking about. You know nothing about me, my history, knowledge or abilities. Again, a completely unnecessary comment.
I don't have to know all these to make this judgement and honestly I don't care either. Simply referring to other scrapers tells me everything. Music video is a totally different animal compared to movie and/or music. For movies, we only need a single title. For music we need a lot more (artist, album, track, etc.), but we can read out this information from the tags to submit proper queries to the content providers. For music video we would also need these, but there are no tags, just the filename.

(2013-03-28 00:36)padfoot Wrote:  As I stated in my original post, I did not expect to find bugs being discussed in the release announcement post. I searched on the Linux support forum for any issues people may have been having with the scraper, as that is the platform I use.
This is not a release announcement thread, this is the thread of the scraper and everything about it belongs here. There is also a search function implemented in the forum and google would've also found this thread.

(2013-03-28 00:36)padfoot Wrote:  Please tell me where I was ranting? My original post above simply states that I was disappointed with the bug report being dismissed as invalid, when it is not, and stated my case as to why.
Not true. You were not created a bug report saying that there is an incorect error message when the filename is not formatted as expected and it would be good to have it fixed if there is a chance. You were instead claiming that the scraper doesn't work. Since this is not the case, the report is invalid - as simple as that.

(2013-03-28 00:36)padfoot Wrote:  I am sorry, but replying to people in that manner will only hurt xbmc in the long run.
Try to report my post again, maybe it will help.

(2013-03-28 00:40)Martijn Wrote:  Ungrateful posts without any helpful information will give you such responses from people who spend many hours trying to provide a great media center experience.

Reporting that post will get you nowhere

It's not only that it will not get him anywhere but I must say it is quite pathetic as well.

Edit: I am sorry for the off-topic from everyone else, this was my last post on this.
(This post was last modified: 2013-03-28 12:48 by olympia.)
find quote
redhalo Offline
Member
Posts: 98
Joined: Feb 2013
Reputation: 0
Post: #28
(2013-03-28 12:46)olympia Wrote:  For music we need a lot more (artist, album, track, etc.), but we can read out this information from the tags to submit proper queries to the content providers. For music video we would also need these, but there are no tags, just the filename.

The mp4 container supports tags, would it be possible to implement queries based on tag info for such files? I personally have a second install of Picard set up to rename and tag my music videos (all mp4), and while I don't always get hits with TheAudioDb scraper it has helped tremendously.

Just looking at how other media management software handles music videos I could see that everyone has the same problems before I even started using xbmc, so I understand where the devs are coming from. Things might be a lot better if there was an online database specifically tailored towards music videos, but those out right now suck.
find quote
Pulfer Offline
Junior Member
Posts: 19
Joined: Mar 2013
Reputation: 0
Post: #29
BTW, what's the upstream git for this scraper? I see version 1.0.3 in xbmc's master @github:
https://github.com/xbmc/xbmc/tree/master...udiodb.com

But version 1.0.4 is availabe from mirrors:
http://mirrors.xbmc.org/addons/frodo/met...diodb.com/
find quote
Pulfer Offline
Junior Member
Posts: 19
Joined: Mar 2013
Reputation: 0
Post: #30
Meanwhile, my adjustments to 1.0.4:

View: http://pastebin.com/5Et0srhY
Download: http://file-store.rosalinux.ru/download/...84c740870d

7 -> 8:
repeat="yes" -> repeat="no"

Because it's better to use first found result (= first album released with the track). For example, when we request Accept - Balls To The Wall, we expect to get Balls To The Wall 1983 album. But when repeat is set to "yes", we get Staying a life 1990 album instead.

16-18:
code removed (replaced with 36,37,38 later)

This is very optional. Prefer album genre over track genre because very few tracks already have genre set. And I don't feel like it's already time to set tracks' genres because genre structure/list seems to be under construction at TheAudioDb.com.

Perhaps the best way should be to request album genre only when track genre is empty. But I didn't find the way to do it (as I'm very new to XBMC, started using it only few days ago, need much to learn about scrapers' possibilities and syntax).

22-24 ; 33-41:
code added

Request additional info about the track from album's data. First get year from album. We should request it here because there is no such data in track info. Of course it may be inaccurate for live music videos (as it will return studio album's year for the track). But it's really better than nothing and works fine in most cases. Then get genre from album.

Perhaps some of these changes are worth of merging in upstream. And some rejected changes still may be useful for users reading this thread.
find quote
Post Reply