Kodi Community Forum
[RELEASE] Rom Collection Browser - Browse and launch emulator game ROMs - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Game Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=291)
+---- Thread: [RELEASE] Rom Collection Browser - Browse and launch emulator game ROMs (/showthread.php?tid=70115)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395

- OldOne - 2010-11-17

malte Wrote:You could do a reimport and check xbmc.log for errors or warnings. Or upload the log file and I will have a look at it. If you only have 3 items, it may be better to delete MyGames.db first and do a clean import.

Hi,

i tried all that, also added 2 more files, but than it only indexed one of the new files,.
Is here someone who uses mame with the RCB? maybe i missed some important thing to do?

- QuinRiva - 2010-11-17

So it seems that the RCB will only import games if an image file exists. So if there is no PathToData/frontbox/%GameName%.jpg the roms aren't added to the database and won't show up in the list.

I've had quite a bit of trouble with the scraper thus far, it seems only about half of the games are successfully being scraped despite the fact that they are listed on mobygames. I manually downloaded the boxfront/boxback/cartridge images and renamed them so they had the exact same filename as the rom name: i.e. roms are in PathToData/roms/%ROMNAME%.zip, images are PathToData/boxfront/%ROMNAME%.jpg.

However I got the same error as jimyx17:

jimyx17 Wrote:Hi malte,

first of all, Thanks! You have done a really good job! Now the bad news, I think I found a bug in dbupdate.py or gamedatabase.py files. When I try to import the full romset of N64, I get an exception with this message:
Code:
"IntegrityError: columns name, romCollectionId are not unique".
xbmc.log

It is definitly having problems with game sequels. Basically I have the following games:
  • Mortal Kombat I
  • Mortal Kombat II
  • Mortal Kombat III
  • Ultimate Mortal Kombat 3

I think that part of the problem is that my files are named:
  • roms\Mortal Kombat II (U) (V1.1).zip
  • boxfront\Mortal Kombat II (U) (V1.1).jpg
  • boxback\Mortal Kombat II (U) (V1.1).jpg
  • cartridge\Mortal Kombat II (U) (V1.1).jpg

Only Mortal Kombat II and Mortal Kombat 3 are showing up in the list & when I select MK2 it launches MK1 and MK3 launches MK2.

I deleted the database and renamed the files to Mortal Kombat 2.xxx and then I encountered a new error:
Code:
20:51:50 T:7936 M:2606964736  NOTICE: RCB_INFO: Searching for game: Mortal Kombat 2
20:51:50 T:7936 M:2606964736  NOTICE: RCB_INFO: More than 1 result found. Try to find best match.
20:51:50 T:7936 M:2606964736  NOTICE: RCB_INFO: Result SearchKey = Mortal Kombat
20:51:50 T:7936 M:2606964736  NOTICE: RCB_INFO: Comparing with Mortal Kombat, ratio: 0.928571428571
20:51:50 T:7936 M:2606964736  NOTICE: RCB_INFO: Result SearchKey = Mortal Kombat 3
20:51:50 T:7936 M:2606964736  NOTICE: RCB_INFO: Comparing with Mortal Kombat 3, ratio: 0.933333333333
20:51:50 T:7936 M:2606964736  NOTICE: RCB_INFO: Result SearchKey = Mortal Kombat II
20:51:50 T:7936 M:2606964736  NOTICE: RCB_INFO: Comparing with Mortal Kombat II, ratio: 0.903225806452
20:51:50 T:7936 M:2606964736  NOTICE: RCB_INFO: Result SearchKey = Ultimate Mortal Kombat 3
20:51:50 T:7936 M:2606964736  NOTICE: RCB_INFO: Comparing with Ultimate Mortal Kombat 3, ratio: 0.717948717949
20:51:50 T:7936 M:2606964736  NOTICE: RCB_INFO: Result url = http://www.mobygames.com/game/snes/mortal-kombat-3

WTF apparrently Mortal Kombat 2 is a closer match to Mortal Kombat 3 then to Mortal Kombat II. Yes I undestand how this can happen but handling of II-->2 and III-->3 should be reversible (i.e. it should use ReplaceValues for both the Game Name and the Lookup results).

I think that would fix the problem.

- QuinRiva - 2010-11-17

So as a temporary fix I have commented out the ReplaceValues line and everything seems to be working fine, but there is still problems with the scraper returning incorrect values with sequels:
PHP Code:
22:06:59 T:5784 M:2614722560  NOTICE: RCB_INFO: More than 1 result found. Try to find best match.
22:06:59 T:5784 M:2614722560  NOTICE: RCB_INFO: Result SearchKey = Super Bomberman
22:06:59 T:5784 M:2614722560  NOTICE: RCB_INFO: Comparing with Super Bomberman, ratio: 0.882352941176
22:06:59 T:5784 M:2614722560  NOTICE: RCB_INFO: Result SearchKey = Super Bomberman 4
22:06:59 T:5784 M:2614722560  NOTICE: RCB_INFO: Comparing with Super Bomberman 4, ratio: 0.888888888889
22:06:59 T:5784 M:2614722560  NOTICE: RCB_INFO: Result SearchKey = Super Bomberman 5
22:06:59 T:5784 M:2614722560  NOTICE: RCB_INFO: Comparing with Super Bomberman 5, ratio: 0.888888888889
22:06:59 T:5784 M:2614722560  NOTICE: RCB_INFO: Result SearchKey = Super Bomberman 3
22:06:59 T:5784 M:2614722560  NOTICE: RCB_INFO: Comparing with Super Bomberman 3, ratio: 0.888888888889
22:06:59 T:5784 M:2614722560  NOTICE: RCB_INFO: Result SearchKey = Super Bomberman 2
22:06:59 T:5784 M:2614722560  NOTICE: RCB_INFO: Comparing with Super Bomberman 2, ratio: 0.888888888889 

Super Bomberman (U).zip is a closer match to any sequel; all the sequels share 16 characters "Super Bomberman ", where as the original only shares 15 (the trailing space does is the additional character).

Note that there is still the general errors with the scraper not returning images for games even it does correctly find and identify the games.

- malte - 2010-11-17

Thank you all for this detailed testing!

I already fixed the IntegrityError. Will upload a new version soon.

About the Sequel handling:

Quote:WTF apparrently Mortal Kombat 2 is a closer match to Mortal Kombat 3 then to Mortal Kombat II. Yes I undestand how this can happen but handling of II-->2 and III-->3 should be reversible (i.e. it should use ReplaceValues for both the Game Name and the Lookup results).

For the string matching algorithm that I use (not my own one) 2 and 3 are closer than 2 an II. No chance to change this with an easy fix.

I implemented this replace mechanism to handle differences between your rom filenames and the search results on the scraped sites. So it would not make sense to replace both values. The problem is that the same site seems to use different patterns to identify games in sequels. I thought, that I could rely that all games in a sequel on the same site are numbered like 2, 3, 4, ... or II, III, IV, ... and not mixed with 2 and III. This makes game matching extremely difficult. You can only do this with changing the rom filename itself (or fix the data at the scraped site).

You can also add your own tokens that should be replaced or removed. This config would replace III and II and removes (U) and (V1.1):
Code:
<scraper name="mobygames.com" replaceKeyString=" III, II, (U), (V1.1)" replaceValueString=" 3, 2,,"/>


Quote:Note that there is still the general errors with the scraper not returning images for games even it does correctly find and identify the games.

Can you give me an example where this happens?

- malte - 2010-11-17

jimyx17 Wrote:Hi malte,

first of all, Thanks! You have done a really good job! Now the bad news, I think I found a bug in dbupdate.py or gamedatabase.py files. When I try to import the full romset of N64, I get an exception with this message:
Code:
"IntegrityError: columns name, romCollectionId are not unique".

As mentioned above I have a work around for this issue. But it will just fix the error during import (means that this error won't stop the complete import now).

The root cause for this error is that RCB gets the wrong game description: You have two games: "Doraemon - Nobita to 3tsu no Seireiseki" and "Doraemon 2 - Nobita to Hikari no Shinden". For some reason, both games are recognized as "Doraemon: Nobita to 3tsu no Seirei Ishi" and RCB tries to import a game with this title twice.

I have left a little fuzziness in game matching so that game names must not match 100% to be imported. I think I will make this fuzzy-factor configurable if someone is not satisfied with the results.

jimyx17 Wrote:I got another exception when I try to run any of my loaded roms but, I think this is related to the first problem or a configuration problem. The error says:
Code:
"RCB_ERROR: unhandled Error in onAction: 'NoneType' object has no attribute 'find'"

I have an idea what happens but I have to look into it again. Seems to be a bug.

carabalb Wrote:@malte: I'm looking into adding my own scraper/parser, how should i format the parser to go through an xml file with multiple games. I tried to copy your local nfo parser but it only recognizes the first entry.

You have to specify the correct root attribute (the xpath to your game entry). For example in thegamesdb scraper it is done like this:
Code:
<GameGrammar type="xml" root="Game">

OldOne Wrote:Hi,

i tried all that, also added 2 more files, but than it only indexed one of the new files,.
Is here someone who uses mame with the RCB? maybe i missed some important thing to do?

Sorry, did not test mame with RCB until now. But I think some users have done it.

QuinRiva Wrote:So it seems that the RCB will only import games if an image file exists. So if there is no PathToData/frontbox/%GameName%.jpg the roms aren't added to the database and won't show up in the list.
You can change this behaviour in the Addon settings: "Don't import games without artwork".

First Post! - ChaosCharza - 2010-11-18

Hey!

Thanx, malte for making a plugin with great potential! I really want it to run, but over the last 24 hours have had multiple problems starting up my own Database; manly the first bug on double imports.

I guess all I want to know if there is a starter kit that someone could post. I've been tinkering with XBMC on my Xbox (now migrated to Linux-AppleTV) and I still cannot get it to run right. It doesn't seem too starter friendly ATM.

Just my 0.02.

Cheers!
ChaosCharza

(BTW, I downgraded off of Dharma for this!)

- malte - 2010-11-18

Sorry, there have been some stupid bugs in 0.7.6. Most of them led to bad image scrape results. Additionally a stupid error in my test scenario why I did not notice them myself;-). I hope they are all fixed now but I want to make some more tests before I upload a new release.

@ChaosCharza:
I guess some of your problems are related to the scraping bugs. A short startup guide can be found in the preview docs: http://code.google.com/p/romcollectionbrowser/wiki/Preview. Not everything done and up to date, but almost complete. Check out the chapter "First use" for a startup.

You downgraded because of RCB? I hope you did not do that;-) It should run fine with Dharma.

- ChaosCharza - 2010-11-18

Meh, it seemed to crash more in Dharma (of course, this was before I rigorously checked the Log File and found the problems). I'll give it another go.

Thanx!

EDIT: Aaaaand... After Installing Dharma (and some tinkering with the Addon Settings and creating some local scrappers) IT WORKS!!!

This is epic! Thanks malte!

EDIT2: I guess one more question:

I'm trying to figure out a seamless way to allow for me to quit out of an emulator. Most seem to use "ESC" to leave, as it's the default mame configuation

The problem is in my setup I do not want to have to have a keyboard hooked up at all times (especially since it's on my roommate's side and he only allows for the device to be there and nothing else!) I want it to be a wireless situation with the only two devices interacting being the Apple Remote and a wireless original Xbox controller.

I figure that the command Joy2Key will most likely be my friend, with a universal setup in place to allow for all my emulators to use it and quit when I press down on the right stick. I know it's mainly a trial and error situation, but if someone could point me in the right direction (or at least a template) that would really help

Thanx!
ChaosCharza

- QuinRiva - 2010-11-20

malte Wrote:About the Sequel handling:

For the string matching algorithm that I use (not my own one) 2 and 3 are closer than 2 an II. No chance to change this with an easy fix.

I implemented this replace mechanism to handle differences between your rom filenames and the search results on the scraped sites. So it would not make sense to replace both values. The problem is that the same site seems to use different patterns to identify games in sequels. I thought, that I could rely that all games in a sequel on the same site are numbered like 2, 3, 4, ... or II, III, IV, ... and not mixed with 2 and III. This makes game matching extremely difficult. You can only do this with changing the rom filename itself (or fix the data at the scraped site).

My understanding is that you are doing a comparison between the game file name and the results from the scraper, thus if mobygames retrurns "Mortal Kombat II" can you not replace the "II" with "2" for the purpose of CRC comparison?

malte Wrote:You can also add your own tokens that should be replaced or removed. This config would replace III and II and removes (U) and (V1.1):
Code:
<scraper name="mobygames.com" replaceKeyString=" III, II, (U), (V1.1)" replaceValueString=" 3, 2,,"/>

It would be great if you could implement wildcards so that all region and version specific details can be retained in the filename --> eg. (*) and [*].

- malte - 2010-11-20

Quote:My understanding is that you are doing a comparison between the game file name and the results from the scraper, thus if mobygames retrurns "Mortal Kombat II" can you not replace the "II" with "2" for the purpose of CRC comparison?

Yes, that is exactly what I am working at. I added some extra sequel handling code. When I don't find a perfect match with the original (rom) name, I check if the game name ends with a digit or roman numeral and start to replace one with the other. So if your game is "Mortal Kombat II" and it does not find a match, I will check if it fits to "Mortal Kombat 2" and vice versa. I do this in code, so you don't have to add these replace options anymore.

The bugs are (hopefully) all fixed now and the sequel handling gives good results. But I have to do some more tests before I will upload the new version.

Code:
It would be great if you could implement wildcards so that all region and version specific details can be retained in the filename --> eg. (*) and[*].
Good idea. I will check what I can do.


Thanks for your input!

- malte - 2010-11-20

Next test version is available: 0.7.7.

Changes:
  • new option: Game Matching Fuzzy Factor - checks how good the game name matches with your rom name
  • added sequel handling in game matching - described in my previous post
  • create file scrapeResult_possibleMismatches.txt - lists all game names that don't match 100% with the rom name
  • Bugfix: giantbomb scraper was broken
  • Bugfix: mobygames scraper did not find boxfronts when there have been images for more than one regions
  • Bugfix: mobygames scraper did not find screenshots when there have been no boxfronts
  • Bugfix: catch IntegrityError
  • Bugfix: fixed bug in handling mutli rom games

I hope that I catched all bugs now.

regards,
malte

- carabalb - 2010-11-20

Hey Malte,

First of all I wanted to thank you for all your hard work in fixing the bugs, I noticed a major improvement in scraping. I did some testing and I found one other issue. It seems like some of the scrapers have uni-code characters which throws this exception:

Code:
14:12:16 T:4704 M:2123845632  NOTICE: RCB_INFO: description file (tokens replaced): http://thegamesdb.net/api/GetGame.php?name=Speedball%202%20-%20Brutal%20Deluxe%20(USA)
14:12:16 T:4704 M:2125914112  NOTICE: RCB_INFO: Searching for game: Speedball 2 - Brutal Deluxe (USA)
14:12:16 T:4704 M:2125914112   ERROR: Error Type: exceptions.UnicodeEncodeError
14:12:16 T:4704 M:2125914112   ERROR: Error Contents: 'ascii' codec can't encode character u'\xfc' in position 31: ordinal not in range(128)

Also, is there any way to apply the replace feature to all of the scrapers?

- ChaosCharza - 2010-11-20

Thanx for a new version, malte! Sounds like it's time for me to upgrade (especially w/ working scrapers). I have a question though: does the order of scrappers matter? What i wanted to do was have my local source find the name, description and crc and then utilize the later scrappers to get artwork. Crc's are unique, and pre-labeling them with the exact name may reduce the "id already in database" error I've been getting with games like "super mario bros + duck hunt" & "super mario bros 2"

As for my Joystick issues, I now use rejoystick with a init.d script at boot. The only complaint i have is how its controlled within XBMC (it's really off in SDL) so I'm currently compiling the latest svn (dharma RC1 FTW) w/o joystick support so I don't have a confusing mess of a configuation (the d-pad on it is mapped in reverse; odd, as it is an original xbox controller).

It should be noted, malte, that my copy of RCB (0.7.6) i was testing in Dharma Beta would freeze often after playing a game and then starting another. I don't know if this is unique to me or not, but I wanted to through it out there. I'll attach a log later, as I am tied up with compiling.

Thanx!

ChaosCharza

- malte - 2010-11-21

carabalb Wrote:It seems like some of the scrapers have uni-code characters which throws this exception

I have to check this. The result list contains the game brütal legend. Seems that one component has a problem with the "ü".

Quote:Also, is there any way to apply the replace feature to all of the scrapers?
Maybe I have to think about this againSmile. At the beginning I wanted to use the replace function to replace sequel numbers (II -> 2). This should be done by console and scraper because it could be different in every scenario. Now that I do the sequel thing in code, the replace function could be used more general. But atm you have to add the replaceKeys and values to every RomCollection and every scraper.

Maybe it will work to add them to the general scraper configuration at the end of config.xml. E.g., if you want to remove [u] it should look like this (note the "," in replaceValueString):
Code:
replaceKeyString="%REPLACEKEYS%, [u]" replaceValueString="%REPLACEVALUES%,"
But I did not test this myself.

ChaosCharza Wrote:I have a question though: does the order of scrappers matter? What i wanted to do was have my local source find the name, description and crc and then utilize the later scrappers to get artwork.
Yes, the scrapers are fired in the order they appear in config.xml. The results of all scrapers will be mixed together. There is already a placeholder on this topic in the wiki but I still have to fill it with life.

One thing you have to keep in mind: If you use (local) game descriptions that hold information for more than one game in one file, you MUST use this scraper as the first one and all following scrapers can only be one-desc-per-file scrapers (all online scrapers are valid here). The config for the complete RomCollection must be "descFilePerGame = False". I hope this was not too confusing...

Quote:It should be noted, malte, that my copy of RCB (0.7.6) i was testing in Dharma Beta would freeze often after playing a game and then starting another. I don't know if this is unique to me or not, but I wanted to through it out there. I'll attach a log later, as I am tied up with compiling.
A log would be good and more info about your system. I know Dharma RC1 and Linux, but is it Live (with or without window manager) or something else?

To be honest: Since I develop RCB there is only very little time to play gamesBig Grin. So it might happen that I loose such kind of problems. But I test launching games on my Windows box quite regularly and did not have any problems so far. My Linux box is still on Camelot, I also don't have problems there.

- jimyx17 - 2010-11-21

Hi Malte,

Again, I have some troubles with scrapers and encoding. Now the problem is with 1080 Snowboarding, this is the log that appears when I try to import it

Code:
File "/home/htpc/.xbmc/addons/script.games.rom.collection.browser/default.py", line 47, in ?
                                                import gui
                                              File "/home/htpc/.xbmc/addons/script.games.rom.collection.browser/resources/lib/gui.py", line 1639, in ?
                                                main()
                                              File "/home/htpc/.xbmc/addons/script.games.rom.collection.browser/resources/lib/gui.py", line 1633, in main
                                                ui = UIGameDB("script-Rom_Collection_Browser-main.xml", os.getcwd(), "Default", "PAL")
                                              File "/home/htpc/.xbmc/addons/script.games.rom.collection.browser/resources/lib/gui.py", line 207, in __init__
                                                self.checkImport(doImport)
                                              File "/home/htpc/.xbmc/addons/script.games.rom.collection.browser/resources/lib/gui.py", line 1231, in checkImport
                                                dbupdate.DBUpdate().updateDB(self.gdb, progressDialog)
                                              File "/home/htpc/.xbmc/addons/script.games.rom.collection.browser/resources/lib/dbupdate.py", line 279, in updateDB
                                                lastGameId = self.insertGameFromDesc(gamedescription, gamenameFromFile, romCollection, filenamelist, foldername, isUpdate, gameId)
                                              File "/home/htpc/.xbmc/addons/script.games.rom.collection.browser/resources/lib/dbupdate.py", line 579, in insertGameFromDesc
                                                gameId = self.insertData(gamedescription, gamename, romCollection, filenamelist, foldername, isUpdate, gameId)
                                              File "/home/htpc/.xbmc/addons/script.games.rom.collection.browser/resources/lib/dbupdate.py", line 766, in insertData
                                                self.possibleMismatchFile.write('%s, %s\n' %(gamename, gamenameFromFile))
                                            UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position 4: ordinal not in range(128)

Please, if I can help you in any way, let me know.

Thanks a lot!