Games library (with database), and extend scraper interpeter/engine to include games?

  Thread Rating:
  • 1 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
EMK0 Offline
Senior Member
Posts: 209
Joined: Oct 2008
Reputation: 0
Post: #16
yea rwparris2 that's what i want. But if someone wants to they could improve it to make it much more user friendly but the basics would do just fine.these are just ideas.
find quote
Terin Offline
Junior Member
Posts: 15
Joined: Nov 2008
Reputation: 0
Location: Houston, TX
Question  Libraries?
Post: #17
This may be a bit out there (or even discussed before) -- but would it be possible to setup a Programs scraper or something like that? I currently use my Xbox as the platform for XBMC, and I was thinking it'd be really cool if I could have information about my games get pulled up and show different sources (i.e. Emulators/Games/Homebrew) all under one little library feature. Something that would even let you manually update it (and maybe check Wikipedia for data with a scraper, i.e. Release Date, Company, etc).

Would be pretty sweet to see Super Mario War or ZSNES show up in the library and get a custom wallpaper for it, etc.

Not sure if Programs only shows up on Xbox -- but would definitely be killer anywhere...
find quote
blittan Offline
Team-XBMC Handyman
Posts: 1,747
Joined: Jun 2004
Reputation: 11
Location: Sweden
Post: #18
not yet, but it's been discussed and I also think there are requests for it on the tracker.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
If you don't have the time to read them, please don't take the time to post in this forum!
For troubleshooting and bug reporting please make sure you read this first.
find quote
Terin Offline
Junior Member
Posts: 15
Joined: Nov 2008
Reputation: 0
Location: Houston, TX
Post: #19
There seems to be a lot of interest in this -- and I'm really interested in it too. I've taken a look at some of the source via SVN at work (blarg on the beginning of the year being so slow). I'm busy working on two major C# projects in my personal time, but I do have a heavy background in C++ as well. Maybe I can do some of it -- but my time is incredibly limited...

Do we have any architects or major players that know this well enough, that they could tell me what all would be necessary to update? I imagine we'd need a ProgramDatabase, ProgramInfo, etc similar to the VideoLibrary, etc. How about linking it in to the rest of the app and playing around with the skinning, etc? I mean, it's not terribly difficult to build a database or anything else from that.

And, help me if I'm a little off here -- but is the persistence of the database (as seen in the constructor) limited to the XBMC's execution; i.e. it will recreate the database each time the app reboots/starts, but persists it in a data format outside of the app, which is then reloaded into the tables? (any reason why?)

Edit:

Furthermore, would it make more sense to call this Program<section; i.e. Database.cpp> or Game<section> ?
find quote
timdog82001 Offline
Posting Freak
Posts: 1,142
Joined: Mar 2006
Reputation: 0
Post: #20
Terin Wrote:Furthermore, would it make more sense to call this Program<section; i.e. Database.cpp> or Game<section> ?

Maybe your asking something else here, but what about a Programs section, with an internal structure similar to the video section, how there are movies and tv shows sections in the video library, which in this case could be Applications and Games?
find quote
Terin Offline
Junior Member
Posts: 15
Joined: Nov 2008
Reputation: 0
Location: Houston, TX
Post: #21
I was referring to the code -- not the layout of the actual GUI. This would most likely build off of what already exists, i.e. the Programs/Applications "tab" or section (or even the "Games" tab).

I just need to get a quick run-through on it. An architect should be able to get this to me relatively quickly and I can just perform the busy-work.

By the way, I was thinking -- it would be really sweet to be able to make scrapers work against ROMs, ala searching for extension of .NES, etc (for Nintendo) and then link it up to a NES Emulator. Just, I guess, kind of like a file-association, like how Windows knows to load .TXT files with Notepad (or something else, if you specify/overwrite that default setting).
find quote
Gamester17 Offline
Team-XBMC Forum Moderator
Posts: 10,523
Joined: Sep 2003
Reputation: 10
Location: Sweden
Post: #22
Terin Wrote:There seems to be a lot of interest in this -- and I'm really interested in it too. I've taken a look at some of the source via SVN at work (blarg on the beginning of the year being so slow). I'm busy working on two major C# projects in my personal time, but I do have a heavy background in C++ as well. Maybe I can do some of it -- but my time is incredibly limited.
Maybe you could work together with Awen on this, see => http://forum.xbmc.org/showthread.php?tid=40715

Terin Wrote:Do we have any architects or major players that know this well enough, that they could tell me what all would be necessary to update? I imagine we'd need a ProgramDatabase, ProgramInfo, etc similar to the VideoLibrary, etc. How about linking it in to the rest of the app and playing around with the skinning, etc? I mean, it's not terribly difficult to build a database or anything else from that.
spiff is our resident database guru, he has offered to act 'mentor' in this area in needed, we also have jmarshall and kraqh3d who also really know their way about the database code, and Jezz_X is our skinning god and jmarshall is the skinning-engine jedi, best is to try to catch them on IRC (#xbmc channel on the freenode network) for real-time chats, the other alternative is the development section here in this forum forum (however that is not always practical for quick/smaller things):
http://wiki.xbmc.org/?title=Appendix_D:_...ment_Notes
http://wiki.xbmc.org/?title=Appendix_D:_...ct_methods

Terin Wrote:And, help me if I'm a little off here -- but is the persistence of the database (as seen in the constructor) limited to the XBMC's execution; i.e. it will recreate the database each time the app reboots/starts, but persists it in a data format outside of the app, which is then reloaded into the tables? (any reason why?)
XBMC uses standard SQL databases, and currently use SQLite as the embedded backend engine, (XBMC currently has one db for videos and one db for audio today, so you would want to add one new additional for 'applications' which would hold programs/games/emulator meta data). Some basic information here:
http://wiki.xbmc.org/?title=The_XBMC_Database
http://wiki.xbmc.org/?title=Category:Inner_Workings
http://wiki.xbmc.org/?title=Category:Development

Happy coding :grin:

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
(This post was last modified: 2009-01-06 20:21 by Gamester17.)
find quote
Terin Offline
Junior Member
Posts: 15
Joined: Nov 2008
Reputation: 0
Location: Houston, TX
Post: #23
I just noticed Awen's code and was thinking I was no longer needed. I need to talk to him and see what all he's done as far as code. I think he's done essentially the exact same thing I wanted to. Just would be nice to have scrapers identify the different extensions and recognize them appropriately/query data on them.

(xbe for Xbox yields a query on the game; nes yields a NES game and an emulator to plug it into; etc; Pretty simple stuff.)

I'll try to post on Awen's thread and see if he needs any help, or the extent of his update/modification.
find quote
tetsuo55 Offline
Member
Posts: 73
Joined: Jan 2009
Reputation: 0
Post: #24
As a game frontend user i would like to add some information.

almost every game has the following graphics available in one way or another:
-Scan of the retail box/photo of the cabinet
-Scan of the Manual
-Scan of the media (cart,cd,...)
-Scan of the PCB (Inside carts or arcade cabinets)

There are some more but these can often be used inside the emulator and do not need to be visable in a frontend

Besides the scans there are also screenshots available:
-Title screen
-Random action shot

Lastly there is written information:
-Creator
-Publisher
-Year
-Device Brand (like nintendo)
-Device Model (like the super nintendo entertainment system)
-Game Name
-Number of players
-Controller Type
-Description of the game
-and so forth...

-------------

I think for XBMC we should limit this to:
ART:
-FanART backdrop(for both individual games and individual systems)
-Retail box artwork/Arcade cabinet artwork(flyers?)
-Random action screenshot
TEXT:
-Name
-Creator/Publisher
-Genre
-Description of the game
-Number of players
-Type of controller
-Type of device
DATABASE:
Make the games searchable by:
-Genre
-Name
-Number of players
-Creator
-Publisher
-Type of device
-Year.

Most importantly i would like to be able to list all the games in one long list regardless of their system(type of device)
find quote
tetsuo55 Offline
Member
Posts: 73
Joined: Jan 2009
Reputation: 0
Post: #25
asphinx Wrote:Oh hell yeah! I thought I was the only one wanting/thinking about how to improve the game section of xbmc.

Here are my demands! Big Grin No just kidding, but seeing as I didn't know people were already working at improving it, I already starting some work myself. On the other end of things though.

I have looked through a lot of already out there gamedb's, but in my opinion most of them are of inferior quality, this in comparison to thetvdb (for tv shows) and for instance the various sources for movie scraping. So here's my suggestion.
A dedicated online db specifically tailored for the needs of the many. Think thetvdb, but for games. Much like thetvdb doesn't offer you the option of downloading a tv show, merely information about it, so would thegamedb (working name) offer a one source solution to everything you might need to know about the games you have. I'm thinking...

- Ratings (PEGI, ESRB, BBFC, OFLC, CERO, USK if it has a rating, it should be possible to scrape it, with transparent .png's to match I thought)
- Posterstyle coverart (already making/have made those anyway)
- Listings of developers, publishers, genre, release dates for different platforms and/or regions
- plot outline, of course. Smile
- for emulated games one could have a flagging system for prefered emulators, which xbmc could use as a reference to locate emulators on HDD (don't know if it's possible)
- etc...

Also much like thetvdb, users could contribute to expand the assets available from thegamedb.

Is this a good idea? I've already started working out a prototype database.. this includes a standardized xml layout, which in turn can easily be translated into an .nfo file. Maybe making that an option for users as well.

The biggest problem I'm seeing so far is that games don't seem to have a unified id tagging system (none that I've found at least.) And I feel that this is something that should be adressed. Sure most of us probably go for naming the game as it should be anyway, and it that case searching based on title would work. But it would be possible to make it a hell of a lot smoother if there was a unified tagging system, maybe even with added parameters for different platforms. (ex. 1234-1 for halo combat evolved on xbox and 1234-2 for halo combat evolved on PC)

Is anyone already making a db?, of so I will gladly contribute anyway I can. In not, well I'll just continue with the work I've done so far.

I think that's a great idea actually!
Unlike mame the amount of information available for other platforms is in terrible shape.

Finding a way to identify games for scraping is actually quite difficult, the best solution i can come up with (should work for emulators)

Each game has one entry in the database, based on its Internation-English name.
A secondary database entry is based on a (strong) hash.

-The scraper should support alternative title names as search option
-The hash is unique can be used to display the name for that region and the system it was meant for.
-When artwork differs between regions it can be tagged as such

--------
Very often most of the information about a game is valid for all the platforms it was ever ported too. This way of doing it removes a lot of redunancy from the database.

Also because there is only one entry for each game, the user can use the database to choose which platform is best for him/her (many games are better on one platform than another due to things like button-limits are sound/video quality)
find quote
Terin Offline
Junior Member
Posts: 15
Joined: Nov 2008
Reputation: 0
Location: Houston, TX
Post: #26
Can we lock/redirect this thread to Awen's thread in development? I think that'd be the best place to include things, otherwise we're linking back and forth.

In regards to your idea, Tetsuo, we may consider using gamefaqs.com as a source for scrapers. I'm fairly pleased with it -- and when I've looked up games there, it's offered suggestions -- something we may consider.

Rather than have a hash, as you suggest, make it work like the video database, where it can suggest other titles. i.e. you type in Dune, and it gives you Dune (1984); Children of Dune ( 2004); Dune (2003) or however it works/worked.

I don't think gamefaqs is going anywhere anytime soon, so it might be good to pull our data from there!

For example, if you search Rudra on GameFaqs, you'll find Rudra No Hihou (Rudra's Treasure). GameFaq's also has an image section for other data on a game... Just a thought.
(This post was last modified: 2009-01-15 19:00 by Terin.)
find quote
asphinx Offline
Team Arcade
Posts: 238
Joined: Nov 2008
Location: Finland
Post: #27
Based on tetsuo55's re-direct I'm assuming it might be better I post here instead. You make a very good regarding gamefaqs Terin, but I still believe that the available data there is too limited, the ratings system alone is limited to AMG & ESRB, now I don't want to be the one to knit pick but all my games are PAL with PEGI rating I would very much like to be able to scrape that as well.

It's one thing what and how much information a user wants to have, but why limit the options to the bare minimum if it's possible to make another option. Now I must confess, in my initial planning I didn't actually consider emulated platforms as much. But seeing as there seems to be an equal "lust" for that I see no reason why it couldn't be included in my previously proposed database. The prototype already has support for alternative titles, if the initial title search fails. One possibility with ID tagging (at least for xbox games) would of course be the XBE ID, but I haven't looked into that as it still only covers xbox games.

I too like the idea about suggestions, either in the form of loose title search. Just like you said Terin, write Halo... and Halo:Combat Evolved, Halo 2 would show up, on every available platform, but prioritizing the ones you have.

One thing I did like about gamefaqs though was "Similar Games", where the results same to be based more on genre than title. Much like you can see apple trailers about upcoming movies, it would be fun to to be able to get "suggestions" about games you might not have/ever played. But this idea is more of a darling, as why suggest a game that you might never be able to play (difficult or impossible to find)
find quote
tetsuo55 Offline
Member
Posts: 73
Joined: Jan 2009
Reputation: 0
Post: #28
Terin Wrote:Rather than have a hash, as you suggest, make it work like the video database, where it can suggest other titles. i.e. you type in Dune, and it gives you Dune (1984); Children of Dune ( 2004); Dune (2003) or however it works/worked.

As soon as you start talking about multi-emulator support that idea stops working very quickly.

A well know example is "Mortal Kombat", this game has been ported to over 10 systems, and almost all of them where released in the same year(often even the same month/day)

I do agree that Gamefaqs is a good starting point.
find quote
natethomas Online
Team Kodi Community Manager
Posts: 3,729
Joined: Apr 2008
Reputation: 63
Location: Kansas
Post: #29
Obviously this current conversation is pretty far away from the current programming step, but it's fun to have, nonetheless.

I've been thinking about the scraping situation, and perhaps this has already been dealt with in a more obvious manner, but it seems to me that, as ROMs are kept in all kinds of different formats and often have multiple system support, developing a scrapping system to deal with all the various kinds of emulators can become tricky.

As such, you have two or three options. You can require that media be stored in specific folders (e.g. the N64 folder, the SNES folder, etc.); you can require that media be named appropriately (e.g. Mortal Kombat.Genesis.rar, Mortal Kombat.SNES.rar, etc.) or you can avoid the entire naming system and use a mythical piece of code to determine what system goes with what game.

I personally am a fan of requiring a rename. XBMC (and probably the "myTV" applications of other Meda Center softwares) along with theTVDB appear to have singlehandedly created a unified TV episode naming structure. (Anyone remember seeing episode files not broken up by season? Seinfeld is an old, obvious example. Seinfeld Ep37?)

Anyway, a requirement of following the basic naming scheme seems like a no-brainer in my head. Any thoughts?
find quote
timdog82001 Offline
Posting Freak
Posts: 1,142
Joined: Mar 2006
Reputation: 0
Post: #30
natethomas Wrote:I personally am a fan of requiring a rename. XBMC (and probably the "myTV" applications of other Meda Center softwares) along with theTVDB appear to have singlehandedly created a unified TV episode naming structure. (Anyone remember seeing episode files not broken up by season? Seinfeld is an old, obvious example. Seinfeld Ep37?)

Anyway, a requirement of following the basic naming scheme seems like a no-brainer in my head. Any thoughts?

I think requiring separate folders makes sense, and I would guess most people already do that. Generally, emulators have their own rom folder inside the program folder so I doubt many people have all their different roms pooled into one giant folder. This would make it easy on the XBMC, you could just Set Sources for the different folders and different systems in the same way you set sources for different folders to pertain to different media types (ie, tv, movies, music videos, etc).

In regards to requiring a rename of all files, I think most roms I've seen via direct download or as part of torrent packs already have correct or close to correct names, and those that don't are usually using well established abbreviations that are fairly universal throughout the emulator/rom world, enough that, iirc, mame, for example, is able to deduce exactly what game it is by the file title. That said, i'm not sure. If i'm mistaken and most roms don't come named in a way that would allow xbmc to use this to ID them correctly, I think requiring a rename of all files would be a poor choice, if avoidable. It seems most emulator users download rom packs which often include thousands of titles. I don't think many users will be willing to spend the countless hours and days that would be required to rename every single file appropriately.
find quote