It's not about the file size of the items in your library. It's about the number of records in the library. SQL Lite does a horrible job with extremely large tables, especially with less than perfect queries. Spiff did a good job, but would be the first to admit that database design and query writing isn't his specialty.
I had another thread related to a better library design (seen here:
http://forum.xbmc.org/showthread.php?tid=28482). I'll give a very simple breakdown of ideal behavior for the library:
Library sections should be able to be added/removed without changing the underlying code in XBMC. Each library section should be able to have any number of customizable fields, with a unique id being the only requirement. Plugins, scrapers, etc should be able to pull/push information into/out of each library section. Finally, the user interface should be able to link into each library section without modifying the underlying code.
This would enable plugins/scripts to create (or use existing) library sections for their own use. AMT-importer could have its own table that it would pull info into, while AMT-plugin would display that info so it matches the current skin. This would also allow multiple scripts/scrapers/plugins to work on the same data. So your main movie data could come from IMDB but posters/covers would come from impawards.com. TV data could come from TheTVDB.com while the TV schedule could come from TV Guide. Adding a section like music videos would be as simple as running the music video setup script, which would create the library section and (optionally) add the music video plugin to your video sources. This plugin would be called with XBMC.CallPlugin("musicvideo", "musicvideos"), where the first parameter is the plugin name and the 2nd is the library section. That would also enable multiple versions of the movie library (for each user, based on ratings), and things of that nature.
Obviously it would take a lot of work, but I really believe XBMC would benefit from something like this. It would open things up quite a bit for developers to be able to store dynamic information which other developers could then make use of.