Pictures database and library mode; scan EXIF and IPTC metadata tags into a SQL db

  Thread Rating:
  • 2 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Thread Closed
Xycl Offline
Fan
Posts: 360
Joined: Feb 2012
Reputation: 7
Location: Germany
Post: #316
(2012-04-06 05:41)mgmartin Wrote:  I've noticed that the browse_folder function scanpath.py tends to bomb out at the stat functions if it encounters any files or folders without read permissions.
This is probably OS-specific and partially a user issue...
HOWEVER - In the name of bulletproofing, would it be possible to insert some exception trapping per-file or per-folder ?
I crudely hacked some try/except blocks around the code to enable it to scan my pics, but this might be handled more elegantly by the original authors.

Thanks again !

Please test it with new version 0.6.6.
File permission errors are now handled when using Windows.

No log no help.
How to submit a log file: XBMC-Wiki

Main page: https://github.com/Xycl
Repository: Xycl Repository
find
mikebzh44 Online
Posting Freak
Posts: 1,790
Joined: Nov 2011
Reputation: 48
Location: Nantes - France
Post: #317
(2012-04-05 19:09)fmronan Wrote:  You can install by repo passion in xbmc or with the zip file too
http://passion-xbmc.org/addons/?Page=Vie...e.mypicsdb
Is it possible to add property for recently or random pictures in the home, like recently movie or other media?
Hello Xycl.

Fmronan, Glass skinner, is looking for a special feature of your script.

Your script is able to return "Recently added" and "Random" images but not as Home properties.

Is it possible to you to set Home properties so skinners could populate widgets, as for "Recently added" and " Random" movies, tvshows, albums ?

Sorry for my english, but, you know, I'm French so ...

find
Xycl Offline
Fan
Posts: 360
Joined: Feb 2012
Reputation: 7
Location: Germany
Post: #318
(2012-04-07 18:40)mikebzh44 Wrote:  
(2012-04-05 19:09)fmronan Wrote:  You can install by repo passion in xbmc or with the zip file too
http://passion-xbmc.org/addons/?Page=Vie...e.mypicsdb
Is it possible to add property for recently or random pictures in the home, like recently movie or other media?
Hello Xycl.

Fmronan, Glass skinner, is looking for a special feature of your script.

Your script is able to return "Recently added" and "Random" images but not as Home properties.

Is it possible to you to set Home properties so skinners could populate widgets, as for "Recently added" and " Random" movies, tvshows, albums ?
Sorry, but what are home properties?
Can you give me an example? Best would be the xml file (and lines) from confluence skin where the recently added videos are shown because I don't know much about skins (only enough to add some home submenus for playlists or sources).
With help of an example it should'nt be to complicated for me to give fmronan the needed pictures.

Update: I can write for him a function like 'RecentlyAdded(n)' which gives back the picture path of the n'th recently added picture.

No log no help.
How to submit a log file: XBMC-Wiki

Main page: https://github.com/Xycl
Repository: Xycl Repository
(This post was last modified: 2012-04-08 15:35 by Xycl.)
find
mikebzh44 Online
Posting Freak
Posts: 1,790
Joined: Nov 2011
Reputation: 48
Location: Nantes - France
Post: #319
How I set Home property with RandomAndLastItems script :

WINDOW = xbmcgui.Window( 10000 )
_setProperty( "PictureDB.%d.Path" % ( _count ), path )

So skinners can use PictureDB.1.Path to PictureDB.10.Path for example.

Sorry for my english, but, you know, I'm French so ...

find
mgmartin Offline
Junior Member
Posts: 4
Joined: Mar 2012
Reputation: 0
Post: #320
Still seems to have issues under Linux.
I have to trap exceptcions in the picentry={...stat...} code to successfully scan the folders.

(2012-04-07 16:50)Xycl Wrote:  
(2012-04-06 05:41)mgmartin Wrote:  I've noticed that the browse_folder function scanpath.py tends to bomb out at the stat functions if it encounters any files or folders without read permissions.
This is probably OS-specific and partially a user issue...
HOWEVER - In the name of bulletproofing, would it be possible to insert some exception trapping per-file or per-folder ?
I crudely hacked some try/except blocks around the code to enable it to scan my pics, but this might be handled more elegantly by the original authors.

Thanks again !

Please test it with new version 0.6.6.
File permission errors are now handled when using Windows.
find
fungify Offline
Senior Member
Posts: 146
Joined: Jul 2010
Reputation: 2
Post: #321
Running with newest version 0.6.6 on Windows Server 2008 R2 and scanning speed has drastically improved. As mentioned previously building the database with about 7000 images from scratch took between 10 and 12 hours to complete. Version 0.6.6 builds almost the same database in 45-50 minutes.

I do still experience two issues.

One is that I still can't select a person with many images attached. In the log a python error is mentioned. Relevant log is posted here: http://pastebin.com/EdMYbcjQ

The other issue is somewhat stranger. Some keywords don't show up - no matter which picture is tagged with them. For example - I'm tagging images with the car featured, but the tag "Hyundai" will not show up - no matter which image is tagged. I'm still testing this issue though.

FFY

EDIT: Updated with relevant info.
(This post was last modified: 2012-04-09 21:33 by fungify.)
find
fmronan Offline
Fan
Posts: 600
Joined: Dec 2010
Reputation: 4
Post: #322
Can't add a new folder with the 0.6.6
I can't browse
[Image: 1333972343-screenshot117.png]
find
Xycl Offline
Fan
Posts: 360
Joined: Feb 2012
Reputation: 7
Location: Germany
Post: #323
(2012-04-09 08:49)mgmartin Wrote:  Still seems to have issues under Linux.
I have to trap exceptcions in the picentry={...stat...} code to successfully scan the folders.

Which directory permission is missing x or r?
I need it to know in order to test the plugin within a vmware linux.
(2012-04-09 13:54)fmronan Wrote:  Can't add a new folder with the 0.6.6
I can't browse

Did you add picture sources?
The browse dialog only shows directories which are added as sources.

No log no help.
How to submit a log file: XBMC-Wiki

Main page: https://github.com/Xycl
Repository: Xycl Repository
(This post was last modified: 2012-04-10 11:36 by Xycl.)
find
Xycl Offline
Fan
Posts: 360
Joined: Feb 2012
Reputation: 7
Location: Germany
Post: #324
(2012-04-09 13:21)fungify Wrote:  Running with newest version 0.6.6 on Windows Server 2008 R2 and scanning speed has drastically improved. As mentioned previously building the database with about 7000 images from scratch took between 10 and 12 hours to complete. Version 0.6.6 builds almost the same database in 45-50 minutes.

I do still experience two issues.

One is that I still can't select a person with many images attached. In the log a python error is mentioned. Relevant log is posted here: http://pastebin.com/EdMYbcjQ

The other issue is somewhat stranger. Some keywords don't show up - no matter which picture is tagged with them. For example - I'm tagging images with the car featured, but the tag "Hyundai" will not show up - no matter which image is tagged. I'm still testing this issue though.

FFY

EDIT: Updated with relevant info.

1) Seems (I'm not sure) to be a problem with GPS coordinates.
Can you PM me the picture "På høreklinikken 27-02-2012\IMAG0029.jpg" or a similar one because I don't own any GPS tagged pictures.

2) Is it possible to PM the database C:\Users\Administrator\AppData\Roaming\XBMC\userdata\Database\MyPicture.db.
I already found some problems when deleting and rescanning pictures. Then sometimes old keywords aren't deleted and the new pictures doesn't get keywords due to the "old not deleted" ones.
You can rename or delete the db and try to rescan your pictures, too.

No log no help.
How to submit a log file: XBMC-Wiki

Main page: https://github.com/Xycl
Repository: Xycl Repository
find
fmronan Offline
Fan
Posts: 600
Joined: Dec 2010
Reputation: 4
Post: #325
ok, sometime I'm stupidTongue
Added 0.6.6 in passion repo
find
fungify Offline
Senior Member
Posts: 146
Joined: Jul 2010
Reputation: 2
Post: #326
(2012-04-10 11:47)Xycl Wrote:  1) Seems (I'm not sure) to be a problem with GPS coordinates.
Can you PM me the picture "På høreklinikken 27-02-2012\IMAG0029.jpg" or a similar one because I don't own any GPS tagged pictures.
PM send.

(2012-04-10 11:47)Xycl Wrote:  2) Is it possible to PM the database C:\Users\Administrator\AppData\Roaming\XBMC\userdata\Database\MyPicture.db.
I already found some problems when deleting and rescanning pictures. Then sometimes old keywords aren't deleted and the new pictures doesn't get keywords due to the "old not deleted" ones.
You can rename or delete the db and try to rescan your pictures, too.
I am actually right now rebuilding the database from scratch (deleted the old one) to see if that might help. Will update if tags / keywords are still not added.

FFY
find
Xycl Offline
Fan
Posts: 360
Joined: Feb 2012
Reputation: 7
Location: Germany
Post: #327
(2012-04-10 12:40)fungify Wrote:  
(2012-04-10 11:47)Xycl Wrote:  1) Seems (I'm not sure) to be a problem with GPS coordinates.
Can you PM me the picture "På høreklinikken 27-02-2012\IMAG0029.jpg" or a similar one because I don't own any GPS tagged pictures.
PM send.

FFY

I sent you a PM.
(2012-04-08 21:45)mikebzh44 Wrote:  How I set Home property with RandomAndLastItems script :

WINDOW = xbmcgui.Window( 10000 )
_setProperty( "PictureDB.%d.Path" % ( _count ), path )

So skinners can use PictureDB.1.Path to PictureDB.10.Path for example.

There is a problem.
The MyPicDB Addon cannot set the properties because it doesn't run when XBMC shows the main menu Frown

No log no help.
How to submit a log file: XBMC-Wiki

Main page: https://github.com/Xycl
Repository: Xycl Repository
(This post was last modified: 2012-04-11 16:05 by Xycl.)
find
fungify Offline
Senior Member
Posts: 146
Joined: Jul 2010
Reputation: 2
Post: #328
Two quick comments / bug reports.

* When adding a folder and the moving to the topfolder (by clickinging ..) the addon sometimes delete the folder that has just been added. I even get the message as if I actually chose this option. This is a new bug that was introduced in 0.6.6 and might be related to the fix you made to dialoges that didn't show?

* When scanning for keywords, it would be great if you would expand the fields you scan. Right now the only keyword field that is scanned seems to be [IPTC]Keywords but that one is far from a standard tag. Different programs seems to use different tags, and IPTC seems to lose users to XMP.

Looking through my images that I've managed with a selection of utilities, I can identify the following tags with keywords:

[IPTC]Keywords
[XMP-microsoft]LastKeywordIPTC
[XMP-microsoft]LastKeywordXMP
[XMP-dc]Subject
[IFD0]XPKeywords

Might I suggest some sort of prioritized reading of tags, so that if one tag is empty, the next one is read?

FFY
find
Xycl Offline
Fan
Posts: 360
Joined: Feb 2012
Reputation: 7
Location: Germany
Post: #329
(2012-04-11 19:59)fungify Wrote:  Two quick comments / bug reports.

* When adding a folder and the moving to the topfolder (by clickinging ..) the addon sometimes delete the folder that has just been added. I even get the message as if I actually chose this option. This is a new bug that was introduced in 0.6.6 and might be related to the fix you made to dialoges that didn't show?

FFY

The complete "Add path" menu must be rewritten.
I got the same message after deleting a folder and then adding one or more folders but never when I only add folders.


(2012-04-11 19:59)fungify Wrote:  * When scanning for keywords, it would be great if you would expand the fields you scan. Right now the only keyword field that is scanned seems to be [IPTC]Keywords but that one is far from a standard tag. Different programs seems to use different tags, and IPTC seems to lose users to XMP.

Looking through my images that I've managed with a selection of utilities, I can identify the following tags with keywords:

[IPTC]Keywords
[XMP-microsoft]LastKeywordIPTC
[XMP-microsoft]LastKeywordXMP
[XMP-dc]Subject
[IFD0]XPKeywords

Might I suggest some sort of prioritized reading of tags, so that if one tag is empty, the next one is read?

FFY

(2012-03-26 22:16)Xycl Wrote:  Suggestion for further development:

1)
There are two tables e.g. Cities & CitiesInFiles for each supported exif, iptc or xmp tag.
Therefore adding new kinds of tags is quite time consuming.
2 tables, 3 database class methods and the menu handling must be added.

Instead of using a table like Cities (key, content) or Categories (key, content) a meta table (key, tag-type, content) would be better because then adding new tags types is only done in file parser. Selecting the tag types from meta table and adding menu items can be done programmatically.
There would be only one menu item "keyword" or "tag" which opens a sub menu displaying all available tag types from meta table.

2)
There should be a mapping table. Then we can join e.g. iptc keyword tag with exif xpkeyword tag. The user would only see one menu item called keyword.

3)
An advanced filter. Perhaps a wizard.
Select pictures from country abc with keyword xyz containing person def. Needs 1) otherwise it's not maintainable.

Any opinions?


I already started implementing part 1).

No log no help.
How to submit a log file: XBMC-Wiki

Main page: https://github.com/Xycl
Repository: Xycl Repository
find
fungify Offline
Senior Member
Posts: 146
Joined: Jul 2010
Reputation: 2
Post: #330
Oh - that's right. I did actually think we had this conversation before, but looking back in the thread I couldn't find the evidence. Good to hear that you are on the case.

Just spend around 10 hours copying tags from [XMP-microsoft]LastKeywordXMP (which all of my pictures had) to [IPTC]Keywords (which only around a third had). MyPicsDB only reads the last tag, and this was why I did not see the correct amount of images under each tag and why some tags where not read at all.

Now it works and I'm a happy camper. I'm just not sure what happens when I add new images, as I still don't know if WLPG writes one or the other tag - though as previously mentioned I actually thought I did.

FFY
(This post was last modified: 2012-04-12 13:03 by fungify.)
find
Thread Closed