Kodi Community Forum
Pictures database and library mode; scan EXIF and IPTC metadata tags into a SQL db - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=26)
+--- Thread: Pictures database and library mode; scan EXIF and IPTC metadata tags into a SQL db (/showthread.php?tid=80845)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27


- solexalex - 2010-10-27

Hi guys !
I need an advice from users.

I made a new scan system for getting pictures inside database.
From an item within the plugin, you can add and remove as many path as you want and of course scan from here for new pictures.
I was thinking to scan pictures every time the plugin starts which may be considered as a good idea, but which gives me some troubles because it sometimes scan even when user goes back to the main plugin menu (not starting it).
Furthermore, when you have several paths to scan, with many many pictures inside, it eats a lot of resource and the whole XBMC becomes slow during the scan, and sometimes it even crashs...
Considering all these, and what you want with it, what to you think about these remarks :
- paths should be scanned only at XBMC start time.
- paths should be scanned only manually at demand
- paths should be scanned automatically at XBMC boot time
- scan should take the hand and users should only wait for scan to finish
- scan should be made totally silently in background (only XBMC getting slow may tell user that scan is in progress)
- scan should be made 'visual' with a discret progress bar showing but XBMC still usable
- scan should be made agressive showing a progress bar and preventing the whole XBMC to do anything else

Thanks to comment any of these remarks with your opinion and needs. It will help me to go the good way.


- mhdebokx - 2010-10-27

solexalex Wrote:- paths should be scanned only at XBMC start time.
»» not realy, because as an user you to wait till it is finished. Perhaps I like to watch an movie or play music. I don't want to wait . . . (soo boring)

- paths should be scanned only manually at demand
»» yep

- paths should be scanned automatically at XBMC boot time
»» not realy, because as an user you to wait till it is finished. Perhaps I like to watch an movie or play music. I don't want to wait . . . (soo boring)

- scan should take the hand and users should only wait for scan to finish
»» not realy

- scan should be made totally silently in background (only XBMC getting slow may tell user that scan is in progress)
»» not realy, I'd like to decide when its done.

- scan should be made 'visual' with a discret progress bar showing but XBMC still usable
»» progress bar yes, but only when the scan is started mannually

- scan should be made agressive showing a progress bar and preventing the whole XBMC to do anything else
»» noooo

see above answers


- pecinko - 2010-10-28

mhdebokx Wrote:see above answers

+1


- fiveisalive - 2010-10-28

solexalex Wrote:Hi guys !
I need an advice from users.

I made a new scan system for getting pictures inside database.
From an item within the plugin, you can add and remove as many path as you want and of course scan from here for new pictures.
I was thinking to scan pictures every time the plugin starts which may be considered as a good idea, but which gives me some troubles because it sometimes scan even when user goes back to the main plugin menu (not starting it).
Furthermore, when you have several paths to scan, with many many pictures inside, it eats a lot of resource and the whole XBMC becomes slow during the scan, and sometimes it even crashs...
Considering all these, and what you want with it, what to you think about these remarks :
- paths should be scanned only at XBMC start time.
- paths should be scanned only manually at demand
- paths should be scanned automatically at XBMC boot time
- scan should take the hand and users should only wait for scan to finish
- scan should be made totally silently in background (only XBMC getting slow may tell user that scan is in progress)
- scan should be made 'visual' with a discret progress bar showing but XBMC still usable
- scan should be made agressive showing a progress bar and preventing the whole XBMC to do anything else

Thanks to comment any of these remarks with your opinion and needs. It will help me to go the good way.

By "boot time", you mean start-up time of the plugin, right? Not booting of the entire computer that XBMC is running on?

I think there should be an option to have automatic scanning (maybe by default off) in settings.
  • If it is set to "off", then only manually by demand, so the scanning dialog box would only appear when you request the update
  • if it is set to "on", then scanning should be done once when the plugin is first entered (but not when you return to the top-menu as it does currently).
I'm happy with the current way it is done in the background, so you can browse around immediately, so that would be the "discrete" visual option. I think retaining the automatic scan is useful because I often use another desktop application (such as f-spot) to add keywords and other kinds of manipulation to the files. Or importing pics to the "Pics" directory that MyPicsDB is monitoring via f-spot or some other program. Then when I want to just browse by date, keywords (e.g. with family or friends), I use the plugin. Having to remember to manually scan would be tedious, so it's good that it will just pick up newly added photos.

But I understand that for other users, they may want to only "refresh" the database at specified times, so in those cases, they could set the automatic scan off.

On Linux it's possible to to tell whether a particular directory has been modified by the timestamp, perhaps you could record the modtime on the directory and only scan a particular subdirectory if it was newer than the last modified time, that way you could speed up scans. This would be especially useful with multiple recursive directories, which you tend to get with digital camera imports in any case.

One issue with the dialogaddscan currently in SVN is that it doesn't play nice with other skins. e.g. on the Alaska skin (which I use) which uses a mostly white background, it still shows up in black background with ugly orange characters and the fonts look out of place. Perhaps Frost could look into getting the current skin settings?


- solexalex - 2010-10-28

Sorry the third sentence is more like :
- paths should be scanned automatically at plugin boot time

Considering your feedbacks (but still waiting for others if you all have something more to request) I'm going to work on this.


- solexalex - 2010-10-29

I changed the 'last 100' item with 'recently added pictures' which shows new pictures added on last scan.
Scan can be made automatically when plugin starts or not with a plugin setting.
It can be made path per path individually and manually from item where you add/remove your path to scan. Or you can scan from here all of your paths if you have more than one in one clic.

Warning about picture updates. If you change any exif meta within a already scanned picture, the infos in the database will not be updated. I'm sorry for that, I have to work on this a little more Sad I'm close to succeed as I already get a sha [secure hash algorythm] in the database. Soon I will check every pics already in database with this sha value. If sha changed for a file, it should mean that this file has changed and so, that the DB has to update it.

I may update the scan too with the info given by fiveisalive to watch for the datetime of the folder. If it changes it should mean that some file anywhere have changed. Need to test before to know if it works on any plateform.

for those that follows closely my dev, it is updated in the svn.


- piipes - 2010-11-06

Hello, and thanks for good plugin.

I try to read, but i can't find, where is 1-5 star photo rating?
I hope i can made those rating too, with remote control quick button and sitting in my divan... Smile

"If you change any exif meta within a already scanned picture, the infos in the database will not be updated."

That is really pooring, when you have large collerction.

Sorry my poor english.


- solexalex - 2010-11-06

Hello

Photo rating is not implemented (yet ??) it looks like photo rating is made from XMP metadatas which I don't handle.
Why my plugin does not handle XMP metas as it looks like the most advanced metas for pictures ? simply because I didn't found any python library to use XMP and because I'm not skilled enough to do it of my own.

As said in a previous message, I can add photo rating INSIDE my plugin, but it will not use the rating such as setted with windows 7 or other picture managers. It will be a 'mypictures database rating only'

As for metadata changes Smile it is supported now. The version is not released yet because I need some more optimisation in all the new features and/or upgrade and/or cleaned up I made.
Every time you will scan your pictures, it will look if a change has been made, if so, the new metas will be updated in the database.


- piipes - 2010-11-06

Ok. Maybe somebody could help you. That 1-5 rates, is maybe most important feature, for good picture collection handling.


- solexalex - 2010-11-06

ho ho ho !
While I was writing that picture rating was an exception for XMP metadatas, I said myself :
"Have I already checked for this ?"
The answer was so simple : "No !!"

I have been driven into error by someone telling me that Windows 7 was using XMP metadatas to rate the pics. It is not true, or it is not only XMP.

So I tried to rate one of my pics, looks in my EXIF library and yes ! Picture rating is handled by exif metadatas :
"Image Rating"

So this gave me some more work... I have to sort pics by rating now !

But, I have to say something more. I always said my plugin was not intended to edit pictures metas. So don't expect to rate your pictures from within XBMC
(It is too dangerous if you modify your metas and the plugin crash, you may loose your pics ! I don't have enough skill and time to make a very strong meta editor !)


Feature request: sorting while scanning - mhdebokx - 2010-11-06

Hi all,

When I'm scanning large collections, like 12.000 picture, I'd like to know where the scanning process is working. At the main screen it shows the actual directory, but this doesn't tell how far the process is.

In my case, I have sorted all my pictures in subdirectories, starting with the year. Eg. 200907 Rhenen or 201005 Red Sea.

Is it possible to pre-sort all directories before scanning ? If so, then I can see on the directory name the process off scanning.


strange scanning speed behavior - mhdebokx - 2010-11-06

While scanning large collections, I see that some picture are scanned very quickly and some pictures very very slowly. I have some pics in the collection made some professional camera (Canon Canon EOS 5D Mark II), and the filesize is more then 10MB (varying up to 23MB) a picture. This is verry verry slow scanning. I find this strange, because the Exif data is just in the beginnen or at the end of the JPG files.

Also pictures made with an Nikon D50 are relatively slow (only 2 till 4MB a picture).

Does somebody else have the same experience ? I think the Exif plugin/script itself isn't aware of large picture handling.


Feature request: excluding directories for scanning - mhdebokx - 2010-11-06

I read that a new scanning mechanismn is being developed for the next version, so that unlimited directories can be added for scanning.

I'd like to see that also directories and those subdirectories can be excluded for scanning. Is this possible ?


- solexalex - 2010-11-06

Hi mhdebrokx !
Don't know which version you are using. My dev version may be different considering time to scan but it should not be so much different.

I don't know how my exif or iptc libraries are working to retrieve exif or iptc meta informations. As I'm not the dev of those I can't tell you how they do their job.
I did my best with my plugin to improve scanning time.
- I selected only exif interesting metas
- I scanned only pics that changed
- ...
One of - I think - most interesting thing when scanning, is the ability to know if a picture changed in any way (exif or iptc meta change and or filename change ...). The only working way to do it was to compute the MD5 or SHA hash value (Now I use MD5 as it is said to be faster than SHA). This hash value will be unique for a given file and will change on any meta change in it.
The problem for large files is that the hash value is computed using the whole set of binary datas. So the biggest the file size, the longest the hash computing will be...

Now is that explain, if you use the last official release as updated through dharma addon update system, you don't have the hash thing and something else is consuming the time.
Like you said, I think this is only due to exif or iptc libraries but once again, I don't know how they work Sad

Considering only the things I can do, do you prefer to easily detect changes in any pictures when updating a folder, or do you prefer to scan only for new or deleted pics ?

----

mhdebrokx Wrote:At the main screen it shows the actual directory, but this doesn't tell how far the process is.
I updated the scan functions so that now you can know as a percent how much pictures are scanned yet.
mhdebrokx Wrote:Is it possible to pre-sort all directories before scanning ? If so, then I can see on the directory name the process off scanning.
once again the scan system changed, the path to the picture that is scanning is not shown anymore... Don't know if it is a good point or bad point for your needs. The best may be to wait a little bit for new version.
mhdebrokx Wrote:I'd like to see that also directories and those subdirectories can be excluded for scanning. Is this possible ?
I may think about it, I think it is doable. Keep this idea somewhere and tell it to me after next release.


- solexalex - 2010-11-06

OK
I added in the code the ability to exclude a path from the scan.

I need to code the gui to set the folders to exclude and the functionnality will be fully working.

Any picture within an excluded path will be deleted from the DB and any picture into any subfolders of the excluded path will be deleted from DB too.
If pictures was not yet scanned into DB, they will never be added if the excluded folder is set.