Pictures database and library mode; scan EXIF and IPTC metadata tags into a SQL db

  Thread Rating:
  • 2 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Thread Closed
solexalex Offline
Skilled Python Coder
Posts: 706
Joined: Jul 2004
Reputation: 6
Smile  Pictures database and library mode; scan EXIF and IPTC metadata tags into a SQL db
Post: #1
2006, October 4th, whodini expose something very interesting as a new feature for XBMC. He asks for a picture management like videos and musics have, storing informations inside a database and accessing at will.

Many of us, and surely a lot of people that never write on forums, are asking themselves "Why the picture section is not used through a database ?" but no answer from team.

Again now I still think it's a must have, but as it is not included inside xbmc itself, I made a plugin for that. It is called "MyPicture Database"

Before continuing, you may want/need to have a look to the original conversation :
http://forum.xbmc.org/showthread.php?tid=22407

As the plugin is now released on the official new XBMC dharma addon repository, I opened a topic in the python developpement section for debugging or just for feature requests.

Features
- Recursive jpg picture scanning (gif, png, bmp, tif ... may be added soon)
- EXIF / IPTC pictures metadata databasing
- By date filtering
- By folders filtering
- By keywords filtering
- local or distant files scanning (need to mount a virtual drive to use network folders)
- A python library behind the system so that developpers can write their own plugins/scripts
find
solexalex Offline
Skilled Python Coder
Posts: 706
Joined: Jul 2004
Reputation: 6
Post: #2
If everything goes well, you can expect a new version soon (maybe Sunday night in France)
-more file extension discovered and added in the database (but IPTC/EXIF will be only for jpg's)
-Number of pics for each filter
ex.: date : 2009-12 (142 pics)
ex.: keyword : Tour Eiffel (231 pics)
find
tslayer Offline
Team-XBMC Developer
Posts: 3,557
Joined: Oct 2003
Reputation: 12
Post: #3
I tried the plugin on linux but get this:


19:09:58 T:140552301729552 M:1088495616 INFO: -->Python script returned the following error<--
19:09:58 T:140552301729552 M:1088495616 ERROR: Error Type: exceptions.ImportError
19:09:58 T:140552301729552 M:1088495616 ERROR: Error Contents: No module named sqlite3
19:09:58 T:140552301729552 M:1088495616 ERROR: Traceback (most recent call last):
File "/home/eric/.xbmc/addons/plugin.image.mypicsdb/default.py", line 31, in ?
import MypicsDB as MPDB
File "/home/eric/.xbmc/addons/plugin.image.mypicsdb/resources/lib/MypicsDB.py", line 51, in ?
from sqlite3 import dbapi2 as sqlite
ImportError: No module named sqlite3
19:09:58 T:140552301729552 M:1088495616 INFO: -->End of Python script error report<--


Thanks.

42.7% of all statistics are made up on the spot

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find
solexalex Offline
Skilled Python Coder
Posts: 706
Joined: Jul 2004
Reputation: 6
Post: #4
It can t load sqlite library wich should be taken from native xbmc dharma.
Dont really know why it does not load in you case.
I will have a look later when i will be back home, but you may try to know if it is the real xbmc dharma beta 1 or another home made compilation in which sqlite would not be compiled...
find
tslayer Offline
Team-XBMC Developer
Posts: 3,557
Joined: Oct 2003
Reputation: 12
Post: #5
It's just from Dharma branch itself. Maybe something isn't being built compared to ppa or perhaps ppa includes something not in the Dharma branch itself?

Good question.

42.7% of all statistics are made up on the spot

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find
tslayer Offline
Team-XBMC Developer
Posts: 3,557
Joined: Oct 2003
Reputation: 12
Post: #6
Looks like this thread: http://forum.xbmc.org/showthread.php?tid=77708

42.7% of all statistics are made up on the spot

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find
tslayer Offline
Team-XBMC Developer
Posts: 3,557
Joined: Oct 2003
Reputation: 12
Post: #7
I guess I should have read the README.linux Smile


.3 $ make -C lib/addons/script.module.pil

.4 $ make -C lib/addons/script.module.pysqlite

42.7% of all statistics are made up on the spot

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find
solexalex Offline
Skilled Python Coder
Posts: 706
Joined: Jul 2004
Reputation: 6
Post: #8
Are you telling me it is working now ? Does any xbmc for linux need something special ?
find
tslayer Offline
Team-XBMC Developer
Posts: 3,557
Joined: Oct 2003
Reputation: 12
Post: #9
It works now.. nothing else needed except proper reading of readme.linux Smile

Though, a lot of my pictures didn't scan in... I'll have to check log later. Lots of the folders (within the plugin) show no pics.

42.7% of all statistics are made up on the spot

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find
tslayer Offline
Team-XBMC Developer
Posts: 3,557
Joined: Oct 2003
Reputation: 12
Post: #10
Okay, just looked quickly at the logs:

Filled with:


10:50:51 T:140232956843792 M:1131466752 NOTICE: MyPicsDB >> /home/eric/mounts/Volume_1/Pictures/0 RamirezGross06/2010-06-01
10:50:51 T:140232956843792 M:1131466752 NOTICE: MyPicsDB >> Ce dossier ne contient pas d'images :
10:50:51 T:140232956843792 M:1131466752 NOTICE: MyPicsDB >> /home/eric/mounts/Volume_1/Pictures/0 RamirezGross06/2010-06-01


But I clearly have files in there:


eric@htpc:~/mounts/Volume_1/Pictures/0 RamirezGross06/2010-06-01$ ls
IMG_6140.JPG IMG_6141.JPG IMG_6142.JPG IMG_6143.JPG IMG_6144.JPG IMG_6145.JPG IMG_6146.JPG

42.7% of all statistics are made up on the spot

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find
solexalex Offline
Skilled Python Coder
Posts: 706
Joined: Jul 2004
Reputation: 6
Post: #11
Okay i know. The problem is known.
It is due to the fact that my lib only scan for 'jpg' down case not JPG upper case. This is not a problem for windows users but uggly for linux users.
This is already repaired and i just need to send the version to the repository. I will not at home to do that before tomorrow.
This new version will make possible the scan of any extension you like. Anyway, only jpg will got iptc and exif infos if available.
Wait til tomorrow and tell me what do you think
find
tslayer Offline
Team-XBMC Developer
Posts: 3,557
Joined: Oct 2003
Reputation: 12
Post: #12
Thanks Smile

42.7% of all statistics are made up on the spot

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find
mhdebokx Offline
Member
Posts: 72
Joined: Dec 2006
Reputation: 0
Location: Netherlands
Post: #13
feature request to implement more EXIF.IPTCdata keywords, like
- Category
- SupplementalCategories
- CountryCode
- Country
- City

thanks in advance
find
solexalex Offline
Skilled Python Coder
Posts: 706
Joined: Jul 2004
Reputation: 6
Post: #14
hi mhdebokx
All of these are already scanned for and included in the pictures database. This is the job of my library mypicsdb.py

Unfortunately I don't really know how to include all of these within a plugin. That's why it is a library and a plugin using my library. I hope other devs could take advantage of my lib and create some scripts with full interface.

I'm thinking right now about your request, maybe can you be more explicit about what you want ? Maybe I can create some other folders like now we have 'by date, by fodler, by keywords, I may add 'by categories', 'by countrycode', ....
(You may be the only one interrested by this but I'm ready to code it for you (maybe can I think about a way to create any filters like the user want.
I.E you want a 'sort by City' folder.
The plugin show you all available fields (ie exif info and.or iptc info) and then create all the subfolders for you (like keywords filter)
City /
..../ Paris
..../ Cape Tribulation
..../ Sydney
......../Darling harbour.jpg
......../Circular Quay.jpg
......../.....
find
Robotica Offline
Banned
Posts: 1,202
Joined: Aug 2010
Post: #15
In the previous topic I reported 2 bugs:
- Strange char in the foldername caused "ERROR: PICTURE: Error loading image F:\Foto's\100ÊSIO\CIMG0276.JPG"
- Time/date values from scan with TWAIN drivers causes problems:
Code:
03:23:34 M: 28311552 WARNING: -->Python Initialized<--
03:23:35 M: 25894912 NOTICE: using pysqlite2
03:23:36 M: 23371776 NOTICE: Q:\plugins\pictures\MyPictures (beta1)\MyPictures.db
03:23:36 M: 23330816 NOTICE: Scan folder :
03:23:36 M: 23347200 NOTICE: ["F:\\Foto's\\xx\\xxxxx\\"]
03:23:36 M: 23330816 NOTICE: F:\Foto's\xx\xxx\
03:23:37 M: 23306240 ERROR: Error Type: exceptions.ValueError
03:23:37 M: 23285760 ERROR: Error Contents: time data did not match format: data=2010.03.24 13.48.00 fmt=%Y:%m:%d %H:%M:%S
03:23:37 M: 23285760 ERROR: Traceback (most recent call last):
File "Q:\plugins\pictures\MyPictures (beta1)\default.py", line 427, in ?
ok = scan_my_pics()#scan lorsque le plugin n'a pas de paramètres
File "Q:\plugins\pictures\MyPictures (beta1)\default.py", line 404, in scan_my_pics
MPDB.browse_folder(chemin,parentfolderID=None,recu rsive=xbmcplugin.getSetting('recursive')=="true",u pdate=False,updatefunc = pupdate)
File "Q:\plugins\pictures\MyPictures (beta1)\resources\lib\MypicsDB.py", line 426, in browse_folder
exif = get_exif(os.path.join(dirname,picfile).encode('utf 8'))
File "Q:\plugins\pictures\MyPictures (beta1)\resources\lib\MypicsDB.py", line 528, in get_exif
tagvalue = time.strftime("%Y-%m-%d %H:%M:%S",time.strptime(tags[tag].__str__(),"%Y:%m:%d %H:%M:%S"))
File "Q:\system\python\python24.zlib\_strptime.py", line 292, in strptime
ValueError: time data did not match format: data=2010.03.24 13.48.00 fmt=%Y:%m:%d %H:%M:%S

Request:
Picture rating from GUI (probably needs XMP to store)
find
Thread Closed