[RELEASE] Texture Cache Maintenance utility - Printable Version +- Kodi Community Forum (https://forum.kodi.tv) +-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33) +--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116) +--- Thread: [RELEASE] Texture Cache Maintenance utility (/showthread.php?tid=158373) Pages:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
|
RE: [RELEASE] Texture Cache Maintenance utility - McButton - 2016-02-05 (2016-02-05, 02:05)Milhouse Wrote:(2016-02-05, 01:08)McButton Wrote: And, lastly...is there a command to find duplicate episodes? Or, does duplicate cover the whole db? Seems like it just returns Movies. Ah. I assumed it picked name. I think all of my movies are scraped via TMDB id. TV shows have some sort of IMDB listing. The episodes are individualized too...but they're kinda unstable. I see shows often that don't have a full listing. It may be difficult by name due to listing extensions for episodes. You'd have to make the utility not only match and ignore the show as a duplicate, it would then have to find duplicated naming within that. Like... TVSHOWNAME - S01E02 - TITLE TVSHOWNAME - S01E03 - TITLE (would maybe match due to same titles and titles are rarely covered with the same season/episode info. You could also do the same to match S01E02 -TITLE & S01E02 - TITLE2 = Match due to the season/episode listing. Depends on what you're looking to find as a duplicate. I say, if nobody has had concern until me, let it ride. I'm over/under a few episodes. I'll figure it out. The Missing Movie scanner works well enough for it. If there isn't an .nfo listed, it's easy to fix....but sometimes.........you have one jerky file that just stays elusive. RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-05 (2016-02-05, 02:52)McButton Wrote: Ah. I assumed it picked name. I think all of my movies are scraped via TMDB id. TV shows have some sort of IMDB listing. The episodes are individualized too...but they're kinda unstable. I see shows often that don't have a full listing. It may be difficult by name due to listing extensions for episodes. You'd have to make the utility not only match and ignore the show as a duplicate, it would then have to find duplicated naming within that. Unfortunately it's not very reliable - not everyone has titles in their filenames. Matching on season/episode might be possible, but then you're having to employ the same kind of regular expressions that Kodi uses to parse season and episode from the filename and they're not trivial (so many different formats). It's a lot of work for potentially very little gain (you might be the second person to ask). (2016-02-05, 02:52)McButton Wrote: The Missing Movie scanner works well enough for it. If there isn't an .nfo listed, it's easy to fix....but sometimes.........you have one jerky file that just stays elusive. Have you tried the texturecache.py "missing" option? Code: texturecache.py missing tvshows <tv source> Code: texturecache.py missing tvshows "my tv shows" RE: [RELEASE] Texture Cache Maintenance utility - guido1138 - 2016-02-05 (2016-02-05, 02:23)Milhouse Wrote:(2016-02-05, 01:26)guido1138 Wrote: As an aside, I've seen the "nc" function, which shows required artwork that is not cached. I wonder if we could have a converse function like "oc" (e.g. over-cached), which shows non-required artwork that is cached. Or perhaps even a sort of modified "C" function that doesn't ignore some artwork removal? This has nothing to do with me or someone else. As I explained clearly in my post, "required" refers to the artwork types defined in the cache.artwork property. That's how you tell what's unnecessarily cached. This is the same the terminology you yourself use in the README so I assumed it would be clear: Code: [nc] Identify those items that require caching (and would be cached by c option) As I also explained my post, a hypothetical "oc" command would be the inverse of "nc", showing cached items which wouldn't normally be cached using the "c" command and according to the cache.artwork propoerty. (2016-02-05, 02:23)Milhouse Wrote: Have a look at "purge" (and "purgetest" for testing) to remove artwork en mass. Give it the partial path for the cached artwork and it will remove them, eg.I already looked at the purge command but since it only scans the texture DB it wont work since most online-scraped artwork doesn't match fixed patterns like your example. That's why I was looking at the "c/nc/C" command family which cross-check the media library against texture DB. Hopefully this is all clear now. RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-05 Thanks for the clarification, I'll think about it, but ultimately if your cache is so small that you can't spare the space for clearart/logo artwork then a) why are you scraping this artwork and b) you should use Path Substitution to relocate the Thumbnails folder to network or USB/SD storage. The "prune" option gets rid of the dross that accumulates over time in the cache, but artwork that is in your media library is, almost by definition, not dross, which is why there's not (currently) an option to bulk remove artwork by type. One option would be to add a prune property that prevents artwork types from being matched in the media library, something like "prune.artwork.delete = clearart, logo, banner" - with that added, next time you run "P" all of your clearart, logo and banner artwork (for any media type - movies, tvshows, whatever) would be deleted from the cache. RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-05 Perhaps if the cache.artwork property is set, eg.: Code: cache.artwork = poster, fanart, landscape, thumb, season.poster then logically, while pruning, any artwork type that wouldn't otherwise be cached by c/C (because it's not in the cache.artwork list) should be removed during the prune - eg. clearart, logo, banner would all be pruned as they're not being cached by c/C, but poster, fanart, landscape, thumb and season.poster would be retained while pruning as they are being cached by c/C. RE: [RELEASE] Texture Cache Maintenance utility - guido1138 - 2016-02-06 (2016-02-05, 11:25)Milhouse Wrote: Perhaps if the cache.artwork property is set, eg.: That would work. It definitely makes more sense to extend the pruning functions ("p/P") instead of modifying a caching function ("nc/oc") as I suggested earlier. But rather than conflating caching and pruning by overloading the cache.artwork property, I think I prefer your previous idea of having a prune-specific property, like: Code: prune.artwork.delete = clearart, clearlogo, discart, banner This has the advantage of being clearer for users by explicitly specifying what to delete instead of "inferring the inverse set" from a cache.artwork property. But honestly, I'd be happy to have the functionality regardless of how the parameters are set. So thanks for giving this some consideration. RE: [RELEASE] Texture Cache Maintenance utility - zosky - 2016-02-13 (2016-02-05, 02:05)Milhouse Wrote: Tvshows don't have imdbnumbers, and I don't think the media library database supports multiple paths for the same tvshow/season/episode so I'm not sure how it would be possible to accurately determine that some random file path is a duplicate of an episode. If you've got any ideas let me know. i sorted this out in bash by concatenating 'showID-season#-ep#' to form a uniqueID the first query makes a list of any uniqueIDs with more then 1 entry. (find the dups) then we loop though the LIST and query again for the path/file.name of each offender hope this gets the creative juice flowing ... Code: #!/bin/bash BTW, thank you for this awesome sauce !! in the last 8yrs i've amassed 7.8gb of thumbnail cache ... now 5.2 after a "quick" prune RE: [RELEASE] Texture Cache Maintenance utility - gjwAudio - 2016-02-18 Hi Milhouse I'm finally getting to Do Battle with mklocal.py, and it's definitely winning Nothing will satisfy the "--local" parameter. Really, three hours of nothing, but... Code: mklocal.py: error: local DIRECTORY /FREENAS/media/ does not exist! ...and all the naming variations I could imagine, to define the NAS as seen "locally" by OpenELEC. I'm sure it's a laughably simple thing I'm missing (being a Me-No-Linux guy), but the laughter stopped a long time ago The Movies source looks like this: but I have no clue (heh-heh, obviously) what "--local" is in terms of the script ? FYI, I PuTTY in to the OpenELEC box to run the scripts (from a Windows desktop), if that's of any significance. Please, help Stop the Madness !! RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-18 "--local" needs to be the smb://FREENAS/media path as it is mounted by the client on which you're running mklocal.py. For a Linux/OpenELEC machine, this would be the mount point. Since this is an smb path, you'd need to mount /FREENAS/media as a CIFS filesystem in OpenELEC, perhaps with autostart.sh or as part of whatever script you're using to run mklocal.py, then use the local mount path as the --local parameter. Something like the following should work: Code: mkdir -p /storage/media change UNAME and PWORD to be a Windows username and password that has read/write access to the media share. Then use "--local /storage/media". When you're done running mklocal.py, use "umount /storage/media" to unmount the filesystem (no, umount isn't a typo). If you were to run the script on Windows, assuming you have /FREENAS/media mapped as your Z drive then "--local Z:\" would work. Or you could use a UNC, ie. "--local \\FREENAS\media". RE: [RELEASE] Texture Cache Maintenance utility - gjwAudio - 2016-02-18 Hi Milhouse Thanks for quick reply. Let me apologize for the Linux-n00b exercise in advance... mkdir = successful, but I'm stuck on the mount command (...let me count the hours ) The operation fails with either this: Code: OpenELEC:~ # mount -t cifs -o username=*****,password=*****,sec=ntlm //FREENAS/media /storage/media or - in desperation - this: Code: OpenELEC:~ # mount -t cifs //192.168.1.175/media/ -o username=*****,password=*****,sec=ntlm /storage/media In the second case (ie: using explicit IP address) the device is recognized alright, but I can't tell what "Invalid arguement" is actually complaining about. I'm curious why it's necessary to use the UNAME and PWORD options... IF permissions on the FreeNAS box are open (yeah, I know... next project). Kodi has full control of "media", and so presumably the scripts running on the Kodi box will have the same authority. Is that view too simple ? Thanks. RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-18 If you don't need the username/password options then drop them - I gave an example mount command, you'll need to figure out the specifics that your setup requires. sec=ntlm usually fixes the "invalid argument" problem, if it doesn't work in this case then please start another thread - sorting out CIFS mount issues is OT for this thread. You could use a systemd mount service but again that's OT for this thread - read the OpenELEC wiki. RE: [RELEASE] Texture Cache Maintenance utility - gjwAudio - 2016-02-19 Hi Milhouse Sorry for the OT... and thanks for the pointer to the systemd thing - that's got me going. I've issued this command, and it's been a long time with no action in the terminal session window... Code: OpenELEC:~/.kodi/userdata # ./mklocal.py --singlefolders --local /storage/media HAW-HAW... doing a Ctrl^C to copy the text from the PuTTY window cancelled the run - Oh Boy is it ever a LEARNING day Now that mklocal.py is halted, should I be able to see the "fixup.dat" file, written to the same directory (ie: userdata), or is it only created at the end of a complete run ? Also: my movies are in single folders, BUT the local artwork is named using the movie-prefix convention. Should I use --singlefolders switch or not if I wish to be consistent and retain that naming ? Thanks (again !) RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-19 mklocal.py is reading from stdin, you need to pipe the output of "texturecache.py jd movies" so that mklocal.py has something to process, ie.: Code: ./texturecache.py jd movies | ./mklocal.py --singlefolders --local /storage/media --prefix smb://FREENAS/media --artwork poster fanart clearlogo clearart banner landscape --verbose --output fixup.dat (2016-02-19, 03:01)gjwAudio Wrote: Also: my movies are in single folders, BUT the local artwork is named using the movie-prefix convention. Should I use --singlefolders switch or not if I wish to be consistent and retain that naming ? Don't use --singlefolders if your artwork includes the movie name prefix. Try a dry-run to be sure. RE: [RELEASE] Texture Cache Maintenance utility - gjwAudio - 2016-02-19 Hi Milhouse Here's me today - - but, I'm determined to learn the correct (and sensible) way to use your wonderful tools ! Firstly... now I realize that what I need is the Three Amigos - working together - to get my library & texturecache into shape (ie: texturecache.py jd /followed by mklocal.py blah-blah /followed by texturecache.py set) I ran the first two steps (with --dryrun), and now am looking through the fixup.dat file. Very interesting. One issue I see, is what's happened in the "Movies_VTS" folder. There's a mess to fix before running this in update mode. The folder structure is: /media/Movies_VTS/Movie-Name/VIDEO_TS/VIDEO_TS.BUP, VIDEO_TS.IFO, etc. A couple "bad things" have happened here by my hand, and running the existing fixup.dat will make things even worse (in the VTS world). As an aside, perhaps this is a funky file-naming case you have not considered yet. Let me illustrate using Defendor (2009)... Artwork Downloader (in the past, before I knew better) was set to save images to the media folders - resulting in NON-movie-name-prefixed files, placed in the Movie-Name folder. However, the associated folders for .actors/ extrafanart/ extrathumbs are found one level down, within the main VIDEO_TS folder. Not cool, it seems to me. Worse... I used to think it "good" to export the Library my to the local folders, and would do this frequently - maybe it was Kodi that put them down there. At least one movie has both prefixed- and NON-prefixed- artwork duplicates. Gee, I don't rememer doing that by hand Checking in fixup.dat, I find the script is planning to rename the Defendor files with prefix alright... BUT it's using VIDEO_TS as the Movie-Name element. Code: { With only 12 movies in VTS format, I can go through them manually to move & rename stuff without too much hassle, but will the scripts be OK with that naming convention in the future ? Best approach (as I see it now) is to remove the Movies_VTS folder from the Library, correct the naming and location anomalies, and then "Scan for new content". In the context of a VTS rip, does it make sense to name the artwork as if there were a single video file (ie: with Movie-Name-prefix) ? In your opinion, are there other practices I should adopt for keeping the artwork under control (I know, this is sorta asking "What do you do"). Thank You again, for ALL your help !! RE: [RELEASE] Texture Cache Maintenance utility and Musicvideos - zimbo - 2016-02-19 Hi, I prefer musicvideos in Kodi Isengard. I'm using Windows 7. My problem is that the artist artwork from the musicvideo isn't shown automatically by default in Kodi under the artist view.. To show my artist artwork I have to add the artist artwork manually with the context menu. It's a lot of work. Fanarts and thumbs are working well and shown from Kodi automatically. I use self created .nfo files for my musicvideos, I doesn't use any scraper for my musicvideos. So there are no musicbrainz information stored in my library. There ist also no "id" in the .nfo file and no "id" in my library. I run python in a portable Version. The path to my portable Python is: C:\Users\xxx\Downloads\kodi texture cache maintenance utility\python-3.5.1-embed-win32\python.exe The Path to the "texturecache.py" and "texturecache.cfg" is: C:\Users\xxx\Downloads\test\ To execute python with the "texturecache.py" I run / type for example in the windows command line: First under C:\Users\xxx> cd Downloads\kodi texture cache maintenance utility\python-3.5.1-embed-win32 Than its shown C:\Users\xxx\Downloads\kodi texture cache maintenance utility\python-3.5.1-embed-win32> Than I type python.exe C:\Users\xxx\Downloads\test\texturecache.py c The "texturecache.py" is working well and the "texturecache.cfg" settings are right. The path to my Kodi database: C:/Users/xxx/Appdata/Kodi/userdata The path to my local media folder (scanned into my library): C:/Users/xxx/Downloads/mp4/ AC-DC/ poster.jpg (artist artwork) AC-DC - Hells Bells/ AC-DC - Hells Bells.mp4 (musicvideo) AC-DC - Hells Bells.nfo (self created .nfo) AC-DC - Hells Bells.tbn (thumbnail) fanart.jpg Adele/ poster.jpg (artist artwork) Adele - Hello/ Adele - Hello.mp4 (musicvideo) Adele - Hello.nfo (self created .nfo) Adele - Hello.tbn (thumbnail) fanart.jpg and so on !! I've read about the "texture cache maintenance utility". Is there a way to populate / add with this tool my poster.jpg (artist artwork) automatically to my library so that the artist artwork (Artist Picture) is shown automatically in Kodi and I have not to do this manually? And if this is possible, please write the code that I have to use. I'm a noob. Please help!! |