[RELEASE] Scraper Editor (Based on ScraperXML open source C# Library) - Help wanted!

  Thread Rating:
  • 1 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
takoi Offline
Fan
Posts: 729
Joined: Oct 2009
Reputation: 10
Location: Norway
Post: #106
v3.x isnt working for me, using mono. it crashes right after selecting a scraper folder.
Code:
System.InvalidCastException: Cannot cast from source type to destination type.
  at ScraperXMLEditor.MainForm.scraperFunctionsBindingSource_CurrentChanged (System.Object sender, System.EventArgs e) [0x00000]
  at System.Windows.Forms.BindingSource.OnCurrentChanged (System.EventArgs e) [0x00000]
  at System.Windows.Forms.BindingSource.<ConnectCurrencyManager>m__1 (System.Object o, System.EventArgs args) [0x00000]
  at (wrapper delegate-invoke) System.EventHandler:invoke_void__this___object_EventArgs (object,System.EventArgs)
  at System.Windows.Forms.CurrencyManager.OnCurrentChanged (System.EventArgs e) [0x00000]
  at System.Windows.Forms.CurrencyManager.ChangeRecordState (Int32 newPosition, Boolean validating, Boolean endCurrentEdit, Boolean firePositionChanged, Boolean pullData) [0x00000]
  at System.Windows.Forms.CurrencyManager.UpdateIsBinding () [0x00000]
  at System.Windows.Forms.BindingManagerBase.PushData () [0x00000]
  at System.Windows.Forms.CurrencyManager.ListChangedHandler (System.Object sender, System.ComponentModel.ListChangedEventArgs e) [0x00000]
  at System.Windows.Forms.BindingSource.OnListChanged (System.ComponentModel.ListChangedEventArgs e) [0x00000]
  at System.Windows.Forms.BindingSource.IBindingListChangedHandler (System.Object o, System.ComponentModel.ListChangedEventArgs args) [0x00000]
  at (wrapper delegate-invoke) System.ComponentModel.ListChangedEventHandler:invoke_void__this___object_ListCha​ngedEventArgs (object,System.ComponentModel.ListChangedEventArgs)
  at System.Windows.Forms.BindingSource.OnListChanged (System.ComponentModel.ListChangedEventArgs e) [0x00000]
  at System.Windows.Forms.BindingSource.ResetBindings (Boolean metadataChanged) [0x00000]
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.BindingSource:ResetBindings (bool)
  at ScraperXMLEditor.MainForm.tvScraperTree_AfterSelect (System.Object sender, System.Windows.Forms.TreeViewEventArgs e) [0x00000]
  at System.Windows.Forms.TreeView.OnAfterSelect (System.Windows.Forms.TreeViewEventArgs e) [0x00000]
  at System.Windows.Forms.TreeView.set_SelectedNode (System.Windows.Forms.TreeNode value) [0x00000]
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.TreeView:set_SelectedNode (System.Windows.Forms.TreeNode)
  at ScraperXMLEditor.MainForm.LoadScraperTree () [0x00000]
  at ScraperXMLEditor.MainForm.NewScraper () [0x00000]
  at ScraperXMLEditor.MainForm.MainForm_Load (System.Object sender, System.EventArgs e) [0x00000]
  at System.Windows.Forms.Form.OnLoad (System.EventArgs e) [0x00000]
  at System.Windows.Forms.Form.OnLoadInternal (System.EventArgs e) [0x00000]

Unhandled Exception: System.ObjectDisposedException: The object was used after being disposed.
  at System.Windows.Forms.Control.CreateHandle () [0x00000]
  at System.Windows.Forms.Form.CreateHandle () [0x00000]
  at System.Windows.Forms.Control.get_Handle () [0x00000]
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control:get_Handle ()
  at System.Windows.Forms.Form.SetVisibleCore (Boolean value) [0x00000]
  at System.Windows.Forms.Control.set_Visible (Boolean value) [0x00000]
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control:set_Visible (bool)
  at System.Windows.Forms.Application.RunLoop (Boolean Modal, System.Windows.Forms.ApplicationContext context) [0x00000]
  at System.Windows.Forms.Application.Run (System.Windows.Forms.ApplicationContext context) [0x00000]
  at System.Windows.Forms.Application.Run (System.Windows.Forms.Form mainForm) [0x00000]
  at ScraperXMLEditor.Program.Main () [0x00000]
find quote
filigran Offline
Senior Member
Posts: 192
Joined: Oct 2009
Reputation: 0
Post: #107
The program crashes for me when trying to run just the selected function.
Noticed it happening when just playing around with the program, so nothing is added. Empty scraper, just started the program.

Steps to reproduce:
1. Start program, add a function
2. Hit tester tab, select added function
3. Hit "Run selected function"

Error code (in swedish, do you need it translated?):
Code:
************** Undantagstext **************
System.NullReferenceException: Objektreferensen har inte angetts till en instans av ett objekt.
   vid ScraperXMLEditor.MainForm.ReplaceBuffers(String strReplace)
   vid ScraperXMLEditor.MainForm.ExpressionCompile(ScraperRegExp sreToProcess)
   vid ScraperXMLEditor.MainForm.ExecuteItem(ScraperRegExp sreToExecute)
   vid ScraperXMLEditor.MainForm.FunctionTimer_Tick(Object sender, EventArgs e)
   vid System.Windows.Forms.Timer.OnTick(EventArgs e)
   vid System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m)
   vid System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Inlästa sammansättningar **************
mscorlib
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3603 (GDR.050727-3600)
    CodeBase: file:///D:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll
----------------------------------------
scraperxmled
    Sammansättningsversion: 3.6.0.0
    Win32-version: 3.6.0.0
    CodeBase: file:///D:/Documents%20and%20Settings/Administrator/Skrivbord/scraperxmleditor/scraperxmled.exe
----------------------------------------
System.Windows.Forms
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Windows.Forms/2.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System/2.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Drawing
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Drawing/2.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Xml.Linq
    Sammansättningsversion: 3.5.0.0
    Win32-version: 3.5.30729.1 built by: SP
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Xml.Linq/3.5.0.0__b77a5c561934e089/System.Xml.Linq.dll
----------------------------------------
System.Core
    Sammansättningsversion: 3.5.0.0
    Win32-version: 3.5.30729.1 built by: SP
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Core/3.5.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
System.Xml
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3082 (QFE.050727-3000)
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Xml/2.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
TechNuts
    Sammansättningsversion: 5.0.0.0
    Win32-version: 5.0.0
    CodeBase: file:///D:/Documents%20and%20Settings/Administrator/Skrivbord/scraperxmleditor/TechNuts.DLL
----------------------------------------
System.Windows.Forms.resources
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Windows.Forms.resources/2.0.0.0_sv_b77a5c561934e089/System.Windows.Forms.resources.dll
----------------------------------------
Accessibility
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/Accessibility/2.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
----------------------------------------
mscorlib.resources
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3603 (GDR.050727-3600)
    CodeBase: file:///D:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll

The first line of this error:
Quote:System.NullReferenceException: Objektreferensen har inte angetts till en instans av ett objekt.
tells me, in english, "The object reference has not been assigned an instance of an object"

There was another crash when opening another scraper that I thought I'd look at for reference, specifically The imdb scraper from svn.
Then I got this error, just by opening it:
Code:
************** Undantagstext **************
System.FormatException: Indatasträngen hade ett felaktigt format.
   vid System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   vid System.Number.ParseDouble(String value, NumberStyles options, NumberFormatInfo numfmt)
   vid System.Double.Parse(String s, NumberStyles style, NumberFormatInfo info)
   vid TechNuts.ScraperLib.Scraper.Deserialize(XElement element)
   vid TechNuts.ScraperLib.Scraper.Load(String XmlPath)
   vid TechNuts.ScraperLib.Scraper..ctor(String xmlFilePath)
   vid ScraperXMLEditor.MainForm.LoadScraper(String xmlFilePath)
   vid ScraperXMLEditor.MainForm.Open_Click(Object sender, EventArgs e)
   vid System.Windows.Forms.ToolStripItem.RaiseEvent(Object key, EventArgs e)
   vid System.Windows.Forms.ToolStripButton.OnClick(EventArgs e)
   vid System.Windows.Forms.ToolStripItem.HandleClick(EventArgs e)
   vid System.Windows.Forms.ToolStripItem.HandleMouseUp(MouseEventArgs e)
   vid System.Windows.Forms.ToolStripItem.FireEventInteractive(EventArgs e, ToolStripItemEventType met)
   vid System.Windows.Forms.ToolStripItem.FireEvent(EventArgs e, ToolStripItemEventType met)
   vid System.Windows.Forms.ToolStrip.OnMouseUp(MouseEventArgs mea)
   vid System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
   vid System.Windows.Forms.Control.WndProc(Message& m)
   vid System.Windows.Forms.ScrollableControl.WndProc(Message& m)
   vid System.Windows.Forms.ToolStrip.WndProc(Message& m)
   vid System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   vid System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   vid System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Inlästa sammansättningar **************
mscorlib
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3603 (GDR.050727-3600)
    CodeBase: file:///D:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll
----------------------------------------
scraperxmled
    Sammansättningsversion: 3.6.0.0
    Win32-version: 3.6.0.0
    CodeBase: file:///D:/Documents%20and%20Settings/Administrator/Skrivbord/scraperxmleditor/scraperxmled.exe
----------------------------------------
System.Windows.Forms
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Windows.Forms/2.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System/2.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Drawing
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Drawing/2.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Xml.Linq
    Sammansättningsversion: 3.5.0.0
    Win32-version: 3.5.30729.1 built by: SP
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Xml.Linq/3.5.0.0__b77a5c561934e089/System.Xml.Linq.dll
----------------------------------------
System.Core
    Sammansättningsversion: 3.5.0.0
    Win32-version: 3.5.30729.1 built by: SP
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Core/3.5.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
System.Xml
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3082 (QFE.050727-3000)
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Xml/2.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
TechNuts
    Sammansättningsversion: 5.0.0.0
    Win32-version: 5.0.0
    CodeBase: file:///D:/Documents%20and%20Settings/Administrator/Skrivbord/scraperxmleditor/TechNuts.DLL
----------------------------------------
System.Windows.Forms.resources
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Windows.Forms.resources/2.0.0.0_sv_b77a5c561934e089/System.Windows.Forms.resources.dll
----------------------------------------
Accessibility
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/Accessibility/2.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
----------------------------------------
mscorlib.resources
    Sammansättningsversion: 2.0.0.0
    Win32-version: 2.0.50727.3603 (GDR.050727-3600)
    CodeBase: file:///D:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll

The first line of this error:
Quote:System.FormatException: Indatasträngen hade ett felaktigt format.
tells me, in english, "The input string had the wrong format." - but the input file was named imdb.xml and was a plain text file. Creating my own scraper, saving it as something.xml and then opening something.xml works fine. So perhaps opening something that wasn't made in the program doesn't work?

Tell me if you need any other info, system specs etc.

Great program otherwise, tried creating a scraper and it made the process alot easier!
(This post was last modified: 2010-02-07 00:09 by filigran.)
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #108
Quote:The program crashes for me when trying to run just the selected function.
Noticed it happening when just playing around with the program, so nothing is added. Empty scraper, just started the program.

Not running just selected function isn't working through my own fault, when i changed the tree structure, i was focused on getting the overall scraper testing to work, and forgot to look over the individual function operations... should be an easy fix. Til then you can always step through using Run selected RegExp, thanks for the heads up on that error, since i never use that particular option it escaped me.

Quote:There was another crash when opening another scraper that I thought I'd look at for reference, specifically The imdb scraper from svn.

I'm having trouble reproducing this error, but the problem may be that all the includes needed for the scraper do not exist in the scraper folder, if using a svn scraper, also make sure that all the includes from svn are also available in wherever you designate the scraper folder (my suggestion is to keep a separate folder updated with a mirror of the svn, to use when opening scrapers from svn, that's how i do it when testing the editor against svn, i have a completely separate SVN folder that keeps updated with svn on a 6 hour basis, Maybe i should put an option for it to copy SVN includes (to the app folder) locally, and user can toggle use of svn includes at their liesure.

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
(This post was last modified: 2010-02-08 04:46 by Nicezia.)
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #109
ventech Wrote:v3.x isnt working for me, using mono. it crashes right after selecting a scraper folder.

I haven't tested against mono in a while, i'll check that out soon, and see that the problem is, it could be as simple as me not accounting for a directory path difference somewhere...

btw, as far as linux goes, i plan to be creating a newer version of scraperxml that will be pure c++, and likewise a new version of scraperxml editor, that will be completely gtk+ based, with no need for mono, however since i'm only learning c++ (and its so much different from c# fundamentally) that may be months in finishing, hopefully since the basic application of code will be same, its simply a matter of either creating, or finding the options i need to do the things in c++

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #110
Features:

* Added SearchStringEncoding for CreateSearchUrl function
* Added CachePersistence capabilities
* Update for conformance with XBMC xml conventions
* Theoretically fixed crash on run selected function
* Miscellaneous bug fixes

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
filigran Offline
Senior Member
Posts: 192
Joined: Oct 2009
Reputation: 0
Post: #111
Nicezia Wrote:I'm having trouble reproducing this error, but the problem may be that all the includes needed for the scraper do not exist in the scraper folder, if using a svn scraper, also make sure that all the includes from svn are also available in wherever you designate the scraper folder (my suggestion is to keep a separate folder updated with a mirror of the svn, to use when opening scrapers from svn, that's how i do it when testing the editor against svn, i have a completely separate SVN folder that keeps updated with svn on a 6 hour basis, Maybe i should put an option for it to copy SVN includes (to the app folder) locally, and user can toggle use of svn includes at their liesure.

Yeah, I haven't tested that, but as it doesn't crash when I use my own scraper, with the thumbnail in the same folder, that's probably it.
find quote
drascom Offline
Senior Member
Posts: 129
Joined: Dec 2008
Reputation: 0
Smile  [SOLVED] repeat function
Post: #112
if my regexp match 2 results like genre. how can i use 2 different row together?
[img=http://pic1.resimupload.com/r7/thumb_913275750.JPG


i wanto use "dram" and " aksiyon" on capture 2 together...

Just click repeat function and it's capture all results.
(This post was last modified: 2010-04-22 15:23 by drascom.)
find quote
hojel Offline
Junior Member
Posts: 11
Joined: Dec 2009
Reputation: 0
Post: #113
I'm making new scraper in Korean.
Ugly thing is that the page is written in euc-kr encoding, not utf-8.
So, I added SearchStringEncoding="euc-kr" to each CreateAlbumSearchUrl and CreateArtistSearchUrl function.

When I run the scraper with pressing 'Run Scraper' button, it seems like encoding query string still in utf-8, not in euc-kr.
I wanna know such encoding conversion is supported in your editor.
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #114
sorry, that was my fault, a omitted statement in the function.

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #115
In one word (and several question marks)

WhatConfusedConfusedConfused??

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
olympia Offline
Team-Kodi Member
Posts: 2,499
Joined: May 2008
Reputation: 32
Post: #116
Hi Nicezia,

are you still maintaining your vital applications?
In other words: do you accept bug reports and will you adopt the editor for the upcoming addon structure?
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #117
fyi i'm not holding back on the info nicezia, just want to avoid you annoy you until stuff is set in stone Smile
find quote
olympia Offline
Team-Kodi Member
Posts: 2,499
Joined: May 2008
Reputation: 32
Post: #118
I don't want to annoy him, just expressed my general interest Tongue
And I've discovered a few bugs which are not related to the changes at all.
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #119
heh, sure, i didn't mean to imply you were annoying him. but if he had gotten instructions from me all the time (as you know yourself we're done quite some changes as shit is developed), then he would probably have been annoyed by now Smile
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #120
its fine spiff. I'm trying to work out some localization and encoding issues, any changes you've made in the scraper process will be pretty quick implementation i believe.

Oh one thing though did fanart get sanitized sanified.. or whatever?
Code:
<fanart preview="">http://foo.com/foo.jpg</fanart>
<fanart preview="">http://foo.com/foo.jpg</fanart>


instead of
Code:
<fanart url="http://foo.com"><thumb>/foo.jpg</thumb></fanart>
Confused

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
(This post was last modified: 2010-06-03 23:06 by Nicezia.)
find quote
Post Reply