[RELEASE] Scraper Editor (Based on ScraperXML open source C# Library) - Help wanted!

flobbes · 2010-01-29 13:00

At the moment im using an empty folder, that works fine.

But I have a problem, using the tester.

THe program crashes when enabling "buffer edit". And it crashes when I load a file from cache and start the testing.

Error enabling buffer edit:
http://paste.ubuntu.com/365114/

Nicezia · 2010-01-29 21:28

uploading 3.6.0 now should fix that problem

also this version fixes all scraper testing (tv show scraper whould work for both tvshow and tvepisode testing now)

flobbes · 2010-01-30 18:36

Nice Buffer edit works now.

When testing my own scrapers, I always get:

Code:

Could not download webpage.

If you were running auto mode it will be halted

Error returned:

Ungültiger URI: Das URI-Schema ist ungültig

The last one translates to:

Invalid URI: The URI-syntax is invalid

Any idea what the problem could be?

Nicezia · 2010-01-30 19:26

i need to see the url its trying to download in the way the scraper returns it in order to know why

and if you're trying to use gzip, that's still something i haven't added as of yet, since i don't truly know how to handle gzipped webpages, still learning when it comes to programming internet interaction.

flobbes · 2010-01-30 23:23

Its a scraper for a german adult site, so not safe for work.

The CreateSearchUrl is:

http://e.orgazmik.ch/film/list/dvd/SRCH....SRCHSTR=\1

The scraper (and my others are working fine) with xbmc, but the editor gives my that error message on all my scrapers.

The whole scraper can be found here:
http://pastebin.com/m72cf8c38

takoi · 2010-02-06 15:57

v3.x isnt working for me, using mono. it crashes right after selecting a scraper folder.

Code:

System.InvalidCastException: Cannot cast from source type to destination type.

  at ScraperXMLEditor.MainForm.scraperFunctionsBindingSource_CurrentChanged (System.Object sender, System.EventArgs e) [0x00000] 

  at System.Windows.Forms.BindingSource.OnCurrentChanged (System.EventArgs e) [0x00000] 

  at System.Windows.Forms.BindingSource.<ConnectCurrencyManager>m__1 (System.Object o, System.EventArgs args) [0x00000] 

  at (wrapper delegate-invoke) System.EventHandler:invoke_void__this___object_EventArgs (object,System.EventArgs)

  at System.Windows.Forms.CurrencyManager.OnCurrentChanged (System.EventArgs e) [0x00000] 

  at System.Windows.Forms.CurrencyManager.ChangeRecordState (Int32 newPosition, Boolean validating, Boolean endCurrentEdit, Boolean firePositionChanged, Boolean pullData) [0x00000] 

  at System.Windows.Forms.CurrencyManager.UpdateIsBinding () [0x00000] 

  at System.Windows.Forms.BindingManagerBase.PushData () [0x00000] 

  at System.Windows.Forms.CurrencyManager.ListChangedHandler (System.Object sender, System.ComponentModel.ListChangedEventArgs e) [0x00000] 

  at System.Windows.Forms.BindingSource.OnListChanged (System.ComponentModel.ListChangedEventArgs e) [0x00000] 

  at System.Windows.Forms.BindingSource.IBindingListChangedHandler (System.Object o, System.ComponentModel.ListChangedEventArgs args) [0x00000] 

  at (wrapper delegate-invoke) System.ComponentModel.ListChangedEventHandler:invoke_void__this___object_ListCha​ngedEventArgs (object,System.ComponentModel.ListChangedEventArgs)

  at System.Windows.Forms.BindingSource.OnListChanged (System.ComponentModel.ListChangedEventArgs e) [0x00000] 

  at System.Windows.Forms.BindingSource.ResetBindings (Boolean metadataChanged) [0x00000] 

  at (wrapper remoting-invoke-with-check) System.Windows.Forms.BindingSource:ResetBindings (bool)

  at ScraperXMLEditor.MainForm.tvScraperTree_AfterSelect (System.Object sender, System.Windows.Forms.TreeViewEventArgs e) [0x00000] 

  at System.Windows.Forms.TreeView.OnAfterSelect (System.Windows.Forms.TreeViewEventArgs e) [0x00000] 

  at System.Windows.Forms.TreeView.set_SelectedNode (System.Windows.Forms.TreeNode value) [0x00000] 

  at (wrapper remoting-invoke-with-check) System.Windows.Forms.TreeView:set_SelectedNode (System.Windows.Forms.TreeNode)

  at ScraperXMLEditor.MainForm.LoadScraperTree () [0x00000] 

  at ScraperXMLEditor.MainForm.NewScraper () [0x00000] 

  at ScraperXMLEditor.MainForm.MainForm_Load (System.Object sender, System.EventArgs e) [0x00000] 

  at System.Windows.Forms.Form.OnLoad (System.EventArgs e) [0x00000] 

  at System.Windows.Forms.Form.OnLoadInternal (System.EventArgs e) [0x00000] 

Unhandled Exception: System.ObjectDisposedException: The object was used after being disposed.

  at System.Windows.Forms.Control.CreateHandle () [0x00000] 

  at System.Windows.Forms.Form.CreateHandle () [0x00000] 

  at System.Windows.Forms.Control.get_Handle () [0x00000] 

  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control:get_Handle ()

  at System.Windows.Forms.Form.SetVisibleCore (Boolean value) [0x00000] 

  at System.Windows.Forms.Control.set_Visible (Boolean value) [0x00000] 

  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control:set_Visible (bool)

  at System.Windows.Forms.Application.RunLoop (Boolean Modal, System.Windows.Forms.ApplicationContext context) [0x00000] 

  at System.Windows.Forms.Application.Run (System.Windows.Forms.ApplicationContext context) [0x00000] 

  at System.Windows.Forms.Application.Run (System.Windows.Forms.Form mainForm) [0x00000] 

  at ScraperXMLEditor.Program.Main () [0x00000]

filigran · 2010-02-07 00:07

The program crashes for me when trying to run just the selected function.
Noticed it happening when just playing around with the program, so nothing is added. Empty scraper, just started the program.

Steps to reproduce:
1. Start program, add a function
2. Hit tester tab, select added function
3. Hit "Run selected function"

Error code (in swedish, do you need it translated?):

Code:

************** Undantagstext **************

System.NullReferenceException: Objektreferensen har inte angetts till en instans av ett objekt.

   vid ScraperXMLEditor.MainForm.ReplaceBuffers(String strReplace)

   vid ScraperXMLEditor.MainForm.ExpressionCompile(ScraperRegExp sreToProcess)

   vid ScraperXMLEditor.MainForm.ExecuteItem(ScraperRegExp sreToExecute)

   vid ScraperXMLEditor.MainForm.FunctionTimer_Tick(Object sender, EventArgs e)

   vid System.Windows.Forms.Timer.OnTick(EventArgs e)

   vid System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m)

   vid System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

************** Inlästa sammansättningar **************

mscorlib

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3603 (GDR.050727-3600)

    CodeBase: file:///D:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll

----------------------------------------

scraperxmled

    Sammansättningsversion: 3.6.0.0

    Win32-version: 3.6.0.0

    CodeBase: file:///D:/Documents%20and%20Settings/Administrator/Skrivbord/scraperxmleditor/scraperxmled.exe

----------------------------------------

System.Windows.Forms

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Windows.Forms/2.0.0.0__b77a5c561934e089/System.Windows.Forms.dll

----------------------------------------

System

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System/2.0.0.0__b77a5c561934e089/System.dll

----------------------------------------

System.Drawing

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Drawing/2.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll

----------------------------------------

System.Xml.Linq

    Sammansättningsversion: 3.5.0.0

    Win32-version: 3.5.30729.1 built by: SP

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Xml.Linq/3.5.0.0__b77a5c561934e089/System.Xml.Linq.dll

----------------------------------------

System.Core

    Sammansättningsversion: 3.5.0.0

    Win32-version: 3.5.30729.1 built by: SP

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Core/3.5.0.0__b77a5c561934e089/System.Core.dll

----------------------------------------

System.Xml

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3082 (QFE.050727-3000)

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Xml/2.0.0.0__b77a5c561934e089/System.Xml.dll

----------------------------------------

TechNuts

    Sammansättningsversion: 5.0.0.0

    Win32-version: 5.0.0

    CodeBase: file:///D:/Documents%20and%20Settings/Administrator/Skrivbord/scraperxmleditor/TechNuts.DLL

----------------------------------------

System.Windows.Forms.resources

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Windows.Forms.resources/2.0.0.0_sv_b77a5c561934e089/System.Windows.Forms.resources.dll

----------------------------------------

Accessibility

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/Accessibility/2.0.0.0__b03f5f7f11d50a3a/Accessibility.dll

----------------------------------------

mscorlib.resources

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3603 (GDR.050727-3600)

    CodeBase: file:///D:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll

The first line of this error:

Quote:System.NullReferenceException: Objektreferensen har inte angetts till en instans av ett objekt.

tells me, in english, "The object reference has not been assigned an instance of an object"

There was another crash when opening another scraper that I thought I'd look at for reference, specifically The imdb scraper from svn.
Then I got this error, just by opening it:

Code:

************** Undantagstext **************

System.FormatException: Indatasträngen hade ett felaktigt format.

   vid System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)

   vid System.Number.ParseDouble(String value, NumberStyles options, NumberFormatInfo numfmt)

   vid System.Double.Parse(String s, NumberStyles style, NumberFormatInfo info)

   vid TechNuts.ScraperLib.Scraper.Deserialize(XElement element)

   vid TechNuts.ScraperLib.Scraper.Load(String XmlPath)

   vid TechNuts.ScraperLib.Scraper..ctor(String xmlFilePath)

   vid ScraperXMLEditor.MainForm.LoadScraper(String xmlFilePath)

   vid ScraperXMLEditor.MainForm.Open_Click(Object sender, EventArgs e)

   vid System.Windows.Forms.ToolStripItem.RaiseEvent(Object key, EventArgs e)

   vid System.Windows.Forms.ToolStripButton.OnClick(EventArgs e)

   vid System.Windows.Forms.ToolStripItem.HandleClick(EventArgs e)

   vid System.Windows.Forms.ToolStripItem.HandleMouseUp(MouseEventArgs e)

   vid System.Windows.Forms.ToolStripItem.FireEventInteractive(EventArgs e, ToolStripItemEventType met)

   vid System.Windows.Forms.ToolStripItem.FireEvent(EventArgs e, ToolStripItemEventType met)

   vid System.Windows.Forms.ToolStrip.OnMouseUp(MouseEventArgs mea)

   vid System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)

   vid System.Windows.Forms.Control.WndProc(Message& m)

   vid System.Windows.Forms.ScrollableControl.WndProc(Message& m)

   vid System.Windows.Forms.ToolStrip.WndProc(Message& m)

   vid System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)

   vid System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)

   vid System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

************** Inlästa sammansättningar **************

mscorlib

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3603 (GDR.050727-3600)

    CodeBase: file:///D:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll

----------------------------------------

scraperxmled

    Sammansättningsversion: 3.6.0.0

    Win32-version: 3.6.0.0

    CodeBase: file:///D:/Documents%20and%20Settings/Administrator/Skrivbord/scraperxmleditor/scraperxmled.exe

----------------------------------------

System.Windows.Forms

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Windows.Forms/2.0.0.0__b77a5c561934e089/System.Windows.Forms.dll

----------------------------------------

System

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System/2.0.0.0__b77a5c561934e089/System.dll

----------------------------------------

System.Drawing

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Drawing/2.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll

----------------------------------------

System.Xml.Linq

    Sammansättningsversion: 3.5.0.0

    Win32-version: 3.5.30729.1 built by: SP

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Xml.Linq/3.5.0.0__b77a5c561934e089/System.Xml.Linq.dll

----------------------------------------

System.Core

    Sammansättningsversion: 3.5.0.0

    Win32-version: 3.5.30729.1 built by: SP

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Core/3.5.0.0__b77a5c561934e089/System.Core.dll

----------------------------------------

System.Xml

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3082 (QFE.050727-3000)

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Xml/2.0.0.0__b77a5c561934e089/System.Xml.dll

----------------------------------------

TechNuts

    Sammansättningsversion: 5.0.0.0

    Win32-version: 5.0.0

    CodeBase: file:///D:/Documents%20and%20Settings/Administrator/Skrivbord/scraperxmleditor/TechNuts.DLL

----------------------------------------

System.Windows.Forms.resources

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/System.Windows.Forms.resources/2.0.0.0_sv_b77a5c561934e089/System.Windows.Forms.resources.dll

----------------------------------------

Accessibility

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3053 (netfxsp.050727-3000)

    CodeBase: file:///D:/WINDOWS/assembly/GAC_MSIL/Accessibility/2.0.0.0__b03f5f7f11d50a3a/Accessibility.dll

----------------------------------------

mscorlib.resources

    Sammansättningsversion: 2.0.0.0

    Win32-version: 2.0.50727.3603 (GDR.050727-3600)

    CodeBase: file:///D:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll

The first line of this error:

Quote:System.FormatException: Indatasträngen hade ett felaktigt format.

tells me, in english, "The input string had the wrong format." - but the input file was named imdb.xml and was a plain text file. Creating my own scraper, saving it as something.xml and then opening something.xml works fine. So perhaps opening something that wasn't made in the program doesn't work?

Tell me if you need any other info, system specs etc.

Great program otherwise, tried creating a scraper and it made the process alot easier!

Nicezia · 2010-02-08 04:38

Quote:The program crashes for me when trying to run just the selected function.
Noticed it happening when just playing around with the program, so nothing is added. Empty scraper, just started the program.

Not running just selected function isn't working through my own fault, when i changed the tree structure, i was focused on getting the overall scraper testing to work, and forgot to look over the individual function operations... should be an easy fix. Til then you can always step through using Run selected RegExp, thanks for the heads up on that error, since i never use that particular option it escaped me.

Quote:There was another crash when opening another scraper that I thought I'd look at for reference, specifically The imdb scraper from svn.

I'm having trouble reproducing this error, but the problem may be that all the includes needed for the scraper do not exist in the scraper folder, if using a svn scraper, also make sure that all the includes from svn are also available in wherever you designate the scraper folder (my suggestion is to keep a separate folder updated with a mirror of the svn, to use when opening scrapers from svn, that's how i do it when testing the editor against svn, i have a completely separate SVN folder that keeps updated with svn on a 6 hour basis, Maybe i should put an option for it to copy SVN includes (to the app folder) locally, and user can toggle use of svn includes at their liesure.

Nicezia · 2010-02-08 04:55

ventech Wrote:v3.x isnt working for me, using mono. it crashes right after selecting a scraper folder.

I haven't tested against mono in a while, i'll check that out soon, and see that the problem is, it could be as simple as me not accounting for a directory path difference somewhere...

btw, as far as linux goes, i plan to be creating a newer version of scraperxml that will be pure c++, and likewise a new version of scraperxml editor, that will be completely gtk+ based, with no need for mono, however since i'm only learning c++ (and its so much different from c# fundamentally) that may be months in finishing, hopefully since the basic application of code will be same, its simply a matter of either creating, or finding the options i need to do the things in c++

Nicezia · 2010-02-08 07:48

Features:

* Added SearchStringEncoding for CreateSearchUrl function
* Added CachePersistence capabilities
* Update for conformance with XBMC xml conventions
* Theoretically fixed crash on run selected function
* Miscellaneous bug fixes

flobbes Senior Member Posts: 140 Joined: Mar 2009 Reputation: 0	2010-01-29 13:00 Post: #101 At the moment im using an empty folder, that works fine. But I have a problem, using the tester. THe program crashes when enabling "buffer edit". And it crashes when I load a file from cache and start the testing. Error enabling buffer edit: http://paste.ubuntu.com/365114/
	find quote

flobbes Senior Member Posts: 140 Joined: Mar 2009 Reputation: 0	2010-01-30 18:36 Post: #103 Nice Buffer edit works now. When testing my own scrapers, I always get: Code: `Could not download webpage. If you were running auto mode it will be halted Error returned: Ungültiger URI: Das URI-Schema ist ungültig` The last one translates to: Invalid URI: The URI-syntax is invalid Any idea what the problem could be?
	find quote