Data mining from a website

[Evan]

Does anyone know how to collect information from a webpage?

Like for example, in my tool, if I select "Dragon" and press the "Get EXP info" button, I want the tool to print "700".
(http://tibia.wikia.com/wiki/Dragon)

How do I go about doing that? I'm hoping there are members that are experienced in this area.

[Blaster_89]

Regular Expressions

C#:

Regex hitpoints = new Regex("</span>(?<hitpoints>[^<]*)<a href=\"/wiki/Hit_Points\"");
Regex experience = new Regex("<br />(?<experience>[^<]*)<a href=\"/wiki/Experience\"");
System.Net.WebClient webclient = new System.Net.WebClient();
string str = webclient.DownloadString("http://tibia.wikia.com/wiki/Dragon");
Console.WriteLine(experience.Match(str).Groups["experience"].Value);

You may want to remove any trailing whitespaces. You might also want to group the patterns together, I just seperated them to make it easier to read.

edit: the patterns can of course be improved, I just tend to read until open/close tag when it comes to HTML out of laziness
i.e. only grabbing numbers would make removing whitespaces superflous

[Evan]

Holy crap, it works.
I totally thought it'd require a LOT more code than that little snippet.

Thanks Blaster_89, now I can do what I really want to do.

[Blaster_89]

Here are the improved patterns:

Regex hitpoints = new Regex(">(?<hitpoints>[^\\D]*)(?:[^<]*)<a href=\"/wiki/Hit_Points\"");
Regex experience = new Regex(">(?<experience>[^\\D]*)(?:[^<]*)<a href=\"/wiki/Experience\"");

?<name> gives the group a name, makes for better readability
[^\\D]* means read until a non-digit character appears
?: means don't capture this group
[^<]* means read until < appears

edit: double backslashes are necessary as escape characters exists, the actual pattern for a non-digit char is \D

[Evan]

Oh okay, thanks a bunch man!
By the way, do you have a list of those other keyword "types", like the \\D you had?

[Blaster_89]

http://msdn.microsoft.com/en-us/library/az24scfc.aspx

[Evan]

Thanks again! You've been a big help

[XtrmJash]

Not sure if TibiaML XML API is still running, but that would be my choice of data mining site. It was provided pretty much for this purpose, and your safest bet would be to download everything and stick it in your own XML or SQL database... Keeping up to date is always a pain though...

[Evan]

Not sure if TibiaML XML API is still running, but that would be my choice of data mining site. It was provided pretty much for this purpose, and your safest bet would be to download everything and stick it in your own XML or SQL database... Keeping up to date is always a pain though...

I would have done some kind of file storage, but "Keeping up to date is always a pain though..." is the reason why I want to do something like this.

[XtrmJash]

I would have done some kind of file storage, but "Keeping up to date is always a pain though..." is the reason why I want to do something like this.

Well that was the best bit about the TibiaML API, it was updated by TibiaML community, so it used to be just a case of acquiring a list of items from one section of it, then acquiring the details from another. The images were also hosted there, so you could just stick it on a site and leave it be. I think Rydan @ XenoBot forums has some method of getting item data, but I guess he made his own database using dat / spr files from Tibia.