Deprecated: The behavior of unparenthesized expressions containing both '.' and '+'/'-' will change in PHP 8: '+'/'-' will take a higher precedence in /home/iano/public_html/tpforums-vb5/forum/includes/class_core.php on line 5842
PHP Warning: Use of undefined constant MYSQL_NUM - assumed 'MYSQL_NUM' (this will throw an Error in a future version of PHP) in ..../includes/init.php on line 165
PHP Warning: Use of undefined constant MYSQL_ASSOC - assumed 'MYSQL_ASSOC' (this will throw an Error in a future version of PHP) in ..../includes/init.php on line 165
PHP Warning: Use of undefined constant MYSQL_BOTH - assumed 'MYSQL_BOTH' (this will throw an Error in a future version of PHP) in ..../includes/init.php on line 165
PHP Warning: "continue" targeting switch is equivalent to "break". Did you mean to use "continue 2"? in ..../includes/functions_navigation.php on line 588
PHP Warning: "continue" targeting switch is equivalent to "break". Did you mean to use "continue 2"? in ..../includes/functions_navigation.php on line 612
PHP Warning: Use of undefined constant misc - assumed 'misc' (this will throw an Error in a future version of PHP) in ..../global.php(29) : eval()'d code(6) : eval()'d code on line 1
PHP Warning: Use of undefined constant index - assumed 'index' (this will throw an Error in a future version of PHP) in ..../global.php(29) : eval()'d code(6) : eval()'d code on line 1
PHP Warning: Use of undefined constant misc - assumed 'misc' (this will throw an Error in a future version of PHP) in ..../includes/class_bootstrap.php(1422) : eval()'d code(4) : eval()'d code on line 1
PHP Warning: Use of undefined constant index - assumed 'index' (this will throw an Error in a future version of PHP) in ..../includes/class_bootstrap.php(1422) : eval()'d code(4) : eval()'d code on line 1
PHP Warning: Use of undefined constant onlinestatusphrase - assumed 'onlinestatusphrase' (this will throw an Error in a future version of PHP) in ..../includes/class_core.php(4684) : eval()'d code on line 6 C# Tibia Scraper
I made a quick Tibia Scraper because I was bored. it's completely open-source. I just threw something together in about 20 minutes and it works great. It first of all scrapes the online list. You can filter it by world, vocation, level and so on. For example, if you want to see all level 34+ druids on calmera, you can do so.
After it scraped the online list, it continues to scrape their profiles pages.
This might take a while depending on how many players you scrape.
If you set it to scrape all worlds, all vocations, all levels, and the online amount is like 13000, then yes it is going to make 13000 webrequests, which will result in a lot of 403 Forbidden errors.
To fix the 403 error, I added the possibility to add a delay between each request (per page).
If you put it to 1 second, it's going to take a minimum of 13000 seconds to scrape it all.
You can try around and see what delay works the best.
It was only recently CipSoft put a block on their web site.
A few weeks ago you were able to make thousands of requests without any problem.
Now you have to slow it down the more requests you make.
Anyways, when it has scraped their profiles, you can save it all to a database.
Instructions for table and column names are included in the application. Takes a minute to set the databse up.
Remember to make the name column unique. This is so that it will UPDATE existing players, and not INSERT them again.
And oh, the world list is extracted from the textfile worlds.txt so that you can quickly update it without recompiling everything, in case cipsoft adds/merges servers.
Of course there's a lot that can be improved or added, but this should do for a simple webscraper if you decide to perhaps collect stats about certain players and so on
Reply With Quote