1
0
Fork 0
Some of my spiders which do some job for me crawling some webs from internet. Also some other tools. Developed in Python
Go to file
Xavier Fontanet b64efee4d5 organize 2024-07-10 18:30:36 +02:00
flat-search organize 2024-07-10 18:30:36 +02:00
links-capture organize 2024-07-10 18:30:36 +02:00
.gitattributes Initial commit 2021-05-15 14:28:05 +02:00
.gitignore include DB and browser open CMD 2023-02-05 12:52:34 +01:00
README.md better explanation of the contents 2024-07-10 18:27:24 +02:00
concurrent_requests.py simplify 2024-07-10 18:13:01 +02:00

README.md

Some Python scripts

Web scraping

This was a personal-use "web spiders" to help me on certain goals such as seeking a new flat with specific characteristics or getting a massive list of edonkey links from a web page.
I've chosen Python due to the complete libraries suite for web scrapping that it offers, and the ease of using them.
But of course, since it's not my main language nor workspace the code quality and design are very poor.

Some time ago, the real state agents fees were paid by the leaseholder, and find a flat rental comming directly from the owner meant saving a lot of money, but this was very hard to find, because the common web platforms only contained real state ads.
This homemade web spider helped me on finding these exceptional deals.

I've also developed one web spider that capture all the eDonkey links from a web-page so that I could copy them to a downloader manager

Other tools

I have here also a multi-threading HTTP GET requests script, that can be used to test the performance of a system under a certain load of concurrency requests. The purpose of its development was to have a metric of reference and later improve the performance.