Learn to write your own Web Crawler (and Scraper)!
by Venkata Ramana (speaking)
Objective
Learn to write a crawler and scraper in python.
Description
Learn to code a basic web crawler and scraper in Python. It helps one know the anatomy of a crawler and scraping the required results in an efficient way.
For many scenarios, frameworks like scrapy and others make your life easier. But having written a crawler yourself lets you fine-tune things for your own needs.
You will be surprised how some algorithms like this bloom help in building a better Crawler. There could be room for more!
We will additionally add some optimizations for getting faster results.
Requirements
- Beginner - Intermediate knowledge of Python.
- Its better to have prior knowledge of python generators, and usage of modules like lxml.
- Laptop with Python and Git Installed.
Speaker bio
Knowledge Engineer. As a developer at Agiliq, i use Python on a day to day basis.