Waf

Intro The people that creates or uses crawlers to fetch information from a web page, know that you can lose access if the website owner add a WAF (Web Application Firewall) to the web page. This will certainly make your life more difficult. Some examples of WAF are: CloudFlare WAF Imperva Cloud WAF AWS/Azure WAF How it work? https://pt-corp.storage.yandexcloud.net/upload/corporate/ww-en/download/PT-devteev-CC-WAF-ENG.pdf Normally sits between the client and the server. Monitor and filters HTTP traffic between server and clients. Contains complex rules to detect malicious traffic. SQL Injections; XSS; TOR; Denial of Service (DoS), etc. Blocks malicious traffic. Story In one of my side projects I maintain some crawlers. Recently one stopped working after the website changed, but that wasn’t the only reason for the crawler to stop working, they also added a WAF to it. ...