At the time of Web Scraping, we need to take care of a lot of stuff like cookies, CSRF Tokens, User-Agent & etc. Check out this article to get an understanding of these. Jacob Padilla covered many of these things with examples.