Javascript-based web logon for page scraping

Question

I'm trying to write a scraping engine for sbrodds.com. Unfortunately I need to login to get the right data. I looked into mechanize and selenium but I am sadly very poor at python and web scraping and am not understanding how to use these.

Here is a screenshot of the Chrome inspect elements page for the login boxes:

Can someone please provide as suggestion as to what technology I should use to perform a login to this site inside Python code? The goal is to eventually load the logged-in page's data into BeautifulSoup.

http://stackoverflow.com/questions/23102833/how-to-scrape-a-website-which-requires-login-using-python-and-beautifulsoup — bryce, Dec 09 '15 at 23:57

score 0 · Answer 1 · answered Dec 09 '15 at 23:59

usually just requests is enough ... why do you need javascript based?

import requests
r = requests.Session()
r.post("http://sbrodds.com/login",{"username":"bob","password":"sagat"})

content = r.get("http://sbrodds.com/some_page").text

something like that at least ... (you might need a csrf_token or something as well)

Javascript-based web logon for page scraping

1 Answers1