0

I've been successfully scraping a website using mechanize, but I've had some problems with the page.open getting stuck (and not giving a timeout error) so I'd like to try and perform the same scrape with Requests. However, I can't figure out how to select a form to enter my login credentials. Here is the working code in mechanize:

# Browser
br = mechanize.Browser()

# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
br.set_proxies({"https": "198.102.28.100:7808", "http": "54.235.92.109:8080"})

# Open Login Page
br.open("https://example.com/login/Signin?")
br.select_form(name="signinForm_0")
br["loginEmail"] = "username"
br["loginPassword"] = 'password'
br.method = "POST"
br.submit()

#Open Page
URL = 'https://example.com'
br.open(URL, timeout=5.0)

I'm unsure how to replicate the br.select_form functionality using Python Requets. Does anyone have any ideas or experience doing this?

user2238685
  • 707
  • 1
  • 6
  • 8

1 Answers1

1

If I not wrong, Selenium is similar to Mechanize, but not Requests. Requests is used mostly to HTTP. Requests is similar to urllib or urllib2 but it is better. You can send request (GET or POST) and read html file from server but you need other modules to get some element on page - BeautifulSoup, lxml, pyQuery

furas
  • 134,197
  • 12
  • 106
  • 148