How to retain the login state in selenium

Question

I use selenium and bs4 to parse web page. And the web page use scan qrcode and verification code to login.

I use WebDriverWait to wait the user to login with

wait = ui.WebDriverWait(driver, 60)         # timeout after 60 seconds, just leave time for user to login

wait.until(lambda driver: driver.execute_script('return isLogin();'))

and after I login, then begin the parsing work.

It's works well, except that each time when I run the script, it open a new broswr window and I need to login.

How to retain the login state to avoid login when each time run the script.

Answer 1

To avoid login every time you can use cookies. For example:

import pickle

from selenium import webdriver 


browser = webdriver.Chrome()
browser.get("http://www.google.com")
pickle.dump(browser.get_cookies(), open("cookies.pkl","wb"))

and download cookies later:

import pickle

from selenium import webdriver


browser = webdriver.Chrome()
browser.get("http://www.google.com")
cookies = pickle.load(open("cookies.pkl", "rb"))
for cookie in cookies:
    browser.add_cookie(cookie)

Answer 2

I've find below two solutions.

1. use broser profile (This not apply to site ask for login when open a new window)

I use firefox, so I set profile in this way.

profile = webdriver.FirefoxProfile('C:/Users/lf/AppData/Roaming/Mozilla/Firefox/Profiles/5fvhqsc9.selenium')
driver = webdriver.Firefox(firefox_profile=profile)

If you want to set a separate profile for your code, use firefox.exe -p in cmd line. Refer to firefox-profile-selenium-webdriver
Note, just use the profile in your code, don't change the default select profile.

2. use cookie (This apply to site ask for login when open a new window)

from selenium import webdriver
from selenium.webdriver.support import ui
import pickle

driver = webdriver.Firefox()
"""
    Cookie can be only add to the request with same domain.
    When webdriver init, it's request url is `data:` so you cannot add cookie to it.
    So first make a request to your url then add cookie, then request you url again.
"""
browser = driver.get('url')

cookies = pickle.load(open("cookies.pkl", "rb"))
for cookie in cookies:
    driver.add_cookie(cookie)
    
browser = driver.get('url')

if driver.execute_script('return !isLogin();'):       #[How to access javascript result in selenium](https://stackoverflow.com/q/58620192/6521116)
    driver.execute_script('openLoginUI();')    
    # [How can I make Selenium/Python wait for the user to login before continuing to run?](https://stackoverflow.com/a/16927552/6521116)
    wait = ui.WebDriverWait(driver, 60)         # timeout after 60 seconds, just leave time for user to login
    wait.until(driver.execute_script('return isLogin()'))
    
pickle.dump(driver.get_cookies(), open("cookies.pkl", "wb"))

---

Reference:
How to access javascript result in selenium
How can I make Selenium/Python wait for the user to login before continuing to run?
org.openqa.selenium.InvalidCookieDomainException: Document is cookie-averse using Selenium and WebDriver

Answer 3

You can use a default profile of a browser

options = webdriver.ChromeOptions() 
options.add_argument("user-data-dir=C:\\Path") #Path to your chrome profile
w = webdriver.Chrome(executable_path="C:\\Users\\chromedriver.exe", chrome_options=options)

You can find a profile path by putting chrome://version/ in browser url.