open multiple webdrivers without login everytime

Question

I am trying to run selenium using ThreadsPoolExecutor. The website requires a login and I am trying to speed up a step in what I am trying to do in the website. But everytime a thread opens chrome, I need to relogin and it sometimes just hangs. I login once first without using threads to do some processing. And from here on, I like to open a few chome webdrivers without the need to relogin. Is there a way around this? PS: website has no id and password strings in the url.

def startup(dirPath):
    # Start the WebDriver, load options
    options = webdriver.ChromeOptions()
    options.add_argument("--disable-infobars")
    options.add_argument("--enable-file-cookies")
    params = {'behavior': 'allow', 'downloadPath': dirPath}

    wd = webdriver.Chrome(options=options, executable_path=r"C:\Chrome\chromedriver.exe")
    wd.execute_cdp_cmd('Page.setDownloadBehavior', params)
    # wd.delete_all_cookies()
    wd.set_page_load_timeout(30)
    wd.implicitly_wait(10)

return wd    


def webLogin(dID, pw, wd):
    wd.get('some url')

    # Login, clear any outstanding login in id
    wd.find_element_by_id('username').clear()
    wd.find_element_by_id('username').send_keys(dID)
    wd.find_element_by_id('password').clear()
    wd.find_element_by_id('password').send_keys(pw)
    wd.find_element_by_css_selector('.button').click()


if __name__ == '__main__':

    dirPath, styleList = firstProcessing()
    loginAndClearLB(dID, dPw, dirPath) # calls startup & webLogin, this is also my 1st login
    # many webdrivers spawned here
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        results = {executor.submit(addArtsToLB, dID, dPw, dirPath, style): style for style in styleList} 

    #Do other stuff
    wd2 = startup(dirPath)
    webLogin(dID, dPw, wd2)
    startDL(wd2)
    logOut(wd2, dirPath)

Any help would be greatly appreciated. Thanks!!

using tokens or cookies got from logging in once you can go to homepage directly using this token or cookie in the header. The tokens would have a timeout normally. — Albin Paul, Aug 21 '19 at 12:17
Will u have some code for me to explore? Thanks. Noob at selenium and threading tbh — J T, Aug 21 '19 at 12:20
you need to find which request is making the login request and analyse its response in the chrome browser (Using networks tab in chrome console). then after you get this you can use `requests` module to first this requests and get the required token or cookie . Here is somewhat [similar answer](https://stackoverflow.com/questions/7022116/how-to-submit-http-authentication-with-selenium-python-binding-webdriver) . — Albin Paul, Aug 21 '19 at 12:26

score 0 · Accepted Answer · answered Aug 21 '19 at 18:31

Like mentioned above, you could obtain the authentication token from the first login and than include it in all the subsequent requests.

However, another option (if you're using basic auth) is to just add the username and password into the URL, like:

https://username:password@your.domain.com

score 0 · Answer 2 · answered Aug 23 '19 at 01:41

ok it looks like there is no solution yet for more complicated websites that do not use basic authentication. My modified Solution:

def webOpenWithCookie(wd, cookies):
    wd.get('https://some website url/404')
    for cookie in cookies:
        wd.add_cookie(cookie)
    wd.get('https://some website url/home')

    return wd

def myThreadedFunc(dirPath, style, cookies):  # this is the function that gets threaded
    wd = startup(dirPath)  # just starts chrome 
    wd = webOpenWithCookie(wd, cookies) # opens a page in the site and adds cookies to wd and then open your real target page. No login required now. 
    doSomethingHere(wd, style)
    wd.quit()  # close all the threads here better I think

if __name__ == '__main__':

    dirPath, styleList = firstProcessing()

    wd1 = startup(dirPath)
    wd1 = webLogin(dID, dPw, wd1)  # here i login once
    cookies = wd1.get_cookies() # get the cookie from here

    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        results = {executor.submit(myThreadedFunc, dirPath, style, cookies): style for style in styleList} # this spawns threads, but each thread will not need login although the compromise is it needs to go to 404 page first.

open multiple webdrivers without login everytime

2 Answers2