0

I'm having trouble logging into a website using Selenium via Python.

I'm new to web scraping and as part of the learning process I'm attempting to web scrape my account activity from American Airlines with Python. This requires logging in, which is where my code (see below) fails. All the form fields are populated, however, when I submit, the page just seems to refresh and clears my entries.

Checks I've perform:

  • Login information is correct. I've manually logged in.
  • I've played around with different lengths for sleep time. No success
  • Cleared form prior to entry. No success
  • Run the code up to (but no including) the submit() line, and then manually click the "Log In" button. Login still fails. This makes me think that the fields are somehow populated incorrectly, but I can't see where the issue is.

Thanks in advance for any help!

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

driver = webdriver.Chrome()
driver.get("https://www.aa.com/homePage.do")

loginId = driver.find_element_by_name("loginId")
lastName = driver.find_element_by_name("lastName")
password = driver.find_element_by_name("password")

time.sleep(2)
loginId.send_keys("my-email-here")
lastName.send_keys("my-last-name-here")
password.send_keys("my-password-here")
time.sleep(2)
password.submit()
narks
  • 3
  • 2
  • 1
    Have you tried using `click()` on the form's submit button instead of `submit()` on an input field ? – Evya Dec 19 '17 at 22:11
  • There is a chance there might be some javascript stuff getting screwed up with how fast the forms get filled in. Can you try putting a few 2 second sleeps in-between sending keys to each of the fields and see if that changes anything? If that makes a difference, you might need to add some explicit waits for whatever is happening to finish. – mrfreester Dec 19 '17 at 22:24
  • @EvyatarMeged yes, still no luck. – narks Dec 19 '17 at 22:27
  • @mrfreester thanks for the suggestions. I tried adding the sleeps between send_keys and also replaced password.submit() with driver.find_element_by_name("_button_go").click(). still not working. could you elaborate on the potential javascript issue? – narks Dec 19 '17 at 22:31
  • @narks sure, let's say there is some validation or some kind of other javascript that is triggered when you enter text in a field. There is a chance the site didn't count on robots filling in the fields and some logic gets jumbled when the javascript functions step on each others toes, causing a failure when you submit the form. If you can slowly step through each step though, that's probably not what's happening here. – mrfreester Dec 19 '17 at 23:18
  • @mrfreester gotcha, thanks – narks Dec 19 '17 at 23:40
  • Some how I am unable to stick on `https://www.aa.com/homePage.do`. `Chrome` and `Firefox` redirects me to `https://www.americanairlines.in/intl/in/index.jsp?locale=en_IN`. Am I missing something? – undetected Selenium Dec 20 '17 at 13:09
  • @DebanjanB odd, it sticks for me and i'm using Chrome – narks Dec 20 '17 at 21:21

1 Answers1

1

I believe that AA and other airlines have sophisticated bot detection that know you are using selenium to find and manipulate elements in the DOM, causing the page to navigate back to login. Even if I do a:

driver.find()

on the webpage and fill out the fields myself it fails, but if I do just a:

driver.get('https://www.aa.com')

and then fill out the fields it will allow me to continue. I have found a few posts on google and reddit that have gone over this if you do some searching. Also Can a website detect when you are using selenium with chromedriver? , goes over some of the ways they might be doing it.

PixelEinstein
  • 1,713
  • 1
  • 8
  • 17
  • I had a similar thought, I think this is a pretty good test for that. Good catch. – mrfreester Dec 20 '17 at 15:54
  • I've begun thinking the same thing, Airlines never cease to be non-user friendly. Thanks for the link, we'll call this the answer. – narks Dec 20 '17 at 21:24