How to implement ajax request using Python Request

Question

I'm trying to log into a website using Python request. Unfortunately, it is always showing this error when printing its content.

b'<head><title>Not Acceptable!</title></head><body><h1>Not Acceptable!</h1><p>An appropriate representation of the requested resource could not be found on this server. This error was generated by Mod_Security.</p></body></html>

For reference my code

from requests import Session
import requests

INDEX_URL = 'https://phpzag.com/demo/ajax_login_script_with_php_jquery/index.php'
URL = 'https://phpzag.com/demo/ajax_login_script_with_php_jquery/welcome.php'
LOGIN_URL = 'https://phpzag.com/demo/ajax_login_script_with_php_jquery/login.php' # Or whatever the login request url is
payload = {'user_email': 'test@phpzag.com','password':'test'}

s = requests.Session()
user_agent = {'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36'}
t=s.post(LOGIN_URL, data=payload, headers=user_agent)
r=s.get('https://phpzag.com/demo/ajax_login_script_with_php_jquery/welcome.php',headers=user_agent,cookies=t.cookies.get_dict())
print(r.content)

May I know what is missing and how can I get HTML code of welcome page from this

UPDATE

I'm trying to get make an API call after login authentication. However, I'm not able to succeed in login authentication. Hence I am not able to get the response of API Call. As per my thought it due to multi-factor authentication it is getting failed. I need to know how can I implement this?

For eg: www.abc.com is the URL of the website. The login is done through JS form submission Hence URL is specified in the ajax part. On the success of that, there is another third authentication party(okta) which will also verify the credentials and finally reach the home page. then I need to call the real API for my task.

But it is not working.

import requests
import sys
class Login:

     def sendRequestWithAuthentication(self,loginDetails,requestDetails):
         user_agent = {'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36'}
         action_url=loginDetails['action_url'] if 'action_url' in loginDetails.keys() else None
         pay_load=loginDetails['payload'] if 'payload' in loginDetails.keys() else None
         session_requests = requests.session()
         if action_url and pay_load:
             act_resp=session_requests.post(action_url, data=pay_load, headers=user_agent,verify=False,files=[ ])
             print(act_resp)
             auth_cookies=act_resp.cookies.get_dict()
             url,method,request_payload = requestDetails['url'],requestDetails['method'],requestDetails['payload']
             querystring=requestDetails['querystring']
             response=session_requests.get(url,headers=user_agent,cookies=auth_cookies,data=request_payload,params=querystring)
             print(response)
             return response.json()

In the above action URL is the API given in the ajax part & in the second request, the URL is the API address for that GET.

In short, may I know how can implement multifactor authentication in python request

My Doubt

Do we need the cookies from the login form page to include in the login request
How to implement multifactor authentication in python request(Here we don't need any pin or something it is done through RSA.)Is there any need of a certificate for login as it now raising unable to validate the SSL certificate

Give a dummy example api that is implement such kind of scenario

score 11 · Answer 1 · answered Jun 01 '20 at 16:37

No, you make it complex.This code worked:

import requests

login_url = "https://phpzag.com/demo/ajax_login_script_with_php_jquery/login.php"
welcome_url = "https://phpzag.com/demo/ajax_login_script_with_php_jquery/welcome.php"

payload = 'user_email=test@phpzag.com&password=test&login_button='
login_headers = {
    'x-requested-with': 'XMLHttpRequest',
    'Content-Type': 'application/x-www-form-urlencoded', # its urlencoded instead of form-data
    'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36',
}
s = requests.Session()
login = s.post(login_url, headers=login_headers, data=payload) # post requests
welcome = s.get(welcome_url, headers=login_headers)
print(welcome.text)

Result:

.....Hello, <br><br>Welcome to the members page.<br><br>

score 2 · Answer 2 · answered Jun 04 '20 at 21:02

TL;DR

Change the part of your code that says data=payload to json=payload, and it should work.

Direct answer to your question

How [does one] implement [an] AJAX request using Python Requests?

You cannot do that. An AJAX request is specifically referring to a Javascript-based HTTP request. To quote from W3 school's AJAX introduction page, "AJAX = Asynchronous JavaScript And XML".

Indirect answer to your question

What I believe you're asking is how to perform auth/login HTTP requests using the popular python package, requests. The short answer— unfortunately, and like most things—is that it depends. Various auth pages handle the auth requests differently, and so you might have to do different things in order to authenticate against the specific web service.

Based on your code

I'm going to make some assumptions that the login page is probably looking for a POST request with the authentication details (e.g. credentials) in the form of a JSON object based on your code, and based on the response back from the server being a 406 error meaning that you're sending data with an accept header that doesn't align with how the server wants to respond.

When using requests, using the data parameter to the request function will send the data "raw"; that is, it'll send it in the native data format it is (like in cases of binary data), or it'll translate it to standard HTML form data if that format doesn't work (e.g. key1=value1&key2=value2&key3=value3, this form has the MIME type of application/x-www-form-urlencoded and is what requests will send when data has not been specified with an accept header). I'm going to make an educated guess based on the fact that you put your credentials into a dictionary that the login form is expecting a POST request with a JSON-formatted body (most modern web apps do this), and you were under the impression that setting the data parameter to requests will make this into a JSON object. This is a common gotcha/misconception with requests that has bitten me before. What you want is instead to pass the data using the json parameter.

Your code:

from requests import Session
import requests

INDEX_URL = 'https://phpzag.com/demo/ajax_login_script_with_php_jquery/index.php'
URL = 'https://phpzag.com/demo/ajax_login_script_with_php_jquery/welcome.php'
LOGIN_URL = 'https://phpzag.com/demo/ajax_login_script_with_php_jquery/login.php' # Or whatever the login request url is
payload = {'user_email': 'test@phpzag.com','password':'test'}

s = requests.Session()
user_agent = {'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36'}
t=s.post(LOGIN_URL, data=payload, headers=user_agent)
r=s.get('https://phpzag.com/demo/ajax_login_script_with_php_jquery/welcome.php',headers=user_agent,cookies=t.cookies.get_dict())
print(r.content)

Fixed (and cleaned up) code:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
   Test script to login to php web app.
"""

import requests

INDEX_URL = 'https://phpzag.com/demo/ajax_login_script_with_php_jquery/index.php'
URL = 'https://phpzag.com/demo/ajax_login_script_with_php_jquery/welcome.php'
LOGIN_URL = 'https://phpzag.com/demo/ajax_login_script_with_php_jquery/login.php' # Or whatever the login request url is

payload = {
    'user_email': 'test@phpzag.com',
    'password':'test'
}

headers = {
    'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36'
}

session = requests.Session()


auth_response = session.post(
    url=LOGIN_URL,
    json=payload,  # <--- THIS IS THE IMPORTANT BIT. Note: data param changed to json param
    headers=user_agent
)

response = session.get(
    'https://phpzag.com/demo/ajax_login_script_with_php_jquery/welcome.php',
    headers=headers,
    cookies=auth_response.cookies.get_dict()  # TODO: not sure this is necessary, since you're using the session object to initiate the request, so that should maintain the cookies/session data throughout the session...
)

print(response.content)

Check out this section of the requests documentation on POST requests, if you scroll down a bit from there you'll see the docs talk about the github API which expects JSON and how to handle that.

Auth can be tricky overall. Sometimes things will want "basic auth", which requests will expect you to pass as a tuple to the auth parameter, sometimes they'll want a bearer token / OAUTH thing which can get headache-inducing-ly complicated/annoying.

Hope this helps!

Thanks - this was extremely helpful to me. While auth on an ajax site is new to me (what i'm using your code as template for), in terms of your #TODO comment, I believe it IS necessary to include cookies in all get requests, even for a session. Requests docs indicate cookies are not persistent here: https://requests.readthedocs.io/en/master/user/advanced/ — thesimplevoodoo, Mar 26 '21 at 16:12
@thesimplevoodoo you're welcome, glad I could help! Interesting. You could always try without the cookies being added in like that and see if it works. If it doesn't work (i.e. you get 403 errors, etc.) then you can add it back in. — masq, Jun 11 '21 at 23:44

Luv · Answer 3 · 2020-05-05T06:47:04.823

1

You are missing the User agent that the server (apache?) requires

Try this:

import requests
from requests import Session

URL = 'https://phpzag.com/demo/ajax_login_script_with_php_jquery/welcome.php'
LOGIN_URL = 'https://phpzag.com/demo/ajax_login_script_with_php_jquery/login.php' # Or whatever the login request url is
payload = {'user_email': 'test@phpzag.com','password':'test'}
user_agent = {'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36'}

s = requests.Session()
x=s.get(URL, headers=user_agent)
x=s.post(LOGIN_URL, data=payload, headers=user_agent)
print(x.content)
print(x.status_code)

edited May 05 '20 at 06:47

answered May 05 '20 at 06:32

Luv

386
4
15

Still the outcome is same you can reproduce my issue by visiting https://phpzag.com/demo/ajax_login_script_with_php_jquery/index.php – Aaditya R Krishnan May 05 '20 at 06:34
But the x.content is empty. How can I get the HTML content of 'https://phpzag.com/demo/ajax_login_script_with_php_jquery/welcome.php' as output – Aaditya R Krishnan May 05 '20 at 06:47
that you have to watch the network tab and then use the session cookie as future headers to follow a redirect to the welcome page. As you said it is ajax, then the page logic is doing it but you can replicate it by looking at the network tab. Your issue was user agent. – Luv May 05 '20 at 06:48
Are you suggesting this ? `p=s.post(LOGIN_URL, data=payload, headers=user_agent) x=s.get('https://phpzag.com/demo/ajax_login_script_with_php_jquery/welcome.php',header=p.header)` – Aaditya R Krishnan May 05 '20 at 06:56
I tried `r=s.get('https://phpzag.com/demo/ajax_login_script_with_php_jquery/welcome.php',headers=x.cookies)` But it result 403 as response – Aaditya R Krishnan May 05 '20 at 08:56
you are running it as a part of session, u do not have to manipulate cookies. you could print(s.cookies.get_dict()) and set explicitly as s.get(..., cookies=s.cookies.get_dict()) – Luv May 05 '20 at 09:26
Can you go through my modifed code in the question. Now I changed as per you suggestion. But it is responded with HTML code of index.page – Aaditya R Krishnan May 05 '20 at 09:37
yes I did, that is a mystery to me. I ve modified it to send everything the browser does but it wont fetch the welcome page. Could you use any other site? Something more standard? – Luv May 05 '20 at 09:41
Yes However my requirement is ajax login using python. Can you try with this website https://makitweb.com/demo/ajax_login/index.php – Aaditya R Krishnan May 05 '20 at 09:45
In concept, you only have to login and use the cookie (session does this for you), or set an authorization header. The link you pasted looks like a single page app where the js code fetches different views. Which means the page can be a skeleton with all the logic and dom manipulation done by js library. What you ve done in your code should work on standard implementations. – Luv May 05 '20 at 09:50

aldokkani · Answer 4 · 2020-05-05T06:44:17.837

0

Take a look at Requests: Basic Authentication

import requests

requests.post(URL, auth=('user', 'pass'))

# If there are some cookies you need to send
cookies = dict(cookies_are='working')
requests.post(URL, auth=('user', 'pass'), cookies=cookies)

edited May 05 '20 at 06:44

answered May 05 '20 at 06:26

aldokkani

853
1
8
17

It's not working You can access the website through `https://phpzag.com/demo/ajax_login_script_with_php_jquery/index.php` – Aaditya R Krishnan May 05 '20 at 06:31
@EdwardArrow This error will appear in some sites if cookies are turned off and mod_security requires cookies to match session data. It is supposed to make things more secure... it ends up just annoying. Especially because web indexing crawlers like googleBot and other search engines do not use cookies, so THEY see this error instead of your site. – aldokkani May 05 '20 at 06:40
So How can overcome this? – Aaditya R Krishnan May 05 '20 at 08:11
Could you please help me with https://stackoverflow.com/questions/62044412/unable-to-perform-api-login-using-python-request-and-post-api-calls @aldokkani – Aaditya R Krishnan May 27 '20 at 18:45
I'm getting an SSL certificate verification with this method – Aaditya R Krishnan May 28 '20 at 07:59