Objective
I am trying to get information from this URL: https://connect.echobotsales.de/#/company/LoD2z50vDE It's a Java-script-enabled website. What I want to do is login into the website and get all the info using python-requests. I want to model the flow of the site in python. The site first goes to the main URL and then calls this "https://connect.echobotsales.de/backend/connect/models.js" to get the info. I just don't want to use selenium for this purpose cause it will take an eternity to get around 100,000 Infos. The site hides some information and is available only when you login. So even if I don't login it should give me some info.
Attempt
Let me know where I am wrong. If you will go to the website and inspect the requests it's sending then you will see that apart from the main URL it's sending a POST request to this URL "https://connect.echobotsales.de/backend/connect/models.js" and you can clearly see that it has all the information. I tried to replicate the same in python. Here is the code
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36',
'x-echo-context': 'salesconnect','origin':'https://connect.echobotsales.de/',
'referer':'https://connect.echobotsales.de/',
'content-type': 'text/plain',
'pragma': 'no-cache',
'referer': 'https://connect.echobotsales.de/',
'sec-ch-ua': """"Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92""",
'sec-ch-ua-mobile': '?0',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'same-origin',
'accept': 'application/json, text/plain, */*',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9,hi-IN;q=0.8,hi;q=0.7,ur-IN;q=0.6,ur;q=0.5,ko-KR;q=0.4,ko;q=0.3',
'cache-control': 'no-cache',
'content-length': '21',
'content-type': 'application/json;charset=UTF-8'}
action2 = {'action':'initCsrf'}
s = requests.Session()
res = s.get('https://connect.echobotsales.de/#/company/LoD2z50vDE')
#print(res.content)
action = {'action':'method','payload':payload}
response = s.post('https://connect.echobotsales.de/backend/connect/models.js',data = str(action2),headers=headers)
print(response.content)
Returns the following:
b'\n<!DOCTYPE html>\n<html>\n <head>\n <title>Echobot TARGET/CONNECT</title>\n <base href="/">\n <meta charset="utf-8">\n <link rel="stylesheet" type="text/css" href="/css/animate.css">\n <link rel="stylesheet" type="text/css" href="/css/bootstrap.css">\n <link rel="stylesheet" type="text/css" href="/css/themes/echobot/style.css">\n <link href="https://fonts.googleapis.com/css?family=Roboto:400,300,300italic,400italic,700,700italic&subset=latin,latin-ext" rel="stylesheet" type="text/css">\n <title></title>\n </head>\n <body class="echobot">\n <div class="blank-container-fluid">\n <div class="row blank-container-fluid-inner fluid-inner-unsubscribe">\n <div id="logo" class="col-xs-4">\n <div class="logoimage"></div>\n </div>\n <div class="logotext col-xs-6">\n <div id="report-text"><span>Es ist ein Fehler aufgetreten! Bitte versuchen Sie es sp\xc3\xa4ter erneut!</span></div> </div>\n </div>\n </div>\n </body>\n</html>\n\n'
[Finished in 1.9s]
I have also tried passing data in the request.post() and passing username and password too.
What I want this code to do is return a 'csrf-token' in its response header just as the site does and then use it to make a second request to the site to get all the info. I just want to replicate the flow of the site. But so far the code is not working and returns an error message "Try again later".
Question Is there any way to login in just using the python-requests module? or to get the info without selenium? Any workaround solution will also be accepted.
