How to get the headers behind a login?

Question

I am trying to create a bot to post message on a plateforme that requires login. I have done several reading online but I can't figure what the exactly the headers in a python requests stand for and how to get them ?

I have logged in manually to the website and post a message while inspecting the console network and succeed to copy the POST request as cURL than translated it to python language. Can someone explain me how the headers work and how to get them ?

The headers I copied from chrome console:

headers = {
    'cookie': 'JSESSIONID=523C2995451CCFC988FA097A85632B9F; csc=dom-use-prodwebapp-179.use.dom.carezen.net1560960000972; vc=9370b249-7a51-4291-a110-02332dffb2ed; n_vis=dom-use-prodwebapp-179.use.dom.carezen.net1560960000972; ROUTE2=d; _ga=GA1.2.1838448005.1560965623; _gid=GA1.2.1576009381.1560965623; mt.v=2.731501753.1560965623252; cto_lwid=44716745-0481-4223-9806-1c8201aa7991; mc_verify=EhAKEmJAM*EZDfeHJUUii1Eux5dvDMmMg3SVX4ULbB0Wo9KwYWs0EFCC67pzqF0bbSgcUFvmuwI11qh3FviCXA..; intl_src=en-us; mt.SFT-MT=1; __ssid=aae0f1f68f4613a358d86a8f63a09de; _RCRTX03=c43b78bb8e3a11e9b59929e665754f78129a93e4862141c5980f155f5ed6f58b; eps="my account login"; _derived_epik=dj0yJnU9dGV3b3FwcER6amNITHQ4QUZiT1J4R0R2clg4MXZRQ18mbj0wVVZBVnZQU3dNS1lpY2JhYnBBV0N3Jm09NyZ0PUFBQUFBRjBMdjhr; mc=EhAKEmJAM%2BEZDfeHJUUii1Eux5dvDMmMg3SVX4ULbB0Wo9KwYWs0EFCC67pzqF0bbSgcUFvmuwI11qh3FviCXA%3D%3D; acs=L*UOHh2C8Xu*9PW02_*Ce6Oew*wRMnFB*b*Qsz7HT9Y.jx4xikhj-4e1e1h; ac=cqPM063k4O*PXdlekY6XiyEVjlI*XVG6roI0_mAVmyg.jx4xikhj-4e1e1h; _sp_id.6a17=bf7adf7e-f56d-4650-bbea-4b53734ea73a.1560965625.8.1561051105.1561010531.241ccbe8-ce5e-4bcf-9b3f-141aabf54e07; sc=R*5vUnwoNgwLU_TVziElnx8JKylwKx5omqzog5VpI9I.; n_tc=7416%7C2416%7C7467%7C7440%7C2127%7C2420%7C1871%7C2048%7C7300%7C7327%7C7375%7C7350%7C7261%7C1831%7C1311; lad="MTU2MTE0NTAzNTUyOQ=="; mt.c-lbx=20; bm_mi=F090A5625A4531D6A7A7BB9138A5C9D4~pdxKboVGxB7WtFWaP+KaJUI1bIO+hVzIOI6nj2028m79PR2Who3cXsYnvNJorc+JQ34bfEBpHtFlwWwDOfyaumh2UGIySCUgyBdw5hBDPArAHcnFCFp5cUDEzqlkAfhapH9eFeRxibJptHA3W5aYc3eNV/3tuOF1FXHkggdo2q+qCxPDTf0Gqeda9uIHN9N0TEABOokZ6+Jrx8hcdXb532A/ZA+bg/+279nh8M2gyGeocabQPdA8dxDZYYl4ASUcE3am4m9N5lEYPwS0knreOeVNkgD0lgcAeV7JapnNomVgOuSaqAnvzHtS3dJXQDfC; JSESSIONID=9D4B710D8379B4937BD45A70627E5298; utag_main=v_id:016b70cd30210012bce5d51584b403079001507100838$_sn:12$_ss:1$_st:1561158651946$ses_id:1561156851946%3Bexp-session$_pn:1%3Bexp-session; frsc=GDyIv*zSQoZ2l6K1jH9Xx1sPNSV3iBJvNL8amV_WBH5sVhc*Q1JpMMRXyfEI0alR-ZFKUaOP26pEkcsUJT0o7NA..; AKA_A2=A; _gat=1; RT=""; amplitude_id_49ee77491a2b775cf48ecf4a55ce04c5care.com=eyJkZXZpY2VJZCI6IjEwZmZlNzk5LWY1NmItNDk4Ny04ZjM2LWIwOGQzYmEzZTYyNFIiLCJ1c2VySWQiOiI0MTM1MjIyOCIsIm9wdE91dCI6ZmFsc2UsInNlc3Npb25JZCI6MTU2MTE1Njg1MTU5MSwibGFzdEV2ZW50VGltZSI6MTU2MTE1OTI2NzQyNywiZXZlbnRJZCI6MjM5LCJpZGVudGlmeUlkIjo3Mywic2VxdWVuY2VOdW1iZXIiOjMxMn0=; ak_bmsc=2AFD7877A475A73646F14141938506F417D70A16856A00009A530D5DEEDD997A~plIddhcnR68Lv1XFm6VH5v+6Vi9hZSBjDY4l4AezuMSv5OC7k9RFzk+DI6ZhTMFpDp6CeyyW/uQhL2pmxZewAmktRfdMBvwfNyiNid/q5USAfuuW+rEBBA2w1VPtTXP3yCptKtS1uij3dnMb3Ua//MwDCbCZXCbGa0bi9xKhn4C2aGldt72vKtalrUzYXB6AvdvfcEtuoll7bZlfojQ5TcZcsq+BqsFULGshbI++j8ShCHMquc3rNXu3OJRHkU4adHJkT8HSv6INdESgDmMMsbTQ==',
    'owasptoken': 'WJUJ-IN87-VT3V-Z4BA-BXXG-VWWJ-9WM2-D44J',
    'origin': 'https://www. example.com',
    'x-care.com-apikey': 'cmtux62opFFZ8Aov8J0aoJ1zRonczZyqP60pTTURdfIx',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    'x-requested-with': 'XMLHttpRequest',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36',
    'accept-encoding': 'gzip, deflate, br',
    'x-care.com-visitid': '9370b249-7a51-4291-a110-02332dffb2ed',
    'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'accept': 'application/json, text/javascript, */*; q=0.01',
    'authority': 'www.example.com',
    'x-care.com-os': 'Desktop',
}

What status code and error message are you getting back from the platform? — Wytamma Wirth, Jun 24 '19 at 00:42
Can you state more clearly what you have now, and what you need? The current wordings sound like "I have the headers, but I don't have the headers". — Imperishable Night, Jun 24 '19 at 00:43
I succeeded to copy manually the header I will update it in my answer. — Solal, Jun 24 '19 at 00:45
I have updated my question. I have a lot of information in the header : how to get the cookies and the token with a python script ? — Solal, Jun 24 '19 at 00:48
I have seen this post for cookies but the lib is not for python 3 https://stackoverflow.com/questions/5606083/how-to-set-and-retrieve-cookie-in-http-header-in-python — Solal, Jun 24 '19 at 00:50

Ivan Vinogradov · Answer 1 · 2019-06-25T09:54:14.607

how to get the cookies and the token with a python script?

Given no details about your target site, I can share some general information about making HTTP requests in python.

You should spend some time to research and find necessary information by yourself, because the way this information is stored heavily depends on your target site.

cookie is a special header. Server sets a cookie for client by sending Set-Cookie header in its response headers. Browser stores all these cookies for this particular domain in form of key-value pairs, which you can obviously observe in your browser's settings. When you perform subsequent HTTP requests (auth for example), browser sends all cookies for this domain in cookie header.
owasptoken and all others are just ordinary HTTP headers. Their respective values can be either hardcoded in js code of the page, or received via ajax requests performed by js. Things like user-agent are provided by your browser. Again, this is a subject for a research.

So to obtain cookies with requests lib you should probably call a GET on your target page and save a cookie from response:

import requests  # pip3 install requests

url = "http://www......."  # probably login page
r = requests.get(url, headers={"User-Agent": "Mozilla/5.0......."})
if r.status_code == 200:
    print(r.cookies)
else:  # request failed
    print(r.status_code, r.text)

It's easier to do that in a single requests session since it will save its state between HTTP requests and send cookies automatically like your browser.

thank you @Ivan for your comment? Just one type: it is `r.cookies` I succeed to get the cookies how can I get the `owasptoken` after login ? — Solal, Jun 24 '19 at 18:48
If you need to get `owasptoken` **after** login, you can look for it in `r.headers` after you made a request. — Ivan Vinogradov, Jun 24 '19 at 20:12

How to get the headers behind a login?

1 Answers1