I am working on my scrapping project which requires login to the page, and two questions just appeared in front of me. Ok, maybe two blocks of questions. (please have mercy with me as I am beginner learning new things and probably I already poorly designed the whole project).
The design is: as the page requires login, I wanted to use python request.Session() method object and with "with context manager" keep to be logged in.
So my first question is:
1, If I want to have, lets say main.php file, which will call python file with login function (the one with context manager), do I have to have the rest of the functions in this same python file (all the functions for the specific scraping) ? more precisely all functions within this "with block" ? What if I want to have each important function in separate python file (one for login.py, another update.py, table.py, etc)? If I call other functions than login function, how the session will be still valid and I wont get logged out ? or should I just call the functions with "s" object somehow ?
Second question:
2, It's related due the sessions. I went through some questions and not sure if I found the correct answer. But when I logged into the page, through network tab on chrome I saw some session ID. But when I checked session ID via "what_in_session = s.cookies.get_dict()",
ID was different. When I moved on the page the session id in network was still same, however, via python was different every time.
Am I doing something wrong ? Or is it correct behavior ? I found this but I am not sure: Why are the IDs different in each request?.
I guess, once the "with request.Session" block is used, I dont need to care about the rest ? Then back to the first question, how to mange these python functions ?
Hope that you'll understand the questions. (if not I can try to rephrase them)
many thanks
