1

I am trying to access to my uni's portal(site with https) and get some data by python.

But keep getting error in final line.

Here is my code to access website and login. Could anyone can clarify what should I check and debug it?

import urllib
import ssl

opener = urllib.request.build_opener(urllib.request.HTTPSHandler(context=ssl.SSLContext(ssl.PROTOCOL_TLSv1)))
urllib.request.install_opener(opener)

loginUrl1 = 'https://www.kau.ac.kr/page/login.jsp?ppage=&target_page=act_Portal_Check.jsp@chk1-1'
reqGetCookie = urllib.request.Request(loginUrl1)
respGetCookie = urllib.request.urlopen(reqGetCookie)
cookie = (respGetCookie.headers.get("Set-Cookie")).split('; ')[0]

loginUrl2 = 'Login authenticating page'
login_info={
    'p_id' : "ID",
    'p_pwd' : "Password"
}
loginForm1 = urllib.parse.urlencode(login_info)
loginForm = urllib.parse.parse_qs(loginForm1)
loginHeader = {
    "Host" : "www.kau.ac.kr",
    "Connection" : "keep-alive",
    "Content-Length" : 138,
    "Cache-Control" : "max-age=0",
    "Accept" : "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "User-Agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
    "Content-Type" : "application/x-www-form-urlencoded",
    "Referer" : "https://www.kau.ac.kr/page/login.jsp?ppage=&target_page=act_Portal_Check.jsp@chk1-1",
    "Accept-Encoding" : "gzip, deflate",
    "Accept-Language" : "ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4",
    "Cookie" : cookie
}
reqLogin = urllib.request.Request(loginUrl2, loginForm, headers = loginHeader)
respLogin = urllib.request.urlopen(reqLogin)

and here is the error:

Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 885, in send
    self.sock.sendall(data)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 886, in sendall
    v = self.send(data[count:])
TypeError: unhashable type: 'slice'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydev_run_in_console.py", line 71, in <module>
    globals = run_file(file, None, None)
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydev_run_in_console.py", line 31, in run_file
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/Python/Excercise/Exam prep/getgrade.py", line 127, in <module>
    respLogin = urllib.request.urlopen(reqLogin)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 162, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 465, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 483, in _open
    '_open', req)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 443, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1283, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1240, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 913, in _send_output
    self.send(message_body)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 889, in send
    self.sock.sendall(d)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 886, in sendall
    v = self.send(data[count:])
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 856, in send
    return self._sslobj.write(data)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 581, in write
    return self._sslobj.write(data)
TypeError: a bytes-like object is required, not 'str'
leaving_traces
  • 87
  • 1
  • 8
  • Your code as posted would not run, also using requests would make life lot simpler for you – Padraic Cunningham Jul 03 '16 at 23:08
  • If you sort out the incorrect imports `urllib.parse.parse_qs(loginForm1)` is causing the error – Padraic Cunningham Jul 03 '16 at 23:13
  • @PadraicCunningham what do you mean it would not run? It runs well just before the last line of code. – leaving_traces Jul 03 '16 at 23:14
  • You are missing imports, anyway use `urllib.parse.urlencode(loginForm).encode("utf-8")` in place of just passing loginForm – Padraic Cunningham Jul 03 '16 at 23:15
  • @PadraicCunningham If I dont't do 'urllib.parse.parse_qs(loginForm1)' then there are 'TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str.' this error occurs. – leaving_traces Jul 03 '16 at 23:17
  • I did not say not to use it, I said to replace loginForm i the request `urllib.request.Request(loginUrl2, urllib.parse.urlencode(loginForm).encode("utf-8"), headers = loginHeader)` – Padraic Cunningham Jul 03 '16 at 23:18
  • @PadraicCunningham Thanks for the comment! I did .endoce("utf-8") then there are no more errors but program runs forever... – leaving_traces Jul 03 '16 at 23:21
  • Is there a particular reason you are doing this using a lower level api than something like requests? http://stackoverflow.com/questions/38110760/how-to-use-python-requests-to-login-to-website-store-cookie-then-access-anothe/38110852#38110852 http://docs.python-requests.org/en/master/ – Padraic Cunningham Jul 03 '16 at 23:21
  • @PadraicCunningham I know that there are the other way around, but for me requests method keep getting errors with SSL verification, thats why I am doing in this way.. – leaving_traces Jul 03 '16 at 23:25
  • As far as hanging goes remove the `"Content-Length" : 138`, if you get ssl errors then that is most likely something on your system that is not well setup – Padraic Cunningham Jul 03 '16 at 23:25
  • @PadraicCunningham If I remove Content- Length then I got `ValueError: Content-Length should be specified for iterable data of type {b'p_pwd': [b'Password'], b'p_id': [b'ID']}` – leaving_traces Jul 03 '16 at 23:27
  • But there should be no dict, you are supposed to encode to bytes after urlencode – Padraic Cunningham Jul 03 '16 at 23:38
  • @PadraicCunningham Yes finally It runs without error!! but after that I did `html = respLogin.read().decode("euc-kr") print(html)` and It prints as same as you go directly through [Login check page](https://www.kau.ac.kr/page/act_login.jsp). which means ID/pwd incorrect. But there are nothing wrong with my info. I guess my program transfer id/pwd data incorrectly.. – leaving_traces Jul 03 '16 at 23:55
  • The issue is some of the posy data is missing based on what I saw in developer tools – Padraic Cunningham Jul 04 '16 at 00:05
  • @PadraicCunningham Oh I think Login succeed with some modification. it prints `` the value `-1123262544` will transfer to next authenticating site. – leaving_traces Jul 04 '16 at 00:12
  • @PadraicCunningham Basically the login process for this site is like below 1.[Login site]:`https://www.kau.ac.kr/page/login.jsp?ppage=&target_page=act_Portal_Check.jsp@chk1-1` 2.[Loginauth iste]`https://www.kau.ac.kr/page/act_login.jsp ` 3.`https://www.kau.ac.kr/page/act_Portal_Check.jsp?chk1=1` 4.`https://portal.kau.ac.kr/portal/PortalLoginSso?seq_id=-1496701033&ppage=` (value that I mentioned) 5.[portal main page] `https://portal.kau.ac.kr/portal/MyPortal_No.jsp` Do you have any recommendation to get to main portal site?? Realy Really appreciate it! – leaving_traces Jul 04 '16 at 00:12
  • I am off the comp untill the morning, I will have a look tomorrow if you have not gotten any working answer in the meantime – Padraic Cunningham Jul 04 '16 at 00:17
  • @PadraicCunningham Yes thank you very much! I am going bed and I've spent on this problem more than a week so I can wait much longer! Thank you – leaving_traces Jul 04 '16 at 00:20
  • @PadraicCunningham Do you have any insight for this login process? I still can't get any working answer – leaving_traces Jul 04 '16 at 14:21
  • one change should be `Login_form = urllib.parse.urlencode(login_info).encode("utf-8")`, forget the parse_qs, I will have a further look in the evening – Padraic Cunningham Jul 04 '16 at 14:25
  • @PadraicCunningham Oh yes that's the thing I did it today... Thank you for helping me! – leaving_traces Jul 04 '16 at 18:10
  • How far have you gotten now? I obviously have no logon details so I cannot check for success – Padraic Cunningham Jul 04 '16 at 18:19
  • @PadraicCunningham Not much progress from last night. when run the code I get some value that I mentioned up there in Javascript and It seems like the value tranferred to seq_id. I want to talk more details with pictures and etc.. but Is there anyway that I can contact you? – leaving_traces Jul 04 '16 at 18:24
  • About the only way would be through linkedin – Padraic Cunningham Jul 04 '16 at 22:46
  • @PadraicCunningham I searched for the LinkedIn by your full name but couldn't find right one. I am Sorry but Would you like to leave your LinkedIn profile link if you don't mind? – leaving_traces Jul 06 '16 at 06:38
  • the link is on my profile – Padraic Cunningham Jul 06 '16 at 11:17

0 Answers0