0

I am trying to automate logging into my cable modems page and scrapping up some information so I can debug dropped internet connections. My Motorola modem's web page is accessed from my local network through this IP address:

https://192.168.100.1

which takes me to:

https://192.168.100.1/Login.html

The browser warns that it is not secure. Ignoring the warning and proceeding anyways gets me to my the login page for the modem. After logging in, I can see various status and log tables that I want to "scape" up and accumulate over time.

I am trying to use requests to login to the modem, then, in the same session access one of the pages. Here is my code:

# URL of my cable modem (same for all modems) - tried these combinations  
LOGIN_URL1 = "https://192.168.100.1/Login.html"  
LOGIN_URL2 = "https://192.168.100.1/cgi-bin/moto/goform/MotoLogin"
LOGIN_URL3 = "https://192.168.100.1/cgi-bin/moto/goform/MotoLogin/frmLogin"

# status pages reachable once logged in
MotoStatusConnection = "https://192.168.100.1/MotoStatusConnection.html"
MotoStatusSoftware = "https://192.168.100.1/MotoStatusSoftware.html"
MotoStatusLog ="https://192.168.100.1/MotoStatusLog.html"

# Default username and password for motorola cable modems - I've changed it :) 
payload = {
    'loginUsername': 'admin',
    'loginPassword': 'motorola',  # default password, I've changed it.
}

# Initiate a post session with login,  then try to get data from a page.
with requests.Session() as s:
    
    # Login into the page
    p = s.post(LOGIN_URL2, data=payload, verify=False)
    print(p.status_code,'\n')
    print(p.text, '\n')
    print(p.headers,'\n')
    
    print('***************************************************************************')

    # Try to get info from another page once logged in
    r = s.get(MotoStatusSoftware, verify=False)
    print(r.text)

And below images is the source of the modem's login page. I think I am not making the proper URL, and need to properly use the "form action=" and the "name=" somehow... or something else?

PS: output of first three prints is consistently:

200

{ "moto/goform/MotoLoginResponse": { "moto/goform/MotoLoginResult": "UN-AUTH" } }

{'Content-type': 'text/html', 'Strict-Transport-Security': 'max-age=60;includeSubdomains', 'X-Frame-Options': 'DENY', 'Cache-Control': 'no-store, no-cache, must-revalidate', 'Pragma': 'no-cache', 'set-cookie': 'Secure; HttpOnly', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Content-Security-Policy': "default-src 'self' 'unsafe-inline' 'unsafe-eval'", 'X-Permitted-Cross-Domain-Policies': 'none', 'Content-Length': '86', 'Date': 'Mon, 21 Mar 2022 19:13:48 GMT'}

Thanks in advance for any help!

enter image description here

enter image description here

enter image description here

Randall Goodwin
  • 1,916
  • 2
  • 18
  • 34
  • I'm fairly certain that the login page sends you a session cookie that you must retain and provide in subsequent calls. As HTTP is stateless the web server (i.e. your router) has no other way of knowing if you are the same user who makes the second call as the one who just logged in. See [this question](https://stackoverflow.com/questions/15778466/using-python-requests-sessions-cookies-and-post) and [this one](https://stackoverflow.com/questions/31554771/how-to-use-cookies-in-python-requests) for ideas. – Selcuk Mar 22 '22 at 02:38
  • 1
    Also it's worth investigating if your modem supports `telnet` or `rlogin` for diagnostics. It might be much easier to access logs that way than trying to scrape HTML. – Selcuk Mar 22 '22 at 02:40
  • Hi - thanks for the quick replies. Checked the two links and tried to prefetch a cookie, but I think my issue is upstream of that. I am not successfully able to POST the credentials and return a session. Getting the "UN-AUTH", so no cookies... For the **telnet** or **rlogin**, I googled it for my modem - a "Motorola Surfboard MB 8611" did not found much, but it may still warrant more searching, as the idea sounds attractive. Thanks again for the comments. Still looking for a complete solution :) – Randall Goodwin Mar 22 '22 at 02:57
  • Can you enable your browser's inspector, go to Network tab and observe what is being actually POSTed when you log in manually? This might give you some hints. – Selcuk Mar 22 '22 at 03:02
  • Added pictures. I am not too familiar with using them. I only saw one "POST" at the end (which seems weird - I would think it would have the first page sent?) – Randall Goodwin Mar 22 '22 at 03:28
  • Are you sure this is the correct request? It goes to `/HNAP1` although the `
    ` tag hints that it should post the payload to `/cgi-bin/moto/goform/MotoLogin`. Maybe there is an Ajax call somewhere. I would turn off Javascript and try logging in again to see if there is a plain HTML fallback.
     – Selcuk Mar 22 '22 at 04:42
  • first I would turn off JavaScript in browser and use it to access modem - to check if it works without JavaScript and if send the same HTML, headers, etc. Next I would try to behave like real user/human - first send GET for main page to get fresh cookies in session. Every connection may need new cookies which can be valid only for short time. And later send POST with form data. – furas Mar 22 '22 at 06:03
  • and you have to send all values which you have in form - I see on screen `` and they also may send some values which you also have to send to model. But if these values are generated by JavaScript then `requests`+`Beautifulsoup` is useless because it can't run JavaScript and you may need s[Selenium](https://selenium-python.readthedocs.io/) to control real browser which run JavaScript. – furas Mar 22 '22 at 06:08
  • sometimes it is simpler to use `DevTools` (tab `network`) to see all values send during login. Some server may check even if form send text from ` – furas Mar 22 '22 at 06:10

0 Answers0