Java scrape website with login using Jsoup

Question

I'd like to print some data (where id="ctl00_cpStud_lblTotalPercentage") from here, I just need to enter the password to access the data.

Password field code:

input name="txtPassword" type="password" id="txtPassword" placeholder="Enter Password" onkeypress="checkcapslockon(event)"

Button code:

input type="submit" name="btnSubmit" value="Submit" id="btnSubmit" class="rc-button rc-button-submit"

Can anyone explain me why this code is not working ? I've tried a lot but nothing is working.

public static void main(String[] args) throws Exception {
       String url="https://erp.cbit.org.in/?__LASTFOCUS=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUKMTExMDk3MjkwOA9kFgICAQ9kFgICBQ8PFgIeB1Zpc2libGVoZGRkApSadXV8hBd7qi9M9MQf24gQFo1JDdpv3rqIRSVoR5Y%3D&__VIEWSTATEGENERATOR=C2EE9ABB&__EVENTVALIDATION=%2FwEdAAUo8HF9hHYWKGGF3Et0JGNxBjpuGLkudYNkCAonVyADt%2B5PVNfdHmla7NuBu7%2FwrMNjemWCTRgEB59HPczIGVNwgWOkgugWB5Cq9dYD7toQNEwZfb2PCk9YCZQ7UhXsjSWufILYgZp8zPh7f7XDtu2a&txtUserName=160117737089&btnNext=Next";             

       Connection.Response loginForm = Jsoup
                       .connect(url)
                       .data("txtPassword", "8055")
                       .data("btnSubmit", "Submit")
                       .method(Connection.Method.GET)
                       .followRedirects(true)
                       .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0")
                       .execute();  

       Connection.Response response = Jsoup.connect(url)
            .method(Connection.Method.POST)
            .cookies(loginForm.cookies())
            .followRedirects(true)
            .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0")
            .execute();

       System.out.println(response);
    }

but im getting below error

Exception in thread "main" org.jsoup.HttpStatusException: HTTP error fetching URL. Status=500, URL=https://erp.cbit.org.in/Login.aspx?ReturnUrl=%2f%3f__LASTFOCUS%3d%26__EVENTTARGET%3d%26__EVENTARGUMENT%3d%26__VIEWSTATE%3d%252FwEPDwUKMTExMDk3MjkwOA9kFgICAQ9kFgICBQ8PFgIeB1Zpc2libGVoZGRkApSadXV8hBd7qi9M9MQf24gQFo1JDdpv3rqIRSVoR5Y%253D%26__VIEWSTATEGENERATOR%3dC2EE9ABB%26__EVENTVALIDATION%3d%252FwEdAAUo8HF9hHYWKGGF3Et0JGNxBjpuGLkudYNkCAonVyADt%252B5PVNfdHmla7NuBu7%252FwrMNjemWCTRgEB59HPczIGVNwgWOkgugWB5Cq9dYD7toQNEwZfb2PCk9YCZQ7UhXsjSWufILYgZp8zPh7f7XDtu2a%26txtUserName%3d160117737089%26btnNext%3dNext%26txtPassword%3d8055%26btnSubmit%3dSubmit&__LASTFOCUS=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUKMTExMDk3MjkwOA9kFgICAQ9kFgICBQ8PFgIeB1Zpc2libGVoZGRkApSadXV8hBd7qi9M9MQf24gQFo1JDdpv3rqIRSVoR5Y%3D&__VIEWSTATEGENERATOR=C2EE9ABB&__EVENTVALIDATION=%2FwEdAAUo8HF9hHYWKGGF3Et0JGNxBjpuGLkudYNkCAonVyADt%2B5PVNfdHmla7NuBu7%2FwrMNjemWCTRgEB59HPczIGVNwgWOkgugWB5Cq9dYD7toQNEwZfb2PCk9YCZQ7UhXsjSWufILYgZp8zPh7f7XDtu2a&txtUserName=160117737089&btnNext=Next&txtPassword=8055&btnSubmit=Submit
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:760)
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:757)
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:705)
    at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:295)
    at jsouptest.JsoupTest.main(JsoupTest.java:44)
C:\Users\Akhil\AppData\Local\NetBeans\Cache\8.2\executor-snippets\run.xml:53: Java returned: 1
BUILD FAILED (total time: 0 seconds)

UPDATED CODE

   @Override
    protected Void doInBackground(Void... voids) {
        try {
            //Connect to the website

            String url="https://erp.cbit.org.in/?__LASTFOCUS=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUKMTExMDk3MjkwOA9kFgICAQ9kFgICBQ8PFgIeB1Zpc2libGVoZGRkApSadXV8hBd7qi9M9MQf24gQFo1JDdpv3rqIRSVoR5Y%3D&__VIEWSTATEGENERATOR=C2EE9ABB&__EVENTVALIDATION=%2FwEdAAUo8HF9hHYWKGGF3Et0JGNxBjpuGLkudYNkCAonVyADt%2B5PVNfdHmla7NuBu7%2FwrMNjemWCTRgEB59HPczIGVNwgWOkgugWB5Cq9dYD7toQNEwZfb2PCk9YCZQ7UhXsjSWufILYgZp8zPh7f7XDtu2a&txtUserName=160117737089&btnNext=Next";
            String url2="https://erp.cbit.org.in/";
            Connection.Response loginForm = Jsoup.connect(url)
                    .method(Connection.Method.GET)
                    .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0")
                    .execute();

            Document doc = loginForm.parse();
            Element e = doc.select("input[id=__VIEWSTATE]").first();
            String viewState = e.attr("value");
            e = doc.select("input[id=__EVENTVALIDATION]").first();
            String eventValidation = e.attr("value");
            e = doc.select("input[id=__LASTFOCUS]").first();
            String lastFocus = e.attr("value");
            e = doc.select("input[id=__EVENTTARGET]").first();
            String eventTarget = e.attr("value");
            e = doc.select("input[id=__VIEWSTATEGENERATOR]").first();
            String viewstateGenerator = e.attr("value");
            e = doc.select("input[id=__EVENTARGUMENT]").first();
            String eventArgument = e.attr("value");

            Document document = Jsoup.connect(url2)
                    .data("cookieexists", "false")
                    .data("txtPassword","8055")
                    .data("btnSubmit","Submit")
                    .data("__LASTFOCUS",lastFocus)
                    .data("__EVENTTARGET",eventTarget)
                    .data("__EVENTARGUMENT",eventArgument)
                    .data("__VIEWSTATEGENERATOR",viewstateGenerator)
                    .data("__VIEWSTATE", viewState)
                    .data("__EVENTVALIDATION", eventValidation)
                    .cookies(loginForm.cookies())
                    .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0")
                    .get();

            Element attendance = document.getElementById("ctl00_cpStud_lblTotalPercentage");
            String at=attendance.text();
            at=at.replaceAll("[^\\.0123456789]","");

        } catch (IOException e) {
            Toast.makeText(WaitingActivity.this, "No Internet Connection", Toast.LENGTH_LONG).show();
        }
        return null;
    }

    @Override
    protected void onPostExecute(Void aVoid) {
        super.onPostExecute(aVoid);
        Toast.makeText(WaitingActivity.this, at, Toast.LENGTH_LONG).show();
    }

The values of viewstate etc. are dynamic - they change each time you load the page. First you have to send `get` request to find these values, and only then you can send the `post` request. See an [example](https://stackoverflow.com/questions/31871801/problems-submitting-a-login-form-with-jsoup) — TDG, Oct 02 '19 at 07:21
What values show I add to the post request? Are VIEWSTATE and EVENTVALIDATION enough to get the document? Or else show I add VIEWSTATEGENERATOR as well? — Akhil Anand Sirra, Oct 02 '19 at 07:37
You must add all the values that you see in the `post` request of the browser, even the empty ones. — TDG, Oct 02 '19 at 09:24
I sent all values in post request but it still isn't printing the post document details after entering password. — Akhil Anand Sirra, Oct 02 '19 at 12:47
Actually it is giving me output in netbeans but not in android studio, Can you tell me why i'll update the code. — Akhil Anand Sirra, Oct 02 '19 at 14:04

Java scrape website with login using Jsoup

UPDATED CODE

0 Answers0