HtmlAgilityPack Post Login

Question

I'm trying to login to a site using HtmlAgilityPack (site:http://html-agility-pack.net).

Now, I can't exactly figure out how to go about this.

I've tried setting the Html form values via

m_HtmlDoc.DocumentNode.SelectSingleNode("//input[@name='EMAIL']").SetAttributeValue("value", "myemail.com");

I then submit the form with

m_HtmlWeb.Load("http://example.com/", "POST");

This isn't working though. It's not logging in or anything. Does anyone else have any other insight?

Thank you

There is a nice an easy way described by Rohit Agarwal and his [BrowserSession](https://stackoverflow.com/a/11286183/1424439) class. With the fix I mentioned [here](https://stackoverflow.com/a/11286183/1424439) it just works fine, as long as cookies are used as session-identifiers. — funkypopcorn, Jul 01 '12 at 23:07
Look at my solution here https://stackoverflow.com/questions/13568933/login-to-website-using-htmlagilitypack/45868450#45868450 — eran otzap, Aug 24 '17 at 18:20

score 20 · Accepted Answer · edited Mar 20 '14 at 05:47

The HTML Agility Pack is used to parse HTML - you cannot use it to submit forms. Your first line of code changes the parsed nodes in memory. The second line does not post the page to the server - it loads the DOM again, but using the POST method instead of the default GET.

It doesn't look like you need to parse the page at all at this point, since you already know the name of the control. Use the HttpWebRequest class to send a post request to the server, with the string email=acb#example.com in the request.

Here's a sample I wrote when I needed something similar:

/// <summary>
/// Append a url parameter to a string builder, url-encodes the value
/// </summary>
/// <param name="sb"></param>
/// <param name="name"></param>
/// <param name="value"></param>
protected void AppendParameter(StringBuilder sb, string name, string value)
{
    string encodedValue = HttpUtility.UrlEncode(value);
    sb.AppendFormat("{0}={1}&", name, encodedValue);
}

private void SendDataToService()
{
    StringBuilder sb = new StringBuilder();
    AppendParameter(sb, "email", "hello@example.com");

    byte[] byteArray = Encoding.UTF8.GetBytes(sb.ToString());

    string url = "http://example.com/"; //or: check where the form goes

    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
    request.Method = "POST";
    request.ContentType = "application/x-www-form-urlencoded";
    //request.Credentials = CredentialCache.DefaultNetworkCredentials; // ??

    using (Stream requestStream = request.GetRequestStream())
    {
        requestStream.Write(byteArray, 0, byteArray.Length);
    }

    HttpWebResponse response = (HttpWebResponse)request.GetResponse();

    // do something with response
}

Also if you are going to download and parse pages beyond the login page, you will likely have to pass the received cookies from the login response in you subsequent requests. See [this question+answer](http://stackoverflow.com/questions/571964/automatic-cookie-handling-c-net-httpwebrequesthttpwebresponse) for more info. — Anders Fjeldstad, Jun 30 '11 at 09:58
@Anders - Good tip! I didn't even know it existed... Thanks! — Kobi, Jun 30 '11 at 10:03
my scenerio is different like Cem answer but not using core need using c# Login url: "www.url.com/login" and download url is : "www.url.com/data/3" but how to achieve from your answer. @Kobi — coder rock, Nov 15 '22 at 14:55

score 4 · Answer 2 · answered Mar 08 '16 at 12:00

If you want to do this with Html Agility Pack. Here's the code.

CookieCollection Cookies = new CookieCollection();
            var web = new HtmlWeb();
            web.OverrideEncoding = Encoding.Default;
            web.UseCookies = true;
            web.PreRequest += (request) =>
            {
                if (request.Method == "POST")
                {
                    string payload = request.Address.Query;
                    byte[] buff = Encoding.UTF8.GetBytes(payload.ToCharArray());
                    request.ContentLength = buff.Length;
                    request.ContentType = "application/x-www-form-urlencoded";
                    System.IO.Stream reqStream = request.GetRequestStream();
                    reqStream.Write(buff, 0, buff.Length);
                }

                request.CookieContainer.Add(Cookies);

                return true;
            };

            web.PostResponse += (request, response) =>
            {
                if (request.CookieContainer.Count > 0 || response.Cookies.Count > 0)
                {
                    Cookies.Add(response.Cookies);
                }
            };

            string baseUrl = "Your Website URL";
            string urlToHit = baseUrl + "?QueryString with Login Credentials";
            HtmlDocument doc = web.Load(urlToHit, "POST");

I don't see how this is an answer. You're assuming a UserName/Password Param exists, which for most security conscious web devs they don't. — maplemale, Feb 23 '18 at 22:38
I agree with @maplemale! User credentials on "query string" is bad idea(even encrypted). This is not an generic solution! — Igor, Mar 27 '20 at 04:46

score 2 · Answer 3 · edited May 24 '17 at 14:56

I spent some hours about this topic and found a very simple solution actually.

I have :

.net core 1.1.2

HttmlAgilityPack 1.4.9.5

login urlLogin: "www.url.com/login".

url for urlData: "www.url.com/data/3" => to get this you should be connected.

Here what I did and it just did work:

HttpClient hc = new HttpClient();

HttpResponseMessage resultLogin = await hc.PostAsync(urlLogin, new StringContent("login=myUserName&password=myPaswordValue", Encoding.UTF8, "application/x-www-form-urlencoded"));

HttpResponseMessage resultPlaylist = await hc.GetAsync(urlData);

Stream stream = await resultPlaylist.Content.ReadAsStreamAsync();

HtmlDocument doc = new HtmlDocument();

doc.Load(stream);

string webContent = doc.DocumentNode.InnerHtml;  => it works

I think It needs to login first your HttpClient then you can send the request you want.

Enjoy

HtmlAgilityPack Post Login

3 Answers3

Linked