Using HttpWebRequest to login to a website

Question

I know there are many similar question here but I tried almost everything with no luck.

Basically, I've never used the POST method before, and I'm struggling to login to any website and maintain the cookies in order to use in next requests.

Here's my code (I'm using Facebook for testing only):

Login method:

protected static CookieContainer Login()
{
    string userName = "username";
    string password = "password";

    ASCIIEncoding encoding = new ASCIIEncoding();
    string postData = "email=" + userName + "&pass=" + password;
    byte[] postDataBytes = encoding.GetBytes(postData);

    HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("https://www.facebook.com/login.php");

    httpWebRequest.Method = "POST";
    httpWebRequest.ContentType = "application/x-www-form-urlencoded";
    httpWebRequest.ContentLength = postDataBytes.Length;
    httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36";
    httpWebRequest.Proxy = null;
    httpWebRequest.AllowAutoRedirect = false;

    using (var stream = httpWebRequest.GetRequestStream())
    {
        stream.Write(postDataBytes, 0, postDataBytes.Length);
        stream.Close();
    }

    var cookieContainer = new CookieContainer();

    using (var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse())
    {
        using (var streamReader = new StreamReader(httpWebResponse.GetResponseStream()))
        {
            foreach (Cookie cookie in httpWebResponse.Cookies)
            {
                cookieContainer.Add(cookie);
            }
        }
    }

    return cookieContainer;
}

Retrieve the homepage after logging in:

void Retrieve()
{
    var req =(HttpWebRequest)WebRequest.Create("https://www.facebook.com");
    req.Proxy = null;
    req.KeepAlive = true;
    req.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36";
    req.CookieContainer = Login();
    string src = "";
    using (var res = (HttpWebResponse)req.GetResponse())
    {
        src = new StreamReader(res.GetResponseStream()).ReadToEnd();
    }
    txt.Text = src;
    wb.DocumentText = src;
}

I also tried to create the CookieContainer first and then assign it to both requests, but still no luck.
I would appreciate if someone can tell me what I'm missing.

Note: As mentioned, I'm using Facebook here just for testing. If required I can post the original website I'm trying to login to.

---

Update:

The actual website I'm trying to automate the login process for is: carrierpoint.com.

I have to say I don't really have much experience with web requests/responses, authentication, etc. I never worked with ASP.NET or any kind of web application, so please excuse my limited knowledge in this area.

So for more info, this is an example of request & response headers for a successful login to this website:

If this isn't just about cookies, I would appreciate if someone can help me figure out what other elements would be needed to submit the correct request and maintain the logged in session.

Also would be great if anyone can suggest some resources as a good start for me to generally understand the correct way(s) to use the POST method and website login & authentication (I mean the general concept to be able to use on other websites).

Answer 1

I can't see any immediately differing methods to the way you're doing it. The login isn't quite so straight forward for us though because we don't have credentials to do the proper testing and quite honestly we can't simply brute force this. I would note two things though:

You're encoding as ASCII. You should encode your data as UTF8.

System.Text.Encoding.UTF8.GetBytes.

You need to url encode your data.

postdata = "loginId=" & WebUtility.UrlEncode("myloginid") & "&password=" & WebUtility.UrlEncode("mypassword")

You may need to Import System.Web.

I would honestly verify Carrierpoint doesn't have some kind of developers API for you. A lot of big businesses have some sort of API they were forced to develop for their operations.

You don't really need to learn HttpWebRequest but for practice I would try using PayPal or Stripe's API's. They're both REST API's for doing CRUD operations with respective accounts in the world of online payments. They're very easy to use and they both use forms of OAuth authentication, and Basic Authentication.

In the case of PayPal you would send a basic authorization request to retrieve an OAuth token:

Dim _req As HttpWebRequest = HttpWebRequest.Create(Base_URL & "/api/v1/token")
_req.Method = "POST"
_req.Headers.Add("Authorization", "Basic " & Convert.ToBase64String(System.Text.Encoding.Default.GetBytes(ClientID & ":" & SecretKey))) 'Convert both credentials into a base 64 string
_req.AuthenticationLevel = System.Net.Security.AuthenticationLevel.MutualAuthRequested 'Sends authorization header immediately on first request

In these scenarios you'll see response strings that are often times in JSON format:

{
"scope": "https://api.paypal.com/v1/payments/.* https://api.paypal.com/v1/vault/credit-card https://api.paypal.com/v1/vault/credit-card/.*",
"access_token": "EEwJ6tF9x5WCIZDYzyZGaz6Khbw7raYRIBV_WxVvgmsG",
"token_type": "Bearer",
"app_id": "APP-6XR95014BA15863X",
"expires_in": 28800
}

After you got your token by parsing that JSON response you would then send it with any request you needed to make to the api. In the example of using PayPal's Payments API for retrieving all payments in a list

Dim _req As HttpWebRequest = HttpWebRequest.Create(Base_URL & "/v1/payments/payment")
_req.Method = Method.ToUpper
_req.Headers.Add("Authorization", "Bearer " & Token) 'The access_token from the previous authentication request
_req.Headers.Add("Accept_Language", "en_US")
_req.Accept = "application/json"

Retrieving responses is always the same as you have done. The only thing you will have to do is parse the data and make sure you know what format the data is in. If you need to parse JSON then use the Newtonsoft NugetPackage. It's very easy to us as well and has a load of documentation. RestSharp is another Nuget Package which contains dozens of helper methods to perform RESTful requests for cases like PayPal and Stripe.

Resources:

https://www.nuget.org/packages/RestSharp/

https://www.nuget.org/packages/newtonsoft.json/

https://developer.yahoo.com/dotnet/howto-rest_cs.html

If you need anymore help do not be afraid to ask.

Answer 2

You should set cookie container for login request. Besides, the code handling the response can be simplified, so the remainder of Login methods looks like:

...
httpWebRequest.CookieContainer = new CookieContainer();

using (var stream = httpWebRequest.GetRequestStream())
{
    stream.Write(postDataBytes, 0, postDataBytes.Length);
    stream.Close();
}

using (var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse())
{
    var cookieContainer = new CookieContainer();
    cookieContainer.Add(httpWebResponse.Cookies);
    return cookieContainer;
}

Answer 3

HttpWebRequest is tricky, and not well suited for web IMO.. You may want to write your helpers, or use nugets.

I can succesfully authenticate to facebook with the code bellow (based on this)

class Program
{
    static void Main(string[] args)
    {
        // Main logic: Login && GetHomePage
        var client = new FacebookClient();
        if (client.Login("userMail", "userPassword"))
            client.GetHomePage();
    }
}

// Specific client
public class FacebookClient : CookieAwareWebClient
{
    public bool Login(string email, string password)
    {
        var loginResult = this.Login("https://www.facebook.com/login.php",
        new NameValueCollection
        {
          { "email", email },
          { "pass", password }
        });

        return loginResult;
    }

    public void GetHomePage()
    {
        // Here's the magic.. Cookies are injected via an overriden
        var webRequest = this.GetWebRequest(new Uri("https://www.facebook.com"));

        string src = "";
        using (var webResponse = (HttpWebResponse)webRequest.GetResponse())
        {
            src = new StreamReader(webResponse.GetResponseStream()).ReadToEnd();
        }
    }
}

Here's the code from linked post, i just added UserAgent string..

// General client
public class CookieAwareWebClient : WebClient
{
    public CookieContainer CookieContainer { get; private set; }

    public CookieAwareWebClient()
      : this(new CookieContainer())
    { }

    public CookieAwareWebClient(CookieContainer container)
    {
        CookieContainer = container;
    }


    public bool Login(string loginPageAddress, NameValueCollection loginData)
    {
        var request = (HttpWebRequest)WebRequest.Create(loginPageAddress);

        request.Method = "POST";
        request.ContentType = "application/x-www-form-urlencoded";
        request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36";

        var parameters = new StringBuilder();
        foreach (string key in loginData.Keys)
        {
            parameters.AppendFormat("{0}={1}&",
                HttpUtility.UrlEncode(key),
                HttpUtility.UrlEncode(loginData[key]));
        }
        parameters.Length -= 1;

        var buffer = Encoding.ASCII.GetBytes(parameters.ToString());
        request.ContentLength = buffer.Length;

        var requestStream = request.GetRequestStream();
        requestStream.Write(buffer, 0, buffer.Length);
        requestStream.Close();

        request.CookieContainer = new CookieContainer();

        var response = (HttpWebResponse)request.GetResponse();
        response.Close();

        CookieContainer = request.CookieContainer;
        return response.StatusCode == HttpStatusCode.OK;                
    }

    // Add cookies to WebRequest
    protected override WebRequest GetWebRequest(Uri address)
    {
        var request = (HttpWebRequest)base.GetWebRequest(address);
        request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36";
        request.CookieContainer = CookieContainer;
        return request;
    }
}

Hope this helps!