Using WebClient or WebRequest to login to a website and access data

Question

I'm trying to access restricted data on a website using WebClient/WebRequest. There is no official API in that website, so what I'm trying to do is simply fill the HTML form and post the values to the server, so I'm logged in.

I tried this and this, but it doesn't look like the upcoming requests are logged in.

The latter example is much more appealing since I obviously prefer WebClient, but legacy WebRequest will do.

Anyway, in the first example I think it did login, but the upcoming requests that access the private data return a page with a message "This is member only content".

How to make a WebClient permanently logged in?

Possible duplicate of [C# Login to Website via program](http://stackoverflow.com/questions/930807/c-sharp-login-to-website-via-program) — RyBolt, Aug 18 '16 at 18:59

Shimmy Weitzhandler · Accepted Answer · 2018-04-12T02:42:30.847

26

Update:

See my comment below.

Here's what I did and it works (credit).

Add this class first:

namespace System.Net
{
  using System.Collections.Specialized;
  using System.Linq;
  using System.Text;

  public class CookieAwareWebClient : WebClient
  {
    public void Login(string loginPageAddress, NameValueCollection loginData)
    {
      CookieContainer container;

      var request = (HttpWebRequest)WebRequest.Create(loginPageAddress);

      request.Method = "POST";
      request.ContentType = "application/x-www-form-urlencoded";

      var query = string.Join("&", 
        loginData.Cast<string>().Select(key => $"{key}={loginData[key]}"));

      var buffer = Encoding.ASCII.GetBytes(query);
      request.ContentLength = buffer.Length;
      var requestStream = request.GetRequestStream();
      requestStream.Write(buffer, 0, buffer.Length);
      requestStream.Close();

      container = request.CookieContainer = new CookieContainer();

      var response = request.GetResponse();
      response.Close();
      CookieContainer = container;
    }

    public CookieAwareWebClient(CookieContainer container)
    {
      CookieContainer = container;
    }

    public CookieAwareWebClient()
      : this(new CookieContainer())
    { }

    public CookieContainer CookieContainer { get; private set; }

    protected override WebRequest GetWebRequest(Uri address)
    {
      var request = (HttpWebRequest)base.GetWebRequest(address);
      request.CookieContainer = CookieContainer;
      return request;
    }
  }
}

Usage:

public static void Main()
{
  var loginAddress = "www.mywebsite.com/login";
  var loginData = new NameValueCollection
    {
      { "username", "shimmy" },
      { "password", "mypassword" }
    };

  var client = new CookieAwareWebClient();
  client.Login(loginAddress, loginData);
}

edited Apr 12 '18 at 02:42

answered Jun 19 '13 at 06:16

Shimmy Weitzhandler

101,809
122
424
632

1

@Anthony, use System.Text.Encoding instead of Encoding if you have the problem with non-static context. – Neshta Oct 01 '14 at 08:18
4

you can't loginData.ToString(), it will just be the type name. – Kyle Gobel Jan 23 '15 at 03:43
As pointed out by Kyle, loginData.ToString() returns "System.Collections.Specialized.NameValueCollection". – Herohtar Jul 06 '17 at 06:03
1

It used to work. Anyway guys, you can use `FormUrlEncodedContent` then `ReadAsStringAsync` instead. Anyway, this answer was good for when it was posted, I'd advise using `HttpClient` instead, providing a `DelegatingHandler` that overrides `SendAsync` to login. I hope to find time to refresh the answer. – Shimmy Weitzhandler Jul 06 '17 at 15:16
4

Anyway I've edited my answer, I hope it's working now. Don't forget that instead of downvoting and getting mad, this is an open website and you can just edit my answer in by that return a favor to the community and for other people looking at this answer in the future. – Shimmy Weitzhandler Jul 06 '17 at 17:16
Is there a way to test if the login was succesfull using this method? – Dennis van Gils Oct 17 '17 at 18:35
1

@DennisvanGils you can check the response message from the server by its HTTP status code etc. – Shimmy Weitzhandler Oct 17 '17 at 22:09
2

@Shimmy That didn't work for me as the server always returned a 200 ok code, however using [this answer](https://stackoverflow.com/a/14074200/5022761) I could check if the container contained a cookie named .ASPXAUTH for the correct site, which did the job – Dennis van Gils Oct 18 '17 at 00:42
what is GetWebRequest? – Sajitha Rathnayake Apr 11 '18 at 03:30
@SajithaNilan this is an old post. Unless there is a reason you specifically want to use `WebClient` I'd advise you to use `HttpClient` instead, overriding the `SendRequest` method to achieve the Login. – Shimmy Weitzhandler Apr 12 '18 at 02:41

score 5 · Answer 2 · answered Jun 19 '13 at 06:02

HTTP is stateless. So, you cannot WebClient permanently logged in. The concept of a session does not exist in HTTP. The server-side technologies such as ASP.NET simulate a stateful behavior through the concept of session using cookie or a query string parameter that gets sent back and forth in every request. Having said that, it is possible to emulate what a browser does from WebClient. If you have access to the website, connect to it using the right credentials and capture the traffic using Fiddler. Then, make sure WebClient sends out the right cookies, request headers, query strings, etc exactly same as the browser.

Using WebClient or WebRequest to login to a website and access data

2 Answers2

Update:

Linked