Automate downloads from password protected website

Question

I need some help with a work project I have been assigned. At the moment we manually go to the site, logon and then download 2 excel files from a supplier's website every month. The files are then loaded into SQL.

We want to automate this process. Now the loading of the files into SQL I can do, but I am not sure how I can automate logging onto the website entering my user details and collecting the files. I mostly deal with SQL and have very little .NET experience, so any code samples would be most appreciated.

Just to confirm. The logon form is on a aspx page. just a basic form with a table containing the username & password fields, the forgotten password link and the logon button

Is it login via a form on the page, or a browser dialog box which asks for username and password? — Mikael Svenson, Apr 10 '10 at 14:55
There are many different ways to password-protect a website (HTTP Basic; login page setting a cookie, ...). What mechanism does your website use? — dtb, Apr 10 '10 at 14:56

score 4 · Accepted Answer · edited May 23 '17 at 12:30

You can either use webclient or httpwebrequest.

Login to the page with HttpWebRequest

How do you login to a webpage and retrieve its content in C#?

Httpwebrequest example:

    HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create("http://sso.bhmobile.ba/sso/login"); 
    req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705;)"; 
    req.Method = "POST";
    req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
    req.Headers.Add("Accept-Language: en-us,en;q=0.5");
    req.Headers.Add("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7");
    req.KeepAlive = true;
    req.Headers.Add("Keep-Alive: 300");
    req.Referer ="http://sso.bhmobile.ba/sso/login";

    req.ContentType = "application/x-www-form-urlencoded"; 

    String Username = "username";
    String PassWord = "Password";

    StreamWriter sw = new StreamWriter(req.GetRequestStream());
    sw.Write("application=portal&url=http%3A%2F%2Fwww.bhmobile.ba%2Fportal%2Fredirect%3Bjsessionid%3D1C568AAA1FB8B5C757CF5F68BE6ECE65%3Ftype%3Dssologin%26url%3D%2Fportal%2Fshow%3Bjsessionid%3D1C568AAA1FB8B5C757CF5F68BE6ECE65%3Fidc%3D1023278&realm=sso&userid=" + Username + "&password=" + password + "&x=16&y=11");
    sw.Close();

    HttpWebResponse response = (HttpWebResponse)req.GetResponse();


    StreamReader reader = new StreamReader(response.GetResponseStream());
    string tmp = reader.ReadToEnd();

    foreach (Cookie cook in response.Cookies)
    {
        tmp += "\n" + cook.Name + ": " + cook.Value;
    }


    Response.Write(tmp);
    Response.End();

Webclient example:

WebClient wc = new WebClient();
wc.Credentials = new NetworkCredential("username", "password");
string url = "http://foo.com";                  
try
{
        using (Stream stream = wc.OpenRead(new Uri(url)))
        {
                using (StreamReader reader = new StreamReader(stream))
            {
                return reader.ReadToEnd();
             }
        }
}
catch (WebException e)
{
        //Error handeling
}

Good answer, except I would drop the "Accept-Encoding: gzip,deflate" header as you would have to unpack the stream as well for this case. And use somemething like Fiddler to sniff how you should send the actual request. — Mikael Svenson, Apr 10 '10 at 15:11
Thank you:- it looks like webclient is what i needed. Will investigate this further — Andrew, Apr 10 '10 at 15:23
@Mikael Svenson: Good point! Yip, fiddler is great and should always be used to check what headers, form fields and hidden fields you need to be sending when trying to mimic browser requests with webclient or httpwebrequest. +1 — Bruce Adams, Apr 10 '10 at 15:25

Automate downloads from password protected website

1 Answers1