I want to extract data from HTML of a website that needs to be logged in. I am searhing the web for 4 days with no clear results.
I need an answer for Asp.net/C#.
The information I collected so far:
- I need to use HttpWebRequest and HttpWebResponse
- I have to work with cookies
- Website is using PHP, JS/AJAX
- postData is like "email=???&password=???&login_now=yes&login_submit=";
- Login button redirects to "http://www.wsName.com/ajax/login.ajax.php"
Login Page Source Code:
<form method="post" action="/" id="login_form" onsubmit="return ajax_submit('#login_form',post_login)">
<table class="login_table" cellspacing="0" cellpadding="0">
<tr>
<td><label for="email">Email</label></td>
<td><label for="password">Password</label></td>
<td></td>
</tr>
<tr>
<td>
<div class='sign_up_error' id="sign_up_error" style='margin-top:0;margin-left:-207px;'></div>
<div class='light_input' style='width:150px'>
<input type="text" name="email" id="email" class="login_field" style='width:142px' tabindex="1" />
</div>
</td>
<td>
<div class='light_input' style='width:150px'>
<input type="password" name="password" id="password" class="login_field" style='width:142px' tabindex="2" />
</div>
</td>
<td>
<span class='button' id="login_button" onclick="ajax_submit('#login_form',post_login);" tabindex="3"><span class='button_border'>Log In <img src='/pics/cf_mini_arrow_right.png'></span></span>
<input type="submit" class="" style="width:0px; height: 0px; overflow:hidden; border: none;" name="submit_login"/>
</td>
</tr>
<tr>
<td><input type="checkbox" name="remember" id="remember" value="1" tabindex="4"/><label for="remember">Remember me</label></td>
<td class='right' style='padding-right:5px;'><a class='weight_normal' href="/forgot-password/">Forgot password?</a></td>
<input type="hidden" name="login_now" value="yes" />
<td><input type='hidden' name="login_submit" id="login_submit" /></td>
</tr>
</table>
</form>
Request Header is like (wsName: WebSite Name):
- Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8
- Accept-Encoding:gzip, deflate
- Accept-Language:tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4
- Cache-Control:max-age=0
- Connection:keep-alive
- Content-Length:59
- Content-Type:application/x-www-form-urlencoded
- Host: wsName.com
- Upgrade-Insecure-Requests:1
- User-Agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
Cookie of Request Header
wsName[css]=medium_text;
cc_cookie_accept=cc_cookie_accept;
- wsName[cid]=f202a09db0f7411532bdc54bb920cca0;
- __gads=ID=7d43a0ff7e0df8dd:T=1481287600:S=ALNI_MbVjhGdrclkVN4uyx0r1R52luOM5g;
- __qca=P0-234787652-1483630026242;
- wsName[fbtst]=fiss;
- wsName[fb_user]=true;
- PHPSESSID=reiqad6gvibjfhoh608iibdis0;
- __utmt=1;
- __utma=114888642.270030171.1481287559.1487259806.1487581174.25;
- __utmb=114888642.7.10.1487581174;
- __utmc=114888642;
- __utmz=114888642.1481287559.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);
- wsName[ajax_info]=2c56dde8b1c491e92421d6feaa790157%3Bed2fc2863c7e88bcb0fc986a4a7afe49
My Code (which doesnt work and taken from here) is like:
string loginURL = "http://www.*wsName*.com/ajax/login.ajax.php";
string loginURL2 = "http://www.*wsName*.com";
string formDataStr = "email=???&password=???&login_now=yes&login_submit=";
//First request: Get the cookies
CookieCollection cookies = new CookieCollection();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(loginURL);
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
//Get the response from the server and save the cookies from the first request..
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
cookies = response.Cookies;
//Second request: POST the form data and recover the cookies from the first request..
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(loginURL);
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies); //recover cookies First request
getRequest.Method = WebRequestMethods.Http.Post;
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36";
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version11;
//getRequest.Referer = "http://www.*wsName*.com/index.php";
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
byte[] byteArray = Encoding.ASCII.GetBytes(formDataStr);
getRequest.ContentLength = byteArray.Length;
Stream newStream = getRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close();
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
string sourceCode = sr.ReadToEnd();
}
I used both links (loginURL, loginURL2), modified and tried many things but at the end i have a very big zero. This is the first time i needed to work with HttpWebRequest and HttpWebResponse. So i think i miss something here.
Please help, thanks in advance.
