What's the easiest way to webscrape behind login page and auto-fill-in forms to get information?

Question

I have access to a manufacturer website.

Once I log in, I can manually enter SKUs into a form, enter a quantity for each sku, and then on submit a new page will load showing YES or NO for each item-quantity.

Since the manufacturer is rather bad at customer service and doesn't provide me with another way to get inventory information, I would like to have a script of some kind log in and check stock quantities on items.

What would be the best way to go about doing this?

Could I have wget do it somehow? It would need access to a list of SKUs to check and also would need to follow an algorithm for checking quantities.

The algorithm could check for qty 10 first. If Yes, double to 20 and check. If no, halve to 10. Etc.

Could I implement such an algorithm with wget?

Or would writing a script to do this be easier in PHP or some other language?

Thanks

EDIT: More info

I will have to handle a COOKIE, plus do the stuff below:

1.) The script needs to log in

Here is the login form:

Note that all the tags are improperly formed (should end with "/>")

<form action="logon.asp"  method="post" name="FmDefault" id="CkIn">    
<table width="90%" border="0" cellpadding="0" cellspacing="0">
<tr>    
   <td colspan="2">
       <div align="right"><img src="images/some_image.gif" width="85" height="20">

         <input name="u_id" type="text" size="15" maxlength="10">

       </div>
   </td>
</tr>
<tr>
    <td colspan="2">
       <div align="right"><img src="images/some_other_image.gif" width="85" height="20">

        <input name="ps_word" type=password size="15" maxlength="10">

       </div>
     </td>
 </tr>
 <tr> 
    <td colspan="2">
       <div align="right">

         <input alt="Login" type="image" src="images/button_image.gif" width="57" height="25" border="0">
        </div>
     </td>
 </tr>
 <tr>
 <td><font color='#ff0000'>

<!---    
<b>ATTENTION: <br>    
</b></font><b><span class="style1"><a href="http://www.example.com/holiday2009.asp" target="_blank" style="color:#00FF00;">CLICK HERE FOR THE <br>    
EXAMPLE 2008 HOLIDAY SCHEDULE!</a></span></b>    
-->    
</td>    
</tr>
</table>
</form>

2.) The script needs to enter item numbers (let's say taken from PHP array $items = array('ITEM1','ITEM2', ...) ) into the 6 "item" inputs (name="item_1", item_2, etc). The qty_1, qty_2 inputs should already be preloaded with default value="1".

The script needs to submit this form and get the results on the next page

Here is the inventory checking form: Note again that all the tags are improperly formed (should end with "/>" instead of closing ... maybe this is old html standard)

<Form action="item-display.asp" method="post" name=Fm>

    <table width="389">

      <tr align=center> 

            <th width=267>Item Number</th>

            <th width=112>Requested<br>Quantity</th>

      </tr>

      <tr align=center> 

            <td><input size=25 type=text name=item_1></input></td>

            <td><input size=5 type=text name=qty_1 value=1></input></td>

      </tr>

      <tr align=center> 

            <td><input size=25 type=text name=item_2></input></td>

            <td><input size=5 type=text name=qty_2 value=1></input></td>

      </tr>

      <tr align=center> 

            <td><input size=25 type=text name=item_3></input></td>

            <td><input size=5 type=text name=qty_3 value=1></input></td>

      </tr>      

      <tr align=center> 

            <td><input size=25 type=text name=item_4></input></td>

            <td><input size=5 type=text name=qty_4 value=1></input></td>

      </tr>      

      <tr align=center> 

            <td><input size=25 type=text name=item_5></input></td>

            <td><input size=5 type=text name=qty_5 value=1></input></td>

      </tr>

      <tr align=center> 

            <td><input size=25 type=text name=item_6></input></td>

            <td><input size=5 type=text name=qty_6 value=1></input></td>

      </tr>

    </table>

  <br>

    <input type=submit value=Inquiry id=S name=S style="font-size:12">

    <input type=reset value=Clear id=R name=R style="font-size:12">

<br>



<input type="Button" value="Export Discontinued Zero Inventory Items" name="discontinued" style="font-size:12" onclick="downloadMe('discontinued_zero_inv.csv');">

<input type="Button" value="Export Discontinued with Inventory Items" name="discontinuedwith" style="font-size:12"  onclick="downloadMe('discontinued_with_inv.csv');">

</Form>

3.) After submitting the above form, the next page will show results and I need to scrape them. How do I make sure I get data from the correct into the right part of my array?

I only need the data from the Available, Date, and Status fields of the HTML table to match the Item Numbers in my array.

Let's say my result array is:

$items = array(
  0 => array(
    'item_number' => 'ITEM1',
    'available' =>' '50+',
    'date' => 'No Date Available',
    'status' => 'Fresh'),
  1 => array(... etc),
  etc...
)

Here is the result page table:

<table>
<tr align=center>
    <th width=120>Item Number</th>
    <th width=350>Item Description</th>
    <th width=40>Qty</th>
    <th width=100>Available&nbsp;
    <img src="/images/tooltip.jpg" width="10" height="10" border="0" onMouseOver="this.T_TITLE='AVAILABLE'; return escape('Availability of product is not guaranteed.')"></th>
    <th width=100>ETA Date&nbsp;
    <img src="/images/tooltip.jpg" width="10" height="10" border="0" onMouseOver="this.T_TITLE='ETA'; return escape('ETA is approximate.')"></th>
    <th width=80>Status</th>
</tr>
    <tr align="center">
        <td width=120>ITEM1</td>
        <td width=350>Large version in red</td>
        <td width=40>1</td>
            <td>50+</td>
            <td>No date available</td>
            <td>Fresh</td>
    </tr>
    <tr align="center">
        <td width=120>ITEM2</td>
        <td width=350>*** Invalid Part No ***</td>
        <td width=40>1</td>
            <td></td>
            <td></td>
            <td></td>
    </tr>
    <tr align="center">
        <td width=120>ITEM3</td>
        <td width=350>Medium in Blue</td>
        <td width=40>1</td>
            <td>12</td>
            <td>2012-05-15</td>
            <td></td>
    </tr>
    <tr align="center">
        <td width=120>ITEM4</td>
        <td width=350>Small Green</td>
        <td width=40>1</td>
            <td>0</td>
            <td>No date available</td>  
            <td>Discontinued</td>
    </tr>
    <tr align="center">
        <td width=120>ITEM5</td>
        <td width=350>Multi Color Oblong</td>
        <td width=40>1</td>
            <td>12</td>
            <td>No date available</td>
            <td></td>
    </tr>
    <tr align="center">
        <td width=120>ITEM6</td>
        <td width=350>Unclear Mottled Annular</td>
        <td width=40>1</td> 
            <td>0</td>
            <td>2012-05-25</td>         
            <td></td>
    </tr>
</table>

Answer 1

Because there is no real code posted, I'll try to outline what your code will need to do:

1. You'll need to send your username / password to the script that the login screen of your manufacturer submits to. You can determine this by looking at the form action tag. You will need to send the username/password using the form field names that are used by the login form. You will also need to send the username/password as the same type of form submission (most likely POST, but could be GET)

2. The response from the server will probably include a cookie. You'll need to capture that and send it back with every page request.

3. You will then need to submit your other forms internally as needed and capture the output. You can do all of this with cURL or there are other libraries that can help parse the HTML responses from the server.

The above should at least point you in the right direction. Once you have some working code (or not working, but should be working ;-), post further questions here and we can help.

Edit: see this link for some working code that gets you through a login page: How to login in with Curl and SSL and cookies