Python POST Request Failing

Question

I'm looking to do a POST request on this site:

http://web1.ncaa.org/stats/StatsSrv/careersearch

The form on the right has four dropdowns. When I run the code below, "School" stubbornly doesn't become selected. There's a hidden input that may be causing the problem, but I haven't been able to fix it. The javascript on the page doesn't seem to have an effect, but I could be wrong. Any help is appreciated:

#!/usr/bin/python

import urllib
import urllib2

url = 'http://web1.ncaa.org/stats/StatsSrv/careersearch'
values = {'searchOrg' : '30123','academicYear' : '2011','searchSport' : 'MBA','searchDiv' : '1'}

data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()

print the_page

How is it failing? Is it raising an error? Returning something you don't expect? — girasquid, Aug 14 '12 at 21:39
value="30123" is not showing up as selected, while the other values I listed are. — hward86, Aug 14 '12 at 21:50
Try this? http://stackoverflow.com/questions/10133899/python-and-mechanize-login-script I like the mechanize part. `b.select_form(nr=2)` is by index. — User007, Aug 14 '12 at 21:51

score 2 · Accepted Answer · answered Aug 14 '12 at 21:54

As you suspected, you're missing a hidden field: doWhat = 'teamSearch' (for submitting the form on the right).

Using these request values works for me:

values = {'doWhat':'teamSearch', 'searchOrg' : '30123','academicYear' : '2011','searchSport' : 'MBA','searchDiv' : '1'}

tijko · Answer 2 · 2012-08-14T22:13:35.063

I used mechanize:

import mechanize
from BeautifulSoup import BeautifulSoup

mech = mechanize.Browser() 
mech.set_handle_robots(False)
response = mech.open('http://web1.ncaa.org/stats/StatsSrv/careersearch')
mech.select_form(nr=2)
mech.form['searchOrg'] = ['30123']
mech.form['academicYear'] = ['2011']
mech.form['searchSport'] = ['MBA']
mech.form['searchDiv'] = ['1']
mech.submit()
soup = BeautifulSoup(mech.response().read())

I know in mechanize the site was asking for searchOrg, academicYear, searchSport, searchDiv in sequence/list form. You should definitely be mindful of the robots.txt.

Python POST Request Failing

2 Answers2