3

I'm developing an ecommerce search engine that allows you to search for products in a lot of ecommerce websites.

How do I approach the matter?

I need an application that will be able to scan websites, parse their HTML and determine which of the images in the website are product images, which are product descriptions, which are product prices.

Would be happy to hear any idea, example.

Thanks in advance.

edit: My question is not how to get the HTML from the websites(which is called screen scraping) but more on how to parse that information and understand which of the html contains the actual data i am looking for, and which is not.

Eytan Levit
  • 473
  • 9
  • 27
  • It's called Screen-Scraping. See the tag. – Cerebrus Feb 06 '09 at 07:08
  • Sounds like http://www.shopbot.com.au/ – Simon Feb 07 '09 at 01:59
  • Seems shops have to pay to get registered on shopbot. Interesting model. http://www.shopbot.com.au/sh-shopregister.html – Simon Feb 07 '09 at 02:10
  • 1
    you can use PYTHON , it is the answer for all web issues (like scan HTML REQUEST & RESPONSES) , so ,you can search for a web spider program written on python then you can compile it as java program(applet) with jython and load it into any free web server. cya –  Nov 21 '10 at 16:57

2 Answers2

2

You may find this thread helpful in your quest. I had outlined the basic steps there. Here's the link to all questions tagged as "Screen-scraping" on SO. Also, lots of material on the web - Google.

Community
  • 1
  • 1
Cerebrus
  • 25,615
  • 8
  • 56
  • 70
1

Most of the sites you'd be scraping (more correctly web-scraping) have partner APIs for "reseller" type deals. For you to circumvent that with screen scraping will quickly find your IP blocked by their traffic servers, and potentially put you in a legal situation.

This is ethically dubious at best.

annakata
  • 74,572
  • 17
  • 113
  • 180
  • How on earth does this get a downvote? I *guarantee* anyone trying what the OP is trying will get shut down fast. – annakata Mar 18 '10 at 20:13