Designing an OOP application in PHP. How to?

Question

I'm a newbie in PHP OOP. I'm trying to get how can I structure this kind of application. This application is for Scraping about 100 different websites.

I have a main Class, "Scrap" that handles the methods that are global to all different websites, and inside the folder "Scripts" I have the Classes for handle particular aspects of the website I'm scraping. I have another folder that is called "Lib" to include external libraries.

Let me explain visually:

I have this file schema:

- Scrap.php
+ Scripts
               - Google.php
               - Yahoo.php
               - Stackoverflow.php
+ Lib
     + libScrap
               - LIB_parse.php
     + phpQuery
               - phpQuery.php
               - others files and folder...

The Scrap.php contains the following:

<?php

// Includes
require('/lib/libScrap/LIB_parse.php');
require('/lib/phpQuery/phpQuery.php');

// Testing Scrap
$testing = new Scrap;
$testing->teste = $testing->getPage('http://www.yahoo.com','','off');
echo $testing->teste; 


class Scrap {

    public function __construct() {
        // do things!
    }

    /*
    * This method grabs the entire page(HTML) on given URL
    * Ex: $htmlgrab->teste = $htmlgrab->getPage('http://testing.com/ofertas/','','off');
    * Returns, the HTML of given URL
    */
    public function getPage($site, $proxy, $proxystatus) {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
        if ($proxystatus == 'on') {
            curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
            curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, TRUE);
            curl_setopt($ch, CURLOPT_PROXY, $proxy);
        }
        curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
        curl_setopt($ch, CURLOPT_URL, $site);
        ob_start();      // prevent any output
        return curl_exec ($ch); // execute the curl command
        ob_end_clean();  // stop preventing output
        curl_close ($ch);
    }

    /*
    * 
    * 
    */
    public function getLinks() {
        // do things!
    }

    /*
    * This method grabs the page title.
    * Ex: <title>This is the page title</title>
    * Returns, "This is the page title"
    */
    public function getTitle() {
        // do things!
    }

}
?>

And inside the folder "Scripts" I will have files like this one:

<?php
require('../Scrap.php');

class Yahoo extends Scrap {

    public function doSomething() {
        // do things!
    }

}
?>

End note: I need to call/instantiate all the classes created in the folder "Scripts" to Scrap the websites. My doubt is about the best method to instantiate about 100 classes.

If you can give me some clues on how to design this.

Best Regards,

Sorry my bad english.

Answer 1

If you want to include all of the files in the script folder, why wouldn't a simple loop suffice? I frequently do the same thing with projects containing many script.

$arr = glob ('scripts/*.php');
foreach ($arr as $script)
    include_once ($script);

UPDATE

As far as initializing each object... the best bet would probably be to declare an object inside of each class... like this...

<?php
require('../Scrap.php');

class Yahoo extends Scrap {

    public function doSomething() {
        // do things!
    }
}

$yahooObj = new Yahoo(); //This is the addition

?>

This way, after you call include_once('yahoo.php') you will also get the $yahooObj object.

Ya dig?

Answer 2

Assuming each class has it's own source file you could consider 'autoloading'. In my own projects I make use of the spl_autoload_register() function to achieve this instead of using __autoload().

Answer 3

I would suggest you to name your Scrap classes like this:

Scrap_Yahoo
Scrap_Google
...

Then you do what Dutchie432 suggested:

$scraps = array();
foreach (glob('scripts/*.php') as $script) {
  $scrap = 'Scrap_' . pathinfo($script, PATHINFO_FILENAME);
  require_once($script);
  $scraps[] = new $scrap();
} 

Then you can do whatever you want with this array/factory of scraps:

foreach ($scraps as $scrap) {
  $scrap->scrap();
}

Then you should define an abstract method scrap() in your Scrap class, and don't forget to make the class abstract:

// file: Scrap.php
abstract class Scrap {
  public abstract scrap();
}

Every class in scripts/* directory will extend class Scrap and defined just this one particular method scrap().

You can go even further and implement Template Method design pattern.

Answer 4

The patterns you are likely looking for are Strategy or Command.

As for setting up the scrapers, well, you have various options. You can hardcode the path to the scrapers into the main Scrape class or load them from a config file or use autoloading or a class map or use a Factory or a combination of these. That's really up to you. It's much more a matter of deciding what works well with your application.

If you are already using autoloading, then make sure your Scrapers can be found. If you want to add an additional autoloader, do so. If you prefer a classmap for the added security and speed, then use a classmap and so on. Discussing all the pros and cons is beyond scope of this question. If you are interested in this, have a look at this blog post about autoloading benchmarks (it's for ZF2 but generally applicable).

Since none of the scrapings are likely to depend on each other. I suggest to look into something like Gearman to run these asynchronously in different processes instead of sequentially in the same script. Your main script would then only create the necessary Workers with the appropriate settings and let them run in a background process. There is some examples in the manual and here is another one from the same site as the benchmarking article