[uf-dev] Re: [uf-discuss] Storing Microformats

[Scott Reynen]

On Sep 17, 2007, at 1:44 PM, Paul Kinlan wrote:

>  My question is: How are people storing
> the data present in microformats so that they can be searched and
> maintained and consumed in applications etc?

(Moved from the -discuss list.)

Back when I did a spider, my database schema looked like this:

CREATE TABLE IF NOT EXISTS `url` (
   `id` int(11) NOT NULL auto_increment,
   `url` text NOT NULL,
   `last_checked` datetime NOT NULL,
   PRIMARY KEY  (`id`)
)

CREATE TABLE IF NOT EXISTS `node` (
   `id` int(11) NOT NULL auto_increment,
   `parent_id` int(11) NOT NULL,
   `url_id` int(11) NOT NULL,
   `html` text NOT NULL,
   PRIMARY KEY  (`id`),
   KEY `parent_id` (`parent_id`),
   KEY `url_id` (`url_id`)
);

CREATE TABLE IF NOT EXISTS `node_property` (
   `id` int(11) NOT NULL auto_increment,
   `node_id` int(11) NOT NULL,
   `name` varchar(255) NOT NULL,
   `value` text NOT NULL,
   PRIMARY KEY  (`id`),
   KEY `node_id` (`node_id`)
)

So the "node" table was basically a DOM tree I used for parsing and  
the "node_property" table was where I put the parsed data for quick  
searching.  I don't know that this was necessarily the best way to do  
it, but I didn't run into any problems during the brief period it was  
running.

Peace,
Scott