[uf-discuss] ufXtract's portable social network parser
glenn.jones at madgex.com
Mon Dec 3 14:34:20 PST 2007
ufXtract's portable social network parser is a combination of the
ufXtract microformats parser and a spider which follows rel="me" links.
It has been designed to extract profiles and friends lists from social
networks and other sites which have microformats support. The parser
returns two main collections of data, all the rel="me" links and any
The parser API
You can set the parser to single single or multiple domains. Currently,
there are limits to the number of pages which will be parsed (20). Each
collection item is given an additional source-url attribute to identify
There is support for both XML and JSON output, for both client and
The parser also uses a version of the representative hCard concept,
which tries to identify the hCard representing the profile owner. The
implementation is a little more complex than described on the
microformats wiki as it extends over multiple pages and domains. This
means you may find multiple representative hCards from one call to the
API, but there should only ever be one per a URL.
I believe there are a number of different ways that this functionality
could be designed into web sites. So I have provided a simple interface
design to demonstrate one possibility. It's a bit of a homage to the
getsatisfaction.com registration page with a few extra twists. I would
Of the sites listed on the demo last.fm and ma.gnolia.com return the
best results. The other sites have differing levels of portable social
network support. It also works well against blogs such as adactio.com or
tantek.com that are marked-up with rel="me" . It's worth trying out the
two depth search levels.
Pages not parsing
You may find on some sites like twitter.com only certain pages are
parsed. These sites often have good microformats support, but parts of
their functionally are locked behind logon's. The parser does not
support authenticated sessions as this would mean asking the user to
pass me their log-in details which is a really bad idea. If I can lay
my hands on a good Open-ID and/or OAuth C# libraries, I will try and
implement some different types of authentication.
This is all research work still under development, I placed it on the
web for others to experiment with. I hope you enjoy playing with it.
More information about the microformats-discuss