RE: http://zz9.dnsq.org

["Keith Doxey" <ukha@xxxxxxx>]

Hi Adam,

I was working on a similar idea last year and had just about perfected the
script that would process the HTML grabbed from Ananova when the b*******
redisigned the site and TOTALLY stuffed my script.

I havent had time to play with the idea anymore but I remember at the time
having concerns about the server logs picking up on "automated"
hits to grab
pages.

If only there was a legitimate way to get this info.

I prefer to control the way the listing appear myself so I want the data in
MY database. This means that I can then use ASP's to do clever stuff like
click on the listing and have the TV change channel.

What would be ideal would be to crack the format of the file that Digiguide
use to store the listings on your machine. That way, you can be a
legitimate
subscriber to Digiguide and get the listings into Digiguide, then run a
script that would parse the file and extract the data to store in your own
database. Digiguide would be suspicious as they are expecting all their
subscribers to grab all the listings every week anyway.

Such a method would be less likely to break as I would imagine the majority
of changes to Digiguide are concerned with the look of the pages rather
than
the data format. It takes quite a bit of work to get a screenscrape to
perform reliably and it only needs a minor change in the format of the HTML
page to totally kill it :-(

Your idea of an XML format could be one way to go, but wouldnt a simple CSV
file in an agreed format take up much less bandwidth. It would also mean
that people could import it directly into Access, SQL, MySQL, Excel etc
without having to do any processing.

Once they had it in a database they can search and retrieve the data anyway
they like.

Still looking for the perfect solution

Keith



-----Original Message-----
From: Adam Stevens [mailto:adam@xxxxxxx]
Sent: 25 January 2002 18:47
To: ukha_d@xxxxxxx
Subject: Re: [ukha_d] http://zz9.dnsq.org




"James Hoye" <james.hoye@xxxxxxx> wrote in message
news:<DNEJIAAIMJGCAICBHBFIKELBCGAA.james.hoye@xxxxxxx>...
>
> Now that IS a neat feature :)  Something I will DEFINITELY be looking
> at once my ADSL is in place :)

You don't really need ADSL, just unmetered... I was running the same
system on unmetered 56k until I got the cable modem - The only time the
system needs to be online is when it picks up the TV listings, and that
only takes a couple of minutes.

> > from missing Eastenders... If only I had a Tivo... :-)
>
> I'm beginning to think that with the TiVo Ethernet port upgrade, TiVo
> webserver giving you control via a browser (don't know exactly how
> much) and a TV guide in a SQL database, the TiVo subscription becomes
> a little bit more redundant ;)

Quite - A friend of mine bought a Tivo a couple of weeks ago, and
although it is very good, it really is overkill for me (as I never get
time to watch the things *I* record, let alone watching things the Tivo
things I'd like!).

> Any chance of the ASP to scrape the TV listings?  I might convert it
> to Cold Fusion as I like it better than ASP!

This really goes against the grain, but I can't give you the exact
code... The reason is that I had a complaint to my ISP from the site I
was picking up listings from, so now go via an anonymous proxy, and keep
the service I use close to my chest (They didn't like the idea of me
being to import TV listings into a database, and therefore, in theory,
being able to resell the data... Not that I'd do that anyway).

However, at the end of the day, it's not difficult.  I use a VBScript
program which is scheduled via w2k's scheduler.  You could do the same
thing in Homeseer by setting it up as an event. (The ASP page you can
see from the site simply reads the local database that the vbScript
program has populated).

Homeseer has a hs.getURL command, which will return the text of the URL
you point it to, however when I first did this, the Homeseer function
wasn't working, so I use a 3rd party COM component called aspHTTP from
www.serverobjects.com (just because the component is written for use via
ASP doesn't mean you can't use it in vbScript or anything else which
supports activeX).

Basically, all you need to do is use one of the above methods to get the
text from a web page which displays a full TV-listing.  Then it's just a
case of parsing the text, and putting the data into the database.

Sorry to be so vague!... I would usually be happy to help more, but I
did have real problems with my ISP (funnily enough, I eventually got
confirmation from them that I *wasn't* breaking their AUP - By it was
touch and go for a while!).

That said and done though, I guess I *could* do an automated bi-weekly
e-mail of the data in XML format either to this newsgroup, or to a
select few personal e-mail addresses? (I'm not really sure on the
legality of this - Any lawyers on the list?! - This is all data which is
freely available on the net, I've just correlated it).

Adam.




For more information: http://www.automatedhome.co.uk
Post message: ukha_d@xxxxxxx
Subscribe:  ukha_d-subscribe@xxxxxxx
Unsubscribe:  ukha_d-unsubscribe@xxxxxxx
List owner:  ukha_d-owner@xxxxxxx

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/