Using Channel Providers Outside the effbot.exe/effnews Application

January 27, 2003 | Fredrik Lundh

This note discusses how to use effnews provider scripts in your own applications.

Loading Providers #

A provider is a Python script that defines two objects, a urls sequence, and a callable simpleprovider object (usually a function).

The following function loads all provider scripts in a given directory (in a default effnews install, the directory is c:\effbot.exe\effnews):

Example: Loading available providers

import os, glob

def getproviders(directory):
    providers = {}
    for file in glob.glob(os.path.join(directory, "*.provider")):
        ns = {}
        try:
            execfile(file, ns)
            for url in ns.get("urls"):
                if isinstance(url, type("")):
                    providers[url] = ns
        except:
            pass
    return providers

The function returns a dictionary, mapping URLs to module namespaces.

Using a Provider #

Once you’ve loaded the provider dictionary, you can download entire web pages, and use the appropriate provider to parse them. Here’s an example, which prints channel information and items to standard output:

Example: Loading data using a provider

import urllib

class Context:

    def push(self, tag, **args):
        print "---", tag
        for key, value in args.items():
            print key, value

providers = getproviders("c:/effbot.exe/effnews")

def loadurl(url):

    provider = providers.get(url)

    file = urllib.urlopen(url)

    if provider:
        c = Context()
        provider["simpleprovider"](c, file.read())

    else:
        ... rss_loader(url, file) ...

    file.close()

The rss_loader function isn’t shown in this example. You can for example use the parser from the effnews toolkit to implement the function.