Open source, collaborative filtering blacklist?

[localzuk]

In light of recent comments on here where people were passing new sites to each other I thought this would be the perfect moment to mention this.

How about setting up a project for schools to obtain blacklists from that are maintained by the schools themselves in some wiki/collaborative form? I think it could be a very useful tool as the lists would be kept up to date by those that deal with the sites themselves on a daily basis.

What do people think?

[tom_newton]

It might be useful for those using straight blocklists - but proxies tend to move about quite a bit. Useful for other "blockable" sites.

[localzuk]

Well that'd be the point really. As things move around quickly, we spot them a lot quicker than the current providers so we would add them to the list quickly.

And as straight blocklists it would be invaluable in my opinion.

[Geoff]

You're reinventing the wheel. Most notably the work at http://urlblacklist.com Now it is commercial, but that's mainly out of necessity. Bandwidth is expensive.

[localzuk]

Well the point is the open nature of it. Why should we be paying for things that we ourselves are constantly updating? The point is the collaboration and school focus rather than a company with profits as the bottom line.

Also, yes bandwidth costs money but then there are always sources of funding for things like this.

There is absolutely no 'necessity' for such a project to cost money for each school. How about a univeristy hosting it? Many host open source projects already... It would barely touch their bandwidth in most cases.

[Geoff]

ok, so we're half way there. What about the legal liabilities? Hands up who wants to shoulder that responsibility?

[Sypher]

Sounds like an interesting idea...

On the purely financial side, when you consider what commercial services like URL blacklist charge, you only need to multiply that by a small number of schools before creating an open list begins to look attractive. As localzuk says, bandwidth isn't expensive on the NEN -- that's part of what it's for.

Given sufficient people contributing, you'd end up with an effective list which reacted very quickly to the types of sites causing problems within schools, including proxy bypass sites -- it only takes one school, monitoring their logs, to pick up on a bypass site and it would be blocked for all schools the next time they pulled the lists (assuming they subscribe to and block that category, of course).

As far as the legal implications go, I'd be interested to hear more about that. I certainly doubt any sane provider of lists offers things like indemnity to schools should they be sued if undesirable content gets through. Anyone know otherwise?

[Geoff]

As localzuk says, bandwidth isn't expensive on the NEN -- that's part of what it's for.

Even when the sites you list start organising DDoS's against your service. After all, that's what happens to anti spam lists?

As far as the legal implications go, I'd be interested to hear more about that. I certainly doubt any sane provider of lists offers things like indemnity to schools should they be sued if undesirable content gets through. Anyone know otherwise?

That wasn't really my point. I was more worried about site owners instigating legal action against the list provider. Again basing my suspicions on the experiences of the anti spam list providers. It's happened several times that they have been shut down directly by a court order generated by some spurious claim by a spammer.

Also, this discussion has reminded me of a secondary issue. There have been incidents in the past where certain named sites have paid money to commercial filtering providers to be exempt from their filters. Despite them clearly belonging in some filtering categories.

[CyberNerd]

In principle it's good idea, but how to maintain the list? domain names change and will need a lot of manual checking. The perfect job -professional pr0n surfer.

[localzuk]

Originally Posted by Geoff

Even when the sites you list start organising DDoS's against your service. After all, that's what happens to anti spam lists?

I seriously doubt that something like that would happen but if it did, if it were hosted at a large institution or if it were hosted on services similar to the uk mirror service and replicated around the country it shouldn't really be an issue.

That wasn't really my point. I was more worried about site owners instigating legal action against the list provider. Again basing my suspicions on the experiences of the anti spam list providers. It's happened several times that they have been shut down directly by a court order generated by some spurious claim by a spammer.

That would just come down to the wording on the site I would think. If it simply outlined that the lists maintain collections of sites that contain eg. pornography, proxies, email services etc... without stating anything negative about them then it shouldn't be an issue. Companies would have very little, if any, comeback in UK law as far as I can tell - the only thing may be libel (ie. 'they say my site contains pornography which is a lie!'), and that would be a stretch.

Also, this discussion has reminded me of a secondary issue. There have been incidents in the past where certain named sites have paid money to commercial filtering providers to be exempt from their filters. Despite them clearly belonging in some filtering categories.

This would prevent such things from happening.

Originally Posted by CyberNerd

In principle it's good idea, but how to maintain the list? domain names change and will need a lot of manual checking. The perfect job -professional pr0n surfer.

I think it would be simply a case of such a project building up a list from existing free lists such as http://squidguard.shalla.de/shallalist.html and then having an easy to use set of tools for schools to submit sites to be added/removed (ie. toolbars for firefox/ie, a quick and easy web front end, etc...). We all spend our time blocking and unblocking sites for our own schools already so it would just be a case of doing it there instead.

[Geoff]

Fair enough. When are you likely to be able to start this project?

[localzuk]

It'd have to be when there were a few people who were willing to help start it up and maybe get some funding and support from somewhere.

To start we'd just need some hosting so we could start planning it and the motivation to do so.

[webman]

Surely the best way to do this is to have one or more schools hosting the files and only allowing downloads out of school hours, eg. a cron job to fetch the lists between 10pm and 7am.

[PiqueABoo]

At this point I think you should forget about the technology, hosting etc. and (along with legal implications) consider how you maintain the integrity of such a list:

Who gets to add things? Who gets to delete them? How do you verify the identity of your list editors? How do you prevent accidental or deliberately harmful changes to the list?

And if you sort that out, what categories will it address?

[localzuk]

I think that sort of thing is what would be discussed once the project was started. However, my ideas were that users of the system would sign up via their official school email address (so in somerset it is a @educ.somerset.gov.uk address). This would help sort out a large amount of possible malicious changes IMO. I think everyone would get to add and remove items but would have to provide a brief reason for each item.

To prevent accidental deletion of items we could implement a limit of items being deleted at once and an easy 'rollback' tool for re-instating such items. Also, a list of recent changes would show users what had changed...

But all those are just ideas. Really it would need somewhere to start discussing these sort of things. Anyone up for providing a spot to host a wiki for discussions to start? I'd prefer not to do it at my school as our connection is bashed enough as it is. Somewhere in Lancashire sounds good - they give good beefy pipes to the high schools up there :P