Open source, collaborative filtering blacklist?

[localzuk]

In light of recent comments on here where people were passing new sites to each other I thought this would be the perfect moment to mention this.

 

How about setting up a project for schools to obtain blacklists from that are maintained by the schools themselves in some wiki/collaborative form? I think it could be a very useful tool as the lists would be kept up to date by those that deal with the sites themselves on a daily basis.

 

What do people think?

[tom_newton]

It might be useful for those using straight blocklists - but proxies tend to move about quite a bit. Useful for other "blockable" sites.

[localzuk]

Well that'd be the point really. As things move around quickly, we spot them a lot quicker than the current providers so we would add them to the list quickly.

 

And as straight blocklists it would be invaluable in my opinion.

[Geoff]

You're reinventing the wheel. Most notably the work at http://urlblacklist.com Now it is commercial, but that's mainly out of necessity. Bandwidth is expensive.

[localzuk]

Well the point is the open nature of it. Why should we be paying for things that we ourselves are constantly updating? The point is the collaboration and school focus rather than a company with profits as the bottom line.

 

Also, yes bandwidth costs money but then there are always sources of funding for things like this.

 

There is absolutely no 'necessity' for such a project to cost money for each school. How about a univeristy hosting it? Many host open source projects already... It would barely touch their bandwidth in most cases.

[Geoff]

ok, so we're half way there. What about the legal liabilities? Hands up who wants to shoulder that responsibility?

[Sypher]

Sounds like an interesting idea...

 

On the purely financial side, when you consider what commercial services like URL blacklist charge, you only need to multiply that by a small number of schools before creating an open list begins to look attractive. As localzuk says, bandwidth isn't expensive on the NEN -- that's part of what it's for.

 

Given sufficient people contributing, you'd end up with an effective list which reacted very quickly to the types of sites causing problems within schools, including proxy bypass sites -- it only takes one school, monitoring their logs, to pick up on a bypass site and it would be blocked for all schools the next time they pulled the lists (assuming they subscribe to and block that category, of course).

 

As far as the legal implications go, I'd be interested to hear more about that. I certainly doubt any sane provider of lists offers things like indemnity to schools should they be sued if undesirable content gets through. Anyone know otherwise?

[Geoff]

As localzuk says, bandwidth isn't expensive on the NEN -- that's part of what it's for.

 

Even when the sites you list start organising DDoS's against your service. After all, that's what happens to anti spam lists?

 

As far as the legal implications go, I'd be interested to hear more about that. I certainly doubt any sane provider of lists offers things like indemnity to schools should they be sued if undesirable content gets through. Anyone know otherwise?

 

That wasn't really my point. I was more worried about site owners instigating legal action against the list provider. Again basing my suspicions on the experiences of the anti spam list providers. It's happened several times that they have been shut down directly by a court order generated by some spurious claim by a spammer.

 

Also, this discussion has reminded me of a secondary issue. There have been incidents in the past where certain named sites have paid money to commercial filtering providers to be exempt from their filters. Despite them clearly belonging in some filtering categories.

[CyberNerd]

In principle it's good idea, but how to maintain the list? domain names change and will need a lot of manual checking. The perfect job -professional pr0n surfer.

[localzuk]

Even when the sites you list start organising DDoS's against your service. After all, that's what happens to anti spam lists?

 

I seriously doubt that something like that would happen but if it did, if it were hosted at a large institution or if it were hosted on services similar to the uk mirror service and replicated around the country it shouldn't really be an issue.

 

That wasn't really my point. I was more worried about site owners instigating legal action against the list provider. Again basing my suspicions on the experiences of the anti spam list providers. It's happened several times that they have been shut down directly by a court order generated by some spurious claim by a spammer.

 

That would just come down to the wording on the site I would think. If it simply outlined that the lists maintain collections of sites that contain eg. pornography, proxies, email services etc... without stating anything negative about them then it shouldn't be an issue. Companies would have very little, if any, comeback in UK law as far as I can tell - the only thing may be libel (ie. 'they say my site contains pornography which is a lie!'), and that would be a stretch.

 

Also, this discussion has reminded me of a secondary issue. There have been incidents in the past where certain named sites have paid money to commercial filtering providers to be exempt from their filters. Despite them clearly belonging in some filtering categories.

 

This would prevent such things from happening.

 

In principle it's good idea, but how to maintain the list? domain names change and will need a lot of manual checking. The perfect job -professional pr0n surfer.

 

I think it would be simply a case of such a project building up a list from existing free lists such as http://squidguard.shalla.de/shallalist.html and then having an easy to use set of tools for schools to submit sites to be added/removed (ie. toolbars for firefox/ie, a quick and easy web front end, etc...). We all spend our time blocking and unblocking sites for our own schools already so it would just be a case of doing it there instead.

[Geoff]

Fair enough. When are you likely to be able to start this project?

[localzuk]

It'd have to be when there were a few people who were willing to help start it up and maybe get some funding and support from somewhere.

 

To start we'd just need some hosting so we could start planning it and the motivation to do so.

[webman]

Surely the best way to do this is to have one or more schools hosting the files and only allowing downloads out of school hours, eg. a cron job to fetch the lists between 10pm and 7am.

[PiqueABoo]

At this point I think you should forget about the technology, hosting etc. and (along with legal implications) consider how you maintain the integrity of such a list:

 

Who gets to add things? Who gets to delete them? How do you verify the identity of your list editors? How do you prevent accidental or deliberately harmful changes to the list?

 

And if you sort that out, what categories will it address?

[localzuk]

I think that sort of thing is what would be discussed once the project was started. However, my ideas were that users of the system would sign up via their official school email address (so in somerset it is a @educ.somerset.gov.uk address). This would help sort out a large amount of possible malicious changes IMO. I think everyone would get to add and remove items but would have to provide a brief reason for each item.

 

To prevent accidental deletion of items we could implement a limit of items being deleted at once and an easy 'rollback' tool for re-instating such items. Also, a list of recent changes would show users what had changed...

 

But all those are just ideas. Really it would need somewhere to start discussing these sort of things. Anyone up for providing a spot to host a wiki for discussions to start? I'd prefer not to do it at my school as our connection is bashed enough as it is. Somewhere in Lancashire sounds good - they give good beefy pipes to the high schools up there

[wil0]

yeah i had this idea a few months ago, posted it on rm communities.

I came to the conclusion that it should allow people who have been verified to add initially proxy addresses to a list and that subbmission would then be checked, by a script searching for the words phproxy, proxy, unblock ect, then added or if fails, would be entered but flagged as not verified, or similar.

The list would then be compiled into a basic text file, and served. scripts would be available for people to add to their smartcache that would add cron jobs to fetch the files daily...

The service could remain free by getting schools to serve the files and make the webserver distrobute the load amongst the mirrors.

Or you could build a huge database, then start charging

 

Then i realised that stuff like protex ect probably does this?

[Mr_T]

Perhaps a collaborative whitelist would be easier to maintain?

[TornUp]

600GB space

6,000gb GB bandwidth

 

at the desposal of this project.. i brough some hosting. but never really thought what i would use it for....

 

id be intrested in building a project like this!

[Guest kerrymoralee9280]

@Localzuk: Great idea - I hope you can get the ball rolling

 

@Everyone else: Why is everyone so negative on this forum - The lad has had an idea that could be worth something to schools and all that we do on here is throw boundaries at him. Give him some support. Don't say you can't do this because..... Say you can't do this because but here is a work around.

[bonald]

You should try opendns.org

 

Use these DNS as your forwarders. Go to their website , setup an account, checkmark the websites categories you want to block and bingo!

[dbrown]

An interesting thread of great relevance to me at the moment.

But I just *have* to complain about,

You're reinventing the wheel.

This must be the most stupid (apologies to geoff, but that's the word) comment ever made in a discussion. Strangely, it nearly always occurs in discussions about IT, where we have some of the most buckled, eccentric, square, rusty, etc and generally unusable "wheels" in creation.

My bank holiday Monday treat was a trip round Coventry Transport Museum (v.v.highly recommended imo). if you think the perfect wheel has been invented, just look at the dozens of different types of wheels there: fortunately the engineers who made them were (are) not shy of "reinventing the wheel" again & again & again .....

Why are IT people always so convinced that their inventions are so perfect that no one can ever improve on them?

[zag]

Good idea, the current companies that do it are slow and expensive.

 

All you need is a good php based website that is user updateable and creates an xml list or something.

 

I have good experience with running user updatable websites so let me know if you need any help. The simple way to do this would be to allow users from this forum at first.

 

You could also have a rating system for blocking sites, so people vote yes or no if they agree with the block.

[klawd]

I think the biggest issue is one of liability.

 

Its no problem setting and maintaining a list, but who is responsible if filtering provided by this list, lets through unsuitable material to pupils.

 

In our case filtering is down by the LA and although it is far from perfect, we have the luxury of deferring completely to them, should the filter let things through.

 

To be able to say that "The filtering is sanctioned and provided by the LA" is a nice escape clause, I think.

[localzuk]

I think the biggest issue is one of liability.

 

Its no problem setting and maintaining a list, but who is responsible if filtering provided by this list, lets through unsuitable material to pupils.

 

In our case filtering is down by the LA and although it is far from perfect, we have the luxury of deferring completely to them, should the filter let things through.

 

To be able to say that "The filtering is sanctioned and provided by the LA" is a nice escape clause, I think.

 

Saying that is not a legal cover though. Legally, the school is still 100% liable.

[richard.thomas]

I've mentioned this in another thread. It would be quite useful if we had somewhere we can just copy + paste our blacklists to and the system automatically removes duplicates.

 

As mentioned though, there would need to be a limit on who can add things. I'm sure non of the members here would do that, but someone could join up to do it. Is there a way of linking it to edugeek so only people that have made a certain amount of posts, or been on here a certain amount of time can contribute?

 

Other contributions would have to be checked to see if it's in the list, and if it isn't moderated before being added.