+ Post New Thread
Page 1 of 2 12 LastLast
Results 1 to 15 of 26
Group Project Thread, Open source, collaborative filtering blacklist? in Coding and Web Development; In light of recent comments on here where people were passing new sites to each other I thought this would ...
  1. #1

    localzuk's Avatar
    Join Date
    Dec 2006
    Location
    Minehead
    Posts
    17,816
    Thank Post
    517
    Thanked 2,473 Times in 1,916 Posts
    Blog Entries
    24
    Rep Power
    836

    Open source, collaborative filtering blacklist?

    In light of recent comments on here where people were passing new sites to each other I thought this would be the perfect moment to mention this.

    How about setting up a project for schools to obtain blacklists from that are maintained by the schools themselves in some wiki/collaborative form? I think it could be a very useful tool as the lists would be kept up to date by those that deal with the sites themselves on a daily basis.

    What do people think?

  2. #2


    tom_newton's Avatar
    Join Date
    Sep 2006
    Location
    Leeds
    Posts
    4,475
    Thank Post
    866
    Thanked 850 Times in 672 Posts
    Rep Power
    196

    Re: Open source, collaborative filtering blacklist?

    It might be useful for those using straight blocklists - but proxies tend to move about quite a bit. Useful for other "blockable" sites.

  3. #3

    localzuk's Avatar
    Join Date
    Dec 2006
    Location
    Minehead
    Posts
    17,816
    Thank Post
    517
    Thanked 2,473 Times in 1,916 Posts
    Blog Entries
    24
    Rep Power
    836

    Re: Open source, collaborative filtering blacklist?

    Well that'd be the point really. As things move around quickly, we spot them a lot quicker than the current providers so we would add them to the list quickly.

    And as straight blocklists it would be invaluable in my opinion.

  4. #4

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,804
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224

    Re: Open source, collaborative filtering blacklist?

    You're reinventing the wheel. Most notably the work at http://urlblacklist.com Now it is commercial, but that's mainly out of necessity. Bandwidth is expensive.

  5. #5

    localzuk's Avatar
    Join Date
    Dec 2006
    Location
    Minehead
    Posts
    17,816
    Thank Post
    517
    Thanked 2,473 Times in 1,916 Posts
    Blog Entries
    24
    Rep Power
    836

    Re: Open source, collaborative filtering blacklist?

    Well the point is the open nature of it. Why should we be paying for things that we ourselves are constantly updating? The point is the collaboration and school focus rather than a company with profits as the bottom line.

    Also, yes bandwidth costs money but then there are always sources of funding for things like this.

    There is absolutely no 'necessity' for such a project to cost money for each school. How about a univeristy hosting it? Many host open source projects already... It would barely touch their bandwidth in most cases.

  6. #6

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,804
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224

    Re: Open source, collaborative filtering blacklist?

    ok, so we're half way there. What about the legal liabilities? Hands up who wants to shoulder that responsibility?

  7. #7

    Join Date
    Jan 2007
    Posts
    5
    Thank Post
    0
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Re: Open source, collaborative filtering blacklist?

    Sounds like an interesting idea...

    On the purely financial side, when you consider what commercial services like URL blacklist charge, you only need to multiply that by a small number of schools before creating an open list begins to look attractive. As localzuk says, bandwidth isn't expensive on the NEN -- that's part of what it's for.

    Given sufficient people contributing, you'd end up with an effective list which reacted very quickly to the types of sites causing problems within schools, including proxy bypass sites -- it only takes one school, monitoring their logs, to pick up on a bypass site and it would be blocked for all schools the next time they pulled the lists (assuming they subscribe to and block that category, of course).

    As far as the legal implications go, I'd be interested to hear more about that. I certainly doubt any sane provider of lists offers things like indemnity to schools should they be sued if undesirable content gets through. Anyone know otherwise?

  8. #8

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,804
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224

    Re: Open source, collaborative filtering blacklist?

    As localzuk says, bandwidth isn't expensive on the NEN -- that's part of what it's for.
    Even when the sites you list start organising DDoS's against your service. After all, that's what happens to anti spam lists?

    As far as the legal implications go, I'd be interested to hear more about that. I certainly doubt any sane provider of lists offers things like indemnity to schools should they be sued if undesirable content gets through. Anyone know otherwise?
    That wasn't really my point. I was more worried about site owners instigating legal action against the list provider. Again basing my suspicions on the experiences of the anti spam list providers. It's happened several times that they have been shut down directly by a court order generated by some spurious claim by a spammer.

    Also, this discussion has reminded me of a secondary issue. There have been incidents in the past where certain named sites have paid money to commercial filtering providers to be exempt from their filters. Despite them clearly belonging in some filtering categories.

  9. #9


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339

    Re: Open source, collaborative filtering blacklist?

    In principle it's good idea, but how to maintain the list? domain names change and will need a lot of manual checking. The perfect job -professional pr0n surfer.

  10. #10

    localzuk's Avatar
    Join Date
    Dec 2006
    Location
    Minehead
    Posts
    17,816
    Thank Post
    517
    Thanked 2,473 Times in 1,916 Posts
    Blog Entries
    24
    Rep Power
    836

    Re: Open source, collaborative filtering blacklist?

    Quote Originally Posted by Geoff
    Even when the sites you list start organising DDoS's against your service. After all, that's what happens to anti spam lists?
    I seriously doubt that something like that would happen but if it did, if it were hosted at a large institution or if it were hosted on services similar to the uk mirror service and replicated around the country it shouldn't really be an issue.

    That wasn't really my point. I was more worried about site owners instigating legal action against the list provider. Again basing my suspicions on the experiences of the anti spam list providers. It's happened several times that they have been shut down directly by a court order generated by some spurious claim by a spammer.
    That would just come down to the wording on the site I would think. If it simply outlined that the lists maintain collections of sites that contain eg. pornography, proxies, email services etc... without stating anything negative about them then it shouldn't be an issue. Companies would have very little, if any, comeback in UK law as far as I can tell - the only thing may be libel (ie. 'they say my site contains pornography which is a lie!'), and that would be a stretch.

    Also, this discussion has reminded me of a secondary issue. There have been incidents in the past where certain named sites have paid money to commercial filtering providers to be exempt from their filters. Despite them clearly belonging in some filtering categories.
    This would prevent such things from happening.

    Quote Originally Posted by CyberNerd
    In principle it's good idea, but how to maintain the list? domain names change and will need a lot of manual checking. The perfect job -professional pr0n surfer.
    I think it would be simply a case of such a project building up a list from existing free lists such as http://squidguard.shalla.de/shallalist.html and then having an easy to use set of tools for schools to submit sites to be added/removed (ie. toolbars for firefox/ie, a quick and easy web front end, etc...). We all spend our time blocking and unblocking sites for our own schools already so it would just be a case of doing it there instead.

  11. #11

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,804
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224

    Re: Open source, collaborative filtering blacklist?

    Fair enough. When are you likely to be able to start this project?

  12. #12

    localzuk's Avatar
    Join Date
    Dec 2006
    Location
    Minehead
    Posts
    17,816
    Thank Post
    517
    Thanked 2,473 Times in 1,916 Posts
    Blog Entries
    24
    Rep Power
    836

    Re: Open source, collaborative filtering blacklist?

    It'd have to be when there were a few people who were willing to help start it up and maybe get some funding and support from somewhere.

    To start we'd just need some hosting so we could start planning it and the motivation to do so.

  13. #13

    webman's Avatar
    Join Date
    Nov 2005
    Location
    North East England
    Posts
    8,406
    Thank Post
    640
    Thanked 961 Times in 661 Posts
    Blog Entries
    2
    Rep Power
    324

    Re: Open source, collaborative filtering blacklist?

    Surely the best way to do this is to have one or more schools hosting the files and only allowing downloads out of school hours, eg. a cron job to fetch the lists between 10pm and 7am.

  14. #14

    Join Date
    Jan 2006
    Location
    Surburbia
    Posts
    2,178
    Thank Post
    74
    Thanked 307 Times in 243 Posts
    Rep Power
    115

    Re: Open source, collaborative filtering blacklist?

    At this point I think you should forget about the technology, hosting etc. and (along with legal implications) consider how you maintain the integrity of such a list:

    Who gets to add things? Who gets to delete them? How do you verify the identity of your list editors? How do you prevent accidental or deliberately harmful changes to the list?

    And if you sort that out, what categories will it address?

  15. #15

    localzuk's Avatar
    Join Date
    Dec 2006
    Location
    Minehead
    Posts
    17,816
    Thank Post
    517
    Thanked 2,473 Times in 1,916 Posts
    Blog Entries
    24
    Rep Power
    836

    Re: Open source, collaborative filtering blacklist?

    I think that sort of thing is what would be discussed once the project was started. However, my ideas were that users of the system would sign up via their official school email address (so in somerset it is a @educ.somerset.gov.uk address). This would help sort out a large amount of possible malicious changes IMO. I think everyone would get to add and remove items but would have to provide a brief reason for each item.

    To prevent accidental deletion of items we could implement a limit of items being deleted at once and an easy 'rollback' tool for re-instating such items. Also, a list of recent changes would show users what had changed...

    But all those are just ideas. Really it would need somewhere to start discussing these sort of things. Anyone up for providing a spot to host a wiki for discussions to start? I'd prefer not to do it at my school as our connection is bashed enough as it is. Somewhere in Lancashire sounds good - they give good beefy pipes to the high schools up there :P

SHARE:
+ Post New Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Open Source XP
    By Dos_Box in forum General Chat
    Replies: 33
    Last Post: 23rd October 2012, 02:09 PM
  2. Open Source Portfolio
    By wesleyw in forum How do you do....it?
    Replies: 1
    Last Post: 13th July 2007, 02:55 PM
  3. Open source alternative.
    By Lee_K_81 in forum *nix
    Replies: 13
    Last Post: 6th February 2007, 11:56 AM
  4. MP support for Open Source
    By Face-Man in forum *nix
    Replies: 13
    Last Post: 8th December 2006, 06:56 AM
  5. Open source in schools
    By kevinmcaleer in forum General Chat
    Replies: 14
    Last Post: 22nd December 2005, 08:52 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •