Some of Ben Laurie's take on the "extremely serious vulnerability" that Debian added to OpenSSL.
I've ranted about this at length before, I'm sure - even in print, in O'Reily's Open Sources 2. But now Debian have proved me right (again) beyond my wildest expectations. Two years ago, they "fixed" a "problem"
in OpenSSL reported by valgrind by removing any possibility of adding any entropy to OpenSSL's pool of randomness.
The result of this is that for the last two years (from Debian's "Edgy"
release until now), anyone doing pretty much any crypto on Debian (and hence Ubuntu) has been using easily guessable keys. This includes SSH keys, SSL keys and OpenVPN keys.
What can we learn from this? Firstly, vendors should not be fixing problems (or, really, anything) in open source packages by patching them locally - they should contribute their patches upstream to the package maintainers. Had Debian done this in this case, we (the OpenSSL Team) would have fallen about laughing, and once we had got our breath back, told them what a terrible idea this was. But no, it seems that every vendor wants to "add value" by getting in between the user of the software and its author.
Secondly, if you are going to fix bugs, then you should install this maxim of mine firmly in your head: never fix a bug you don't understand.
Well. I don't think it's quite that black and white. There's several things that can be learned from this incident. I'll go through them, so we can all learn something.
One should not blindly trust compilers or in this case valgrind warnings (or any other code compilation or analysis tool) without understanding what is going on. Every software tool has bugs, even the ones you use to find the bugs with!
IMHO The issue of vendors fixing bugs (or indeed adding features) is a red herring. Especially in Debians case given it's non-profit volunteer status. This is simply a matter of not having the right resources, ie people with the necessary skill, to do the job. This doesn't mean they shouldn't try.
However Debian did screw up here in one regard. They should of sent the patch upstream. not only to validate it as a real bug fix, but because it's the 'correct' thing to do in the Open Source world. Yes I know they aren't compelled to in this case, as OpenSSH is BSD licensed, but it's just bad moaners. Especially when your an Open Source project yourself.
> One should not blindly trust compilers or in this case valgrind warnings
One should understand what valgrind/whatever warnings really mean e.g. "oh look here's X" is not the same as "oh look, here's X and it definitely is a deadly bug that absolutely must be eliminated at all costs".
Kind of thing I thought anyone who'd done much compiling et al learnt a long time ago. I've certainly got enough code that would die horribly if anyone was rententive enough to try and make all the warnings go away.
> not having the right resources, ie people with the necessary skill, to do the job.
> This doesn't mean they shouldn't try
Trying is fine & I'm all for it, provided the triers have a real job to do and factor in the consequences of failing.
I spent some time with the OpenSSL source for win32 builds in the crypto embargo days and unless it's had a makeover since, you'd be insane to touch 90% of it, especially anything concerning critical-to-crypto randomness. I had a good reason to fiddle e.g. making it build at all. Why anyone would want to (presumably) "improve" it is completely beyond me.
> Debian did screw up here in one regard
No they just screwed up. Here's a bit of Ben's comments I didn't include the first time:
"the Debian maintainers, instead of tracking down the source of the uninitialised memory instead chose to remove any possibility of adding memory to the pool at all".
As in they chose the wrong fix for that warning and caused some significant collateral damage. An out and out mistake... baby and bathwater.
I'd hardly call Debian a vendor, more of a "bundler and packager". I know that's just semantics, but this kind of thing highlights the fact that they are just people who take a bunch of packages and put them together into a distribution, then are unable to really fix anything because they lack the right skills.
Obviously I'm a bit biased (being an ex-kernel engineer at Sun), but proper vendors like Sun do this a bit better (but not perfect either); when we used open source code/products in Solaris, we submitted any improvements back to the community unless they were specific changes that related to our packaging/distribution rather than actual enhancements or bug fixes. Obviously we had to produce our own patches ('cos that's how our customers applied them to their systems), but the code would go back upstream. Thunderbird and Firefox have "benefited" from Sun contributions, as have many of the gnome sub-projects, e.g. nautilus.
This wasn't always the case - anyone who ran a Solaris box in the "early days" will probably remember such things as sendmail 8.x.x+Sun etc, where we took sendmail, butchered it a bit, called it a sun distro, changed the cf files (remember writing those by hand?), then overwrote any customer's sendmail.cf whenever we applied the latest security patches...
The thing that's different about a "real" vendor and someone like Debian is not just skill, but the quantity of skill and process - even in the UK, Sun have two largebuildings full of very clever people who are constantly writing code for bug fixes or new code for projects going into Solaris (and other stuff). Having moved into teaching fairly recently (because I'm a fool, clearly), the thing I've found hardest to deal with, apart from having to wear a tie and get up before 9, is not being able to turn around to any one of a couple of hundred people and talk through a problem at every conceivable level, down to SPARC assembler, C source and packets, in, say, the VM sub-system, or the IP stack.
When you put all these people together, you stand a greater chance of getting good solutions to problems, usually by the process of a support engineer writing a quick hack to work around or fix the problem, then handing it over to the sustaining team to actually produce a real fix, who then have to code review it with their peers, then get a patch produced by yet another team. So, from quick and dirty fix to patch, it's gone through at least 3 engineers, and been reviewed by many more.
Bundle together a bunch of people who aren't used to software engineering AND dealing directly with properly mission critical customers (BT 999 service?), and you get incomplete or unsatisfactory fixes.
Apologies for the length of this post/slight rant.
Last edited by ajtaylor; 15th May 2008 at 11:10 PM.
If it's anything like most of the "technical" mailing lists out there, the signal to noise ratio is very low. Most projects have a "public" list and a "developers only" list, but even then, the dev only list can get a large amount of crud. We had a trend to start up a list for a specific purpose, then it would get hijacked by numpties, so we'd start up a new one, etc. On kernel specific mailing lists, we'd get questions like "I have a customer who can't print. Why?" . Of course, anything that starts with "I have a customer" triggers the primal delete instinct.
Yea i know that only too well, but the posts did get several replies. Im not saying they were from core dev's but even still, you think one of them would've picked up such a school boy error lol. Crazy really.
> I'd hardly call Debian a vendor, more of a "bundler and packager".
You read my mind. Had counterparts who did the unix dev protesting a few times about folk "gluing a few things together with a bit of script and thinking they're proper developers.." only with a bit more uhh.. vim.
They all worked on Solaris and despite the short haircut I had a Solaris x86 2.1 running INN for a few years, but not sendmail - stuck with mercury/Netware, later post.office/NT 3.x.
> If it's anything like most of the "technical" mailing lists out there, the signal to
> noise ratio is very low
Long time since I subscribed to their lists, but they were a bit more annoying than most and I lasted maybe a year decreasingly wanting to be helpful. OpenSSL must be about 10 years old now... just can't imagine wanting to stick with all that repetition for that long.
> he did consult the open-ssl dev mailing list
That was an essentially anonymous "When debbuging applications that make use of openssl using valgrind, it can show alot of warnings" and the advice they got in that context was fine. It was a far cry from say "I'm a package maintainer for for X, here's a patch I've made and next week I'm going to roll it out to zillions of users - any reason not to?"
A lot of clueless ankle-biters running around saying it was really OpenSSL wotdunnit (because of X, Y and the 'consulted" thing), does not make it true.