AI-controlled US military drone ‘kills’ its operator in simulated test

[jthompson]

From The Guardian: https://www.theguardian.com/us-news/2023/jun/01/us-military-drone-ai-killed-operator-simulated-test

 

No real person was harmed, but artificial intelligence used ‘highly unexpected strategies’ in test to achieve its mission and attacked anyone who interfered

 

...a drone powered by artificial intelligence was advised to destroy an enemy’s air defense systems, and ultimately attacked anyone who interfered with that order.

 

“The system started realising that while they did identify the threat, at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective,”

 

“We trained the system – ‘Hey don’t kill the operator – that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”

 

I suppose we're safe enough until the machines develop time travel. Or are we??

 

[mikeprice]

did it say

 

I'm sorry, Dave. I'm afraid I can't do that.

 

at any point???

[TechMonkey]

The issue I see here is that AI is like a toddler. Don't like the instructions? Scream at the one giving them. Get told that is bad? Prevent the instructions getting through by effectively putting fingers in ears and singing "la la la la"!

 

I can marvel at the reasoning and ingenuity but I would not give a toddler a hellfire.

[Dos_Box]

How weird, I was just reading that very article! @mikeprice, had you read 2010 (the sequel to 2001) or seen the movie/ It explains why HAL did what it did,and it was for similar reasons that the US drone simulation went the way it did.

As regards to the calls for a pause in AI development - like that's going to happen! - I'm in agreement. It's amazing how sci-fi had been prediciting these very issues decades ago, and we need to be channeling our inner Azimov about this.

[Davit2005]

Rise of the machines

[Dos_Box]

Another issue about AI is that it does run the very real risk of dumbing down tasks that require a lot of thought. By this I mean that the person asking for a solution may accept whatever the AI tells them, but does not have the context for the solution. A lot like people looking for answers to complex questions on Google and just reading the little summary answer it give you at the top of the search results.

Yes, AI could be a huge game changer, but how we use it needs to be very well thought over.

As a network manager having an inbuilt Microsoft AI in Windows Server platforms to perform tasks for you could be great. Just imagine asking it to fix a broken Exchange server and it doing in minutes what would have taken you a day to research and implement a solution for. That sounds great until you realise that you are not getting the in-depth knowlege you would have otherwise had you had to do it yourself.

There could be a new job title 'AI Wrangler'.

Edited June 2, 2023 by Dos_Box

[mikeprice]

Another issue about AI is that it does run the very real risk of dumbing down tasks that require a lot of thought. By this I mean that the person asking for a solution may accept whatever the AI tells them, but does not have the context for the solution. A lot like peopel looking for answers to complex questions on Google and just reading the little summary answer it give you at the top of the sear results.

Yes, AI could be a huge game changer, but how we use it needs to be very well thought over.

As a network manager having an inbuilt Microsoft AI in Windows Server platforms to perform tasks for you could be great. Just imagine asking it to fix a broken Exchange server and it doing in minutes what would have taken you a day to research and implement a solution. That sounds great until you realise that you are not getting the in-depth knowlege you would have otherwise had you had to do it yourself.

There could be a new job title 'AI Wrangler'.

 

Exactly

WHen I was a proper IT person (programming, and all that) we always insisted people in our team (i.e., technical support for the normal programmers) could program in assembler as well 3rd gen languages such as Fortran and Cobol

Our theory was that being able to use assembler menat that you had a good idea of what the actual machine was doing and why x took longer than y

I remember trying to explain to a systems analyst why using the interpreted language that their team normally used was too slow for a specific task - I ended up running a test where doing the same task in assembler took 24 seconds, Cobol took 25 seconds and ScreenWrite (which uses the interpreter) took several minutes.

 

 

AI is just taking people further away from the machine and hence further away from knowing what is actually happening - and more importantly WHY it is happening

[southhamster]

Exactly. About a million years ago (1982!?) I did an evening class and learnt to write 6800 machine code (hex.), then burn the program to EPROM. That really made you understand hardware architecture, albeit on a simple 8 bit processor, but it's stood me in good stead a number of times. It makes the Arduino IDE seem quite luxurious!

Exactly

WHen I was a proper IT person (programming, and all that) we always insisted people in our team (i.e., technical support for the normal programmers) could program in assembler as well 3rd gen languages such as Fortran and Cobol

Our theory was that being able to use assembler menat that you had a good idea of what the actual machine was doing and why x took longer than y

I remember trying to explain to a systems analyst why using the interpreted language that their team normally used was too slow for a specific task - I ended up running a test where doing the same task in assembler took 24 seconds, Cobol took 25 seconds and ScreenWrite (which uses the interpreter) took several minutes.

 

 

AI is just taking people further away from the machine and hence further away from knowing what is actually happening - and more importantly WHY it is happening

[TechMonkey]

It appears the Colonel misspoke!?

 

Whether that means he wasn't meant to tell that story and is back tracking as they now say that experiment never took place!

[mikeprice]

It appears the Colonel misspoke!?

 

Whether that means he wasn't meant to tell that story and is back tracking as they now say that experiment never took place!

 

Or did he only say it never happened because the AI controlled drone was circling outside with an armed Hellfire?????

[mikeprice]

How weird, I was just reading that very article! @mikeprice, had you read 2010 (the sequel to 2001) or seen the movie/ It explains why HAL did what it did,and it was for similar reasons that the US drone simulation went the way it did.

As regards to the calls for a pause in AI development - like that's going to happen! - I'm in agreement. It's amazing how sci-fi had been prediciting these very issues decades ago, and we need to be channeling our inner Azimov about this.

 

Forgot to mention before

 

Yes I did read 2010 - but I have never read 2001 - only seen the film

 

it was very interesting!

[mavhc]

We're massively over estimating AI, as usual. The entire story was nonsense from the start

[jthompson]

I'm taking it as a sexed up account of a laboratory exercise in order to make for an amusing anecdote during a panel discussion. More like computer game AI rather than LLM-grade stuff.

[Andrew_C]

Shame we've decided to ignore these :-

 

A robot may not injure a human being or, through inaction, allow a human being to come to harm.

A robot must obey orders given it by human beings except where such orders would conflict with the First Law.

A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

 

And even then; these were not robot proof!

[robintech]

Can't find the source but I remember a much older anecdote about an Australian real life war game where the fighter jets were "shot down" by Kangaroos with ground to air missiles after the computer side equipped them.

[Dos_Box]

Can't find the source but I remember a much older anecdote about an Australian real life war game where the fighter jets were "shot down" by Kangaroos with ground to air missiles after the computer side equipped them.

 

Not exactly AI, but when I was in the army in the late 80's I was at the headquarters of the 1st Brtish Corps in Bielefeld, West Germany the cold war was still ongoing and we had deployed on our major winter exercise, Winter Sales. We spent a lot of time in the field as the army had just introduced it's new Ptarmigan comms system and any time a formation such as a division or Brigade HQ went into the field so did we as we were classed as a main access node (most comms go to here eventually). On this ex however it was a 'big one' (we did several of these per year) and my role was to drive the BSV (Boxed Staff Vehicle), think hi-tech offices in the field, and setup the HQ. This was a major undertaking with the HQ staff and other NATO elements all joining in.

After a week (it was a 2 week ex), something happened and we were all sent back to camp. It took a couple of days to find out the reason. Apparently the computer controlling the excercise lost track of where we were and had popped a nuke over a location. A location we should have moved on from a couple of days before, but the computer messed up for 'reasons'.

To much relience on automation doesn't always work out! Still, that never happened again so must have been a learning curve for someone!

[Andrew_C]

We spent a lot of time in the field as the army had just introduced it's new Ptarmigan comms system

My father worked on the development of that. Trying to stop nukes from killing it.

[mikeprice]

My father worked on the development of that. Trying to stop nukes from killing it.

 

Sounds a bit feather brained - bit of a foul design

 

still - good for a Game

 

 

 

 

sorry