If it's anything like the fun and games I've had with one of their damned language packs, you need to use a slightly different syntax to remove it:
Get-AppxPackage | Where-Object {$_.PackageFullName -eq "***PACKAGE FULL NAME***"} | Remove-AppxPackage
(Obviously removing ***PACKAGE FULL NAME*** and replacing it with the package's full name!)