CTF from some MIS are not UTF-8 encoded?

[psydii]

We've had an interesting issue recently. Some CTFs contain characters that are coded differently between UTF-8 and ISO8859-1. When opened they are treated as UTF-8 by say, VS Code or SIMS 7, but the special characters (being coded for ISO8859-1) don't render and cause the import to SIMS to fail.

 

If I tell Code to reopen the file as ISO8859-1 the characters render correctly.  So, my guess is that either there is an MIS out there exporting things with the wrong encoding, or something is happening when Secure Transfer is offering us the file.

Anyone got thoughts on / experience with this?

 

 

[Ditto]

Well over 5 years ago since I worked with CTFs, but some things stick! Have a look at the header record:

 

 

The specification https://assets.publishing.service.gov.uk/media/663a42884d8bb7378fb6c31f/CTF_24_technical_specification_v1_.pdf also has some info about coding.

 

Let me know how you get on.

Edited October 6, 2025 by Ditto

[Steve21]

Normally it's a SIMs vs Bromcom thing - Not everything SIMs accepts/exports in CTF Bromcom will accept, for example Bromcom won't accept ` characters where SIMs does etc. So some names that are entered like O`Reilly Bromcom won't accept until you change it to O'Reilly etc (Note the difference in `')

Normally if you open the CTF it'll tell you inside it which MIS it was created by

Steve

Edited October 6, 2025 by Steve21

[Ditto]

15 minutes ago, Steve21 said:

Normally it's a SIMs vs Bromcom thing - Not everything SIMs accepts/exports in CTF Bromcom will accept, for example Bromcom won't accept ` characters where SIMs does etc. So some names that are entered like O`Reilly Bromcom won't accept until you change it to O'Reilly etc (Note the difference in `')

Normally if you open the CTF it'll tell you inside it which MIS it was created by

Steve

Yep, and the encoding should be  in there too:

Edited October 6, 2025 by Ditto

[psydii]

I skim read that yesterday, but but missed the "can be excepted as long as..." paragraph.

 

So right now the main issue is that some MIS's are generating ISO-8859-1 files but tagging them UTF-8, and when parsed as UTF-8 the file contains invalid characters.

 

When I get a moment (lol)  I'll have a poke at a fresh download of one of the files and change that value to see if a file that actually declares itself iso-8859-1 is accepted by SIMS. Though maybe @SIMS_Customer_Support might know/be able to point me to a KB where that sort of thing is documented?

 

My mind is being awakened to a new layer of possible horrors and errors. Bad enough that so many imported names have trailing spaces, but characters that render the same but are in fact different in a system that at least on some level demonstrates an inability to handle that as a human would need it to... oh gods. 

 

 

 

 

[Ditto]

I know the feeling well!

 

<Imaged generated by ChatGPT from my instructions - the ones sneaking out the bottom aren't quite right 🤣>

[SIMS_Customer_Support]

On 07/10/2025 at 09:30, psydii said:

I skim read that yesterday, but but missed the "can be excepted as long as..." paragraph.

 

So right now the main issue is that some MIS's are generating ISO-8859-1 files but tagging them UTF-8, and when parsed as UTF-8 the file contains invalid characters.

 

When I get a moment (lol)  I'll have a poke at a fresh download of one of the files and change that value to see if a file that actually declares itself iso-8859-1 is accepted by SIMS. Though maybe @SIMS_Customer_Support might know/be able to point me to a KB where that sort of thing is documented?

 

My mind is being awakened to a new layer of possible horrors and errors. Bad enough that so many imported names have trailing spaces, but characters that render the same but are in fact different in a system that at least on some level demonstrates an inability to handle that as a human would need it to... oh gods. 

 

 

 

 

Hi @psydii

 

I can certainly arrange for a case to be logged and referred to our Tech team to understand if there is a KBA available. Please can you confirm the organisation you are from and also your email address. I look forward to receiving your response.

 

Best wishes

Sally 

 

 

[coolhands]

Interesting as I’ve had one recently that causes sims to crash. I will check it out in more detail tomorrow.

[Ditto]

On 07/10/2025 at 09:44, Ditto said:

I know the feeling well!

 

<Imaged generated by ChatGPT from my instructions - the ones sneaking out the bottom aren't quite right 🤣>

Hmmm - what happened to my image?

[Marci]

BC hasn’t the first clue about consistent  UTF8 decoding/encoding… hence plain apostrophes break support document templates regularly, and £ sounds are prefixed ^A in funding tables on pupil’s profiles etc and spaces get x0200’d in new report domains when taken out as live feeds.

Edited October 9, 2025 by Marci