22 replies to this topic

[ScipioAemilius]

Since the update of the forum, all the old threads in the french subforum (I haven't checked the other languages) have been affected by a strange disease : all the special caracters (é, è, ê, à, ù, etc) are shown as é or è (and many others). The new thread doesn't seem to be affected (from what I saw).

[loco]

I'm afraid that's a bug to do with the encoding used on the database during the restore, where the only current solution would be to manually edit the affected posts.

[ScipioAemilius]

:(

I kinda hoped that wouldn't be the case... Oh well... Some more work for me, it seems!

[syao]

Hmm, looks like a double utf8 conversion, and chinese/japanese forums are completely broken... There _may_ be a fix, if you have a copy of the database you want to test it on. There's a possible solution at http://mysql.rjweb.o...oc.php/charcoll (scroll to the section: Fixing "double encoding").

The convert(convert(convert())) trick should do it, but you'd better try on a copy of the db before. The query's gonna take a looont time though, but the earlier it's done, the less posts will be broken by it (any posts made now before it's fixed, will have garbled text after the fix).

Depending on how much stuff is broken, it may be worth a try.

[ScipioAemilius]

I've just finished fixing the Porting guides in french... :(

I guess I'll have to fix it one more time, it is a bad day!

[syao]

Oh... well, it should be possible to exclude specific threads by ID by editing the query, so maybe there will be no need :3

[loco]

Your suggestion is interesting and you are right, that seems to be the issue. The problem is, our database is over 200 megabytes now, it might just be easier doing the edits manually. I'll filter through some of the posts as well when I can, that should reduce the work load a little. Encouraging frequent posts to fix their own posts could speed things up a little more.

[syao]

To help out people fixing their posts:

http://dotalux.com/fixdoubleutf8.php

Paste your (broken) post in the text field, press the button, copy the fixed text below.

I tested it with a few posts in the japanese section, and it seems to work fine.

The script is just a quick wrapper for the triple convert() trick.

[syao]

Is there something still wrong with the database character encoding settings by any chance?

I tried fixing the posts in the Japanese section, and they seem to remain broken. The only thing I was able to fix was the topic title.

I paste "誰もいないの？？", the preview shows correctly, but after submitting, the post will look like "誰もいないの？？" again

[ScipioAemilius]

I noticed that I needed to add a character (a space in the middle of a word, or a letter somewhere) for my copy/paste to have effect... Else I would always get the same unmodified post. Maybe you should try it?

[syao]

Good catch! That does it. I guess there's some check to save time on a database update when overwriting a post with the same text.

[ScipioAemilius]

I just wanted to report that the same problem showed up again. I'll slowly fix the french subsection, but it should be done with others too (or at least notice the problem).

[devilhunter]

yes, sadly the database import from the phpBB board nuked pretty much everythin -non-unicode.
Russian section got hit the hardest.

If you can, correct some more obvious mistakes in posts.
Edit:
But you're just a normal user now.
We currently have a bit of trouble here and there with some code, afterwards, we'll implement submoderators again.

[ScipioAemilius]

Well, I will modify what posts I made...

Edit : I seem to have some problems with the compatibility of Sayo's tool and the new forum layout...

[devilhunter]

Quote

Edit : I seem to have some problems with the compatibility of Sayo's tool and the new forum layout...

Perhaps a little in detail?
But it's best if you ask syao in person.

[ScipioAemilius]

Well, the fixer will only show so many characters. On the older forum, somehow, copy/pasting would also take into consideration the characters that didn't show up... but on this version of the forum, I get about a line and a half of text pasted... Plus I get to copy/paste the dark background (and dark letters, which shows all black). I can get around that last problem by hitting shift+option+command+V.

Quote

Normal copy/paste on a 5 paragraphs text :
Comme ça n'a aucun lien avec le fil actuel, tu aurais pu créer un nouveau fil de discussion, et en plus, je ne suis pas sûr de saisir vraiment ta question, mais bon ... Si je ne réponds pas tout-

shift+option+command+V on a 5 paragraphs text :
Comme ça n'a aucun lien avec le fil actuel, tu aurais pu créer un nouveau fil de discussion, et en plus, je ne suis pas sûr de saisir vraiment ta question, mais bon ... Si je ne réponds pas tout-

Edit : when pasting the first one, it did show as a black line...

Also, there is a problem to copy from a quote here to paste on Sayo's tool : the quote thing will not show unicode text, but already "modified" text, which is going to become a pain to decode and repost (broken / disappearing links, quotes not shown as quotes, etc)

[devilhunter]

Yeah the letter formatting is a pain, i actually had to reinstall the finished IPB from scratch because search wasn't functioning because of a different character set. That wasn't funny.

[ScipioAemilius]

Well, I wish you luck and courage with the transition!

[thanoulas]

What's funny is that the Russian section now displays "Ελλάδα" at the link in the sections which means "Greece" and it's written in Greek.

[devilhunter]

oops
(i may have copied over the wrong word... )

Edit: fixed
thanks for noticing

[syao]

The Russian section is now taken care of, as well as the Japanese and Chinese sections.

The fix procedure isn't fully automated (it requires user validation since, depending on the post contents, it might require to pass zero, one, two or three times through the encoder), so I can't do it globally for the entire forum, but only by post ID.

If there's any post with garbled text, though, please report it through the Report button and I'll give it a fix (I'll be processing the queue every 24-48 hours, depending on how much stuff gets reported).
[Sorry mods, you're gonna get a bunch of extra emails, please ignore them. If it's too much, feel free to complain and we'll find another solution ]

[ScipioAemilius]

I'm back from a 2 weeks trip to Italy, and I just saw the latest postson the forum... Well, french has a lot of special characters (of course, not as much as russian, chinese or japanese, but still, many accentuated letters : àâèéêîïùç). I'd appreciate it if you could go through this section as well, it could save me a few hours of work, but if, somehow, you can't or don't have time for this, then I'll get to it in a few days (still need to get rid of jetlag, and to do two weeks of stuff that has been hanging around )

[syao]

Alright, I'll handle the French section as during this week. Once I'm done I'll reply here so you can give it a quick look to see if everything was converted correctly :3