This pertains to bags (http://www.webgui.org/bugs/tracker/import-of-packages-with-international-text-is-broken http://www.plainblack.com/bugs/tracker/http-proxy--syndicated-content-with-international-text-corrupted) and some others.

My apologies for long message, simply, I wanted to describe in one place all symptoms, which can help to correct these bugs.

Around two years back I've wrote about that that unless put by force to set of the connection to the base, that given in the base can be written not in Unicode, but, for instance, in latin1. (Can be a different variants.) Then this has not found supports beside community, as not actual. Many users use the Latin alphabet so beside them this does not render the essential influence upon functioning.

I was need to recode all database on servers, which I support, and every upgrade I was have to add every time line $dbh->do("SET NAMES utf8"); after connection to database:

           my $dbh = DBI->connect($dsn,$user,$pass,{RaiseError=>0,AutoCommit=>1 });
           $dbh->do("SET NAMES utf8"); 
            unless (defined $dbh) {
                        $session->errorHandler->error("Couldn't connect to database: $dsn");
                        return undef;
            }

There is one more thing, which caused discomfort, but possible was work:

In Russian translation I could not install russian locale. As soon as I set ru_RU, In some assets given were displayed in not correct charset. I was need to set en_US
This does not depend on that, was added line " $dbh->do("SET NAMES utf8"); " or not.

Before fall-winter packages were imported/exported orderly and httpProxy & SyndicatedContent worked orderly.

I do not remember exactly, since what versions ceased to work. Possible, thereafter, what have migrated to 2-nd JSON.

Insofar I litter, as from versions 7.5.1 have solved to provide in correspondence to charset data, which are written to the database.

The charset was setting directly when connecting by turning on the flag.
mysql_enable_utf8 => 1

When use of such way Russian translation is displayed in wrong charset (screen shot is attached). Herewith text of the site itself is displayed it is correct.

For reproduce it You need to set in Your and visitor's profile Russian language.

If I use the old way ( $dbh->do("SET NAMES utf8"); ), that translation works orderly:

    my (undef, $driver) = DBI->parse_dsn($dsn);
    my $dbh = DBI->connect($dsn,$user,$pass,{RaiseError => 0, AutoCommit => 1,
#        $driver eq 'mysql' ? (mysql_enable_utf8 => 1) : (),
    });
        $dbh->do("SET NAMES utf8"); # my line instead of commented line
            unless (defined $dbh) {
                        $session->errorHandler->error("Couldn't connect to database: $dsn");
                        return undef;
            }

I understand that these belongings can not and must not depend on each other, but there is fact and I can not find him explanation.

In description of this module is described that both ways of the set connection charset must be equivalent on result:

 (http://search.cpan.org/~capttofu/DBD-mysql-4.007/lib/DBD/mysql.pm about mysql_enable_utf8):

".......Additionally, turning on this flag tells MySQL that incoming data should be treated as UTF-8. This will only take effect if used as part of the call to connect(). If you turn the flag on after connecting, you will need to issue the command SET NAMES utf8 to get the same effect.

This option is experimental and may change in future versions....."

So,

When to use SET NAMES utf8:

• The Russian translation works orderly
• httpProxy & SyndicatedContent - works not it is correct
• Import/export package - breaks the charset.
• Use ru_RU locale in russian translation breaks the charset.

When turning on the flag mysql_enable_utf8:

• The Russian translation - breaks the charset
• httpProxy & SyndicatedContent - works orderly.
• The Import/export package - breaks the coding in 7.5.11 (tested on WebGUIdemo) though I seem that in 7.5.10 or 7.5.9 worked orderly  (tested on WebGUIdemo too).  Possible, this presently problem in demo only.

Graham or JT, please, upload Russian translation to the translation server (http://www.plainblack.com/uploads/cb/MO/cbMOw8pkZUoSY2FG9BqOuw/Russian.tar.gz).

Vladimir Vitkovsky
http://www.webgui.uanet.biz/
http://web-octopus.com
http://www.transport.su/

 It seems that Translation is not displayed so, that multibit symbols are displayed as singlebit.

Translation is displayed so, that all text encoded twice

As experiment I have tried to add line "utf8::decode($output);" to /data/WebGUI/lib/WebGUI/International.pm:

sub get {
        my ($self, $id, $namespace, $language) = @_;
        $namespace = $namespace || $self->{_namespace} || "WebGUI";
        $language = $language || $self->{_language} || $self->session->user->profileField("language") || "English";
        $id =~ s/$safeRe//g;
        $language =~ s/$safeRe//g;
        $namespace =~ s/$safeRe//g;
    my $cmd = "WebGUI::i18n::".$language."::".$namespace;
    WebGUI::Pluggable::load($cmd);
    our $table;
    *table = *{"$cmd\::I18N"};  ##Create alias into symbol table
        my $output = $table->{$id}->{message};
        $output = $self->get($id,$namespace,"English") if ($output eq "" && $language ne "English");
        utf8::decode($output);   # my line
    return $output;
}

All translation text became be displayed orderly with mysql_enable_utf8 => 1. Nearly all work orderly, but if I try to edit some assets I see blank page (Data Form, Page Layout, etc.)

It is likely that Perl has a lot of "reefs" when work with unicode...

Vladimir Vitkovsky
http://www.webgui.uanet.biz/
http://web-octopus.com
http://www.transport.su/

Hello Graham,

I've saw your changes in translationserver.cgi and tried to apply this. Seems it solved problem.

And that pleasantly, I now without problems can in file of the translation to set ru_RU or uk_UA.

Translation files have partly plane unicode text and partly text like: "\x{411}\x{435}\x{437}\x{43f}\x{435}\x{43a}\x{430}". Both formats work orderly. 
All files have line "use utf8;" - seems, this that helped to force to display orderly translation.
Now there is no need to in that line, which I added as experiment to /data/WebGUI/lib/WebGUI/International.pm. All work and with it and without it.

Even more, I tried to put the line (use utf8;) in old translation files and it too worked orderly.

Does really necessary text in such (\x{411}\x{435}) format only?

Than I tried to put the line (use utf8;) to 7.4 translation files - whole translation became be displayed in crossover coding.
Than I have changed the lines (from "SET NAMES utf8" to "mysql_enable_utf8 => 1") and all became be displayed orderly. Even more, bug with HTTP Proxy (http://www.webgui.org/bugs/tracker/http-proxy--syndicated-content-with-international-text-corrupted#sduZ8mPtKU27rVnbHuYuKg) - disappeared, but appeared for 7.4 bug with Syndicated Content as I wrote for 7.5.15 (http://www.webgui.org/bugs/tracker/syndicatedcontent-brokes-cyrillic-text#eYynbirf_bSWC4lCV5XgwA)

Vladimir Vitkovsky
http://www.webgui.uanet.biz/
http://web-octopus.com
http://www.transport.su/