Posted on: 03/02/08 01:34pm
By: trampoline
I have my Config.php set to unicode character set so assume my database must also be Unicode, however when I paste unicode into a story (this is an asian font) it looks OK but then when I publish the coding goes... and I get a line of questionmarks What am I missing here ???
I guess I am wondering if I can mix character sets.... ????
Re: Unicode
Posted on: 03/02/08 01:45pm
By: Dirk
Is this a fresh or an existing install? If it's an existing install and you only switched to UTF-8 now, it's possible that your account is still using whatever non-UTF-8 character set you were using before.
Go to "My Account", switch to another language, save, switch back to your preferred language, save. Try again.
bye, Dirk
Re: Unicode
Posted on: 03/02/08 01:55pm
By: trampoline
I am definately using the english UTF8 Character set as the caching engin is saving it as such and the SQL database reports as a UTF8 format, but if I paste a mixture of english and Tibetan the Tibetan just comes out as question marks.....
I guess if Config.php says unicode than it must be a unicode database...
Re: Unicode
Posted on: 03/02/08 02:09pm
By: Dirk
Could it be a font issue? I.e. the font used in the text input area is different from the one that's used to display the text?
Also, we are talking about Geeklog 1.4.1, aren't we?
bye, Dirk
Re: Unicode
Posted on: 03/02/08 02:12pm
By: trampoline
That is very possible,
I can see the font when it is pasted it llooks fine, only when saving does it disapear...
I am using 1.4.1b2.... I hope that is OK...????
I will upgrade it's just I have so much custom code now....
Re: Unicode
Posted on: 03/02/08 02:24pm
By: Dirk
You could also check what's in the database.
The change you'd need for proper UTF-8 handling is in 1.4.1b2, so you're lucky. But why are you using a beta version?
bye, Dirk
Re: Unicode
Posted on: 03/02/08 02:38pm
By: trampoline
Because the updates in 141 b2 were essential for the site and I then did a load of custom code...
I will upgrade shortly to 1.4.1 proper...
the database it's self shows as a UNICODE database...
Re: Unicode
Posted on: 03/02/08 05:58pm
By: Laugh
You may want to wait and upgrade to 1.5 when it comes out (hopefully soon) so you don't have to redo all your custom code again.
Re: Unicode
Posted on: 03/04/08 05:55am
By: trampoline
I may just do that..
When would that be out ? do you know ???
The database is a UTF8 Unicode Database
connection collation utf8_unicode_ci
So possibly it's the beta release code that is the problem...
So if 1.5 is coming soon I will wait...
Thanks...
Re: Unicode
Posted on: 03/04/08 11:48am
By: Laugh
Dirk could asnwer this better than I, but I would hope in a month or two.
They are mainly bug fixing at the moment.
Re: Unicode
Posted on: 03/04/08 11:50am
By: trampoline
Will 1.5 have any additions from Geeklog 2 in it ?
Or is Geeklog 2 going to be entirely different..?
Re: Unicode
Posted on: 03/04/08 02:23pm
By: Dirk
GL2 is a completely different beast. 1.5 will mainly feature the things implemented for the
Summer of Code[*1] (while we're already busy preparing for the next instance ...).
bye, Dirk
Re: Unicode
Posted on: 03/04/08 04:40pm
By: trampoline
Will the database be upgradeable to geeklog 2 or will it be totally seperate ?
I would love to upgrade a whole site to geeklog 2 when it arrives...
is there any ETA yet ?
Will this be a completely new animal on the CMS / Blog Scene ??
Re: Unicode
Posted on: 03/05/08 12:37am
By: r_f_o_t
Quote by: trampolineWill the database be upgradeable to geeklog 2 or will it be totally seperate ?
I would love to upgrade a whole site to geeklog 2 when it arrives...
is there any ETA yet ?
Will this be a completely new animal on the CMS / Blog Scene ??
See Tony's
post here.[*2]
HTH,
Chuck
Re: Unicode
Posted on: 03/05/08 11:04am
By: samstone
I remember 1.4.1b2 had a lot of issues, but don't remember if it had issues with Unicode.
Have you check if the table collation is also utf-8? Because database can be utf-8 but the tables might not be the same.
Edit: Now I remember that if you didn't create the database as utf-8 from the very beginning, your tables won't change to utf-8 by merely changing the database's collation.
Peace!
Sam
Re: Unicode
Posted on: 03/05/08 11:12am
By: trampoline
Correct !!!
You are right the database is unicode but the table collation is latin1_swedish_ci
So what on earth do I do now ???
I am certain I did get unicode text in there once so I must have accidentaly changed it.
How do I change it back ???
Re: Unicode
Posted on: 03/05/08 01:36pm
By: samstone
As far as I remember from my experience, you didn't change anything. The pre GL 1.4.1 works with Unicode fine with the none Unicode collation. Don't ask me why . Starting with 1.4.1b2 the developers did something to make GL more Unicode compatible, which somehow made it stop working with none-unicode tables.
Well, I remember those nightmare days. Depending on the sites I had, for some sites I had to change every table one by one to Unicode collation; took a lot of patience. For others, I just recreated new databases with Unicode from scratch.
Again, another potential problem is, depending on the way your server is configured. GL installation might still create Swedish collation on the tables at installation. You have to edit my.conf to make the new databases and tables created automatically in Unicode. In my case, that broke some older sites, so I had to reverse it. I think PHP5 and MySQL5 would not have this problem because they can be purely utf-8. I have PHP5, but dare not yet upgrade to MySQL5 worrying that some sites might be affected, but I might do it when I can afford some unforeseen down time.
In any case, the easiest method is to change the table collations one by one. For instant solution, just change the story table to utf-8.
Hope this helps!
Sam
Re: Unicode
Posted on: 03/05/08 01:53pm
By: trampoline
I just want to change the Story & Forum postings to Unicode then maybe the whole site,
So you think the transition in SQL 5 is easier ?
Would I do this with sql Administrator ?
I guess I am asking is it possible to do a straight conversion fairly easily.
What did you mean by going to SQL 5 does that make this easier or does sql 5 just handle unicode anyway ?
Thanks for any help...
Re: Unicode
Posted on: 03/05/08 03:10pm
By: trampoline
So I now have SQL Administrator up and running.
I have the table gl_stories and I have gone to the panel where I can set the default character set..
I have backed up and would like to change the default to UTF-8 Unicode
and Collation to what ?
utf8_general_ci
or
utf8_unicode_ci
?
Do I do this for the whole story table ??
I guess I can do this for any part of the systems I want to use Unicode for ???
Looks dangerously easy...
Re: Unicode
Posted on: 03/05/08 04:50pm
By: trampoline
OK so I just did it as I could not wait to see what happened.
I changed the gl_stories Table into
UTF-8 Unicode
with collation set to
utf8_unicode_ci
Still no difference ...
Unicode not showing...
Re: Unicode
Posted on: 03/06/08 07:39pm
By: trampoline
Quote by: samstoneAs far as I remember from my experience, you didn't change anything. The pre GL 1.4.1 works with Unicode fine with the none Unicode collation. Don't ask me why . Starting with 1.4.1b2 the developers did something to make GL more Unicode compatible, which somehow made it stop working with none-unicode tables.
Well, I remember those nightmare days. Depending on the sites I had, for some sites I had to change every table one by one to Unicode collation; took a lot of patience. For others, I just recreated new databases with Unicode from scratch.
Again, another potential problem is, depending on the way your server is configured. GL installation might still create Swedish collation on the tables at installation. You have to edit my.conf to make the new databases and tables created automatically in Unicode. In my case, that broke some older sites, so I had to reverse it. I think PHP5 and MySQL5 would not have this problem because they can be purely utf-8. I have PHP5, but dare not yet upgrade to MySQL5 worrying that some sites might be affected, but I might do it when I can afford some unforeseen down time.
In any case, the easiest method is to change the table collations one by one. For instant solution, just change the story table to utf-8.
Hope this helps!
Sam
Well Stupidly I actually upgraed to 1.4.1b as I thought Unicode support would be better, I naturally assumed with a unicode databse and unicode set as the language that the site would be unicode compatible......
As you pointed out I do indeed have swedish collation under a unicode database and I have tried changing the gl_stories table in one go to unicode, it changes fine still works but the site still does not accept Unicode for stories... So what am I doing wrong ???
Thanks for any pointers....
Re: Unicode
Posted on: 03/06/08 10:03pm
By: jmucchiello
Does this issue occur with newly made stories or just old stories or both?
Re: Unicode
Posted on: 03/07/08 05:19am
By: trampoline
Quote by: jmucchielloDoes this issue occur with newly made stories or just old stories or both?
Both
Re: Unicode
Posted on: 03/07/08 10:53am
By: samstone
It seems that you are using MS SQL. I have not experience with that. I was talking abou MySQL.
A fresh installation is much easier than trying to find out how and where to fix it, since I don't know how you site and database is set up.
I would recommend that you back up your database, and also make another backup without the tables but only data, create a new database with utf8_general_ci, run a fresh installation using GL.1.4.1. and insert the backup data to the new database. That will give you a clean utf-8 site.
The problem with this is that you might not be able to insert the data if the fields are different (e.g. if you have some plugins previously, you might have to install the pulgins first before you upload the data). You can also migrate the data filed by field to avoid conflict. I have been there and done that, and I can tell you that there is not easy way. As long as you keep a backup database, you can try different ways until it works.
If you want pure utf-8, also configure the my.conf and php.ini to use utf-8. The reason I mentioned MySQL5 is because it creates utf-8 tables by default, paring with PHP5, it is more international friendly, but that's another subject.
Peace!
Sam
Re: Unicode
Posted on: 03/07/08 11:12am
By: samstone
One encouragement is that you won't have this problem from now on, as starting from 1.4.1, you can create pure utf-8 websites with GL.
Peace!
Sam
Re: Unicode
Posted on: 03/07/08 11:19am
By: trampoline
I am using MYSQL on a Mac OS X server.
I actually did install the whole site again at 1.4.1 as I thought it was Unicode Compliant, I deleted the old site Now the site is large and I cannot do that so
I absolutely have to make this Unicode compatible...
So you think if I create a new database and put the data back into it is will work ??? at least it is worth a try.
The database is its self is unicode just the tables seem not to be...
Re: Unicode
Posted on: 03/07/08 11:25am
By: mevans
Here is an article I wrote on how I
convert sites to UTF-8[*3] . The tools I reference are Windows based, but I'm willing to bet you can find OS X versions too (you could even update the wiki entry with the OS X tools if you like
). Not sure if this will help you out or not, but it might be worth a look.
Thanks!
Mark
Re: Unicode
Posted on: 03/07/08 11:45am
By: trampoline
Quote by: mevansHere is an article I wrote on how I convert sites to UTF-8[*3] . The tools I reference are Windows based, but I'm willing to bet you can find OS X versions too (you could even update the wiki entry with the OS X tools if you like ). Not sure if this will help you out or not, but it might be worth a look.
Thanks!
Mark
Thanks Mark I will try that.. I now have a busy site so it may not be so easy..
in config.php
it looks like this and has done sice I created the site...
Text Formatted Code
$_CONF['language'] = 'english_utf-8';
$_CONF['default_charset'] = 'utf-8';
There are only Unicode languge profiles no non unicode profiles,
the database is Unicode but the tables are not...
The server system is
OS X 10.4.11
apache 1.3.33
SQL 4.1.22
php 5
Re: Unicode
Posted on: 03/07/08 11:50am
By: mevans
There are only Unicode languge profiles no non unicode profiles,
the database is Unicode but the tables are not...
I think this is where the problem exists. If you can dump the tables, edit the charset per the article and then run them through iconv and then reload, life should be good.
Good luck!
Mark
Re: Unicode
Posted on: 03/07/08 11:55am
By: trampoline
Quote by: mevansThere are only Unicode languge profiles no non unicode profiles,
the database is Unicode but the tables are not...
I think this is where the problem exists. If you can dump the tables, edit the charset per the article and then run them through iconv and then reload, life should be good.
Good luck!
Mark
OK I will try sounds a bit scarry !!!!
I'll have to find the things in OS X to do it will post back on progress if any .....
I actually converted a table to UTF8 with SQL Administrator so surely new articles would be Unicode compliant ?
I am just wondring if there is more to this than just the database as I have set the stories database to Unicode and new stories do not come out right.
I am sorry I am very very slow....
Re: Unicode
Posted on: 03/07/08 07:49pm
By: samstone
Quote by: trampolineI am using MYSQL on a Mac OS X server.
I actually did install the whole site again at 1.4.1 as I thought it was Unicode Compliant, I deleted the old site Now the site is large and I cannot do that so
I absolutely have to make this Unicode compatible...
So you think if I create a new database and put the data back into it is will work ??? at least it is worth a try.
The database is its self is unicode just the tables seem not to be...
Upgrading to 1.4.1 does not automatically change the table collation. You need a clean installation and before you hit install, you need to create a database with utf8_general_ci collation.
What's your MySQL version? Is it version 4.*? If it is and If I remember it right, you will still end up with the tables created in sweedish_general_ci upon installation, if you don't edit the my.conf to tell it to use utf-8.
You can try Mark's method. I did, but didn't work for my situation because there were other problems with my database fields that I had to correct manually, and I don't quite understand the iconv part which took a long time to execute.
Good luck!
Sam
Re: Unicode
Posted on: 03/08/08 04:34am
By: trampoline
Upgrading to 1.4.1 does not automatically change the table collation. You need a clean installation and before you hit install, you need to create a database with utf8_general_ci collation.
What's your MySQL version? Is it version 4.*? If it is and If I remember it right, you will still end up with the tables created in sweedish_general_ci upon installation, if you don't edit the my.conf to tell it to use utf-8.
You can try Mark's method. I did, but didn't work for my situation because there were other problems with my database fields that I had to correct manually, and I don't quite understand the iconv part which took a long time to execute.
I deleted the old database created a new one which was utf8_general_ci
hen installed Geeklog but as you correctly pointed out even though I set the config.php to use utf-8 my tables are infact sweedish_general_ci
So it looks like have the same problem...
What manual things did you do to put it right ???
Re: Unicode
Posted on: 03/08/08 10:10am
By: Laugh
Keep posting your results, this is something I a am going to have to do soon and I can see myself running into similar problems.
Thanks
Tom
Re: Unicode
Posted on: 03/08/08 10:50am
By: trampoline
Quote by: LaughKeep posting your results, this is something I a am going to have to do soon and I can see myself running into similar problems.
Thanks
Tom
I will but it will be a while possibly as I actually fell quite at sea with this...
So i will post back when I get some result..
Re: Unicode
Posted on: 03/08/08 11:12am
By: samstone
Please read my postings. I have repeated this a couple of times.
You can do three things,
1) After the initial installation you can change collations of the tables and fields manually one by one.
2) Set the my.conf for the MySQL to create tables in UTF-8 by default before installation. This way you don't have to change the collations one by one. However, this requires that you have access to the server root to edit the my.conf. If your site is hosted by someone else, you are not likely be given the rights to do that. If you have access to my.conf, the following is my setting:
[mysqld]
set-variable = max_connections=500
safe-show-database
character-set-server=utf8
[client]
default-character-set=utf8
or
3) Upgrade to MySQL5 and PHP5. Again this requires that you own the server. If you are on a hosted site, transfer to another host that provide MySQL5 and PHP5 with UTF-8 set as default.
Peace!
Sam
Re: Unicode
Posted on: 03/08/08 11:17am
By: trampoline
Hi Sam I did see your post...
I asked a question...
I run my server and could upgrade SQL to 5 possibly..
I have php5 already and have had for a long time...
Your point 3
Just by upgrading to sql 5 will this actually resolve the issue ?
Your poit 2
my.conf ... is this the sql conf file ?
I will alter it as suggested I assume this is OK to do with sql 4 ?
Your point 1
I did alter the gl_stories table manually do I need to do the field as well ??
I did not realise I will do that then...
Re: Unicode
Posted on: 03/08/08 12:02pm
By: samstone
Quote by: trampolineHi Sam I did see your post...
I asked a question...
I run my server and could upgrade SQL to 5 possibly..
I have php5 already and have had for a long time...
Your point 3
Just by upgrading to sql 5 will this actually resolve the issue ?
Your poit 2
my.conf ... is this the sql conf file ?
I will alter it as suggested I assume this is OK to do with sql 4 ?
Your point 1
I did alter the gl_stories table manually do I need to do the field as well ??
I did not realise I will do that then...
I haven't upgraded to SQL5 myself, but I read on their website that it works smoother with utf-8 than the v.4, in fact it's utf-8 by default as far as I remember what I read. I am about to upgrade when I can afford the unforseen downtime. Upgrading to it would solve the problem permanently.
Yes, my.conf is MySQL config file. It's usually in the /etc directory. This would my MySQL4 to use utf-8 by default. In my case when I changed it, it broke the other none utf-8 sites, so I had to revserse it back. What I did was I changed the my.conf and installed the GL, and changed the my.conf back after the GL installation so that others sites wouldn't be affected. Now I have upgraded all other sites, so I can keep this configuration. In your case, if you don't have other sites to worry about, you can keep this configuration for good.
Yes, you need to change the collations of the fields or it wouldn't take effect. But, since you have access to my.conf, I would just delete the current database and reinstall it with a new my.conf setting so that they will automatically be set to utf-8.
Peace!
Sam
Re: Unicode
Posted on: 03/08/08 12:06pm
By: trampoline
So If I delete the database (I guess after saving it)
Then add the code you suggest into my.conf
then import the data again into the new database that will just sort it all out ???
Re: Unicode
Posted on: 03/08/08 01:58pm
By: samstone
Quote by: trampolineSo If I delete the database (I guess after saving it)
Then add the code you suggest into my.conf
then import the data again into the new database that will just sort it all out ???
After the second step you need to install GL, which will create the tables and fields in utf-8.
Then import the data, but by data it must only be the data, not including the table structure because if you import the tables and fields along with data, it will replace the new tables and fields and you are back to square 1.
Peace!
Sam
Re: Unicode
Posted on: 03/08/08 05:24pm
By: trampoline
Quote by: samstoneQuote by: trampolineSo If I delete the database (I guess after saving it)
Then add the code you suggest into my.conf
then import the data again into the new database that will just sort it all out ???
After the second step you need to install GL, which will create the tables and fields in utf-8.
Then import the data, but by data it must only be the data, not including the table structure because if you import the tables and fields along with data, it will replace the new tables and fields and you are back to square 1.
Peace!
Sam
I think I am unfamiliar with how to export just data, how do you do that ?
I have SQL Administrator but that is it...
Re: Unicode
Posted on: 03/10/08 01:42am
By: samstone
What's your server control panel? Is it CPanel? Do you have a PHPmyAdmin?
Sam
Re: Unicode
Posted on: 03/10/08 05:25am
By: trampoline
I have OS X Server 10.4.11
SQL Administrator
Navicat
I could install PHPMYAdmin
Re: Unicode
Posted on: 03/10/08 09:49am
By: samstone
I don't know about SQL Administrator, but PHPmyAdmin would let you easily dump the data the way you want it, or you have to use the commandline.
Sam
Re: Unicode
Posted on: 04/02/08 11:01am
By: trampoline
OK I have done it,
It took 20 minutes..
I used Mark's tutorial
at http://www.gllabs.org/wiki/doku.php?id=geeklog:utf-8
I downloaded ICONV with Darwin ports on my OSX machine.
I Used BBedit to change the Latin to utf-8 encoding (96) occurances in the text file.
Verified the output with ICONV
Uploaded the result to the new database
Changed the database that the site uses...
Done...
Thanks Everyone...