1
trabis
Migration script for non utf8 sites
  • 2011/9/14 21:31

  • trabis

  • Core Developer

  • Posts: 2269

  • Since: 2006/9/1 1


Hi,

I need help testing a migration script that changes your database/tables/fields charset and collation.

File can be dowloaded here:
http://code.google.com/p/xuups/downloads/detail?name=dbupdate.zip

Don't use this script on a live site!!!

Place the dbupdate.php on your XOOPS root (same directory as mainfile.php);

Edit the file and change the config settings(if needed):
//START CONFIG
$old_charset 'latin1';
$old_collation 'latin1_general_ci';

$new_charset 'utf8';
$new_collation 'utf8_general_ci';
//END CONFIG


Point your browser to dbupdate.php and execute it

You should get a page with SQL statements.
Open phpmyadmin and select your XOOPS database.
Open SQL tab and enter the statements you got from dbupdate.php
Execute.
Wait, may take some time if database is to big.
Now your database should have been migrated.
Edit your mainfile.php(xoops < 2.5) or xoops_data/data/secure.php(xoops > 2.5) and define XOOPS_DB_CHARSET accordingly
Run your xoops site and see if modules content is ok(news items, forum posts, etc).
Your language files will have to be updated to the new charset also. Ignore those errors.

Report back, please.

Ps:
Migration tips were taken from this page:
http://codex.wordpress.org/Converting_Database_Character_Sets

I've tested migration from utf8 to latin1 and then back from latin1 to utf8 and it worked ok for me.





2
deka87
Re: Migration script for non utf8 sites
  • 2011/9/15 5:03

  • deka87

  • Friend of XOOPS

  • Posts: 1125

  • Since: 2007/10/5


hm, I guess we've already had one like a year ago. at least i had one
Mind anchors

3
Mamba
Re: Migration script for non utf8 sites
  • 2011/9/15 18:37

  • Mamba

  • Moderator

  • Posts: 11366

  • Since: 2004/4/23


Can you share a link to your code?
Support XOOPS => DONATE
Use 2.5.10 | Docs | Modules | Bugs

4
trabis
Re: Migration script for non utf8 sites
  • 2011/9/15 19:32

  • trabis

  • Core Developer

  • Posts: 2269

  • Since: 2006/9/1 1


Quote:

deka87 wrote:
hm, I guess we've already had one like a year ago. at least i had one


Was it this one?´
https://xoops.org/modules/newbb/viewtopic.php?post_id=321964#forumpost321964

That script does not change collation on the fields, only changes database and tables. Look at your table fields and see if the collation was modified to uft8.

The one I propose is a merge of the core upgrade script and the suggestions given by wordpress. There is an open bug on sourceforge stating that the core upgrade script is not finished/implemented, that is why I'm working on this.

5
trabis
Re: Migration script for non utf8 sites
  • 2011/9/16 22:24

  • trabis

  • Core Developer

  • Posts: 2269

  • Since: 2006/9/1 1


Script did not worked good if I converted latin1 to utf8. Only worked good if I converted utf8 to latin1 and then back to utf8

I fixed the script and added one extra config:
http://code.google.com/p/xuups/downloads/detail?name=dbupdate_1.1.zip

I also noticed that my instructions were incorrect, please test it as follows:

Place the dbupdate.php on your XOOPS root (same directory as mainfile.php);

Edit the file and change the config settings(if needed):
//START CONFIG
$doBinary false//If updated strings are not readable, try set this to 'true'

$old_charset 'latin1';
$old_collation 'latin1_swedish_ci';

$new_charset 'utf8';
$new_collation 'utf8_general_ci';
//END CONFIG


Point your browser to dbupdate.php and execute it

You should get a page with SQL statements.
Open phpmyadmin and select your XOOPS database.
Open SQL tab and enter the statements you got from dbupdate.php
Execute.
Wait, may take some time if database is to big.
Now your database should have been migrated. Look at your tables and check if fields content are readable.
Edit your mainfile.php(xoops < 2.5) or xoops_data/data/secure.php(xoops > 2.5) and define XOOPS_DB_CHARSET accordingly
Your language files will have to be updated to the new charset also. If you don't have utf8 language files, change your language to english.
See if content is readable.

If not, upload your database backup, set option $doBinary = true, and repeat the steps.

Report back, please.

6
Mamba
Re: Migration script for non utf8 sites
  • 2011/9/16 22:38

  • Mamba

  • Moderator

  • Posts: 11366

  • Since: 2004/4/23


Trabis, could you provide test tables with non-UTF content, that we could use for testing?

It would be easier for me to just install test tables and test it, instead of trying to create non-UTF content

But I'll do it anyway
Support XOOPS => DONATE
Use 2.5.10 | Docs | Modules | Bugs

7
Anonymous
Re: Migration script for non utf8 sites
  • 2011/9/17 10:44

  • Anonymous

  • Posts: 0

  • Since:


Great work! .. anyone tested it with xoops 2.4.5 ?

8
Tarik
Re: Migration script for non utf8 sites
  • 2011/9/17 20:12

  • Tarik

  • Not too shy to talk

  • Posts: 170

  • Since: 2010/2/3 1


lately i have converted a database encoding converter to a sort of module o.O
i'll translate it and upload it.
Some people like what you do,
-- some people hate what you do,
---- but most people simply don’t give a damn.

9
Tarik
Re: Migration script for non utf8 sites
  • 2011/9/17 20:34

  • Tarik

  • Not too shy to talk

  • Posts: 170

  • Since: 2010/2/3 1


here is a link to the module
https://github.com/Xo-Aminoss/Module_xotools
basically what it do is to create a new database with the new encoding (empty) and clone the data from the origin to the new and converting them this way you have the old as backup and it can at the end check if there is any data loss
Some people like what you do,
-- some people hate what you do,
---- but most people simply don’t give a damn.

10
trabis
Re: Migration script for non utf8 sites
  • 2011/9/17 21:01

  • trabis

  • Core Developer

  • Posts: 2269

  • Since: 2006/9/1 1


Quote:

Tarik wrote:
here is a link to the module
https://github.com/Xo-Aminoss/Module_xotools
basically what it do is to create a new database with the new encoding (empty) and clone the data from the origin to the new and converting them this way you have the old as backup and it can at the end check if there is any data loss


Awesome Tarik! Will try :)

Login

Who's Online

222 user(s) are online (151 user(s) are browsing Support Forums)


Members: 0


Guests: 222


more...

Donat-O-Meter

Stats
Goal: $100.00
Due Date: Apr 30
Gross Amount: $0.00
Net Balance: $0.00
Left to go: $100.00
Make donations with PayPal!

Latest GitHub Commits