xoops forums

redheadedrod

Home away from home
Posted on: 2013/1/6 20:27
redheadedrod
redheadedrod (Show more)
Home away from home
Posts: 1296
Since: 2008/2/26
#1

Proposal for a "universal language" system.

Not sure why but I had a thought pop in my head a couple days ago about languages. I think I may have come up with a concept that should be relatively easy to implement and should make xoops much more powerful for language stuff.

The actual details would have to be worked out but I wanted to present this idea and see if there was any interest in it at all.

With the current system I see a major problem.

Each module needs to have its language files built from the ground up and included with the module. When the module is released the language files have to be translated by others at some point. Since MOST modules are released with an English language file this really is more important for anyone wanting to support another language on their site.

I have been kicking around the thought of how much easier it would be if you could have some sort of "universal" language library that could be used by module developers and it would be part of the system. The considerations are that if a module developer can use constants that were defined in a universal library then their module would not need translation to any of the supported languages. The problems I saw was it can be resource demanding to try to accomplish all of this. So I came up with a proposal that I think will work but needs some of the details figured out.

The idea is we have a database table setup. This table consists of basically 5 fields.
Core:
This would be a boolean field that would only tell if the entry is core or not. If it is core then it should not be changed since it is needed by the core.

Language:
Standard Xoops Language name (English, Spanish, etc)
This will determine which language a constant is for.

Constant Name:
Name of the Constant to be used

Constant Value:
Value for the Constant to be set to

Module using:
This will be an array of Boolean values. Each module would have a bit set for this. If a module uses this constant then the field will be true.


The idea is that the system then would be preloaded with the Core Language values when it is installed. You could also build preload "language packs" that cover a series of modules if you so desire.

Each module then would have a list of ALL the constants it needs so that at install time it can check against the database for those already loaded and only install constants it needs that are not already loaded. It would also set the "module using" field accordingly to show it is being used.

When a module is removed it would again look through the language definitions and compare against the database for any of those it installed that were not used by any other modules it would then remove those.


You could then add a maintenance module to the admin screen that would allow any of the Constants to be changed. This module would allow changes to be made to Core entries, Existing non-core entries and non-existing entries. You would likely only let the webmaster or main admins modify the "core" entries. You could have moderators or main admins modify any non-core existing entries and then allow most trusted users to add to the library for any non-existing entries. You could also build into the module versioning and some other cool features to make sure you have ways to roll back changes in case of malicious users. Basically this would allow an easy method to gain translations for things. This module could also list languages that need conversions etc. You could then use this module to build language packs etc.

I haven't worked out the full details yet how it would work for the module side but the idea is that the system would build the files for the module to use for the language translations. The maintenance module that allows editing of the Constants would then rebuild these files when needed. If a module developer ONLY used core constants and perhaps some language packs then they would not need to translate it. You could have the system basically just build files similar to the ones currently in existence and go that route or go some other route. The files could be built at the time the module is initially installed then again any time the constants have been changed.

I think it would work well and would allow much less work to be needed in translations.

Anonymous

Posted on: 2013/1/6 21:30
Anonymous
Anonymous (Show more)
Posts: 0
Since:
#2

Re: Proposal for a "universal language" system.

The idea sounds great, but I have some doubts. Translations are in most cases module specific, and can't be substituted by generic phrases. I also suspect there could be problems with 'generic' phrases not fitting in a local language. As we have one word for snow but Eskimos have 50...

I translated ExtGallery the past days and found out the module maker reused some language tags at more places in admin. In some cases this gave me a headache as the reuse did not fit in Dutch. As language used in a sudden context can be something completely different in another language.

trabis

Core Developer
Posted on: 2013/1/6 21:45
trabis
trabis (Show more)
Core Developer
Posts: 2268
Since: 2006/9/1 1
#3

Re: Proposal for a "universal language" system.

With the move to INI files and the extra parsing that will be required for each file, we will have to add a cache layer. We will be fetching the translations from the cache, not from the files. Between this cache layer and the file layer we can/should add a database layer.
As you propose, we can fetch the files, and save entries on db. This will allow the admin to edit the db entries and get assistance on translating entries not yet translated. This assistant can propose translations based on similar entries.
As this entries are saved/updated, cache files are generated. It is this cache files that we will be loaded when a loadLanguage() method is triggered.

redheadedrod

Home away from home
Posted on: 2013/1/6 22:21
redheadedrod
redheadedrod (Show more)
Home away from home
Posts: 1296
Since: 2008/2/26
#4

Re: Proposal for a "universal language" system.

@Flipse
As mentioned in my proposal the developer should use common phrases where possible but in cases such that you mention it would be a simple matter to change things. If you needed specialized wording in a module you could EASILY change the name of the constant in the module and use that instead. The maintenance module would need to support adding new constants for such an occasion. But you may find that it shouldn't need to. It would work the same as current modules do. Just that you have a little better access to the constants from the system and don't edit the files directly.

@Trabis
Sounds like the idea I have here would be a good fit with the new system.

Hopefully a way can be worked out to share the translations like I mentioned here. I laid out an easy way to maintain and hopefully it or something like this can fit in with what your doing so we can have a strong language system. I don't know if any other CMS is using a centralized language library like this.

irmtfan

Module Developer
Posted on: 2013/1/7 5:12
irmtfan
irmtfan (Show more)
Module Developer
Posts: 3419
Since: 2003/12/7
#5

Re: Proposal for a "universal language" system.

Yes with INI files we will have many opportunities.

1- increase/decrease number of definitions for one entry based on the local language.
Quote:

flipse wrote:
I translated ExtGallery the past days and found out the module maker reused some language tags at more places in admin. In some cases this gave me a headache as the reuse did not fit in Dutch. As language used in a sudden context can be something completely different in another language.

more examples for increase:
Quote:

DCrussader wrote:
The user is user without Gender only in English, in rest languages there are... (3 forms) "I accept ToS" - don't have other meaning then I (but for English only, when female/male/boy accepts terms).

despite the increase that you mentioned sometimes we need to decrease them
example for decrease: In english we have "He" and "She" but in persian we just need one word. as same for plural countable words in english we said "1 comment" and "X comments" but in persian both needs a same expression.

2- auto diff tools

3- theme based definitions: eg in a help desk system like xhelp: "Submit a Ticket" for theme 1 and "Send an Owl" for theme 2 (maybe Harry Potter theme)
themes/YOUR_THEME/modules/xhelp/language/YOUR_LANG

Edit:
I propose this before many times but I like to repeat:
IMO we should move every customization to "themes" folder.
Now "module templates" can be customized for each theme. we should do it for images/icons/js/css files and even translations.
the benefits are huge:
- users dont need to look at every places to do customization.
- users dont need to worry about their customized images/icons/js/css/translations with every module update. users dont need to touch core and module files (I mean "modules" folder) at all.

4- module based definitions. I mean different translations for each clone of the same module like publisher.

5- developing languages/translations tools. I mean an advance tool with all features. even it can import translated files from some outside sources like Transifex, sourceforge, ... Also can find/update the changes in the SVN.


Mamba

Moderator
Posted on: 2013/1/7 14:17
Mamba
Mamba (Show more)
Moderator
Posts: 10817
Since: 2004/4/23
#6

Re: Proposal for a "universal language" system.

Few comments:

@ redheadedrod
The beauty of the current system is that is simple, and each module is independent from another. Having an extra table to keep list of which module uses which "constant" would add extra level of complexity that would make XOOPS more open to something going wrong.
So it would be better to just offer a "dictionary" or "library" of translations, similar to "library of images". And the same way as module developer can select and use an image from a central repository, they could use, if they want, an existing translation. This would be much simpler.

And we could have a translation team to add the most popular translations to the central "dictionary", the same as we would do with images.

@ irmftan

I think, it's better to keep themes for customization related to "look & feel".

All 'localization assets", i.e. translations, images, and CSS files, should go to /languages folder.

As a user, I don't want to download my "German" files from several different folders. I want to download just one folder, copy it to one location and I am done.

As a translator, I don't want translate files in /language folder, and then go to /themes folder, and do it there.

As a designer who is working on templates in /themes, I don't want to have some language files floating around

So I think, this would be much simpler:

/language folder for all localization related files
/themes folder: for all design related files

It will also make it easier for XOOPS to find the right files. If you change the language in Preferences, it will look only for files in /language folder.

If you change a template set, it will look only in /themes folder.

I just like to keep things simple
Support XOOPS => DONATE
Use 2.5.10 | Docs | Modules | Bugs

Peekay

XOOPS is my life!
Posted on: 2013/1/7 14:55
Peekay
Peekay (Show more)
XOOPS is my life!
Posts: 2335
Since: 2004/11/20
#7

Re: Proposal for a "universal language" system.

Quote:

Mamba wrote:

And we could have a translation team to add the most popular translations to the central "dictionary", the same as we would do with images.


That may be OK for generic phrases like 'Date of Birth' but not 'Breed of Dog'. I think a central dictionary would simply create a long list of phrases that nearly mean what the developer wants to say... but not quite. If you are going to have to create one custom definition, you might as well do the whole lot.

Surely the overhead of parsing module language files must be incredibly small?.

Mamba

Moderator
Posted on: 2013/1/7 15:12
Mamba
Mamba (Show more)
Moderator
Posts: 10817
Since: 2004/4/23
#8

Re: Proposal for a "universal language" system.

Quote:
That may be OK for generic phrases like 'Date of Birth' but not 'Breed of Dog'.

Of course!

It would focus on XOOPS related generic stuff ONLY like:

- Add a file
- Are you sure to delete this file?
- Yes
- Database was updated successfully

etc.
Support XOOPS => DONATE
Use 2.5.10 | Docs | Modules | Bugs

redheadedrod

Home away from home
Posted on: 2013/1/7 15:41
redheadedrod
redheadedrod (Show more)
Home away from home
Posts: 1296
Since: 2008/2/26
#9

Re: Proposal for a "universal language" system.

Quote:

Mamba wrote:
Quote:
That may be OK for generic phrases like 'Date of Birth' but not 'Breed of Dog'.

Of course!

It would focus on XOOPS related generic stuff ONLY like:

- Add a file
- Are you sure to delete this file?
- Yes
- Database was updated successfully

etc.

Yes this would be the main focus of this idea.

@Mamba:
This would NOT add complexity at the module level. The beauty is that in those places you CAN and DO use generic phrases as quoted above it would save a lot of work. The compilation of the language file would only be done at two different times. At module installation and any time the database is updated with a new translation that would affect the module. The purpose for keeping track of what module uses what constants is to make sure you can "recompile" those words. To the user or module developer it would actually be the same or simpler. It really would only be a system thing that could be controlled totally by the admin. the maintenance module would allow for editing of the words and when someone does a translation it would be much easier than it is now.

Also, the intention of the "language packs" I mentioned above are two fold. Between core releases you may want to add more entries and you may have developers that have prefered phrases they use through out their modules. Allowing for a "Language pack" would help between releases and such. You could incorporate the language pack into the next core release very simply.

For module specific tags you could still do as people have done and use a prefix that makes the "term" specific to the module. Then it would just load the module specific entries in to the database. You need to keep track of what modules use what entries so that when you remove a module you only remove the entries that are no longer being used in case those entries are used by other modules as well.

@irmftan:
My home Language is English and I took Spanish 20+ years ago in Highschool so I am not very familiar with the intricacies of different languages so please excuse my ignorance. I wonder if having an "over ride" table would work in your case. This would be where you have another table that could contain "other" entries when the stock/generic term doesn't really apply for your application which would over ride the stock term for that module. This should hopefully handle any of those weird quirks. Anything outside of that and it really would be up to the module developer. Someone Fluent in English but aware of these differences in languages could write up a short document for developers to consider when writing modules to keep the modules as language transparent as possible. I don't think you can otherwise program for EVERY possibility. Things can otherwise get more complicated then necessary.

redheadedrod

Home away from home
Posted on: 2013/1/7 15:44
redheadedrod
redheadedrod (Show more)
Home away from home
Posts: 1296
Since: 2008/2/26
#10

Re: Proposal for a "universal language" system.

If in fact this looks like something that might be of interest I would be willing to try putting something together to show what I mean. It may take a little while and wouldn't be in time for 2.6 Beta but depending on how the new .INI system is setup it could be easily incorporated for 2.6.1 and in the mean time worked out as a plugin/extension for 2.6.0.

Rodney