OK, here's my 2 cents for this discussion.

What I see as the task of this QA team is to lead the way to preserve and improve the quality of the XOOPS modules. This needs to be implemented on several levels IMHO.

- Setting the standards
QA is a community effort too, and we should make it as open as possible. But for a good QA process, standards need to be set, and protocols as well. The QA team should be responsible in setting (and motivating) those standards.
The most difficult one to set is coding standards. They could range from using the PEAR Coding Standards for code notation, to PHPDocumentor code comments, to using the XOOPS API to it's fullest extend, to using only secure code, to keeping the code lean and fast and easy on the queries. But these require a LOT of knowledge from the coder, and a LOT of documentation from the XOOPS teams. That, I'm afraid, is still a bridge too far.
But it's not just coding standards, it's also presentation, packaging, file structure, user information etc. Only XHTML 1.0 Transitional output in the templates (the ThemeForge could play a role in this), complete predefined structure for module packages, so they install easy, documentation, language definitions, readme files, license files, changelogs, uncopyrighted images, credits to all who helped out, support website reference, bug reporting link, feedback link etc. All these should be part of a fully certified module. We need lists of Must Haves and Nice to Haves. And then we need to show everyone how to make modules according to those standards.

- Showing the way.
Along with the developers and designers, a sample module would be a good thing, IMHO. But again, this should be a community effort. Therefore I suggest a sample module project with a list of things to do and to be added, and anyone can add their bits and pieces. The QA team will supervise and help out as well.

- Providing the tools.
Third task IMHO is to provide the tools and procedures so that anyone, and I mean anyone, can perform a series of tests on a module or a piece of code and see if it works, and meets some of our standrads. These should be workable, and reporting should be easy enough to make some initial conclusions. The smoketests are the first (and best) example, the sample module is another.

- Last but definately not least: security.
I also think the QA team should have a few people dedicated and knowledgable about security issues. They should keep an eye out for important reports and issues, and dig into the system to see if XOOPS is vulnerable. The japanese community seems highly competent for this task, maybe some collaboration on this would be great.

Like I said, my 2 cts, but it's not just random thoughts. I have thought this thru before, but it's nothing final yet. Let us know what you think!

Herko

I would like to ad to the list of Marco's

I use some steps as you probably do when I test/debug modules. It's a quick contribution to our smoketest methodology :
Light explanation :
0/ Validate xoops_version file
>> 0a/ Instalation Help / Readme are included etc

1a/ Test module installation
1b/ Test module upgrade (refresh in admin)
1c/ Test update script (i.e no data are lost)
2/ Play with module under debug mode
- Test admin configuration
>> with a scale of user usability eg: 1=Easy 2 =Medium 3=Hard

- Test use of smarty variables
- Test bloc configuration


3/ Validate translation directories
it's a quick list, I know, but it's the "methodology" I usually use... I'm shure you make other/better tests too !

4/ >> Test user interface ( with various Browsers operating systems )

>> XHTML 1.0 Transitional template output

----------------------------------------------------
Couple Question for the Module Developers

1 ) Has there been any decision made on the developer side of the CSS issue of each module having seperate Css files which means when " a basic normal user " adds a module they then have to try and find the right file to edit to get colour right .

( my 1 cents worth )
Themes and modules sort of run hand in hand in my mind so realy the impact on themes should be part of the QA .
Maybe we should use a number of standard themes wether they are the ones bundeled with XOOPS package or what the Theme Forge say are up to stardard . Then the resuls incorperated in the certification that the module compatible with X theme's .
With a scale (1)= Compatable (2)= Possible Minor Template or CSS Modification Needed .



Cheers
Peter

My 1/2 cent,

1. A good application not only need a strong core with good attachements but also needs to perform good under stress conditions. Thus while developing and coding we should also look into how much load its generating on server.

Test using tools like loa runner and winrunner has to be done to find out if XOOPS core or anyother module is suffering from memory leak, process zombies or untrapped threads.

If we could tackle those undsired outcomes then we can have better performance at high loads.

2. A standards documetnation has to be released so that while coding those standards are maintained and followed strictly.

3. Usage of stable versions for production must me communicated. This will reduce the ill feelings in newbies in relate to unperforming modules.

4. Module naming conventions has to be defined show that it carries meanings.


5. Each developer should create a set of test cases to justify that the module performs as expected. This tests should be executed by QA team to check if the module meets the given test cases.

I would like to give my input here, although it wont be very technical. Will the QA team basically be working with the module developers, i.e testing beta versions to see what improvements(bug and functionality wise) needs made and also testing production/stable modules for bugs and functionality.

Also will there be a certain level or goal of buglessness(not sure if thats a word) before a module will be made 'xoops qaulity assured'

There are a lot of modules out there, in the repository for example that are i think still considered beta, or when i have used them are very buggy and dont work correctly. As was said before it would be good to test them under diffrent circumstances, usage, etc.

Speaking from personall experiance, when i jsut got into XOOPS I downloaded a few modules that I though hey I could use this on my site this would be cool, but they turned out buggy or faulty and I dont have the knowledge to fix any code or anything like that, for many people IMO they want the fast fix, an easy way to update and maintain a website-which XOOPS is ideal for but for newbies like I once was they are finding problems with modules they want to use and are perhaps unsure on how to resolve the problem.

Basically, are we working towads the goal of having a certain standard for modules which will be released to the community as stable?

Cheers

Martyboy

Quote:The QA team is primarily for production/stable modules - the tests are for a stamp of approval, not another level of bugtesting

Quote:I would assume that this would be one of the criteria to judge from

Quote:Hopefully the QA team will be big enough to have enough resources to perform this - or arrange for others to do so and report their findings back to the QA team and the author

Quote:That's a big yes!

One more thing, and I think that if quality assurance is about the quality of the module and thus, increasing near-universal usability, then this qualifies:

Module Documentation.

I cannot tell you how often I have found myself at loggerheads with a module, because while the functionality is in there, undoubtedly, I could not seem to pry it out.

One thing that we can advocate for is documentation aimed at admins, to help them get that module up and running in a reasonable time frame. I mean doing what it is purported to do, not just installed and sitting there, looking promising.

That would be a real quality boost.

Also, and this could be a toughie: A standard on screen and operational metaphors. Cut down on the "Tower of Babel" factor.


Of course, I do understand the native language pitfalls in all of this.

QA has to watch out for wanting to create the Perfect Modules... IMHO it should use the QA certification as a positive force: this is the way to do it!, and not a negative force: this is not the way to do it!. Thus, QA should be about insiring and teaching others, to flatten the learning curve. It should be about showing the way, or at least showing how others paved the way. That will probably mean that no module is perfect, but many parts of many modules just might be close enough

Herko

Quote:

I would add here, that the code savvy group of the QA should look at the beta/rc modules to put those back on track, before being torn down by the QA process if their module doesn't follow the correct XOOPS coding practices.

You've been doing a lot of that on the dev site Mith, giving hints and help to other projects, well the QA Coding Group should do the same.

Quote:

The classification system still must be somewhat strict. I agree that you just don't want all coders to quit if the QA Group is too strict and coems out as arrogant, but you can't let everyone pass just to be nice. An excellent module will probably be very rare to achieve, but it must still remain the goal. lol

The points raised in this thread have been good ones, but the thread was supposed to be about writing smoketests. (See the initial post.)

If you want to diverge from the topic under discussion, please feel free to start a new thread. In the new thread, you can quote or link to an existing post if needed.