2
*Edit: Ooops sorry, I've misread. In any case, I hope you're not talking about giving everyone ftp access! If you do, they should not be able to delete/change permissions on directories for which they're not the owner, should they? Original post follows:*
Depends a little on your servers setup, but if they can up, then they can also delete. From the point of view of file permissions.
Now, if they can upload only through some php script, such as a XOOPS gallery, then it's probably the user who runs the webserver who has the appropriate permissions and ownership, not the world or the person who's looking at the gallery through a browser. Meaning that your visitors can only delete if the php script provides that option to them. They hopefully don't have ftp access anyway.
In any case: It's not about chmod, but about tuning the permissions for your gallery, which, I assume, integrates with xoops. If you're talking about ftp, then it's a different story.