Is there a maximum amount of user groups that XOOPS will allow?

I believe the only restraints you have is that set by your server/hosting spec.

I have one site with tons of groups, I have different level admins, and different level subscriptions.

So the short answer I believe is no, there are no maximum ammounts of user groups.

EDITED:
Simple look at MySQL shows:
explain dbname_groups;
+-------------+----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+----------------------+------+-----+---------+----------------+
| groupid | smallint(5) unsigned | NO | PRI | NULL | auto_increment |
| name | varchar(50) | NO | | | |
| description | text | NO | | | |
| group_type | varchar(10) | NO | MUL | | |
+-------------+----------------------+------+-----+---------+----------------+

Group ID is a smallint, the number range is -32768 32767. However it's unsigned therefore the shift is done from negative numbers towards positive side. Therefore without problems you can have at least 65535 groups.

I can't imagine having that many groups.