How do I prevent mass registrations? For example, if someone using a bot to register 5000 accounts in an hour.

You can prevent this by requiring new users to validate their email addresses.

Then they have to check their email and click the activation link to complete registration.

Admin>System>Preferences

You could also theoretically implement some kind of CAPTCHA test (write the letters from the image in the box) to require new users to go through a bot-human image validation test.

but that would take a bit of code tweaking.

GI-JOEs Protector module may help too.