Not sure if this counts as a module - but how on earth do I install this? For a beginner there's surprisingly little helpful guidance as most of it assume technical knowledge of terms and things that I don't understand. Does anyone have a straightforward step-by-step guide for installing it?

It is a graphical library for the Apache/PHP environment. Installation of it is reserved for web servers adminstrators.
Why do you need it?
Most modules (and captcha) can work with other more ubiquitous libraries as GD2.

It's for a friend who wants it, and we have the web server admin stuff, but we just don't know how to do it. He said something about thumbnails not being made in proper colour in GD/GD2 or something like that?

For the install, you better ask these questions on the ImageMagick website.

About which module and version are you talking? Making thumbnails is by definition a lossy conversion. You loose detail and colours are a generalisation of the compressed pixels. There are several methods to obtain the result and much is also dependend on which the module developer has choosen. The result may also vary depending on the kind of the original graphic (colours, brightness, contrast). Also the ratio for sizes (go for round numbers) is a factor.
I'm not sure if changing graphic library will always help.