7
This works fine (just tested on Ubuntu 18.04):
git clone https://github.com/XOOPS/XoopsCore.git yourdir
cd yourdir
cp composer.json.dist composer.json
./xoops_lib/composer.phar self-update
./xoops_lib/composer.phar install
You should not be in
xoops_lib when you run composer install. Also, XoopsCore does not contain a composer.lock file, so you might have better results starting from a fresh clone.
Installing composer globally is preferred over depending on the included composer.phar, but for now it will work if updated.
The message
"Endroid Installer did not detect a compatible project type" is just informational. It is telling you that it didn't find a Symfony project to add itself to, which is as it should be.
For now, XoopsCore is targeted to developers only, and it is expected that developers have or will build their own proficiency with composer.