mental ray is a renderer developed by mental images >
http://www.mentalimages.com <, a company from Berlin, Germany.
It indeed is very expensive when purchased seperately (their standalone renderer for instance) but it is now integrated in maya as it is in XSI, or at least it will be in maya 5, to be released sometimes in may. Right now (maya 4.5) it can't render SubD's, paint effects or particles, but it will in 5.
There's also a mental ray for 3Ds max available (about 5 k I believe).
If you want to see some great VFX rendered with mental ray, rent FightClub and Panic Room.
I might add that mental ray finally introduces radiosity and HDRI to maya, some features which are long part of LightWave...