Today I noticed a strange omission in the documentation for the GLU function
gluLookAt. As the specification says, the function is designed to place the camera at a certain point in the scene, point it at a certain other point, and roll it such that a certain given vector points upward in the view.
First, the function computes a front vector, F, by subtracting the eye point from the centre point. This normalized version of this front vector is called f and is the vector that should be mapped to the −z axis. To find the side vector, the one that maps to the x axis, the cross product between the normalized front vector and the normalized up vector UP' is computed. Both vectors have unit length, but the user may have specified an up vector that is not perpendicular to the front vector, so the result s might not have unit length. If we used it like this, then the resulting matrix would include a scale component in the x direction, resulting in a scaled scene. Hence, the side vector s has to be normalized as well, which is exactly what the Mesa source code for
gluLookAt does in line 134:
/* Side = forward x up */
cross(forward, up, side);
gluLookAt documentation does not mention this! It says “s = f × UP'”, and follows by plugging this s straight into the resulting matrix M. If you implement the algorithm precisely as stated in that manual page, like I did, you will end up with an incorrect matrix.
Note that, after computing the side vector, the ‘official’ up vector is recomputed as the cross product between the side vector and the front vector. If you did it correctly and not follow the documentation literally, both have unit length. Since these are guaranteed to be perpendicular you should end up with a unit-length up vector that does not have to be normalized afterwards; and indeed the Mesa code does not do this. But if you did follow the documentation, then your up vector will also be wrong, resulting in a scene that is scaled in the y direction as well.
I am trying to get in touch with the OpenGL people about the problem in the documentation. I'm curious to see what will come of it.