Solution
I shall take a numerical example because then I can explain both the principles and how the working out goes.
Suppose the position vector of P from the origin O is OP = p = (2, -1, 2).
Suppose that the vector equation of line L is   r = (-1, 0, 7) + t (4, 1, -2).
The shortest distance of P from L is given by the length of the perpendicular from P to L. Suppose this perpendicular meets L at H. Then we want to find the length of PH.
There are two ways we can set about this.

Method (1)   Using the dot product

Since H lies on L we can say that   OH = h = (-1, 0, 7) + t (4, 1, -2) = (-1 + 4t, t, 7-2t)
for some value of t which we need to find.
Also, vector PH = -p + h = - (2, -1, 2) + (-1 + 4t, t, 7 - 2t) = (-3 + 4t, 1 + t, 5 - 2t).
But PH is perpendicular to the direction vector (4, 1, -2) of line L.
So the dot product of vector PH and (4, 1, -2) is zero.
So (-3 + 4t, 1 + t, 5 - 2t).(4, 1, -2) = -12 + 16t + 1 + t - 10 + 4t = - 21 + 21t = 0   so   t = 1.
Therefore OH = h = (3, 1, 5) and vector PH = -p + h = (1, 2, 3).
Its length |OH| is given by the square root of (1 + 4 + 9) = 3.74 to 2 sf.

Method (2)   Using the equation of a plane.

Just as we found in method (1), we have h = (-1 + 4t, t, 7-2t).
Now PH lies in a plane which is perpendicular to line L. So the direction vector (4, 1, -2) of L is perpendicular to this plane. Therefore it is a normal vector to the plane. Also, P lies in the plane.
Using the equation of a plane of r.n = p.n, with r = (x, y, z), we get
(x, y, z).(4, 1, -2) = (2, -1, 2).(4, 1, -2)   so   4x + y - 2z = 3.
But H also lies in this plane so we can say

return to the list of problems page