This paper addresses the problem of accurately and automatically recovering the epipolar geometry from an uncalibrated stereo rig and its application to the image matching problem. A robust correlation based approach that eliminates outliers is developed to produce a reliable set of corresponding high curvature points. These points are used to estimate the so-called Fundamental Matrix which is closely related to the epipolar geometry of the uncalibrated stereo rig. We show that an accurate determination of this matrix is a central problem. Using a linear criterion in the estimation of this matrix is shown to yield erroneous results. Different parametrization and non-linear criteria are then developed to take into account the specific constraints of the Fundamental Matrix providing more accurate results. Various experimental results on real images illustrates the approach.