How to teach machine to perceive 3D world from unlabeled videos? We will present new solution via incorporating Structure-from-Motion (SfM) into self-supervised model learning. Given RGB inputs, deep models learn to regress depth and correspondence. With the two inputs, we introduce a camera localization algorithm that searches for certified global optimal poses. However, the optimality is “half-done”. Only poses are optimized, while depth and correspondence remain unchanged. We then leave their fine-tuning as self-supervision with known poses. We introduce a technique that utilizes NeRF to triangulate pseudo-groundtruth for depth and correspondence models. The scheme is initially verified in seconds duration video. On SfM, the method significantly outperforms COLMAP in localization accuracy. On self-supervision, even with seconds duration videos, the algorithm enhances the State-of-The-Art supervised depth and correspondence models.