In this work, we propose a tightly coupled monocular object and plane SLAM to build a more accurate, meaningful and dense map. More importantly, it demonstrates that scene understanding and SLAM can improve each other in one system. To do that, we first propose an efficient 3D object detection without shape priors and graphical layout inference without box room assumptions. Then we propose a bundle adjustment system to jointly optimize camera poses with objects and planes. They can provide additional geometric, long-term scale, and semantic constraints to improve SLAM estimation. Dynamic object movements are also modeled explicitly to achieve 4D mapping and improve the estimation. Experiments on some public TUM, KITTI and collected datasets show that the proposed algorithm can get the state-of-the-art monocular camera localization accuracy and also improve the 3D object detection.