Tremendous efforts have been made on image segmentation but the mask quality is still not satisfactory. The boundaries of predicted masks are usually imprecise due to the low spatial resolution of feature maps and the imbalance problem caused by the extremely low proportion of boundary pixels. To address these issues, we propose a conceptually simple yet effective post-processing refinement framework, termed BPR, to improve the boundary quality of the prediction of any image segmentation model. Following the idea of looking closer to segment boundaries better, we extract and refine a series of small boundary patches along the predicted boundaries. The refinement is accomplished by a boundary patch refinement network at the higher resolution. The trained BPR model can be easily transferred to refine the results of other models as well. Extensive experiments show that the proposed BPR framework yields significant improvements on the semantic, instance, and panoptic segmentation tasks over a variety of baselines on the Cityscapes dataset.