Objective: Research of catheter-based polarization sensitive optical coherence tomography (PS-OCT) is a challenging field. In this paper, we present a new polarization determination method, similar Mueller matrix (SMM) method, for a catheter-based PS-OCT system using a standard clinical catheter probe with an outer diameter of 0.9 mm. Methods: The SMM method can remove the diattenuation and depolarization compositions by polar decomposition. By constructing the similarity between the measured Mueller matrices and sample matrices, the phase retardance of the sample can be determined from the trace of the measured matrices. Results: In the experiments, we find that images processed by the SMM method without any averaging or phase correction have a better polarization contrast of multiple biological tissues than those by the Jones matrix based method. We also preliminarily achieve phase retardance imaging of the ex vivo porcine cardiac blood vessel. Conclusion: Compared with the Jones matrix based method, the presented SMM method can provide a more robust birefringence imaging of biological tissues under low signal-to-noise ratio, depolarization, diattenuation, and phase instability. Significance: The SMM method has a potential to become a widely accepted polarization determination method for catheter-based PS-OCT.