Spatial variations of Cr, Cu, Hg, Ni, Pb, and Zn in the surface sediments from 34 stations of the Kaohsiung coastal zone southwestern Taiwan were studied to address the current pollution status, sediment quality, and potential ecological risk. The study revealed that the concentrations of sediment metals in Kaohsiung Harbor were alarmingly high compared to the other region of Kaohsiung coast. The concentrations of Cr, Cu, Hg, Ni, Pb, and Zn in the harbor sediments were as high as 351, 247, 1.93, 61.8, 60.9, and 940 mg kg−1, respectively. The current situation of metal pollution was assessed by different pollution indices and results showed moderate to severe enrichment of Cu, Hg, and Zn in the harbor sediments. According to the degree of contamination, pollution load index, and contamination severity index, the sediments from the inner Kaohsiung Harbor show high degree and high severity of metal contamination, while the rest of Kaohsiung coastal areas show uncontaminated or low-level pollution. Results of mean ERM quotient and potential ecological risk index also indicated that the harbor sediments posed a 49% probability of biological toxicity and very high ecological risk. The toxic units indicated that the negative biological effects of the six metals in the harbor sediments were Zn > Cu > Cr > Ni > Hg > Pb. In contrast to Kaohsiung Harbor as a trap where considerable amount of anthropogenic metal loadings accumulated in sediments, low metal concentrations were observed in most Kaohsiung coastal sediments. It probably resulted from the limited fine-grained sediment deposition. In the wave-dominated Kaohsiung coastal zone, fine-grained sediments associated with polluted metals tend to be easily resuspended and transported offshore via waves and wave-induced currents. The results of this study can provide valuable information for river and coastal zone management.