Colorectal cancer usually appears in polyps developed from the mucosa. Carcinoma is frequently found in those polyps larger than 10mm and therefore only this kind of polyps is sent for pathology examination. In consequence, accurate estimation of a polyp size determines the surveillance interval after polypectomy. The follow up consists in a periodic colonoscopy whose frequency depends on the estimation of the size polyp. Typically, this polyp measure is achieved by examining the lesion with a calibrated endoscopy tool. However, measurement is very challenging because it must be performed during a procedure subjected to a complex mix of noise sources, namely anatomical variability, drastic illumination changes and abrupt camera movements. This work introduces a semi-automatic method that estimates a polyp size by propagating an initial manual delineation in a single frame to the whole video sequence using a spatio-temporal characterization of the lesion, during a routine endoscopic examination. The proposed approach achieved a Dice Score of 0.7 in real endoscopy video-sequences, when comparing with an expert. In addition, the method obtained a root mean square error (RMSE) of 0.87mm in videos artificially captured in a cylindric structure with spheres of known size that simulated the polyps. Finally, in real endoscopy sequences, the diameter estimation was compared with measures obtained by a group of four experts with similar experience, obtaining a RMSE of 4.7mm for a set of polyps measuring from 5 to 20mm. An ANOVA test performed for the five groups of measurements (four experts and the method) showed no significant differences (p<0.01).