Telegram keeps your previous profile pictures in a separate album, visible to everyone.

You can see them by opening the settings panel, and clicking/touching your current profile picture there. From this album you can delete them individually.

To me this is kind of a hidden UI. Maybe it's common knowledge, but I decided to post just in case someone's not aware of this feature, and doesn't want an old picture publicly visible.