Most objects in GeoGebra can be deleted by a simple right-click and then select the point at all. This is also the pen tool, where you just have to right click on the pencil line and select the menu item 'Delete'.