When you delete an attachment, it still shows in the ask comments, so deleting it only removes it from the attachment area and not from the task completely.
This is not a bug but by design.
Task activity is supposed to keep record of all the changes in the task. And hence the file would stay as a history in the task activity.
But I can still click and access this attachment which is a problem if you attach something that is incorrect. This can lead to mistakes since the wrong data is still accessible. I fully understand the need to show action history and I see no problem with keep a message showing the file was attached then deleted, but if deleted, it should no longer be accessible.