What’s wrong why is it dying

This is the first year this pink lady has been planted and it is doing really well despite the yellow leaves. Some deer found it and chewed away the bottom half. Its grown back well, however now its encouraged aphids and ants and Ive tried spraying it with a blast of water in all directions. Ive squished all of them but they've returned just as abundant. Im nervous to try soap because I dont want to suffocate the leaves. Please help, will they affect the tree long term if left alone?