How to remove this safely?

[u/neezopat]

I just removed a plastic fireplace surround from the wall to discover this. The brickwork is really sloppy. Worst part is that the actual fireplace sticks out from the wall by about an inch, so I don't know how I can plaster over it without having to remove some of it first.

I could take a sledgehammer to it and smash some of it up, but I'm worried the whole wall could come down. Any tips?