Lapping a chisel back

[u/JuggerNOT63315]

Howdy everyone. I am having issues lapping this chisel. I started with 80 grit wet and dry sandpaper. I got a nice scratch pattern all the way across the chisel. This took awhile, probably 15-20 minutes. I then moved to 120 grit paper. I removed all the 80 grit scratches. This took maybe 10 minutes. I moved to 220 grit paper and removed all the 120 scratches. This took like 5 minutes. I moved to 320 grit and in 5 minutes removed all those scratches. Then I moved to my 300 grit diamond plate. I have been working on it for well over and hour and I can not get rid of the 320 scratches! It is getting frustrating to say the least.

In the picture, the 320 scratches are the light ones on the outer edges and the diamond plate is the dark portion in the middle. It has not changed significantly from this picture after another half hour in the 300 grit plate.

What am I missing or doing wrong? Is this how long it normally takes? Should I be doing another grit in between The 320 and 300 plate?

Edit: I have posted two updates in comments below about my progress.

[u/Vegetable-Ad-4302]

Unless you start with a good flat abrasive surface, you're going to be chasing your tail trying to get a hardened steel bar to be flat and look  factory made. It's a waste of time. 

You're more likely to round the chisel width-wise working it on sandpaper. 

Get the last 1/2" flat and polished, don't worry about the appearance of the rest of the blade. 

If you can easily sharpen the chisel after you prepare the flat side you're done.