Getting huge amount of latency while transfering packets between two devices connected by WiFi Direct

Hi all,
I am adding multiplayer functionality to my 2-player sports game which currently works on single player mode with AI opponent. I used Android's WifI peer to peer API to achieve this.

So far devices are getting connected, data gets transferred as expected. Only issue is that I face a huge amount of latency and since this is a real time game, this latency breaks the game completely.
I am passing touch inputs (X,Y) in packets since my model is deterministic . I am also following Gaffer on Games blog for tips and info on UDP transfer.

      As per the blog, I checked the Round Trip Timme of my packets and they tend to oscillate between 300 to 1000ms (ideally it should be below 250ms). I tried reducing the rate at which packets are sent, reduced the packet size as well (40 bytes from 400bytes) , yet just cant figure out which it still lags.

Any advice , help would be greatly appreciated.

Thanks.

Hi,
No it's udp transfer. Probably would have faced more latency had I used tcp