Hi Sajal,
did you already contact your basis-guys to trace the performance of the ADS itself?
Sounds to me, that the connection of the portal is not that good and maybe this is one of the problems.
Also check on the interface to the form. what takes the time... the driver-program fetching the data or even the form itself.
Additional to that you should have a look inside the form and see how much scripting is inside. Sometimes there is a lot of unnecessary source inside and out of that, you didn't share the form (if it is a SAP-delivery) I cannot get more in detail with my answer here.
Hope it gives you a clue where to start with your journey.
~Florian
PS: If you use the search with keywords "ADS + trace" you find a lot of useful information.