Wireguard : ping from server to client doesn't work without adding endpoint in the peer section. What could be going wrong?

I have a wireguard configuration; client is a server on digital ocean and server is somewhere else. Not in digital ocean. I have the below wireguard config for server and client:

Server side:

[Interface]
Address = 10.15.15.1/24
ListenPort = 51824
PrivateKey = ********private key of server
PostUp = ip link set wg5000 master TEST1

[Peer]
PublicKey = *******pub key of client
AllowedIPs = 10.15.15.0/24
PersistentKeepalive = 25

client:

[Interface]
Address = 10.15.15.2/32
ListenPort = 51824
PrivateKey = ******privaet key of client

[Peer]
Endpoint = public_ip_of_server:51824
PublicKey = *************pub key of server
AllowedIPs = 10.15.15.0/24
PersistentKeepalive = 25

Ping from client to server: ping 10.15.15.1 times out ping from server to client: destination host unreachable.

wg show command on server:

   interface: wg5000 
    public key: ************************** 
    private key: (hidden) 
    listening port: 51824 
    peer: ********************************** 
    allowed ips: 10.15.15.0/24 
    persistent keepalive: every 30 seconds

client:

interface: wg5000   
publickey:****************************************   
private key: (hidden) 
listening port: 51824 
peer: *******************************
endpoint:public ip of server:51824   
allowed ips: 10.15.15.0/24
transfer: 0 B received, 444 B sent persistent keepalive: every 30 seconds

Can any one help me understand how to make this work without adding endpoint on peer section in server?