You can check your dedicated/vps server for open dns relay by following below methods. It tells you if your server allows dns recursion or not. Follow these steps:
Login to your server via ssh with root login details
Execute following commands:
dig @yourServerIP google.com
if it returns ANSWER:0 that means there is no recursion allowed on the server. Otherwise recursion is allowed on the server.
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 37803 ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
Above output shows NO RECURSION (ANSWER: 0)