In order to copy files through scp when the remote server use a non standard port for ssh, use the -P switch. Do not use the ssh lower key -p switch, that is reserved for “preserve” which keeps the file timestamps and permissions.

Example:

scp -P 8080 {user}@{host}:/var/www/mysite mysitelocal