Here is a small script to retry scp in python, i added few exceptions. You can use them or replace all of them with single generic exception.
#!/usr/bin/python def scp_Retry(self, cmd): while True: try: run = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE) logger.info('Executing: '+cmd) out, err = run.communicate() print(out,err) logger.info('SCP StdOut: '+out) if err == "": print('copied*****') break elif "Operation timed out" in err: raise RuntimeError('Connection timed out') elif "Network is unreachable" in err: raise ValueError('Network Error, con connectivity') elif "Permission denied (publickey,password,keyboard-interactive)" in err: raise BaseException('Ssh is not ready yet ...') except RuntimeError as e: logger.error("SCP StdErr: " + err) except ValueError as e: logger.error("SCP StdErr: " + err) except BaseException as e: logger.error("SCP StdErr: " + err) print('retrying') time.sleep(5) return True scp_cmd = 'scp -B -o "StrictHostKeyChecking no" -o LogLevel=ERROR -i /Users/gil/.ssh/server.pem /Users/gil/Desktop/ca.crt firstname.lastname@example.org:/home/ubuntu/' self.scp_Retry(core_reg_ca_scp)
Very important thing to keep in mind is "scp" options, reason is i worte the above scirpt when i was automation in aws cloud, where i create a vm and immediately scp few files, Since vm will not be ready before min or so, i'll keep retrying and get those different types of Excpetion.
First exception is for connection timed out, my script will start pushing files to newly created vm immediately after creating even before scp port is open, so i get "connection timed out" for few seconds.
Second error for internet connectivity, if you are running the script from a computer with out internet connectivity, then you'll see this.
Third one is very important, After scp port is ready and before vm is ready to validate the ssh key we are using. Here what happens is you'll be prompted for the password even you are using ssh key. VM will be in this state for couple of seconds or more.
-B option will avoid prompting for password at above mentioned state, advantage of using this option is you'll not be blocked prompting for password, this option will times out the connection and our script will retry again and we'll succeed at later retries.
"StrictHostKeyChecking no" This will disables Host Key Checking, so that you don't have to press "yes" while connecting to new vm for the first time.
"LogLevel=ERROR": This will disable all log messages show up while ssh/scp. Reason i use is i'm using Popen.communicate, i'm capturing stdout and stderr, to check if stderr is None or not. If i don't use this option, even successful scp, warning messages will be piped into stderr and my if condition will fail even after successful scp. Below is the warning message you see , while you do scp for the first time.
example ": Warning: Permanently added 'x.x.x.x' (ED25519) to the list of known hosts.\r\n" gil...