Windows Server Backup - how can I haz notifications?



  • So, TIL (thankfully "only" via the Exchange server's drives filling with logs) that my Windows Server Backups had been failing for quite a while due to the iSCSI target drives going offline. Best I can determine after the fact, the outage coincides with some maintenance done on the iSCSI server around the time the backups stopped working.

    Bearing in mind that I'm obviously the one at root fault for not paying closer attention to my backups... does anyone have some suggestions for how to get better notifications (especially on backup failure) from WSB? I thought I had e-mail reports working with this PowerShell script running on a triggered task (On event - Log: Microsoft-Windows-Backup, Source: Microsoft-Windows-Backup, Event ID: 4 or 5)

    # Set up variables
    $SmtpServer = sanitized 
    $SmtpPort = sanitized
    $FromEmail = sanitized
    $ToEmail = sanitized
    $OutputFile = sanitized
    
    # Create backup results file
    $LastBackup = (Get-Date (Get-WBSummary | select "LastBackupTime" | ft -hide | out-string)) - (New-TimeSpan -Minutes 5)
    Get-WinEvent -LogName "Microsoft-Windows-Backup" | Where {$_.timecreated -ge $LastBackup} | Sort-Object TimeCreated | Format-Table TimeCreated, Message -AutoSize -Wrap
    echo $LastBackup
    
    # Construct and send email
    $Msg = new-object Net.Mail.MailMessage
    
    $smtp = New-Object System.Net.Mail.SmtpClient($SMTPServer, $SMTPPort);
    $computer = gc env:computername
    $Msg.From = $FromEmail
    $Msg.To.Add($ToEmail)
    $Msg.Subject = "## Backup Complete on " +$Computer +" ##"
    $Msg.Body = "The backup for " +$Computer+" has completed. Please see below for backup results.`n`n" + $(Get-Content $OutputFile | Out-String)
    #$Smtp.Send($Msg)# Set up variables
    $SmtpServer = *sanitized*
    $SmtpPort = *sanitized*
    $FromEmail = *sanitized*
    $ToEmail = *sanitized*
    $OutputFile = *sanitized*
    
    # Create backup results file
    $LastBackup = (Get-Date (Get-WBSummary | select "LastBackupTime" | ft -hide | out-string)) - (New-TimeSpan -Minutes 5)
    Get-WinEvent -LogName "Microsoft-Windows-Backup" | Where {$_.timecreated -ge $LastBackup} | Sort-Object TimeCreated | Format-Table TimeCreated, Message -AutoSize -Wrap
    
    # Construct and send email
    $Msg = new-object Net.Mail.MailMessage
    
    $smtp = New-Object System.Net.Mail.SmtpClient($SMTPServer, $SMTPPort);
    $computer = gc env:computername
    $Msg.From = $FromEmail
    $Msg.To.Add($ToEmail)
    $Msg.Subject = "## Backup Complete on " +$Computer +" ##"
    $Msg.Body = "The backup for " +$Computer+" has completed. Please see below for backup results.`n`n" + $(Get-Content $OutputFile | Out-String)
    $Smtp.Send($Msg)
    

    So, if someone can save me from yet another long day of fruitless searching The Googles for the right way to do this, I'll give you a dozen 🍪

    (Alternatively, suggestions for solutions that provide better e-mail notification and don't cost the $700+ per server of Backup Exec are also welcome)


  • Grade A Premium Asshole

    I have not really looked over the script, but what was the issue that you had? Was it sending an email that said backups were fine? Or were you not getting an email at all?



  • In this particular case, we didn't get any e-mail at all (so, again, obviously, I should have been checking). The task on the Windows Servers that should fire the PS script shows a last run date of the last time the backup worked.

    And apparently I can't look back through old notifications because those are sorted into a folder that has auto-pruned since the last time it worked, but my recollection is that even when the e-mailing was working, we were having problems where the notification e-mail was only showing the "backup history" log up to the last date I fiddled with and manually ran the PS script on the server. It would never update to actually show the most recent backups.



  • So, apparently, the root cause of me not getting any e-mails for this failure is that there's one event ID for success and about eleventy billion for failure, and the blog post I originally went off of didn't really mention that (and/or I missed it, but I choose to blame a random stranger from the Internets).

    So, I have tentative hope I'll get some sort of message on the next failure.


  • Grade A Premium Asshole

    @izzion said:

    In this particular case, we didn't get any e-mail at all

    There ya go. If you get the email when it is successful, just keep an eye out for that. You really need to be checking in on the server pretty regularly anyway. We do not have any notifications setup for any of the servers that we are contracted to maintain. We login to each of them on a schedule and scan the event logs, check the backup status and where possible and applicable we will verify the integrity of the backups. A backup that is not tested may not be a backup at all.



  • I'm not by any means a PowerShell expert, but looking at your script, you seem to have a large number of parameters which are all set to the value "sanitized". If I had to start debugging the script, I'd probably start from there...



  • Actually, it's a new discofeature that sanitizes passwords and other sensitive information. I can see it the way I entered it, but all you see is "sanitized". It even works for your Discourse forum password, mine is sanitized



  • The hunter2 thread is ⬆ ↩ ↔ that way.


  • Java Dev

    Is it? I think mine was sanitized, but I'm not sure - discourse keeps me logged in. Could also have been 2hunter or hunter3.



  • Looks like you can maybe do everything with just a scheduled Windows task. However filtering on Windows Event logs is really limited because it only supports a small subset of XPath.



  • @hungrier said:

    The ******* thread is ⬆ ↩ ↔ that way.

    The Belgium thread?



  • I guess part of my original goal was to get the actual logs via e-mail (ala how BE typically notifies), but given that that isn't working, I suppose at least getting a generic "success" or "fail" notice would be an improvement ^^

    Last night's messages came through with the backup log entries from May 1. Whoo.


Log in to reply