Hi All,
My next couple of blogs will be a series of blogs where I will be explaining on how to use PowerShell for the monitoring of critical infrastructure. I will be releasing a blog every day that will touch on how to monitor specific software components, but also network devices from Ubiquity, third-party API’s and Office365. I will also be showing how you can integrate this monitoring in current RMM packages such as Solarwinds N-Central, Solarwinds RMM MSP and even include the required files to import the monitoring set directly into your system.
Requirements:
- (Optional) ShadowProtect SPX
- (Optional) IASO Backup
- (Optional) BackupExec 12 or higher
- PowerShell v3 or higher
Creating the monitoring sets:
This blog will contain multiple monitoring sets for multiple backup sets. For some of these we’re using event-log viewing to check when the last backup happened and what the state was, for others we’re using the included module the backup supplier has given.
Monitoring ShadowProtect Jobs
Unfortunately ShadowProtect is one of those products that do not truly believe in the power of automation, but I’ve found that you can monitor the jobs simply by filtering for the eventID related to the ShadowProtect jobs. The message always contains “Warning”, “Error” or “failed”
1
2
3
4
5
6
7
8
9
10
11
12
|
try{
$ShadowProtectLog = Get-EventLog -logname Application | Where-Object {(Get-Date $_.TimeWritten) -gt ((Get-Date).AddHours(-24))}
} catch {
$ScriptError = "Query Failed: $($_.Exception.Message)"
}
foreach($logentry in $ShadowProtectLog){
if($logentry.Message -match "Warning"){ $ShadowProtectStatus += "Backup at $($logentry.TimeGenerated) has a warning"}
if($logentry.Message -match "Error"){ $ShadowProtectStatus += "Backup at $($logentry.TimeGenerated) has an error"}
if($logentry.Message -match "Failed"){ $ShadowProtectStatus += "Backup at $($logentry.TimeGenerated) has failed"}
}
if (!$ShadowProtectStatus) { $ShadowProtectStatus = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }
|
Monitoring IASO/MAX-Backup/MSP-Backup
IASO/MAX/MSP-Backup suffer from pretty much the same fault as ShadowProtect, but they do give us much better presentable logs. We only have to look for 2 states in the logs they create to check the backup status “[E]” which stands for backup errors and warning, and the string “NotStarted” which is logged when VSS errors are found.
1
2
3
4
5
6
7
8
9
10
11
12
|
try{
$TodayFormatted = Get-Date -format yyyy_M_d
$IASOLog = get-content "C:\ProgramData\MXB\Backup Manager\logs\BackupFP\BackupFP_$($TodayFormatted).log"
} catch {
$ScriptError = "Query Failed: $($_.Exception.Message)"
}
foreach($line in $IASOLog){
if($line -match "NotStarted") {$IASOStatus += "'n$($line)"}
if($line -match "[e]") {$IASOStatus += "'n$($line)"}
}
if (!$IASOStatus) { $IASOStatus = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }
|
Monitoring BackupExec Backups
We rarely use backupexec anymore, but I know some MSP’s stuggle with monitoring it. The following script uses the “BEMCLI” which is the PowerShell suite given to us by Symantec. The upside about this monitoring method is that the alerts only get cleared after you acknowledge them from the BackupExec console. To do this you open the BackupExec console and click in the bottom left of the status bar to view/clear the alerts.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
#Setting default CLI
import-module bemcli
#Getting Alerts
$Alerts = Get-BEAlert
#Looping through the alerts and setting them.
foreach($Alert in $Alerts){
switch ($Alert.Category)
{
JobWarning{$JobWarning = "TRUE - $($Alert.Message)" }
JobFailure{$JobFailure = "TRUE - $($Alert.Message)"}
JobCancellation{$JobCancellation = "TRUE - $($Alert.Message)" }
CatalogError{$CatalogError = "TRUE - $($Alert.Message)"}
SoftwareUpdateWarning{$SoftwareUpdateWarning = "TRUE - $($Alert.Message)"}
SoftwareUpdateError{$SoftwareUpdateError = "TRUE - $($Alert.Message)" }
DatabaseMaintenanceFailure{$DatabaseMaintenanceFailure = "TRUE - $($Alert.Message)"}
IdrCopyFailed{$IdrCopyFailed = "TRUE - $($Alert.Message)"}
BackupJobContainsNoData{$BackupJobContainsNoData = "TRUE - $($Alert.Message)" }
JobCompletedWithExceptions{$JobCompletedWithExceptions = "TRUE - $($Alert.Message)"}
JobStart{$JobStart = "TRUE - $($Alert.Message)"}
ServiceStart{$ServiceStart = "TRUE - $($Alert.Message)"}
ServiceStop{$ServiceStop = "TRUE - $($Alert.Message)"}
DeviceError{$DeviceError = "TRUE - $($Alert.Message)"}
DeviceWarning{$DeviceWarning = "TRUE - $($Alert.Message)"}
DeviceIntervention{$DeviceIntervention = "TRUE - $($Alert.Message)"}
MediaError{$MediaError = "TRUE - $($Alert.Message)"}
MediaWarning{$MediaWarning = "TRUE - $($Alert.Message)" }
MediaIntervention{$MediaIntervention = "TRUE - $($Alert.Message)"}
MediaInsert{$MediaInsert = "TRUE - $($Alert.Message)"}
MediaOverwrite{$MediaOverwrite = "TRUE - $($Alert.Message)"}
MediaRemove{$MediaRemove = "TRUE - $($Alert.Message)"}
LibraryInsert{$LibraryInsert = "TRUE - $($Alert.Message)"}
TapeAlertWarning{$TapeAlertWarning = "TRUE - $($Alert.Message)"}
TapeAlertError{$TapeAlertError = "TRUE - $($Alert.Message)" }
IdrFullBackupSuccessWarning{$IdrFullBackupSuccessWarning = "TRUE - $($Alert.Message)"}
LicenseAndMaintenanceWarning{$LicenseAndMaintenanceWarning = "TRUE - $($Alert.Message)"}
default{$OtherErr = "TRUE - $($Alert.Message)" }
}
}
|
Monitoring other backup software & VSS Errors
Most backup application these days use VSS to create snapshots and we can monitor these to check if the snapshots have succes status or failures. The same issue as before occurs – Microsoft does not have a PowerShell VSS module or log that we can capture so we’ll have to use the eventlog that registers VSS snapshots
1
2
3
4
5
6
7
8
9
10
11
|
try{
$VSSLog = Get-EventLog -logname Application -Source VSS | Where-Object {(Get-Date $_.TimeWritten) -gt ((Get-Date).AddHours(-24))}
} catch {
$ScriptError = "Query Failed: $($_.Exception.Message)"
}
foreach($logentry in $VSSLog){
if($logentry.EntryType -eq "Warning"){ $VSSStatus += "`nVSS Snapshot at at $($logentry.TimeGenerated) has a warning"}
if($logentry.EntryType -eq "Error"){ $VSSStatus += "`nVSS Snapshot at at $($logentry.TimeGenerated) has an error"}
}
if (!$VSSStatus) { $VSSStatus = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }
|
Hope you’ve enjoyed this blog, the next one will be about monitoring windows performance and unexpected shutdown states. 🙂
Downloads for RMM packages:
N-Central 11.0+ – BackupExec Monitoring
