Performance and Troubleshooting - Windows System Tuning

Articles based on common support questions.
Post Reply
varghesesa
Posts: 28
Joined: Thu Jul 11, 2019 9:52 pm

Performance and Troubleshooting - Windows System Tuning

Post by varghesesa » Thu Feb 18, 2021 7:35 pm

Introduction

Ok! You completed the Onboarding section. The IP Camera Connections article showed you how to leverage ONVIF to setup cameras easier, how to get your cameras connected with PTZ/Audio controls and how to optimize your CPU usage with dual streams. The Camera Stream Optimization article explained how to optimize the connection between each of your cameras.

Unfortunately, the software is still slow, crashes or is unstable. Apologies for frustration. It's challenging with Blue Iris because there are so many external dependencies that could cause the software crash. Below are a few of the many external dependencies:
  • Hardware drivers from Intel or NVidia for decoding live streams
  • Camera vendors to provide reliable streams with supported encoding.
  • Gmail or other email servers for email alerts
  • Apple / Google platform for mobile app push notifications
  • Custom integrations via http request and mqtt interfaces
All these 3rd party dependencies can lead to instability.

Windows Event Viewer
The Windows Event Viewer confirms application crashes and the cause. In the BI Help file, follow instructions in the Troubleshooting and FAQ section to examine the Windows Event Viewer. If you confirm BI is crashing, send the required information along with the ticket.

Windows issues/conflicts

Antivirus software

Your Security Software (Windows defender, McAfee, BitDefender etc) or a Security Software update is interfering with Blue Iris.
Symptoms: Performance issues, Crashes, Software will not start, Auto-update is blocked are all symptoms due to improper software security exemptions.

Below is an example of robust Software Security Exemptions for Windows Defender. Similar exemptions need to be provided for your Security Software.
  • Note BlueIris process and file is excluded
  • Note the BI folders for storage are excluded: G:\BlueIris
  • Note bvr file types are excluded
Exclusions.PNG
Exclusions.PNG (11.44 KiB) Viewed 1809 times

See Help file, Topic is "Security Software Exemption", for details associated with other vendors (McAfee, Malware Bytes etc). If you are experiencing instability, the easiest step to take is to just turn off your security software and observe over time if the issues go away. The Performance and Troubleshooting webinar is also helpful. https://www.youtube.com/c/BlueIrisSupport

Windows BitLocker is another security measure provided with Windows 10 which allows encryption of the hard drive. BI performs constant reads/writes to storage. Most users have not enabled BitLocker.

Firewall Settings

There is no way BI can talk with your cameras or the web interface or the mobile apps unless BI is allowed access to the network.

Windows allows the ability to categorize networks as public or private. My server running BI is connected directly to my switch containing all the cameras and does not move. I'm not concerned about whether I categorized my network correctly since I know it is physically safe on my private switch. Thus I keep Firewall settings simple and allow access to all networks, the public and private network as seen below.

You may have BI running on a laptop that you use at Starbucks and other networks. In those cases, labeling networks as private / public and allowing BI access to private networks is more of a concern that should be addressed.

firewall.PNG
firewall.PNG (15.8 KiB) Viewed 1315 times

BI Service Issues

Running Service as LocalSystem
While discouraged, it is possible to run a Windows service under the account "localsystem" by leaving username/password blank when selecting to run BI as a service (image below). LocalSystem may not have the access rights for certain functions. A common issue is alerts to run batch files or MQTT commands may not get executed when the service is run under the LocalSystem account. Running BI as a Windows service under the account "LocalSystem" can also cause instability because localsystem may not have the needed privileges for NAS devices or sending alerts. See Help file, Topic "Running as a Service" for details.
localsystem.PNG
localsystem.PNG (5.06 KiB) Viewed 1765 times
Follow steps below to fix:
  • Basic idea: Disable and re-enable the service to run under a Windows account instead of localsystem.
    Create a local windows account with admin privileges.
    Global settings -> Startup tab ->Run as a Windows service. Uncheck selection -> Restart BI.
    Go back to Global settings -> Startup tab -> Run as a Windows service. Re-check the selection.
    Enter your own local Windows account (w/ admin privileges) in pop-up dialog
    Restart BI once again as a service.
  • If you are using a NAS device for storage, the Windows username/password used to run the service has to be identical to the credentials on the NAS device otherwise writing to the NAS device will not work.
1057 Error
If you get the 1057 error, i.e. “Failed to install service (1057)”, the username chosen above to run the service does not exist or is invalid. Follow instructions above to create a local Windows account with admin privileges. Run Windows Services -> Right click Blue Iris Service -> Properties -> Log on tab. Specify username / password.

1069 Error
The 1069 Windows error occurs when the password associated with the Windows username associated with the service has changed. Run Windows Services -> Right click Blue Iris Service -> Properties -> Log on tab -> reset the password.
1069 error.png
1069 error.png (10.92 KiB) Viewed 1869 times

Auto-update fails because update program cannot stop the service. Following error message appears.
blueirisservice.png
blueirisservice.png (5.34 KiB) Viewed 1869 times
  • Stop the Blue Iris Service manually. From the Windows search bar, type services and bring up the Services console.
  • Select the Blue Iris service.
  • Select "Stop service"
  • Start BI.
  • Return to BI. Global settings -> Startup tab -> Run as a Windows service. Check the selection.
  • Restart BI. BI should start as a service.

Auto-update fails because update program cannot stop BI. Following error message appears. (Similar issue as above)
update error.PNG
update error.PNG (48.1 KiB) Viewed 1067 times
  • Stop Blue Iris manually.
  • Restart the machine.
  • Find where you downloaded the update or just go to blueirissoftware.com and download the latest version.
  • Install/start the latest version and you should be good to go.


Web server has incorrect IP Address
On rare occasions, for some reason Windows will start the BI service before initializing the network card/stack and attaining an IP address. This results in the BI web server binding to an improper IP address.

The only way to really know if this is happening to you is by going to the Global settings -> Web server tab. You need to check the WAN/Internet address and confirm it is indeed the WAN IP Address provided by your ISP provider. I know! This unfortunately requires a lot of networking knowledge from the novice user.

The temporary fix is to restart the service. The permanent fix is to configure Windows to delay the start of the BI service.
service_delay.PNG
service_delay.PNG (16.15 KiB) Viewed 1137 times


What has changed?
The biggest question to ask yourself when observing instability is what has changed?
  • Was there a Windows update?
  • Was there an update to your Antivirus or Firewall software?
Common causes for crashes are a Windows update caused driver instability for the hardware acceleration. Windows 10 now provides an easy way to roll back updates, System Restore. To turn on System Restore.
  • From the Windows search prompt (lower right) -> Search for "restore" -> Select "Create a restore point"
  • System Properties Dialog pops up. System Protection tab should be active. See below.
    system restore.PNG
    system restore.PNG (20.11 KiB) Viewed 1764 times
    In this example, this computer has 3 internal storage drives (C, D, G). C is the windows drive.
  • Select the C drive and select "Configure" button.
  • In the System Protection dialog, "Turn on system protection" for the C drive. (See below)
    I also specify 10% for Disk Space Usage. User discretion regarding setting a value or leaving defaults.
    system protection.PNG
    system protection.PNG (13.91 KiB) Viewed 1764 times
Once System Restore is activated, rolling back is simple.
  • Go back to System Protection tab in System Properties.
    From the Windows search prompt (lower right) -> Search for "restore" -> Select "Create a restore point"
  • Select System Restore button
  • Restore Wizard will walk you through going back on a Windows update.
    You can create restore points and you should. Simply choose the Create button routinely, e.g. monthly, when you feel the system is stable. Windows automatically does so as well with every Windows update, making going back easy.
    restore wizard.PNG
    restore wizard.PNG (20.37 KiB) Viewed 1764 times
If you are convinced the latest BI update is causing the instability, the Rolling back BI version article will provide guidance on how to go back to your previous version. After rolling back, confirm stability is restored. If so, submit a ticket following instructions below.

Software will not start

If the software does not start, the only remedy is to adjust the registry until the software does start. The Software Startup Issues article will walk you through necessary steps.

Immediate crashes
Immediate crashes are those that occur within ~5 minutes of starting the software and frequently crashes. This symptom is almost always indicative of hardware incompatibility or issues with a new camera.

ToDos:
  • Turn off hardware acceleration for all cameras. Global Settings -> Cameras tab -> Hardware accelerated decode = No.
    Does stability return? If so, was there a Windows update recently? If a Windows update is the issue, then you may need to leave hardware acceleration (HA) off until there is a Windows resolution. Or you may have to revert back to the previous Windows version.
  • Did you add a new camera? Maybe the conflict is with the hardware and that one particular camera. Simply disable the camera and turn HA back on. Does stability return?
  • Any clues in the status log?
    Go to Status -> log tab and click on open file button (bottom). Find out what BI was doing before the server restart. Copy / paste 10-15 lines before the crash in the ticket. Not necessary to send the entire file.

    Common mistake: Users send a screenshot of the Status log. See below. This provides all the events AFTER the crash. In order to find out what happened BEFORE the crash, you need to "Open file" (button at bottom), but first you have to "Save to file" (checkbox at bottom). "Save to file" saves the logs to files. Then scroll to the bottom of the file and copy/paste last 10-15 lines.
    status log restart.png
    status log restart.png (117.75 KiB) Viewed 1173 times
If you turned off HA for all cameras, now you can narrow down the problem to the Windows driver or an issue with a particular camera.
  • Turn off HA for all the cameras at the camera level by selecting NO in Camera settings -> Video tab -> Hardware decode. Make sure "Limit decoding unless required" and "Also BVR" are unselected.
  • Turn HA back on for all the cameras at the global level. Global Settings -> Cameras tab -> Hardware accelerated decode. Choose appropriate setting. HA is still off since it is turned off at the camera level from previous step.
  • Start turning HA back on for each camera (Hardware decode = Default) and see if the instability is due to a certain camera(s).


Intermittent crashes
Intermittent crashes are ones that occur over time, perhaps 2-3 days.

Does CPU/RAM utilization grow over time? Security software (Windows firewall/defender, McAfee etc) having BI file/s open for scanning is a common cause. Please double-check the BI folders and BI executable are exempted from your security software (Windows firewall/defender, McAfee etc). The Performance and Troubleshooting webinar is a great resource in addition to the Security Software Exemption section in the help file. Check out our webinars in our YouTube Support Channel when you have free time. https://youtube.com/c/BlueIrisSupport.

Continue observing status -> log for errors. An example of the logs leading to clues was when we received a ticket for frequent crashes. The user reviewed the monthly logs and noticed suddenly on 5/8, BI was receiving frequent write errors. The Status Log Error article provides details. A review of the security exemptions resolved the issue.
write error.png
write error.png (64.69 KiB) Viewed 345 times

You could also create Status alerts any time an error or warning appears in the log. (Status -> Log tab -> Status alerts button). If you tend to be near the BI server, this allows you to examine the machine near the time of errors in order to gather clues to instability prior to a possible crash.



Final thoughts
It's possible that we introduced a memory leak or adversely affected CPU usage. However, these types of errors would typically result in 100s - 1000s of tickets immediately and we would often be contacting you and all the other users to make everyone aware of the problem. So in general, while there could very well be an issue with BI on your system, we need some patience to diagnose the issue. In the interim, if the instability makes the system unusable, rolling back your BI version or Windows update may be the interim solution to get your system running again.

Identifying the root cause for crashes can be a mystery. We have known users that uninstalled/reinstalled BI, deleted and recreated cameras, reinstalled windows then BI, reinstalled phone apps, unplugged/replugged their hard drives and things miraculously started working again.

A very viable next step at this point is to simply rollback Windows using System Restore as described above. If you want to test a little bit before reinstalling Windows, you could run sfc /scannow or DISM to see if the hard drive or Windows has issues. You would need to search on YouTube for guidance. We are not Windows expert.

Windows also provides many ways to recover and restore windows in case of software failures or hardware failures. If you are running BI on a standalone machine with no other apps, you should become familiar with these capabilities. From above, you should prepare for possible problems and be ready if BI stops working.
  • Be sure to create a Windows ISO file on a thumb drive to use if and when needed
  • With Windows, you can also create a Recovery drive
  • With Windows you can also create a System Image to use for recovery
It's good to be aware of all the options, so you know the next step in case of software or hardware failures.

Status Log Errors
Review Status Log Errors - Problems and Resolutions for known problems and resolutions.

Users can create alerts (push, email, sms) when there are system log errors to help diagnose the issue. All the different types of alerts are located in Status -> Log tab. Status Alerts button. See Help for details (i.e. hit the Help button in the dialog)

Furthermore, users can also create alerts for each and every camera that is causing issues (e.g. no signal). Camera settings -> Watchdog tab has details.

Submitting ticket - Isolate the Issue
Troubleshooting and isolating the issue is key to identifying the root cause and resolving the issue. Attempt to isolate the issue by systematically disabling cameras, features, etc until there is stability. What does that mean?

Think of Blue Iris as blocks or modules that work together.
  • If you recently added a new camera, does turning it off resolve the issue?
  • If you turn off hardware acceleration for all the cameras, does stability return?
  • Can you then start turning hardware acceleration on for each camera to isolate the camera that is in conflict with your hardware acceleration?
  • If you turn off cameras from a particular vendor, does stability return?
  • Set the shield to red, i.e. let the cameras stream but no trigger, record or alert etc settings are active. Does stability return?
  • If stability does not return after turning the shield to red what else is on? This is a big clue.
    Are you streaming to YouTube or website?
    Are you streaming to your Home Automation system?
    Are you posting to a website?
    Are there inputs to BI besides cameras? Any DIO devices?
  • Is auto-cycle on? Group cycle or camera cycle? Turn it off. Does stability improve?
  • Are you uploading files to the cloud via FTP?
  • Are you constantly streaming video to a Web Interface, iPad, via mqtt or Webcasting? Does stability improve if you turn it off?
  • If stability does return after turning the shield to red, this is also a big clue. Start turning things on like activating triggers, recording and alerts per camera to isolate which cameras or settings are causing the issue.
  • If you are using Deepstack, turn Deepstack off. Does stability return?
  • Do you frequently see crashes right after common activities in the status log?
  • Do you notice crashes when someone connects remotely via the web interface or mobile apps?
One example of a user helping us find a bug was with audio playback on mobile devices. By happenstance, he put the camera in a group and noticed audio playback started working. That turned out to be a big clue.

Another user started with a generic ticket like many just describing the issue, e.g. "constantly missing notifications". I provided the generic follow-up questions, e.g. did you check the Status -> log for errors. He took the follow-up questions seriously, rolled up his sleeves and was able to discover the "constantly missing notifications" was actually from only one camera because the BI server would crash when Deepstack was running and when an object was found. If an object was not found, the cancel alert worked fine.

In summary, we love customer support that leads to the narrowing down of the problem (usually with reproducible steps) so engineering can then determine where in the software to look and identify the problem. We also realize everyone does not have that level of curiosity or time to help. No worries. In that case, the next best step is often to report the issue and either revert back to the last stable version or wait for the next release. The BI Versioning System and Rolling back BI Version article will provide details. You may want to consider changing the BI Update setting from "All available major and minor updates" to "Critical and highly stable updates only".

The Windows crash reports are useful in confirming BI caused the issue. However, the codes usually imply BI accessed memory incorrectly. BI does so many times a second depending on how many cameras you have so the report in and of itself will never result in identifying the root cause.

Needed ticket info
  • Include support info in body of email. Global settings -> About tab -> Email support button. Copy/paste information in body of email.
  • Any clues in the status log?
    Go to Status -> log tab and click on open file button (bottom). Find out what BI was doing before the server restart. Attach the most recent logs file.
  • Details on your Server setup.
    If your setup is anything more than BI running on a Windows 10 machine, we need to know.
    Are you running BI on a VM? Details including VM vendor.
    Do you have h/w acceleration? NVidia? Intel? Other? Details.
    Are you running Deepstack?
  • Cameras setup
    How many cameras? Make/model.
    Screenshot of Status -> Cameras tab.
  • Integrations
    Any DIO inputs besides cameras? Speakers? Motion sensors?
    Any unique streaming to 3rd party destinations like websites (YouTube), home automation systems etc?
    Are you using the mobile apps or web interface? How many users?
  • Windows Information
    Include the event viewer crash info. The Troubleshooting and FAQ section in Help provides details. Copy / paste information into body of email. Any other clues seen in Event viewer? For example the 1001 error as seen below always precedes the BI crash error. If so, copy/paste information into body of email as well.
    event log.PNG
    event log.PNG (24.35 KiB) Viewed 2593 times
    This user noticed many BI Event Errors (Event ID = 1001) prior to the BI Crash (Event ID = 1000). All very important clues.
    bi event errors.PNG
    bi event errors.PNG (50.7 KiB) Viewed 1842 times
  • Any other clues or information that you may consider helpful. This is where you share details regarding isolating the issue.
    Can you reproduce the crash? If so, what steps are required to do so.
    Do the crashes occur at the same time daily?
    Anything change with the machine, Windows or your BI settings?
    Does the issue only occur with a particular camera?
    Does the issue occur when connecting via a web browser?
    Does the issue occur only when recording?
    Does the issue occur only with a push notification?
Things to double check. See if stability returns before submitting a ticket.
  • If you are running BI as a service, confirm the Windows account running the service has admin privileges.
  • If you are using a NAS device for storage, please confirm the Windows username/password used to run the service has identical credentials on the NAS device otherwise writing to the NAS device **will not work**.
  • Confirm you have the proper exclusions for your firewall and antivirus software. See Security Software Exceptions in help for details.
    Exclusions.PNG
    Exclusions.PNG (11.44 KiB) Viewed 1809 times
Post Reply