Performance and Troubleshooting - Windows System Tuning

Articles based on common support questions.
Post Reply
varghesesa
Posts: 32
Joined: Thu Jul 11, 2019 9:52 pm

Performance and Troubleshooting - Windows System Tuning

Post by varghesesa » Thu Feb 18, 2021 7:35 pm

Introduction

After installing BI, it's always good to go through this article just to confirm Windows settings are not in conflict with BI functionality.

Blue Iris is challenging because there are so many external dependencies that could cause issues with BI. Below are a few of the many external dependencies:
  • Hardware drivers from Intel or NVidia for decoding live streams
  • Camera vendors to provide reliable streams with supported encoding.
  • Gmail or other email servers for email alerts
  • Apple / Google platform for mobile app push notifications
  • Custom integrations via http request and mqtt interfaces
All these 3rd party dependencies can lead to instability.

Windows settings

Antivirus software

Your Security Software (Windows defender, McAfee, BitDefender etc) or a Security Software update is interfering with Blue Iris.
Symptoms: Performance issues, Crashes, Software will not start, Auto-update is blocked are all symptoms due to improper software security exemptions.

Below is an example of robust Software Security Exemptions for Windows Defender. Similar exemptions need to be provided for your Security Software.
  • Note BlueIris process and file is excluded
  • Note the BI folders for storage are excluded: G:\BlueIris
  • Note bvr file types are excluded
Exclusions.PNG
Exclusions.PNG (11.44 KiB) Viewed 1930 times

See help file topic "Security Software Exemption" in the Help file as well. This section provides additional details associated with other vendors (McAfee, Malware Bytes etc). If you are experiencing instability, the easiest step to take is to just turn off your security software and observe over time if the issues go away. The Performance and Troubleshooting webinar is also helpful. https://www.youtube.com/c/BlueIrisSupport

Windows BitLocker is another security measure provided with Windows 10 which allows encryption of the hard drive. BI performs constant reads/writes to storage. Most users have not enabled BitLocker.

Firewall Settings

There is no way BI can talk with your cameras or the web interface or the mobile apps unless BI is allowed access to the network.

Windows allows the ability to categorize networks as public or private. My server running BI is connected directly to my switch containing all the cameras and does not move. I'm not concerned about whether I categorized my network correctly since I know it is physically safe on my private switch. Thus I keep Firewall settings simple and allow access to all networks, the public and private network as seen below.

You may have BI running on a laptop that you use at Starbucks and other networks. In those cases, labeling networks as private / public and allowing BI access to private networks is more of a concern that should be addressed.

firewall.PNG
firewall.PNG (15.8 KiB) Viewed 1436 times

System Restore

Starting with Microsoft Windows XP and still available in Windows 10 there is a recovery tool called ‘System Restore’. Essentially this tool takes snap-shots of critical files at certain points in time (e.g. software installations, system updates, etc. as well as at your request.) These system restore points are accumulated and if needed will often (but not always) return a PC to its operating status at the selected point-in-time.

Since it is part of the Operating System and easy to use, we highly recommend using it. See Windows Tools article for details.



Running BI as a Service

Running BI as a service is a very popular feature. Running as a service means BI is always running in the background. One advantage of doing so ensuring BI is running. Furthermore, Windows will automatically restart the service if the application crashes. The "Running as a Service" topic in Help explains all the details associated with setting up BI to run as a service. Below are known issues based on past tickets.

Running Service as LocalSystem
While discouraged, it is possible to run a Windows service under the account "localsystem" by leaving username/password blank when selecting to run BI as a service (image below). LocalSystem may not have the access rights for certain functions. A common issue is alerts to run batch files or MQTT commands may not get executed when the service is run under the LocalSystem account. Running BI as a Windows service under the account "LocalSystem" can also cause instability because localsystem may not have the needed privileges for NAS devices or sending alerts. See Help file, Topic "Running as a Service" for details.
localsystem.PNG
localsystem.PNG (5.06 KiB) Viewed 1886 times
Follow steps below to fix:
  • Basic idea: Disable and re-enable the service to run under a Windows account instead of localsystem.
    Create a local windows account with admin privileges.
    Global settings -> Startup tab ->Run as a Windows service. Uncheck selection. Restart BI.
    Go back to Global settings -> Startup tab -> Run as a Windows service. Re-check the selection.
    Enter your own local Windows account (w/ admin privileges) in pop-up dialog
    Restart BI once again as a service.
  • If you are using a NAS device for storage, the Windows username/password used to run the service has to be identical to the credentials on the NAS device otherwise writing to the NAS device will not work.


1057 Error
If you get the 1057 error, i.e. “Failed to install service (1057)”, the username chosen above to run the service does not exist or is invalid. Follow instructions above to create a local Windows account with admin privileges. Run Windows Services -> Right click Blue Iris Service -> Properties -> Log on tab. Specify username / password.



1069 Error
The 1069 Windows error occurs when the password associated with the Windows username associated with the service has changed. Run Windows Services -> Right click Blue Iris Service -> Properties -> Log on tab -> reset the password.
1069 error.png
1069 error.png (10.92 KiB) Viewed 1990 times

Auto-update fails because update program cannot stop the service. Following error message appears.

blueirisservice.png
blueirisservice.png (5.34 KiB) Viewed 1990 times
  • Stop the Blue Iris Service manually. From the Windows search bar, type services and bring up the Services console.
  • Select the Blue Iris service.
  • Select "Stop service"
  • Start BI.
  • Return to BI. Global settings -> Startup tab -> Run as a Windows service. Check the selection.
  • Restart BI. BI should start as a service.


Auto-update fails because update program cannot stop BI. Following error message appears. (Similar issue as above)

update error.PNG
update error.PNG (48.1 KiB) Viewed 1188 times
  • Stop Blue Iris manually.
  • Restart the machine.
  • Find where you downloaded the update or just go to blueirissoftware.com and download the latest version.
  • Install/start the latest version and you should be good to go.


Web server has incorrect IP Address
On rare occasions, for some reason Windows will start the BI service before initializing the network card/stack and attaining an IP address. This results in the BI web server binding to an improper IP address.

The only way to really know if this is happening to you is by going to the Global settings -> Web server tab. You need to check the WAN/Internet address and confirm it is indeed the WAN IP Address provided by your ISP provider. I know! This unfortunately requires a lot of networking knowledge from the novice user.

The temporary fix is to restart the service. The permanent fix is to configure Windows to delay the start of the BI service.
service_delay.PNG
service_delay.PNG (16.15 KiB) Viewed 1258 times



Instability / Crashes

Windows Event Viewer
The Windows Event Viewer confirms application crashes and the cause. In the BI Help file, follow instructions in the Troubleshooting and FAQ section to examine the Windows Event Viewer. If you confirm BI is crashing, review the Windows Events article to understand the issue.



What has changed?
The biggest question to ask yourself when observing instability is what has changed?
  • Was there a Windows update?
    If you suspect a Windows update is causing instability, use System Restore to remove the update.
    See Windows Tools article for details.
  • Was there an update to your Antivirus or Firewall software?

If you are convinced the latest BI update is causing the instability, the Rolling back BI version article will provide guidance on how to go back to your previous version. After rolling back, confirm stability is restored. If so, submit a ticket following instructions below.

If going back does not restore stability, make sure to update to the latest version to continue troubleshooting.
BI provides some cool features to save the previous version of the software in case you need to roll back quickly and/or want to change update schedule to latest critical. You may want to consider changing your auto-update schedule at this time to avoid future instability.



Minimum information for any ticket
Submitting a ticket stating your BI server is crashing will not lead to any resolution. At the very least, we need to know what errors are found in the Status -> Log and the details from the Event viewer.

Go to the Status dialog (button with chart and arrow in top left of console) to understand how the server is performing.
status button.png
status button.png (11.29 KiB) Viewed 48 times
  • Status -> Log: Any errors? See Status Log Errors article for error details. The Help button in the Status dialog has additional information also.
    status log restart.png
    status log restart.png (117.75 KiB) Viewed 1294 times
  • Review the status->log BEFORE the crash so we know what BI was doing prior to the crash.

    Common mistake: Users send a screenshot of the Status log (above). This provides all the events AFTER the crash. In order to find out what happened BEFORE the crash, you need to "Open file" (button at bottom), but first you have to "Save to file" (checkbox at bottom).
    "Save to file" saves the logs to files.
    Status -> Log -> Open file opens the log file once created (you may have to wait for next crash). Search for all "Restart after unexpected..." events. Do you see common tasks before a crash?
  • Status -> Cameras: Are the camera streams healthy. See Camera Stream Optimization article for details. Help button has additional information also.
  • Status -> Clip Storage. Any red errors? See corresponding Help file by pushing Help. The Storage and Recording webinar shares best practices. https://www.youtube.com/c/BlueIrisSupport/videos

Software will not start

If the software does not start, the only remedy is to adjust the registry until the software does start. The Software Startup Issues article will walk you through necessary steps.


Immediate crashes
Immediate crashes are those that occur within ~5 minutes of starting the software and frequently crashes. This symptom is almost always indicative of hardware incompatibility or issues with a new camera or hardware issues with your server (e.g. hard drive). Keep in mind, if this were a BI issue, there would be 100s - 1000s of tickets and we would most likely contacting you.

ToDos:
  • Any clues in Status as explained above?
  • Turn off hardware acceleration for all cameras. Global Settings -> Cameras tab -> Hardware accelerated decode = No.
    Does stability return? If so, was there a Windows update recently? If a Windows update is the issue, then you may need to leave hardware acceleration (HA) off until there is a Windows resolution. Or you may have to revert back to the previous Windows version. See System Restore section in Windows Tools article.
  • Did you add a new camera? Maybe the conflict is with the hardware and that one particular camera. Simply disable the camera and turn HA back on. Does stability return?
If you turned off HA for all cameras, now you can narrow down the problem to the Windows driver or an issue with a particular camera.
  • Disable all your cameras.
  • Turn HA back on for all the cameras at the global level. Global Settings -> Cameras tab -> Hardware accelerated decode. Choose appropriate setting.
  • Enable each camera one by one and confirm stability.
The camera that causes instability is the culprit. Turn off HA for the camera. If stability returns, the issue is HA.
If stability does not return, simplify camera encoding settings and check if stability returns.

Intermittent crashes
Intermittent crashes are ones that occur over time, perhaps 2-3 days. These are tough issues that often leads to major software improvements.

Does CPU/RAM utilization grow over time? Security software (Windows firewall/defender, McAfee etc) having BI file/s open for scanning is a common cause. Please double-check the BI folders and BI executable are exempted from your security software (Windows firewall/defender, McAfee etc). The Performance and Troubleshooting webinar is a great resource in addition to the Security Software Exemption section in the help file. Check out our webinars in our YouTube Support Channel when you have free time. https://youtube.com/c/BlueIrisSupport.

Continue observing status -> log for errors. An example of the logs leading to clues was when we received a ticket for frequent crashes. The user reviewed the monthly logs and noticed suddenly on 5/8, BI was receiving frequent write errors. The Status Log Error article provides details. A review of the security exemptions resolved the issue.
write error.png
write error.png (64.69 KiB) Viewed 466 times


Final thoughts
It's possible that we introduced a memory leak or adversely affected CPU usage. However, these types of errors would typically result in 100s - 1000s of tickets immediately and we would often be contacting you and all the other users to make everyone aware of the problem. So in general, while there could very well be an issue with BI on your system, we need some patience to diagnose the issue. In the interim, if the instability makes the system unusable, rolling back your BI version or Windows update may be the interim solution to get your system running again.

Identifying the root cause for crashes can be a mystery. We have known users that uninstalled/reinstalled BI, deleted and recreated cameras, reinstalled windows then BI, reinstalled phone apps, unplugged/replugged their hard drives and things miraculously started working again.

If you want to test your hardware before reinstalling Windows, you could run sfc /scannow or DISM to see if the hard drive or Windows has issues. You would need to search on YouTube for guidance. We are not Windows/hardware experts. Posting a question to the forum is a good idea as well.

A very viable next step at this point is to simply rollback Windows using System Restore or re-install Windows using the many Windows options. The Windows Tools article provides options and details.



System Alerts
Users can create alerts (push, email, sms) when there are system log errors to help diagnose the issue. All the different types of alerts are located in Status -> Log tab. Select "Status Alerts" button. Hit the Help button in the dialog for details.

Furthermore, users can also create alerts for each and every camera that is causing issues (e.g. no signal). Camera settings -> Watchdog tab has details.

If you tend to be near the BI server, this allows you to examine the machine near the time of errors in order to gather clues to instability prior to a possible crash.


Submitting ticket - Isolate the Issue
Troubleshooting and isolating the issue is key to identifying the root cause and resolving the issue. Attempt to isolate the issue by systematically disabling cameras, features, etc until there is stability. What does that mean?

Think of Blue Iris as blocks or modules that work together.
  • If you recently added a new camera, does turning it off resolve the issue?
  • If you turn off hardware acceleration for all the cameras, does stability return?
  • Can you then start turning hardware acceleration on for each camera to isolate the camera that is in conflict with your hardware acceleration?
  • If you turn off cameras from a particular vendor, does stability return?
  • Set the shield to red, i.e. let the cameras stream but no trigger, record or alert etc settings are active. Does stability return?
  • If stability does not return after turning the shield to red what else is on? This is a big clue.
    Are you streaming to YouTube or website?
    Are you streaming to your Home Automation system?
    Are you posting to a website?
    Are there inputs to BI besides cameras? Any DIO devices?
  • Is auto-cycle on? Group cycle or camera cycle? Turn it off. Does stability improve?
  • Are you uploading files to the cloud via FTP?
  • Are you constantly streaming video to a Web Interface, iPad, via mqtt or Webcasting? Does stability improve if you turn it off?
  • If stability does return after turning the shield to red, this is also a big clue. Start turning things on like activating triggers, recording and alerts per camera to isolate which cameras or settings are causing the issue.
  • If you are using Deepstack, turn Deepstack off. Does stability return?
  • Do you frequently see crashes right after common activities in the status log?
  • Do you notice crashes when someone connects remotely via the web interface or mobile apps?
One example of a user helping us find a bug was with audio playback on mobile devices. By happenstance, he put the camera in a group and noticed audio playback started working. That turned out to be a big clue.

Another user started with a generic ticket like many just describing the issue, e.g. "constantly missing notifications". I provided the generic follow-up questions, e.g. did you check the Status -> log for errors. He took the follow-up questions seriously, rolled up his sleeves and was able to discover the "constantly missing notifications" was actually from only one camera because the BI server would crash when Deepstack was running and when an object was found. If an object was not found, the cancel alert worked fine.

In summary, we love customer support that leads to the narrowing down of the problem (usually with reproducible steps) so engineering can then determine where in the software to look and identify the problem. We also realize everyone does not have that level of curiosity or time to help. No worries. In that case, the next best step is often to report the issue and either revert back to the last stable version or wait for the next release. The BI Versioning System and Rolling back BI Version article will provide details. You may want to consider changing the BI Update setting from "All available major and minor updates" to "Critical and highly stable updates only".

Needed ticket info
  • Include support info in body of email. Global settings -> About tab -> Email support button. Copy/paste information in body of email.
  • Describe the issue.
    Were you able to isolate the issue? Can you reproduce the crash? If so, what are the reproducible steps?
    Do the crashes occur at the same time daily?
    Anything change with the machine, Windows or your BI settings?
    Does the issue only occur with a particular camera?
    Does the issue occur when connecting via a web browser?
    Does the issue occur only when recording?
    Does the issue occur only with a push notification?
  • Ticket summary
    This section (in bullet list preferably) lists everything you tried and the results. This information provides the needed clues to determine what to test next.
  • Details on your Server setup.
    If your setup is anything more than BI running on a Windows 10 machine, we need to know.
    Are you running BI on a VM? Details including VM vendor.
    Do you have h/w acceleration? NVidia? Intel? Other? Details.
    Are you running Deepstack?
  • Cameras setup
    How many cameras? Make/model.
    Screenshot of Status -> Cameras tab.
  • Integrations / Use cases
    Any DIO inputs besides cameras? Speakers? Motion sensors?
    Any unique streaming to 3rd party destinations like websites (YouTube), home automation systems etc?
    Are you using the mobile apps or web interface? How many users?
  • Windows Information
    Include the event viewer crash info. The Troubleshooting and FAQ section in Help provides details. Copy / paste information into body of email. Any other clues seen in Event viewer such as other frequent Errors.
Post Reply