Instability and Troubleshooting

Symptoms to Resolution

It's easy to state symptoms:
My BI server started crashing.
My recordings are missing.
No alerts are being sent.

Much harder to know how to investigate and self-correct or provide the needed information for support to help.

The articles here are frameworks on how to think about the issue and get to resolution.
Post Reply
varghesesa
Posts: 90
Joined: Thu Jul 11, 2019 9:52 pm

Instability and Troubleshooting

Post by varghesesa »

Introduction
Blue Iris is challenging because there are so many external dependencies that could cause issues with BI. Below are a few of the many external dependencies:
  • Windows
  • Server hardware
  • Networking
  • Hardware drivers from Intel or NVidia for decoding live streams
  • Camera vendors to provide reliable streams with supported encoding.
  • Gmail or other email servers for email alerts
  • Apple / Google platform for mobile app push notifications
  • Custom integrations via http request and mqtt interfaces
All these 3rd party dependencies can lead to instability.

Some users mistakenly believe sending us their Logs and/or Event viewer details will allow us to resolve their issue. The event viewer details are primarily used to confirm the crash was indeed due to Blue Iris.

Any known issues are documented. This article explains how to resolve unknown issues. If the issue is unknown, i.e. BI is crashing in your environment, isolating the issue is key to resolution.

The Blue Iris status log webinar walks through this article if you prefer to listen instead.


Routine maintenance
Check server health section for any known issues. See if resolving those issues restores stability.


Version control
If you believe a recent update introduced instability, fall back to the prior version and see if stability returns. Fortunately, BI provides a lot of functionality in regards to version control.

If the instability makes the system unusable (more than one crash per day), rolling back your BI version or Windows update may be the interim solution to get your system running again. When you have time to investigate, then reinstall the latest update and proceed through article.


Windows tuning article
The place to start is with the Windows tuning article to confirm there are first no conflicts between BI and Windows. In this article, walk through all the checks. Double check your Anti-virus and Firewall exemptions.

If you have not done so already, consider running BI as a service. By doing so, Windows will automatically restart BI when it runs into issues. This gives you time to further investigate instability while also keeping the server running. See Running as a service section in Help.


Startup
If the software does not start or crashes quickly, the only remedy is to adjust the registry until the software does start. The Software Startup Issues article will walk you through necessary steps.


Instability
Windows Event Viewer
The Windows Event Viewer confirms application crashes and the exception associated with the crash. In the BI Help file, follow instructions in the Troubleshooting and FAQ section to examine the Windows Event Viewer. If you confirm BI is crashing, review the Windows Events article to understand the issue.

From this article you should be able to determine:
  • Which version of BI first introduced instability.
  • The exception associated with the crash. Understanding of the exception

Stability
This section explains how to restore stability.
Third party dependencies
Removing known instability is the first step.
Hardware acceleration
Temporarily disable hardware acceleration, especially if you see GPU errors in the logs. This is often a source for instability.

Cameras
Disable / disconnect cameras that lose connectivity frequently.
After searching the logs over the last few days and paying attention to what preceeded the unexpect stops. I have discovered a camera that seems to be rebooting constantly when there is no light in the area. I have disabled that camera and the unexpected stops have not occured for the last 2 days.

Network
A reliable network is a huge factor in BI running correctly. We are not networking engineers. We outline some known pitfalls in the Network tests section.

CPU Utilization
If CPU utilization is >65%, this article explains ways to optimize your server.

RAM Utilization
This article explains how to deal with RAM issues.


Re-install windows
Identifying the root cause for crashes can be a mystery. We have known users that uninstalled/reinstalled BI, deleted and recreated cameras, reinstalled windows then BI, reinstalled phone apps, unplugged/replugged their hard drives and things miraculously started working again.

Customer anecdotal
So after concluding that BI was simply unuseable due to it's random, frequent crashes, in desperation I uninstalled BI. To be clear I did the Windows uninstall, then deleted the leftover BI folders from the C:\ drive and then used the Windows regedit app to delete the registry items for Perspective Software in HKEY_Current _User/software and HKEY_Local_Machine/software. Once this was done I then re-installed BI5 and re-configured the cameras, etc.
Now BI seems to be stable. It has not crashed randomly for two days now which is a huge improvement over it's previous behaviour.
I will monitor the situation going forward but the initial signs are good.

If you want to test your hardware before reinstalling Windows, you could run sfc /scannow or DISM to see if the hard drive or Windows has issues. You would need to search on YouTube for guidance. We are not Windows/hardware experts. Posting a question to the forum is a good idea as well.

A very viable next step at this point is to simply rollback Windows using System Restore or re-install Windows using the many Windows options. The Windows Tools article provides options and details.


Isolate the issue
At this point, it is safe to suspect something with your usage or environment is causing instability.

Root cause

At this point, you have to decide whether you want to help engineering identify the root cause. Not everyone has the technical expertise to help engineering get to the root cause.

The Isolate Issue article provides an example of providing the right information to quickly identify the root cause. If you have a good handle of BI functionality and IT, we appreciate any and all support.

The Server crashes ticket is another example of a good ticket that actually led to resolution.

If you do not have this skillset, no worries. BI provides some cool features to save the previous version of the software in case you need to roll back quickly and/or want to change update schedule to latest critical. You may want to consider changing your auto-update schedule at this time to avoid future instability. Version Control article has details.


If you are still reading, you are interested in helping identify the root cause. See Isolate the issue - Framework article.
Post Reply