RAM Utilization

Symptoms to Resolution

It's easy to state symptoms:
My BI server started crashing.
My recordings are missing.
No alerts are being sent.

Much harder to know how to investigate and self-correct or provide the needed information for support to help.

The articles here are frameworks on how to think about the issue and get to resolution.
Post Reply
varghesesa
Posts: 90
Joined: Thu Jul 11, 2019 9:52 pm

RAM Utilization

Post by varghesesa »

Introduction
We occasionally receive tickets that RAM keeps growing while BI is running. Occasionally there is a RAM leak introduced by BI and if so we will notify you. If not, more often than not, hardware issues or drivers cause the leak.

Instability_gotcha 2.png
Instability_gotcha 2.png (15.06 KiB) Viewed 19176 times

Troubleshooting
Version control
If BI created a memory leak, we usually know quickly and notify users to update the software. If you believe a recent update has caused the issue, go back to a previous version and see if leak goes away. This is very helpful information.

Anti-virus software
Otherwise, the usual culprit is Anti-virus software.
All the content in this article is good to review. Pay attention to the Anti-virus and Firewall sections.

Hardware acceleration
Other sources for leaks are drivers. Turn off hardware acceleration and see if RAM stops growing. You may need to update your drivers. Alternatively, go to each of your cameras and simplify the encoding and observe if the leaks go away.

Other hardware
Below are examples of RAM and disk drive failures that lead to RAM issues.

Customer anecdotals:
Antivirus update and software reinstall.
The Memory issue seems to have gone. I did some cleanup in BI, rebooted some cameras, and checked Firewall / AV. I’m not sure what fixed it. But… till next time. Thanks.
Reinstalled software
OK, so I deleted everything BI off the computer, I also reformatted the drive I save footage to. I then reinstalled BI and it seems to be working... below are the data from about an hour apart. So is the RAM issue (and it appears the CPU usage dropped as well) related to the storage drive filling up? or was it the fresh install of BI that fixed it?

Update for you. The storage has been maxed out for 5 weeks now. Small increase of CPU usage which dropped the last 2 weeks. RAM usage has stayed below 3gs.
I believe that would eliminate everything other than the corrupted update possibility which re installing would have solved.

Thanks for your help
Faulty hard drive
I have recently upgraded my PC (specification attached) and haven’t really been able to get Blue Iris to work properly since ☹
In the most recent logs I am getting out of memory errors (Exception code: 0xc0000005). However this is a brand new top spec gaming PC with 16GB RAM. The CPU Never goes over 15% and the RAM 5G as far as I can see.

Blue Iris also regularly crashing out, I have attached the Application Error that seems to cause it (Exception code: 0xc0000005). So far since purchasing the PC I haven’t been able to get Blue Iris to work properly on it ☹

Thanks for the attempt at helping, but upon further investigation by me this has actually turned out to be a faulty hard drive. Once BlueIris was started it kept creeping up to 100% usage and then maxing the RAM out.

New PC company is sending replacement disk.
I have been trying to troubleshoot a problem that I've had for a month or two. I started getting Signal: network retry, Signal: restored (these were only a few seconds long between the retry and restore), and DeepStack errors like timeouts and restarts in the logs. Some alerts wouldn't play back. Then the memory would creep up and the server would become so unresponsive I would need to do a hard reset.

I finally discovered in the system event logs some bad block errors on the disk. Never had any write errors in the BI logs. I finally swapped the local drive where I keep the BI data and everything has been back in business again. Is there a reason BI couldn't detect write errors? It took a month before I figured it out and went through so many settings changes, reinstalls, and tweaks. What proved it was when I tried a full format and it never finished. I even ran a chkdsk and it didn't show any bad sectors. The health was good in the disk utility. I tried reseating the cables, etc. Then finally the system couldn't even boot unless I unplugged the drive.
Bad RAM
Update on this ticket.

I did some more digging and have determined that the “chirping” issue is due to hardware memory errors. (I was able to verify this with PassMark’s MemTest86) I’m hopeful the playback stuttering and other issues I’ve been having with Blue Iris are also related to bad RAM.

The chirp was coming from the buzzer on the motherboard.

I don’t think it had anything to do with BI other than when BI was running the system was having RAM issues (which were triggering a BIOS level fault that alerted by chirping the buzzer) . No errors in any windows logs. Process monitor did show lots of buffer overflows, which led me to test the ram.

Logs
Check the logs for errors.
Storage
Storage and RAM issues often go hand in hand. Check storage.

Isolate the issue
If not the above, you need to isolate the feature / functionality in order to discover the source of the leak. See if it only happens when camera X is enabled or a particular feature is used.


Submitting a ticket
The bottom line is if we are aware of a memory issue, we are the first to let you know. The above documents the usual culprits, many of which are due to hardware failures. Is it possible that your unique environment has introduced a memory issue? Possible. This is why the final steps are to check the logs for errors and isolate the issue. If you have additional information regarding log errors we can help. If you can isolate the issue, we are thankful and happy to investigate further.

Replying to a ticket after being forwarded to this article
If your response after reading above is "Yes, I checked all that and the issue still remains." or something to that extent, then you may want to keep running the last stable version or identify a local reseller that can provide support. The good tickets sub-forum shows the kind of user expertise that leads to resolution.

As you can see from the above customer anecdotals, there are some pretty tech saavy users that know Windows and how to diagnose hardware issues. I also listed the Tools they used to identify root cause below. If you do not have the hardware expertise, perhaps visit a local PC shop that can evaluate your hardware.

Tools
Tech saavy users have used the following tools to identify RAM issues.
passmark software: RAM tests
Process explorer: May provide hints based on errors.
Post Reply