Diagnosing Computer Crashes

DougLorenz
Posts: 5
Joined: Wed Aug 19, 2020 3:33 pm

Diagnosing Computer Crashes

Post by DougLorenz »

Hi, I've been using Blue Iris for a few months now, and it has been working great with one problem... Every few days the computer it is running on just stops working, completely freezing up and requiring that the computer be powered off and back on again. Obviously, this stops the recording and renders the whole system useless until I notice the problem and restart the computer.

The computer is a HP ProDesk with an Intel Core I5-8500T, 16GB of RAM, and a 500GB Samsung 970 EVO NVME drive. I only have 3 cameras at the moment, so the computer is handling the load fine. The system consists of a fresh install of Windows 10 and Blue Iris with nothing else installed.

Trying to figure out my next step. Is there some diagnostic software that I can be running to identify the root cause of the crashes? Additionally, is there some piece of hardware that can power cycle the computer if it goes dark for more than 5 minutes or so?

Doug
Matts1984
Posts: 496
Joined: Fri Apr 10, 2020 1:12 pm
Location: Maryland, USA

Re: Diagnosing Computer Crashes

Post by Matts1984 »

I'm guessing we're talking about a full computer lockup - not BSOD, but the next closest thing - and only a hard power cycle helps.

I'd look at System logs. My gut says this is a driver though.

As for a hardware system that would power cycle... not that I'm aware of as an out-of-the-box system. You might be able to set up some sort of network based monitor to find when a particular service/port stops responding and then and then has the ability to control a smart plug - but you're probably looking at some custom coding or like a RaspberryPi like project.
Blue Iris 5.8.8.x | Server 2022 VM | Xeon X5650 @ 2.67GHz - 12 Cores | 12GB RAM | 7TB RAID | Sophos UTM WAF | 4x SV3C 5MP Bullet A | 1x SV3C 5MP PTZ HX | 1x SV3C 5MP Bullet HX | 1x SV3C 5MP Dome HX | 2x Amcrest 5MP Bullet
DougLorenz
Posts: 5
Joined: Wed Aug 19, 2020 3:33 pm

Re: Diagnosing Computer Crashes

Post by DougLorenz »

Basically, the system just stops what it was doing. If the screen was on at the moment of death, it stays on, but frozen at that last moment. Mouse, keyboard, drive lights, nothing changes. Nothing I've tried can bring the system back out from this other than a manual power cycle.

I don't know what program or log files are available that could help me diagnose this further. Any suggestions?
User avatar
TimG
Posts: 2098
Joined: Tue Jun 18, 2019 10:45 am
Location: Nottinghamshire, UK.

Re: Diagnosing Computer Crashes

Post by TimG »

Take a look at "Reliability History". Type it into the search bar. It keeps a record of failures. That was one of my biggest helpers in my recent pc fiasco.

Are you using Acronis True Image ? Reddawg pointed out that the ransomware protection slowed his pc down. It was a major factor on mine too. Ransomware turned off.

I also had a pci-e satellite tuner card with issues. That broke com ports, usb etc as well as just not working properly. Removed from pc and smashed to pieces with a sledgehammer. Very satisfying.

I moved BI5 from my AMD 8350 server due to these issues. It is presently running on an Intel i5-3330 with 16GB RAM and is solid.

1. I have BI5 automatic updates off. Weekend job.
2. I have Win10 updates postponed for two weeks. I will do them manually over the weekend rather than come back to a badly crashed server.
3. Sub-streams. Now I know why people used to think that BI5 was a cpu hog. Cpu down from 18% to 9%.

The BI5 server is showing 10/10 in Reliability now, and it has been up since last weekends Win10 updates. BI5 is running as a service, and all is well.

Let us know if your Rel History shows anything.
Forum Moderator.
Problem ? Ask and we will try to assist, but please check the Help file.
User avatar
TimG
Posts: 2098
Joined: Tue Jun 18, 2019 10:45 am
Location: Nottinghamshire, UK.

Re: Diagnosing Computer Crashes

Post by TimG »

Just realised, do you only have one hard drive ?

I keep BI5 storage and database away from the Windows drive. I have a second 250GB SSD just for the BI5 New folder and the db, and a 14TB mech hard drive just for the Storage.
Last edited by TimG on Sun Aug 23, 2020 8:47 am, edited 1 time in total.
Forum Moderator.
Problem ? Ask and we will try to assist, but please check the Help file.
Matts1984
Posts: 496
Joined: Fri Apr 10, 2020 1:12 pm
Location: Maryland, USA

Re: Diagnosing Computer Crashes

Post by Matts1984 »

Thats pretty cool on the Reliability History - I can confirm it exists in Server 2019 as well (which is essentially Win 10 with some extra features anyway). Thanks for sharing that.

While I'm not sure it's causing the stability issue - yes I'd definitely recommend having a separate hard drive, at least logically, for any database like function, ie massive amounts of video files. There are a ton of read/writes going on that are harder on the drive than they'd typically be used to. If a drive fails, you're only talking about losing recordings. Add a new drive and you're back up in minutes (minus some historical video files, oh well). If it's your system drive, and it fails - well now you're starting over from scratch. Also I've seen too many times where some application fills the drive to max capacity and you lose all capability other than a direct console access to sort it out. I try and keep the minimum on C:

I'd still bet this is some driver related issue (or something like TimG's pci-e card). Hopefully the Reliability History can help point you in the right direction. Looks like it's System Event Logs with a graphical interface and some intelligence to showing you stuff that might actually be relevant. Once you find the issue:
GIF.gif
GIF.gif (146.07 KiB) Viewed 9037 times
Blue Iris 5.8.8.x | Server 2022 VM | Xeon X5650 @ 2.67GHz - 12 Cores | 12GB RAM | 7TB RAID | Sophos UTM WAF | 4x SV3C 5MP Bullet A | 1x SV3C 5MP PTZ HX | 1x SV3C 5MP Bullet HX | 1x SV3C 5MP Dome HX | 2x Amcrest 5MP Bullet
DougLorenz
Posts: 5
Joined: Wed Aug 19, 2020 3:33 pm

Re: Diagnosing Computer Crashes

Post by DougLorenz »

Great info, and I'll start digging into it today and see where it leads...

BTW, unless some sort of product like this already exists, I was thinking that taking a Sonoff switch and reprogramming the ESP8266 inside to listen for a heartbeat and powercycle the computer if it doesn't get it, combined with a program on the Blue Iris computer to send the heartbeat to the Sonoff every few seconds... Obviously, there's a bit of programming involved, but the total hardware cost should be under ten bucks.
DougLorenz
Posts: 5
Joined: Wed Aug 19, 2020 3:33 pm

Re: Diagnosing Computer Crashes

Post by DougLorenz »

IAmATeaf wrote: Fri Aug 21, 2020 8:36 pm I should learn to read Image

For freezes like that I’d install software to monitor the temp of the CPU and run some memory tests. Bad memory is something that would cause lockup’s so make sure you run tests on it using some decent software.
I ran the system's memory test, and it came back fine. However, that's where the term "decent software" comes into play. I really don't know what to use for a full system diagnostic. If anyone has any suggestions, I'm listening.

I'm running the system and the "New" storage on a ludicrously fast NVME drive (Samsung 970 EVO Plus) in the hopes of overcoming any storage bottleneck issues. I can add a SATA drive, but that really seems counterproductive.
User avatar
reddawg
Posts: 145
Joined: Sun Jun 30, 2019 11:29 am

Re: Diagnosing Computer Crashes

Post by reddawg »

TimG wrote: Fri Aug 21, 2020 12:55 pm Take a look at "Reliability History". Type it into the search bar. It keeps a record of failures. That was one of my biggest helpers in my recent pc fiasco.
Thanks Tim, I did not know about "Reliability History". That's actually very useful. I like that it shows App crashes and Windows updates.
Blue Iris v5.3.9.10 | Win10 x64 version 22H2 | Dahua IPC-HFW2100, Amcrest IP2M-841W, Hikvision MINI PT DS-2CD2F52F-IS, Edimax IC-3030iWn | Intel i5-2500 CPU, 8GB Ram, Samsung 860 EVO 512GB SSD, WD Black 1TB HD.
User avatar
TimG
Posts: 2098
Joined: Tue Jun 18, 2019 10:45 am
Location: Nottinghamshire, UK.

Re: Diagnosing Computer Crashes

Post by TimG »

DougLorenz: Does Reliability History give you any info ? You can click on previous days and it gives you the errors from that day. You can then double click on each event to see what it is about.

Other than that, as as been said before, it's likely to be drivers or hardware. You just have to rule them out one by one. For example:

1. If it freezes once a day, and you suspect overheating, take the side off the pc and leave a fan blowing air into it. Did that change the time to the crash ?
2. Are there any exclamation marks in "Device Manager" ?
3. Does this pc have built in diagnostics at BIOS level ? If so, run them all.

Please note that if you think I am trying to teach you to suck eggs, it is only because I am trying to help you, but I don't know how techy you are !

I have used software such as Sisoft Sandra to do burn-in tests on faulty pc's, but if you use that, be aware that it can finish off a failing pc. Reliability History, if it can help, is a very low risk, and easy way to see what is happening to your pc.

As they used to tell Sparky Tim down t'pit "If you can't fix it with a hammer, it's an electrical problem.". Such wise words :lol:
Forum Moderator.
Problem ? Ask and we will try to assist, but please check the Help file.
Post Reply