5.5.5.11 DeepStack: timeout

carteriii
Posts: 11
Joined: Tue Sep 28, 2021 3:23 pm

5.5.5.11 DeepStack: timeout

Post by carteriii »

I upgraded to 5.5.5.11 and every alert then showed a message of, "DeepStack: timeout" (see image below) and consequently no alerts are ever confirmed. I reverted to 5.5.4.5 and all such errors went away, implying there is a regression of some sort in the integration between 5.5.5.11 and DeepStack.

I realize this sounds unlikely, so in defense of myself :=), I did not touch any DeepStack or other setting when upgrading to 5.5.5.11, and I tried restarting BlueIris, stopping & restarting DeepStack, and rebooted the entire computer to see if that resolved the "DeepStack: timeout" problem. Nothing resolved the problem, except/until reverting to 5.5.4.5 and now things are working smoothly again. I really believe 5.5.5.11 has a problem with its DeepStack integration.
blue-iris-deepstack-timeout.png
blue-iris-deepstack-timeout.png (90.05 KiB) Viewed 2471 times
carteriii
Posts: 11
Joined: Tue Sep 28, 2021 3:23 pm

Re: 5.5.5.11 DeepStack: timeout

Post by carteriii »

Update: Now I'm getting DeepStack timeout messages even with 5.5.4.5, so my original assertion that 5.5.5.11 had a problem must not be correct. Something else has changed for me after the upgrade & reverting back. I'm running the CPU version of DeepStack and don't see my CPU pegging, and I haven't touched the models in a long time (I am running the dark.pt and license-plate.pt models).

I am seeing a lot of "DeepStack has been restarted" messages after a timeout. Any suggestions on how to get more detailed logging from DeepStack to know what's going on?

DeepStack: Alert cancelled [DeepStack: timeout] 15011ms
DeepStack has been restarted
pqRec
Posts: 21
Joined: Sat Jan 15, 2022 1:45 am

Re: 5.5.5.11 DeepStack: timeout

Post by pqRec »

Can you browse to the DS web server instance (default is http://127.0.0.1:82 ) ?

If not, open task manager > details > right click the labels at the top (eg Name PID Status CPU etc) > Select Columns > near the bottom select Command Line > OK

For the "server.exe" process, you should see something like C:\DeepStack\server\server.exe -VISION-SCENE=False -VISION-DETECTION=True -ADMIN-KEY= -API-KEY= -PORT=82

If that's all there and you still can't browse to it, try changing the port number or updating DeepStack - a new version was released a couple of months ago.
User avatar
YrbkMgr
Posts: 587
Joined: Sun Nov 24, 2019 12:56 am
Location: Chicagoland

Re: 5.5.5.11 DeepStack: timeout

Post by YrbkMgr »

first things first. Gear icon -> AI -> DeepStack section. Post that dialog. If you see "Stop Now" button, click test in browser. Can you start it from here? Let us know how that goes.
  • "Whenever I take something apart to fix it and put it back together again, I end up with like six really important looking pieces left over" -Tim Allen
  • If you know what your after, you'll recognize it when you see it.
carteriii
Posts: 11
Joined: Tue Sep 28, 2021 3:23 pm

Re: 5.5.5.11 DeepStack: timeout

Post by carteriii »

DeepStack settings listed below. I have been able to stop & start deepstack, and I have been able to see the default page with no problem. Last night I became convinced that the timeout was not due to DeepStack taking too long, but because it wasn't getting called at all. As I mentioned, I could see the status page just fine, so I thought perhaps that the BlueIris update (& rollback and then update again) combined with possibly a Windows 10 update that I also ran could have broken the Windows firewall settings (I never touched DeepStack). I finally tried re-installing DeepStack, and only because I saw a newer version available (2022.01.01) vs. what I had previously installed (2021-09.01), I went with the newer version. The installation did prompt me with a Windows alert to give permission (which may or may not be relevant) but after that, DeepStack appears to be working again.

I realize (& feel bad) that I claimed the DeepStack version was likely the issue, forgetting that I also ran Windows updates that could also have been a contributing factor.

For what it's worth, I am running the combined.pt and dark.pt models (mentioned in other threads here and on ipcamtalk.com), but until the updates I ran this weekend, everything has been running fine with no timeouts. I'll be watching closely today since there wasn't much activity last night.
deepstack-settings.png
deepstack-settings.png (19.37 KiB) Viewed 2443 times
MikeLud
Posts: 84
Joined: Sun Mar 21, 2021 3:29 am

Re: 5.5.5.11 DeepStack: timeout

Post by MikeLud »

Try only using one instance.
User avatar
YrbkMgr
Posts: 587
Joined: Sun Nov 24, 2019 12:56 am
Location: Chicagoland

Re: 5.5.5.11 DeepStack: timeout

Post by YrbkMgr »

Cool. Thanks for the follow up. I'm still using v5.5.4.5 - updates have been buggy so unless there's a compelling reason to update, Ima hang back on that.
  • "Whenever I take something apart to fix it and put it back together again, I end up with like six really important looking pieces left over" -Tim Allen
  • If you know what your after, you'll recognize it when you see it.
User avatar
YrbkMgr
Posts: 587
Joined: Sun Nov 24, 2019 12:56 am
Location: Chicagoland

Re: 5.5.5.11 DeepStack: timeout

Post by YrbkMgr »

MikeLud wrote: Sun Mar 06, 2022 4:56 pm Try only using one instance.
Agreed.
  • "Whenever I take something apart to fix it and put it back together again, I end up with like six really important looking pieces left over" -Tim Allen
  • If you know what your after, you'll recognize it when you see it.
carteriii
Posts: 11
Joined: Tue Sep 28, 2021 3:23 pm

Re: 5.5.5.11 DeepStack: timeout

Post by carteriii »

Will do (going back to 1 instance). If it matters, I have 8 cores on this Radeon 5900 and have been running with more than one instance for a while and thought it was ok until this recent issue with 100% timeouts. I have also explicitly tested the ports (e.g. 82 & 83) to make sure they are responding. Is there a known issue with more than 1 instance, or just a matter of caution & testing to see if that might be part of the problem?

@MikeLud, I also want to offer a big public thank you for all you've done with the combined model!
carteriii
Posts: 11
Joined: Tue Sep 28, 2021 3:23 pm

Re: 5.5.5.11 DeepStack: timeout

Post by carteriii »

Things are working sometimes, but I still have big timeouts, sometimes followed by the message that, "DeepStack has been restarted". If I wasn't paying close attention and looking at the BlueIris logs, I might not even know, instead having just missed what should have been an alert. I'm pasting a screenshot of a small section of my log from this morning.

You'll note a few instances of a timeout that include the ms duration (with my 20-second timeout), and in a few places there is a timeout without any time given. It seems that timeouts initiated from an ONVIF/EXTERNAL event show the time (e.g. 20016ms), but timeouts initiated by MOTION_A do not give the time, presumably implying the logging is just a bit inconsistent.

Is there a more detailed log that I can get from BlueIris, showing the specific commands & payload? I'm curious to see if there is a pattern of what generates a timeout. When BlueIris starts DeepStack, is there any way to see more detailed logs on the DeepStack side of things, to see if the call is even making it to DeepStack?

Would those of you reading this also please do a check of your logs to see if the timeout or restarting of DeepStack happens for you? Just use the filter and look for "timeout" or "restart".

One other question about the logs. What is the buffer size or duration shown in the UI, which in turn determines the count shown in the 2nd column? For example, in my screenshot below the count of restarts is 342 but the number of times it appears in the text log is far greater. So if/when any of you check, just realize the log in the UI isn't showing everything you might have (if you are saving the text file of the log).
deepstack-timeout-restart-logs.png
deepstack-timeout-restart-logs.png (65.05 KiB) Viewed 2415 times
Post Reply