Deepstack not ready for prime time

[jbardi]

I've tested Deepstack with Blue Iris for the past day or so, and it is definitely not reliable for serious security. We live in a very rural area on 3 1/2 acre lot, and we have 21 cameras around the property. We have a variety of wildlife on and around the property, to include horse, deer, cow, jack rabbit, cotton tail, skunk, raccoon, prairie dog, red fox, bobcat, coyote, mountain lion, wolf, etc

On a typical day, we have many animals caught on all 21 cameras, to the point I can watch their full path from camera to camera to camera, yet since I activated Deepstack, and set it up to detect person along with every animal Deepstack can currently track, I have had only 2 detections.

They were both of a big feral tom cat that roams our property, yet he was only detected on 2 cameras. I reviewed all the other cameras continuous recording footage at that same time frame and found the cat was clearly seen, very large and in full view of 6 other cameras, and the log shows the motion alerts on those cameras were canceled by Deeptack claiming it found nothing.

Not only was it only detected on just 2 cameras, but Deepstack tagged the cat as a bird on both cameras. Cat is one of the animals Deepstack claims to be trained to see, yet it said it was a bird at 83%.

It wasn't dark, or obscured but rather just strutted slowly across the cameras and Deepstack was unaware of what it was.

I can only imaging how many other wildlife was skipped during my testing. I am not even confident it will detect a person when it really matters.

I've disabled Deepstack integration and am back to simply relying on continuous recording and Blue Iris built in motion detection for my alerts. Better to have more alerts of things I don't care about than to have no alerts of things I do care about.

[p3ter]

It seems you are using BlueIris as an advanced Trail Cam rather than for 'serious security', (unless those Jack Rabbits you are so interested in are stealing your stuff...) Just kidding - I think my use of Blue Iris pretty much matches yours, but I'm not so quick to conclude that Deepstack is "not ready for prime time"... I am still getting a feel for where it might fit in my workflow though. (after only 48 hours of testing)

The goal for me with detection has always been reducing the number of 'non interesting' alerts, while also having a general level of confidence that I am not missing anything 'interesting'. I have worked on the Motion Sensor configuration to tune object sizes, zones, travel paths etc, and I am happy that the current level of Alerts are 'mostly interesting'. It is almost impossible however with existing tools to give those Alerts a 'Threat Level', they only reflect 'things I am interested to be notified about', which may or may not also include "potential threats".'

So what can Deepstack add to that? For me, it enables a new level of information "interesting and identified", and even (at a bit of a stretch) "Interesting and a Potential Threat". So I think from a usabililty/worflow perspective it would be nice to see the option to treat AI as an 'additive' step, adding an additional layer of increased classifying quality to an alert i.e. "even more interesting" rather than the current 'subtractive' step i.e. "On second thoughts, AI didn't find anything, please ignore"
I was very impressed by how easy it was to set up, but the instant outcome for me was that I have moved my main Alert review workflow to starting in the 'Cancelled Alerts' view. It's counter-intuitive, but if you want to treat Deepstack as an additive/qualitative step (instead of a subtractive, 'cancel' step), then everything you USED to see, is now in 'Cancelled Alerts', whereas things that are both Alerts and Identified are now in 'Alerts'
In other words - if 'Filter By' -> 'Cancelled Alerts' was possible to rename to 'AI Unidentified Alerts', and 'Alerts' was renamed to 'AI Identified Alerts', it would (for my use case) be a lot more intuitive, and exactly match how I am trying to use it right now.

Also remember that the list of objects Deepstack can identify is pretty limited (and many of them are entirely useless in an outdoor CCTV setting!), and it will not learn on its own...Out of the box, Deepstack can identify a Teddy Bear and a Toothbrush, but it can't identify a Jack Rabbit or a Bobcat... So if you were expecting all of your local wildlife to be accurately identified, then maybe it was oversold to you, and of course you will be disappointed...

Here's what deepstack can identify:
person, bicycle, car, motorcycle, airplane, bus, train, truck, boat, traffic light, fire hydrant, stop_sign, parking meter, bench, bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe, backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket, bottle, wine glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donot, cake, chair, couch, potted plant, bed, dining table, toilet, tv, laptop, mouse, remote, keyboard, cell phone, microwave, oven, toaster, sink, refrigerator, book, clock, vase, scissors, teddy bear, hair dryer, toothbrush.

From that list, and in a nature setting, I have personally only chosen people and vehicles as my AI filter. So what I am thinking is that at a remote, natural summer house location, the local fauna is 'interesting but not a threat' while People and Vehicles are more likely to be a threat. Of course like any new technology, how this is developed depends on how we use it... (which is why I bothered to explain my thought process here!)

Also remember that none of us will be 'training' Deepstack (unless you use the 'Faces' feature) - therefore unless you are prepared to invest very significant time in a whole new product (Deepstack) and a whole new skill set, you should assume that while Deepstack might be described as 'Intelligent', it absolutely stopped learning to identify objects at the time you installed it on your computer. It will never become more intelligent (in the area of learning your local wildlife) without a very significant investment on your part.

from the authors:

"The promises of Artificial Intelligence are huge but becoming a machine learning engineer is hard."

A possible future for CCTV AI (but one that might require a bigger R&D team than Blue Iris has) would be that every time you saw something interesting in your 'Cancelled Alerts' view, you could click inside the bounding box surrounding the trigger object, and manually identify it from a list of objects (standard ones, and custom ones)... Then Blue Iris would use that image snippet to train your own instance of Deepstack...

An AMAZING (and now we are at 'pie in the sky' level of dreaming) possibility would be if all Blue Iris customers automatically shared their (anonymized) manually tagged/identified objects with a central server, and all of our systems would learn from the work done by others... i.e. I have never seen a raccoon (I live in Sweden!) but if we all contributed to the identification of objects from our video streams, your work in pointing at a raccoon and tagging it as a raccoon would contribute to my BlueIris install being able to identify a racoon in future. There are of course huge privacy risks assoicated with this, so I would not expect that it could be reasonably achieved, but it is a fun thought! And I am sure Deepstack makers would LOVE the thought of thousands of people around the world sharing the work of training their AI for them!

[terk]

The goal for me with detection has always been reducing the number of 'non interesting' alerts, while also having a general level of confidence that I am not missing anything 'interesting'.

This has been what made me interested in DeepStack as well, I was seeing too many false alerts due to shadows or bushes moving on the lawn. I have my external cameras recording 24/7 just in case something is missed (either due to DeepStack not recognizing it or BI not seeing the motion therefor not engaging DeepStack for that instance) however since switching from the AI Tools implementation with a 1 minute cool-down and only one image per detection to the integrated DeepStack I feel even more confident in DeepStack as it is now capable of analyzing multiple images per detection and so far I haven't found it to miss people around our house as that is the only detection I have enabled on most of the cameras however I do have Dog and Cat enabled on a few as well, but haven't noticed any detection for those yet.

I have never seen a raccoon (I live in Sweden!)

Exactly, from my understanding DeepStack originates in Nigeria which explains why so many of the animals it can currently detect many of us would consider zoo animals since I'll never see a giraffe, lion, or elephant walking around my year, it also probably explains why bobcat or raccoon aren't on the list currently. I do have raccoons roaming around here at night which is one of the reasons I added Cat and Dog as detection types for some of the cameras hopping to see them at least if they didn't cause a bunch of alerts and thinking those would be the most likely to match profiles in the list.

[ibrujo]

Hi
I just read these posts and my main issue is with bushes and plants being captured by BlueIris. My other big issue is sunrise, At sunrise the cameras trigger an alert.
Were you able to solve your issue?
If so could you please provide me with the information on how to solve this issue.
Thank You
Mario

[p3ter]

@ibrujo I don't really think your question is relevant to Deepstack, it sounds like a general object detection question. Anyway - there is no single or simple answer since all requirements and environments are different, but my method to reduce false positives is to focus on 'zones and hot spot' settings in combination with the 'Object Crosses Zones' setting. For example the simplest way to avoid bushes would be to create a Zone A where you paint everything apart from the bushes, then add an Object Detection trigger of 'Object Crosses Zones - A' In other words, you are telling Blue Iris to ignore motion which happens outside of Zone A, so the zone acts like an inverted mask which ignores motion in the non-painted areas.

If you are looking for 'predictable' motion (a car driving up/down a road, a person walking up/down a pathway) you can for example paint half your road/path as Zone B, then paint the other half as Zone C, and set up the 'Object Crosses Zones' as "B-C" (or B>C, if you only want to detect one way traffic).

I currently have min object size 200, min contrast 15, min duration 1.0, which on my cameras is enough to trigger on a cat or a decent sized bird - but due to the 'Object travels' 200 pixels setting, and the 'Object Crosses Zones' settings, I only see birds/cats travelling up/down the path/road, which leads to much fewer false posistives, since the object needs to be travelling a significant distance, and in an 'interesting' direction to be relevant.

Again, there is basically unlimited variation in cameras, scenes, lighting, location, and what people want to trigger vs what they don't want to trigger - so there is never going to be one answer - but the principle for me has been to try to ignore ALL motion in uninteresting areas, then try to limit triggers to motion moving in a specific direction/path for a specific distance.

[YrbkMgr]

In addition to p3ter's excellent comment... consider the following.

It isn't clear to me what problem are you trying to solve - too many alerts or too many triggers? The answer to that drives where to focus your efforts.

If you're concerned about too many camera triggers (recordings), ask yourself why? In my view too many recordings is only a problem if you have limited storage. In that case, let Clips and Archiving manage storage - set it and forget it. I learned this the hard way after manually trying to delete what I didn't need, and generally screwing the pooch by doing so. In my view, there is no benefit to obsessing about the number of clips created.

What IS a problem is false alerts. That is to say, Alert Notifications from unimportant events - generally you want to be notified (text, email, sound, etc.) when events of interest occur. In the days before DeepStack integration, the game was to fine tune the trigger tab/motion sensor in an effort to find the sweet spot to reduce false alerts and be notified of events of interest.

That's where DS helps. DeepStack is the traffic cop that decides whether to notify or otherwise trigger an event. It's a "go/no go" tool. Record to your hearts content, but alert me if you recognize these objects.

In general, my recommendation is don't worry about the number of recordings - let it do it's thing... and focus on getting notified when you should.

[SolarEclipse]

I finally gave up on Deepstack. Though it did function for a time approximately a day after a system reboot it'd fail and just not analyze anything. Then I'd reboot, it'd work for a day, then have to repeat.
So, I finally turned it off. Though it was nice that it could identify cars, trucks, dogs, people, etc. I could do the same myself like I was before I had Deepstack.

[YrbkMgr]

Sometimes it's better to set it aside for a while and come back to it.