Camera Stream Optimization

Details on the five major steps to configure and deploy Blue Iris.
1. Connecting IP cameras.
2. Configuring your home or office network for Remote Access via a web browser or the phone apps.
3. Setting up Storage and Recordings.
4. Creating Triggers and Alerts.
5. Creating Schedules and Profiles
Post Reply
varghesesa
Posts: 20
Joined: Thu Jul 11, 2019 9:52 pm

Camera Stream Optimization

Post by varghesesa » Thu Feb 04, 2021 3:18 pm

** If you are reading this article, we assume you went through all the necessary steps in the "IP Camera Connections" article to connect your cameras. **

Introduction
This article documents the many pitfalls with camera settings and helps you self-correct.

If you prefer to watch the webinar associated with this article, checkout our YouTube channel. Webinar name: Camera Connections and Streams.

Reference: If you want to learn more about how streaming and encoding works, a great reference article was created by IPVM titled "H.264 vs MJPEG - Quality and Bandwidth Tested". If you are not familiar with IPVM, they are a great resource for research and information regarding surveillance and security cameras.

Issue
We receive tickets associated with the camera streams being choppy, lagging in time, stuttering, loses signal or no signal error followed up with, however, the camera is up and working fine in the camera vendor's app.

The issue here is the vendor app is pulling streams from a proprietary port and protocol. The app is also an active solution, i.e. a user opened the app and wants to view a camera so the camera wakes up and delivers a quality, bandwidth optimized stream to the user.

Blue Iris (BI), like all VMS systems, is passively viewing all the time. Because we are a third party solution, we have access to different ports and settings on the available ports are crucial for BI to deliver a quality user experience.

Diagnosis

BI provides two ways to understand the health of your camera streams. For an individual camera, Camera settings -> General tab provides all the relevant information for an individual camera.

camera_stats_optimized.png
camera_stats_optimized.png (228.82 KiB) Viewed 411 times

Main stream stats:
  • Resolution of camera: 2.1 MP
  • Frames per second (fps): 31.76
  • Key frame interval per second: 0.70 fps
  • Bit rate: 381.9 kB/s
Sub stream stats (if connected):
  • Resolution of camera: 0.3 MP (640 x 480)
  • Frames per second (fps): 29.77
  • Key frame interval per second: 0.60 fps
  • Bit rate: 42.4 kB/s
Frames per second and Key frame interval per second have the most bearing on smooth streams. For most people, an FPS value above 12 fps and a key frame interval above 0.5 and ideally 1.0 are good guidelines for a good streaming experience.

To get a holistic view of the performance of all your cameras, go to Status -> Camera tab.
camera_log_optimized.png
camera_log_optimized.png (59.57 KiB) Viewed 2536 times
  • Name: Name of camera
  • IP: IP Address of cameras
  • MPixels: Camera resolution main stream/sub stream
  • HA: Hardware Acceleration (I = Intel, N = NVidia)
  • Elapsed: Length of time camera has been running
  • FPS/key: frames per second / key frame interval per second for main stream
  • Bitrate: bitrate for main stream
  • Sub FPS/key: frames per second / key frame interval per second for sub stream
  • Sub Bitrate: bitrate for sub stream
Status counters
  • Motion: Motion represents the number of motion events, not necessarily leading to a Trigger or Alert.
  • Trigger: Trigger represents the number of trigger events, when there was sufficient motion to trigger,
    or the camera was triggering in another way. If this cell is black, the camera is currently in
    the triggered state.
  • Alerts: Alerts represents the number of times that a trigger resulted in one or more alerts fired—
    emails, push notifications, alarms etc.
  • Clips: Clips represents the number of files created. If this cell is black, the camera is actively
    recording.
  • Posted represents the number of frames sent via FTP or saved to disc according to settings on
    the Post page in camera settings.
  • Webcast: Webcast represents the number of frames viewed by web server or app users.
  • NoSignal: NoSignal represents the number of times the camera signal was lost. The signal may have
    been immediately restored causing no loss of video, or it may have been out for longer.
    These events are logged to the Log page in status.
Troubleshooting
At this point, you should know what your cameras are doing and where they are performing poorly.

Keep in mind, users often set hardware acceleration on globally and later notice some cameras have switched from Camera settings -> Video -> Hardware decode from Default -> No. BI detects errors decoding your camera streams using your hardware and changes the setting behind the scenes. The steps discussed below may help resolve the issue. Alternatively, driver updates may occasionally resolve the issue. And at times, the only solution is to leave the setting at No and let BI decode the camera stream via software. As long as the CPU utilization is reasonable, the No setting should be fine.


Camera encoding settings

BI only works with the H.264 and H.265 standards. Camera manufacturers have tweaked their encoding to optimize bandwidth for their apps. H.264+, H.265+ (note the + sign) Smart Codec etc are all proprietary encodings that can only be used by the vendor apps. When customers report the cameras work on their app but not in BI, the encoding is often times the reason. The No Signal Error is almost always due to the fact that the cameras are using a proprietary encoding.

Also best to start with H.264 encoding before moving to H.265. Also, choose H.264 Main/Baseline instead of High if given the choice. Provide BI the easiest connection available first and increase the complexity if and when needed. Finally, it just may not be possible for your hardware video acceleration (hwva) to handle the camera's encoding perhaps due to old drivers. In that case, you may just need to turn hwva off for a particular camera.

To summarize, many camera settings can affect the video quality seen in Blue Iris. Think of these settings as levers that can be adjusted to provide the optimal video stream.
  • H.264 vs H.265
  • H.264/265 Encoding Profiles: Baseline -> Main -> High etc
  • HWVA. On or off?
  • Key frame interval
  • FPS

Video Streaming Issues

Examples of video distortions

Gotcha 1: Green bars

Encoding-HighToBaseline.png
Encoding-HighToBaseline.png (173.34 KiB) Viewed 41 times
Fix/lever: Simplify encoding. In this case, we changed the camera's encoding from H.264 High to H.264 Baseline. Alter any of the above Camera encoding settings until streams work.


Gotcha 2: Ghost images
ghost.png
ghost.png (174.73 KiB) Viewed 30 times

Ghost images occur when BI cannot decode the intermediate frames (p-frames / b-frames) coming from the camera. The image on the left, i.e. the actual scene was in the same mp4 video and would flash every 1s which was the key frame. In the ghost image (intermediate frame) you can kind of make out the mailbox behind the man walking. I focused in on the ghost image in the second image so you could see the correlation to the original scene. The image is not to scale.
Fix/lever: Turned off hwva. In this case, the encoding/decoding incompatibility was between the camera and Intel QuickSync. For some reason, the camera stats were saying the key frame ratio was 30 (mjpeg stream). It was almost as if the H.265 encoding was interpreted by the hwva as an mjpeg stream and thus rendering each intermediate frame as if it were a key frame. When hwva was turned off, the key frame ratio was 1 just as the camera settings show (GOP 1 I-frame / 1s).
The user also stated when exporting the recording to mp4, the default setting is to re-encode using BI software which worked fine. Another clue, the hwva was causing the issue. The user ended up using his mobile device to create the video which is different from exporting from the server to share the issue.


Gotcha 3: Pulsating jaggies, cameras are stuttering or routinely glitching

pulsating jaggies.png
pulsating jaggies.png (200.56 KiB) Viewed 35 times
Symptom: The video starts out fine but consistently every 1s or 2s, i.e. key frame interval, the image gets very pixelated. The key symptom is the fact this happens in a consistent interval like every 1s or 2s implying BI or the stream is having issues processing the key frame.

Example 1: This often implies the camera resolution cannot be handled by the graphics card.
Fix/lever: Turn off hwva

Example 2: The issue can be due to the camera settings. The camera was set to Constant Bit Rate (CBR) instead of Variable Bit Rate (VBR). Many cameras have a default key frame rate set to 0.5 which means a new key frame (full frame) is sent every 2s. If the CBR setting is too low, then the camera needs to perform image optimizations to get the frame across the network. Those optimizations lead to the glitches. So simply changing the camera setting to VBR instead of CBR should resolve the issue. In general, setting your cameras with VBR is a better user experience. CBR should only be considered if you have network bandwidth issues.



Examples of Stream quality issues


Gotcha 4: Choppy streams

Symptom: Choppy streams, i.e. the live view or the playback is not smooth or the seconds on the time overlay are not sequential. These issues are almost entirely due to camera encoding settings. If the cameras are speaking Spanish and BI is speaking English, neither party can understand each other resulting in a poor experience.
Fix/lever: Look at the key frame ratio in Camera -> Stats. Anything less than 0.5 may provide a bad experience. Set Key frame interval = fps, e.g. 15.


Gotcha 5: Camera streams are lagging or sluggish or high latency
Almost always, camera streams are sluggish because BI is not receiving enough key frames (i-frames). As stated above, key frame ratios less than 0.5 lead to sluggish cameras.

Rule of thumb: Make sure the key frame (i-frame) interval on your camera equals the FPS for the camera. The only reason you would consider increasing the interval is if network bandwidth is becoming an issue.

Furthermore, keep settings simple such that all the streams have the same FPS. Choosing between 30 fps or 15 fps is a user preference. Setting all streams to have the same fps makes time reconciliation easier for BI. Turns out certain vendors like Hikvision set different fps (10, 15) for different streams. Different fps settings unnecessarily complicate multiple streams for BI.

Examples:
  • Camera fps = 30. key frame interval = 30 (1.0 fps). Can consider key frame interval as high as = 60 (0.5 fps)
  • Camera fps = 15. key frame interval = 15 (1.0 fps). Can consider key frame interval as high as = 30 (0.5 fps)
Every camera vendor has a different UI. However, observe the default settings for the i-frame interval for each of the streams for the camera vendor (below). All the streams are 15 fps. However the i-frame interval is 30/60/120! The only setting that may provide an ok user experience is the main stream and most users will record D2D for the main stream. If dual streams are connected, the live view will probably be choppy, slow, sluggish! The i-frame interval for the two streams chosen to be used by BI for below example needs to be lowered to 15.
key-frame-sample_optimized.png
key-frame-sample_optimized.png (80.17 KiB) Viewed 2533 times

Gotcha 6: No signal / Loss of signal

The Status -> Camera tab as mentioned above maintains No Signal stats so you can easily get an idea on how big is the problem. Furthermore, Camera settings -> Watchdog allows you to set alerts when a camera losses signal. This is a good tool to understand the scope of the problem and investigate the issue at the time it occurs.

GPU / Hardware Incompatibility
There is nothing more frustrating than doing a BI update and all your camera streams stop. The below was a real ticket. :) The customer was convinced the issue was with the BI update.

no-signal_optimized.png
no-signal_optimized.png (65.33 KiB) Viewed 408 times

Rolling back to the previous version or last stable version did not resolve the issue. The customer was convinced the issue was with the BI update. Instead of trouble-shooting, he waited for the next update, hoping for a fix.

Unfortunately, no fix came. The customer then revisited my trouble-shooting email a few weeks later. I first asked the customer to turn off cameras and to observe whether other cameras started working. I suspected the CPU was being overworked. No improvement after disabling all the cameras but one.

experiment-1_optimized.png
experiment-1_optimized.png (57.58 KiB) Viewed 405 times

My next suggestion was to turn off hardware acceleration. After doing so for the non-working cameras, all came back online!
experiment-2_optimized.png
experiment-2_optimized.png (141.94 KiB) Viewed 408 times
We suspect the root cause was probably a Windows update or GPU update that disrupted the camera streams. Moral to the story is BI is dependent on many external drivers and software. All these 3rd party dependencies can lead to instability. Second take away is turn off hardware acceleration if a camera or group of cameras constantly has no signal and see if issue goes away. Hardware acceleration is one of the few settings that can affect many cameras at once.

FYI, the software now turns off HA automatically for a camera if it determines the stream is not compatible and this is logged to Status->log as well. If you turned on HA and noticed in Camera settings -> Video tab, Hardware decode is set to NO, the software made the change based on errors. If the CPU load is fine, then you can leave HA off for a camera. If you want to fix the problem, you can go to the camera settings on the camera and adjust the encoding to a more simple format that may start working with your hardware. A simple example is if the camera was set to H.264 High encoding, adjust the setting to H.264 Baseline. If you were using H.265, switch to H.264.

Networking
If issue persists, the problem is with the camera settings or BI has issues with the camera stream. First, determine the health of the stream by first looking at the status -> camera stats as mentioned above. Is the FPS > 12 fps? If not, the camera is not sending enough frames to BI. Double check your camera settings. Next, this could be due to the network, the network settings or the router or the ethernet card on the BI server.

I had a user with 5 cameras, most of which were high reolution (4 MP+). The issue was the router which was eventually replaced. The problem was not obvious. 4 of the cameras worked for the most part (but they too would lose signal on occasion). The 8 MP camera (PoE) was the most problematic. Even more strange the lower resolution wireless cameras worked better than the 8 MP camera. The user had no issues after replacing his router.
Troubleshooting network issues is beyond BI support.


Camera settings
Is the key frame interval < 0.5? Then BI is not getting enough key frames as stated above. Key frame interval may be a camera setting. Key frames are also affected by camera encoding settings. Again, BI only supports H.264 (baseline preferably) or H.265 encoding. As mentioned earlier, other encodings are problematic.

A good test for lost signal is to connect to the camera via the vendor's phone app. If the BI connection re-establishes as soon as the vendor's phone app connects to the camera, this is a good indication that the encoding setting on the camera to BI is not following the H.264/H.265 standards as mentioned earlier. The camera app is re-establishing a "live" connection to the camera, basically waking the camera up.

Other Third Party Apps
Download the VLC player or another 3rd party player to your BI machine and connect to the cameras. Do they continue to play when BI states lost signal? This is really good information to know. If you cannot resolve the issue, open a ticket with support. If another third party app can access a camera, so should BI.

Narrow down the issue by turning functionality off
Turn the shield from green to red. Everything is turned off recordings, alerts etc. The cameras remain connected and stream live video. Do the cameras start working again? This could imply the CPU is overloaded when running normally, i.e. storage, remote connections, triggers and alerts etc are enabled.

Camera hardware
Loss of signal could also be due to maybe the camera being overloaded. If you are pulling dual streams, does the fps go up if BI only pulls one stream? If you can pull the main stream or the sub stream correctly, but not both together, you may need to reach out to the camera vendor for more information. Foscams seem to frequently have this issue.

Finally, when there is a loss signal, check Status -> logs to see if BI reports any errors.

RTMP vs RTSP Protocol
99% of cameras only support the RTSP protocol. On occasion, a vendor will support RTSP and RTMP. Reolink is one such vendor and the RTMP protocol seems to provide a healthier stream.
Checkout the Reolink Gotchas article for best practices on connecting Reolink cameras.

Camera gotchas
BI Support has also been compiling and sharing other gotchas associated with camera settings which we have uncovered through trouble-shooting with customers.

Final action
It's possible you tried all the checks above and still no luck. It could very well be your router (networking equipment) just cannot handle the traffic. With all these very high resolution (8MP) cameras becoming more and more pervasive, there is an increased load on your networking equipment. Closed a ticket yesterday after a user discovered the core issue was his modem/router could not handle the traffic from his Reolink 810A causing No Signal errors on BI(see below).
No, only one camera was initially causing trouble. I have 4 reolink and the amcrest. Interestingly, the wireless reolink worked great. The poe 810A was the most problematic,..and I think also the highest resolution. Hence, perhaps some timeouts or something with my old / bad modem. No problems since I replace the modem.

You may also want to consider deleting and re-adding the camera. I have customer testimonials stating their skepticism to doing so but it has worked on occasion. Some mysteries with Windows and software will never be solved.

Further Optimizations
At this point, you should have good camera connectivity. Now you are ready to optimize your server further.

Record Direct to Disk
Camera settings -> Record tab -> Video file format and compression -> Direct-to-disc should be selected. Keep in mind, if you have configured dual streams, D2D is the only way to store both streams. If the re-encode option is selected with dual streams, the sub stream will only be stored. When playing back clips or alerts, you will notice this distinction.

D2D.PNG
D2D.PNG (19.67 KiB) Viewed 2530 times

Confirm BI still works well during playback of clips and / or alerts.

Hardware Acceleration
Hardware acceleration can be turned on/off globally. Global settings -> Cameras -> Hardware accelerated decode
HA.PNG
HA.PNG (34.29 KiB) Viewed 2529 times

Hardware acceleration can also be turned on/off at the camera level. Or if you have an Intel CPU with a NVidia graphics, for example, you could mix and match which cameras use which hardware to balance the load. At the camera level, the setting is Camera settings -> Video tab -> Hardware decode. As mentioned above, the Status -> Camera tab will display which hardware is being used by which camera.

BVR.PNG
BVR.PNG (29.1 KiB) Viewed 2529 times

With dual streams and hardware acceleration, the need for "Limit decoding unless required" and "Also BVR" are largely no longer needed since you get such HUGE CPU savings with these two new features. They remain in the software for backwards compatibility for those users that have not upgraded their hardware or updated their software.

Limit decoding gotcha: Keep in mind if you have a key frame ratio less than one, you will notice artifacts with the Limit decoding setting. In particular, a common key frame ratio of 0.5 means a key frame is sent to BI every 2s. Limit decoding to save CPU usage tells BI to only decode key frames and ignore the rest. By doing so, the camera stream will only update every 2s in the live view. You many not notice this, unless you have the clock overlay. You will suddenly notice the clock increments every other second instead of every second!

Also BVR gotcha: If you see artifacts in the playback, turning off Also BVR may be a quick fix.

Choosing hardware acceleration is as much an art as a science

Example 1: GPU improves quality.

You never know if your hardware acceleration will work in your environment until you test with your cameras. For example, observe the image from this high resolution camera from the camera's app. The colors and contrast are very vibrant.

camera app.jpg
camera app.jpg (100.22 KiB) Viewed 514 times

After connecting the camera to BI and using the default software decoding for playback, observe the image from BI. The colors look less vibrant with lower contrast.

software decode.jpg
software decode.jpg (92.67 KiB) Viewed 514 times

Now observe the playback with the NVidia decoder. The color and contrast looks equivalent to the camera app!

Nvidia NVDEC.jpg
Nvidia NVDEC.jpg (104.48 KiB) Viewed 514 times

Because the Nvidia card provided a better stream, the customer also turned on Video settings -> Also BVR. This was a case where the Also BVR setting improved was enabled because the quality and thus user experience was improved, not because the customer was hoping for CPU savings.


Example 2: GPU leads to artifacts.

Sometimes, the GPU decoding can lead to artifacts.

gpu-artifacts_optimized.png
gpu-artifacts_optimized.png (112.83 KiB) Viewed 514 times

Because the hardware acceleration was decreasing the quality and user experience, this customer chose to turn off hardware acceleration for this camera. The server could handle the additional CPU load and the customer was more concerned about the quality of the user experience.


Some final thoughts

This article progressively moves you through the steps needed to optimize camera connections.

Camera settings -> D2D Recordings -> Hardware acceleration.

When troubleshooting, work backwards, simplify the problem and see if the streams improve. By working backwards, you can identify the issue.

Turn off hardware acceleration -> Switch recording back to re-encode -> Adjust camera settings
  • Turn off hardware acceleration. Camera settings -> Video tab -> Hardware decode. Set to No. Does the live video / streams improve?
  • Switch back to Re-encode in recording settings. Does playback improve?
Next steps / Submitting a Ticket
At this point, if you cannot get a good quality stream, provide the following information:
  • Make / Model of camera.
  • Identify any errors in the Status -> log.
  • A screen shot of your camera stats. Status -> Camera tab.
  • The article mentions many checks to resolve the issue. List out what you tested and results. In particular, what are the video stream settings in your camera (encoding, fps, i-frame interval etc). This gives us better insight as to what may be the issue.
  • Send the .reg file for one of the problematic cameras. (Camera settings -> General tab -> Export button) Not necessary to send all the cameras .reg file. Fixing one often leads to cut/paste fixes for the others. There should be no privacy concerns. Unless the cameras are on the internet and we know the WAN address, camera access is impossible.
  • Put the camera on the internet for remote access/testing. We would be happy to do some remote testing.  Could you please send a WAN address for this camera for testing purposes with necessary ports, usually just 80 and 554 (RTSP) and 8999(ONVIF if available).  Don't forget a temp login as well. If you put the camera on the internet we can take a look.  This video explains how to do so. To be clear, we need direct access to the camera, not the BI web interface. 
Post Reply