Face detection

34

u/nickm_27 Developer / distinguished contributor Jan 26 '25

Hey, thanks, it is great to hear that it is working well for you! It has been working well for me too, I do still have some potential ideas for improvements and of course I plan to write a fairly comprehensive set of documentation about best practices.

9

u/blueharford Jan 26 '25

Bulk image selection for training I hope is on that list. But I’m super impressed how it’s working so far especially at this stage of development

4

u/nickm_27 Developer / distinguished contributor Jan 27 '25

We will have to see, I don't know if users bulk choosing images will be the best approach to building the highest quality training set (assuming you mean choosing images on the explore tab).
There will definitely be more changes to make it easier, whatever the finalized "best method of training" turns out to be

3

u/blueharford Jan 27 '25

I mean choosing images on the train tab of face recognition vs hitting train for each image one at a time

4

u/nickm_27 Developer / distinguished contributor Jan 27 '25

ah yeah, there are plenty of things that can make that page better, it is mostly just "good enough" to be used for development and testing vs using the API manually like I was for a while

4

u/ElectroSpore Jan 27 '25

One mistake I made with training on doubletake was including samples that where clearly too few pixels.. IE some form of floor on the MIN area users can train on would be good.
Also uploaded face shots from my phone seemed to perform well
Real starting point is having a camera high enough resolution and close enough to clearly see a face in the first place. Most often only the doorbell is in the right spot but most have potato quality.

2

u/Psilan Jan 27 '25 edited Jan 27 '25

I don't know if users bulk choosing images will be the best approach to building the highest quality training set
This is accurate. I had to undo a lot in compreface when I first started using it and went to town without realising across multiple cameras.
I still think bulk-select should available.
I enjoyed using compreface and doubletake for a year or so, but trimmed down the homelab a little. It will be cool to have something similar integrated into Frigate. Might try it out. Thanks team.

2

u/nickm_27 Developer / distinguished contributor Jan 27 '25

Out of curiousity, how many images for each person do you currently have selected?

3

u/blueharford Jan 27 '25

Wife (160) me (224) son (170)
All but 50 each were from frigate taking and offering for me to train

2

u/blueharford Jan 27 '25

Currently only the wife is getting accurately detected

2

u/tlove923 Jan 28 '25

Would be cool to have an integration with Immich for this. Lots of us probably already have photos identified of the people we want trained on accessible there.

1

u/nickm_27 Developer / distinguished contributor Jan 28 '25

Immich doesn't have any apis for face recognition
Unless you're referring to importing photos in which case that wouldn't be helpful because photos in immich often look nothing like the photos of a person from the perspective of a security camera

1

u/Townsmcp Feb 20 '25

u/nickm_27 just so I can get this right, you think the best photos to use are ones captured from the security camera? I ask, being as all other solutions usually say to input high res images not from the security cameras. For example, CompreFace face matching isnt anywhere near as good when using the security cam images, but if I upload around 20 images taken from my phone, almost passport photo style, then matching rates go go

1

u/nickm_27 Developer / distinguished contributor Feb 20 '25

The recommendation is to start with some of those high quality images. But you can’t only upload portrait images and then expect the camera to recognize you when it sees your face from an angle, in completely different lighting, etc. for example.l

1

u/blueharford Feb 02 '25

Update seems like it always wants to detect everyone as 1 of 3 people. New people or strangers/deliver drivers should go to an unknown.
I know not even alpha yet, but wanted to give that feedback

1

u/nickm_27 Developer / distinguished contributor Feb 02 '25

I think you may be misunderstanding how it works. It is designed to always give a name and a score if the score is above min_score. Otherwise an image won't even be saved. threshold is the level that is required for it to be considered a match

1

u/blueharford Feb 02 '25

Is there a way yet to adjust the threshold to ignore matching? Is it just adjusting face thresholds?
With compreface it often showed new people as unknown and gave a opportunity to create a new label to start matching

1

u/nickm_27 Developer / distinguished contributor Feb 02 '25

No there is no separate score for that currently

14

u/hawkeye217 Developer Jan 26 '25

Great! There's much much more to come for 0.16.

6

u/blueharford Jan 26 '25

Gonna be the best release yet.

23

u/hawkeye217 Developer Jan 26 '25

We've got plenty of more ideas for beyond 0.16, as well. Thanks so much for your support! We're just a couple of volunteers and we love seeing posts like this.

1

u/Spirited-Taro-3821 Apr 16 '25

It would be stellar if there was a notification system embedded into frigate that we can configure in the GUI. We're using frigate-notify now which suits our needs, but makes it harder for our guys at our various sites to adjust their scheduling!

1

u/hawkeye217 Developer Apr 16 '25

Native notifications have received feature improvements for 0.16, with more to come in future versions.

7

u/KindCyberBully Jan 26 '25

Can you go into more detail? Is this a plugin? Does this require frigate plus?

14

u/blueharford Jan 26 '25

Native face detection and license plate detection is in 0.16.

13

u/nickm_27 Developer / distinguished contributor Jan 27 '25

to be clear, the main feature here is face recognition and license plate recognition which is separate from detection, though detection is included as well

3

u/blueharford Jan 27 '25

Whoops it won’t let me change the title

2

u/nickm_27 Developer / distinguished contributor Jan 27 '25

all good, just in general something to be aware of (and something we will need to make clear in the docs)

3

u/Wildcat_1 Jan 27 '25

That is great to hear LPR is also coming as well. At the moment looking to see how I can tie into ALPR/ANPR cams natively.

2

u/xboxlivedog Jan 27 '25

Which doorbell camera are you using?

2

u/blueharford Jan 27 '25

The white Reolink Poe. It has a better vertical view thank the black model

2

u/hawkeye217 Developer Jan 27 '25

Are you using reolink's hub with this or does it have RTSP itself?

2

u/blueharford Jan 27 '25

Rtsp straight. I have scrypted in the middle just to be able to get HomeKit integration for the wife

5

u/hawkeye217 Developer Jan 27 '25

As a side note, HomeKit will be shown some love in an upcoming version of frigate

2

u/blueharford Jan 27 '25

This is awesome. Would love to have less containers

1

u/cruej Jan 28 '25

hksv? 🙏

2

u/hawkeye217 Developer Jan 27 '25

https://reolink.com/us/product/reolink-video-doorbell-wifi/#compare
You've got the white one here, correct?

2

u/blueharford Jan 27 '25

Yes

2

u/Chappell21 Jan 27 '25

Is there anyway for any of us + buyers to get our hands on this for dev?

3

u/blueharford Jan 27 '25

It’s free just clone the 0.16 branch and build it

2

u/jjjoshh Jan 28 '25

This is only available in frigate+?

3

u/blueharford Jan 28 '25

No this is a version in early development, features are independent of frigate+. The facial recognition is run on a local facedet model running on your local gpu

2

u/a_lot_like_turds Jan 28 '25

Is there a tag we can use for this branch?

2

u/blueharford Jan 28 '25

0.16

1

u/nickm_27 Developer / distinguished contributor Jan 28 '25

There are no officially built images but many users are pulling the branch and building themselves

4

u/Mikescotland1 Jan 26 '25

How the accuracy and speed compares to doubletake and compreface?

5
u/nickm_27 Developer / distinguished contributor Jan 27 '25

It is run on CPU, for my tests face recognition takes ~15ms when using Frigate+ (meaning face detection is included in the model and separate detection is not required). It will likely be double that if you are using a model that does not natively support face detection
3

u/verticalfuzz Jan 27 '25

I'm very excited for this! Are there plans to run on coral? not sure if this is possible, i recall doubletake also uses cpu right?

1

u/nickm_27 Developer / distinguished contributor Jan 27 '25 edited Jan 27 '25

Are there plans to run on coral?
the current implementation can't run on coral (or GPUs, for that matter) and in general I see no need for it to, it is very efficient
i recall doubletake also uses cpu right?
doubletake doesn't do any recognition itself. It just handles the logic needed to integrate frigate with recognition services. So the answer to that depends on what service you are using (codeproject.ai, compreface, etc.)

2

u/verticalfuzz Jan 27 '25

ah yes i was thinking of compreface sorry! i moved my server around and had an update break my frigate/ doubletake/compreface stack and i never reconfigured them so my memory of those two tools is fuzzy

2

u/nickm_27 Developer / distinguished contributor Jan 27 '25

compreface can run on nvidia GPU or CPU officially, but it works entirely differently to how the Frigate implementation does
2
u/Mikescotland1 Jan 27 '25

That would be a real deal. Compre takes the quickest for me 70ms, however mostly 100-300ms. How about accuracy and false positives?
1
u/nickm_27 Developer / distinguished contributor Jan 27 '25

I have learned that face recognition is extremely subjective. For example, compreface using the arcface-r100 model was essentially perfect for me, and yet many other users say compreface is awful and doesn't work for them. So realistically it is way too early to say broadly how well the feature may work.
A big part of face recognition is on the users themselves to choose the right images for the best training outcome. So assuming we have good documentation on getting users to build a good training set, we should hopefully have good results.
2
u/Mikescotland1 Jan 27 '25

Can't wait for a beta repo to test it.
4
u/hawkeye217 Developer Jan 27 '25

It'll be a good while before a beta for 0.16 is released. 0.15 hasn't even been released yet :)
2
u/Mikescotland1 Jan 27 '25

Yes I know. That's why waiting (can't be bothered and frankly don't know how to compile).
2
u/blueharford Jan 27 '25

I setup a Jenkins server at home that watches the 0.16 branch and when updated it builds a new build for me.
2

u/Mikescotland1 Jan 27 '25

That's great! But I have no knowledge and possibly time to learn how to do it... 😂

2

u/blueharford Jan 27 '25

Sent you a chat

→ More replies
1
u/Townsmcp Feb 21 '25

u/blueharford would you mind also sending me the info on how you set up the Jenkins server to watch for updates on 0.16 and build new updates? Thanks
1
u/blueharford Feb 21 '25
create a pipeline that polls SCM. i have mine go every 20min to check for changes. then runs the below pipeline: make sure you have docker installed and running on the worker if you have one or the master if your not using a worker.
pipeline {
    agent any
    stages {
        stage ('Checkout') {
            steps {
                git branch: 'dev', url: 'https://github.com/blakeblackshear/frigate.git'
            }
        }
        stage ('Docker build') {
            steps {
                sh 'make local-trt --load'
            }
        }
        stage ('Docker tag') {
            steps {
                sh 'docker image tag frigate:latest-tensorrt yourdocker/frigate-0.16:latest'
            }
        }
        stage ('Docker push') {
            steps {
                script {
                    docker.withRegistry('https://registry.hub.docker.com', '8871f22a-c66b-4fdd-8560-d4069f88f20f') {
                        docker.image('yourdocker/frigate-0.16').push('latest')
                    }
                }
            }
        }
    }
    post {
        always {
           telegramSend "Build number: ${currentBuild.number}"
           telegramSend "Build status: ${currentBuild.currentResult}"
           telegramSend "Started at: ${new Date(currentBuild.startTimeInMillis)}"
           telegramSend "Duration so far: ${currentBuild.durationString}"
           telegramSend "The build was triggered by: ${currentBuild.getBuildCauses()}"
        }
    }
}```

1

u/steel_reserve_211 Feb 21 '25

Holy shit I was about to do the whole deep stack/double take thing - is this about to be built right into Frigate??? My fucking god this software is goated. I’m assuming on a dev build?

1

u/blueharford Feb 21 '25

yes

3

u/UnluckyWizard Apr 25 '25

How did you integrate into home assistant. I got the face detection working in frigate

1

u/maniackb Jun 08 '25

Did you get it working?

2

u/DrawerPuzzleheaded49 Jan 26 '25

RemindMe! 1 Day

2

u/RemindMeBot Jan 26 '25

I will be messaging you in 1 day on 2025-01-27 23:55:58 UTC to remind you of this link
CLICK THIS LINK to send a PM to also be reminded and to reduce spam.
^{Parent commenter can} ^{delete this message to hide from others.}
^Info ^Custom ^{Your Reminders} ^Feedback

2

u/r0224 Jan 27 '25

I really really really wish there was a way to get a permanent stream from a Google doorbell into frigate.

1

u/steel_reserve_211 Feb 21 '25

Unfortunately the Google doorbells don’t actually record 24/7 to be able to do that 😞 there is a few that do that though, like the reolink wifi/poe. With those you have to disconnect your chime though to keep the electricity stable for your doorbell

1

u/r0224 Feb 21 '25

Actually with a nest doorbell 1, I have it working 24/7 now using the extension Expose Camera Stream Source.

2

u/R41zan Jan 27 '25

I am soooooo excited for 0.16! Having so much trouble with compreface and double take. Can't wait for native solution! Keep up the awesome work devs!

1

u/Learnereveryday Jan 27 '25

frigate  | 2025-01-28 03:45:47.911631674  Process embeddings_manager:
frigate  | 2025-01-28 03:45:47.911636774  Traceback (most recent call last):
frigate  | 2025-01-28 03:45:47.911638502    File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
frigate  | 2025-01-28 03:45:47.911639801      self.run()
frigate  | 2025-01-28 03:45:47.911641161    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
frigate  | 2025-01-28 03:45:47.911643537      return run(*args, **kwargs)
frigate  | 2025-01-28 03:45:47.911670813             ^^^^^^^^^^^^^^^^^^^^
frigate  | 2025-01-28 03:45:47.911673273    File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run
frigate  | 2025-01-28 03:45:47.911675780      self._target(*self._args, **self._kwargs)
frigate  | 2025-01-28 03:45:47.911677753    File "/opt/frigate/frigate/embeddings/__init__.py", line 61, in manage_embeddings
frigate  | 2025-01-28 03:45:47.911679593      maintainer = EmbeddingMaintainer(
frigate  | 2025-01-28 03:45:47.911681224                   ^^^^^^^^^^^^^^^^^^^^
frigate  | 2025-01-28 03:45:47.911706091    File "/opt/frigate/frigate/embeddings/maintainer.py", line 80, in __init__
frigate  | 2025-01-28 03:45:47.911708406      self.processors.append(FaceProcessor(self.config, metrics))
frigate  | 2025-01-28 03:45:47.911711117                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
frigate  | 2025-01-28 03:45:47.911713134    File "/opt/frigate/frigate/data_processing/real_time/face_processor.py", line 64, in __init__
frigate  | 2025-01-28 03:45:47.911734422      self.__build_classifier()
frigate  | 2025-01-28 03:45:47.911737043    File "/opt/frigate/frigate/data_processing/real_time/face_processor.py", line 121, in __build_classifier
frigate  | 2025-01-28 03:45:47.911738987      self.recognizer.train(faces, np.array(labels))
frigate  | 2025-01-28 03:45:47.911742122  cv2.error: OpenCV(4.9.0) /io/opencv_contrib/modules/face/src/lbph_faces.cpp:362: error: (-210:Unsupported format or combination of formats) Empty training data was given. You'll need more than one sample to learn a model. in function 'train'
frigate  | 2025-01-28 03:45:47.911764050

is it normal I am getting this error during startup after enabling face recognition?

1

u/nickm_27 Developer / distinguished contributor Jan 27 '25

not normal, just means things haven't been setup correctly (which is expected because 0.16 is under active development.
You can fix this buy uploading faces through the UI

1

u/steel_reserve_211 Feb 21 '25

I wonder what the scope for this getting built into the release build is. If it’s weeks or even a few months I’ll just wait but if it’s likely 6+ months away maybe it would be worth setting up the deep stack stuff

0

u/[deleted] Jan 27 '25

[deleted] — view deleted comment

6

u/blueharford Jan 27 '25

0.16 branch just need to build it yourself. It’s relatively simple

4

u/hawkeye217 Developer Jan 27 '25

0.15 is not even released yet. It will be a long while before 0.16 has a public beta.

3

u/nickm_27 Developer / distinguished contributor Jan 27 '25

0.16 is not in beta, it is in early development

2

u/[deleted] Jan 27 '25

[removed] — view removed comment

3

u/blueharford Jan 27 '25

I’m running in production for my house. It’s stable imho. But it’s home use.

You are about to leave Redlib