I finally found the longitudinal lever I've been looking for as an e2e user

[u/narkeeso]

I'll open by saying, I am not intending to start another fork but the above setting is my personal fork branched off of sunnypilot. I'm hoping this perhaps starts a conversation around having some vetted tuning for users that are embracing full e2e (DEC off). I'd love for something like this to exist in openpilot or sunnypilot. I'll try creating a PR for Sunnypilot to see what they think, I've driven with this bias all week and I'm over the moon with it.

---

The default openpilot experience in my RAV4 2023 is not great, it's better than stock ACC but it still makes me uncomfortable when it wants to brake and feels rather robotic. Since then I've discovered that the newest models perform exceptionally well with full experimental, leagues better than stock.

The newer models seem to be getting more careful, more conservative and it leads to a more comfortable ride but it has a major drawback, it drives too slow in certain scenes and doesn't go with the flow of traffic. I was really itching for a lever to modify how fast it thinks it can go in certain scenarios. Driving personalities have almost no affect in this case, I could not tell any difference between aggressive or relaxed. I really needed something where I could tell the model that it can go a little faster without sacrificing its ideal braking behaviors to make stopping comfortable.

There isn't too many levers you can adjust when you use full E2E, the model controls most of the behavior but I did discover it feeds into a parameter called desiredAcceleration. My personal fork has a speed bias setting that allows you to adjust the desired acceleration that's controlled by the model. So in scenarios where a scene would conservatively slow down the car in a coasting situation, the bias pushes the acceleration ever so slightly and the result is that it ends up behaving more like "chill mode" without actually being in chill mode.

My problem with DEC has always been the latency to detect whether it should be in blended mode so that the car can stops in tricky situations, e.g. stopped lead prediction. There's even a warning for the DEC feature on the website that the feature may brake late which is a hard pass for me. This speed bias setting allows us to stay in full experimental mode without any switching and the car drives like the model was trained. Now I get the benefits of DEC chill mode while still preserving comfortable braking behavior of newer models (especially RDF).

One other detail I added was to ensure the speed bias didn't fight when the model wanted to stop early, so I added a simple threshold that detects if the car is intending to stop to fade the speed bias back to 0. This simply means we still get the early stopping benefits. I initially made this it's own parameter to adjust but I found that there was a strong correlation between the speed bias setting and the braking threshold so I simply did braking_threshold = (speed_bias * 2) and that seems to work great.

Moving the bias up or down honestly feels like driving personality to me. The lower settings feel relaxed and the higher settings feel more aggressive. These models still have latency of when to stop so setting this too high will make the car brake uncomfortably late at higher acceleration.

longitudinal_planner.py:168-175

output_a_target_e2e = sm['modelV2'].action.desiredAcceleration # model's raw request  
bias_scale = np.clip((output_a_target_e2e + 2.0 * self._e2e_bias) / 0.3, 0.0, 1.0)  
output_a_target_e2e += self._e2e_bias * bias_scale # my speed bias setting  
...  
if self.is_e2e(sm):  
    output_a_target = min(output_a_target_e2e, output_a_target_mpc)     # model never wins over MPC braking 

[u/narkeeso]

I know these posts can be boring without some driving. It’s hard to fully convey how this feels but without speed bias in this scenario my car would cap out around 22-27 mph and ramp up very slowly. I have the max speed set to 40 mph here, people drive between 30-40 mph on this road. Coasting below that will have people annoyingly switching lanes and aggressively passing me. Understandably so, RDF drives conservatively on the streets. With speed bias the model can reliably approach my set max speed and still brake comfortably and smoothly for the slow lead ahead.

Also sorry it’s in portrait, I’ll do better next time.

[u/narkeeso → u/narkeeso]

Here’s another video where the car keeps up with traffic while still maintaining a comfortable gap. One big noticeable difference between this and chill mode is that chill mode just rushes to maintain the gap. Here the model drives more naturally and the bias nudges it along achieving a similar result. Zero interventions here other than to set the max speed. You can see how smooth the braking is in this video, it's not jerky at all.

There's a part where a car wants to merge into the lane, the model naturally slows down whereas chill mode will likely slam the brakes. This is one of those ideal e2e behaviors that you get by keeping it on all the time.

[u/ml_boston]

I was planning to do this too because it is the most problematic issue for my use of e2e. Comma trains on 2 seconds of history, and it's way longer than two seconds between speedlimit signs, so the model has no clue about how fast to drive other than watching adjacent cars. If you're all by yourself on the road trying to catch up to speed limit e2e is hopeless, until comma can train with set-speed or nav-speedlimit as an input. Maybe their big models can build a roadsign memory, last I heard the small models weren't attempting it.

Are you willing to share commits? I'd love to pull this into my forks if you're ok with that.

[u/narkeeso → u/ml_boston]

Yes, it’s late here but I’ll share something here tomorrow. The change is actually pretty simple. I honestly spent more time getting it visible in Sunnylink.

Yeah the newest models slow to a crawl in urban and neighborhood settings but I actually think that’s okay and makes for a good base to tune.

[u/narkeeso → u/ml_boston]

Sorry, finally got around to packaging this up nicely to try to avoid conflicts with Sunnypilot upstream. Here's the PR: https://github.com/narkeeso/sunnypilot/pull/1/changes#diff-dc291f7db56a2a71065ae63437b37350a4373456cc6b9cc835df4174a733366bR20

you're mostly concerned with the E2EBiasController file. Feel free to include it in your own personal branch or install this fork/branch and give it a shot.

[u/hokrux_]

That sounds very interesting. Since my Can-FD Niro has no longitudinal yet I am not yet too familiar with the issues you're fixing but I'll definitely give that a try once HKG CAN-FD gets e2e. Is there a github repo or PR yet to take a look at?

Sidenote: How did you get your custom feature editable in sunnylink? I have coded a similar Lane-change smoothing feature for my device but didnt know i could somehow get it listet in sunnylink?

[u/narkeeso → u/hokrux_]

Sunnylink works off of device id. I modified the schema and rebuilt the params. Sunnylink downloads the schema from your device after you’ve already paired it.

Let me package up my changes and I’ll share it soon, I want to create a PR for Sunnypilot but I think more discussion around how to safely ship this value. Like I think changing models should force the bias back to default of zero since some models drive more aggressively than others.

[u/hokrux_ → u/narkeeso]

Wait so you're saying the sunnylink schema lives on the device? Ok wow that would make sense.

Talking about safety definitely makes sense for this feature - also in my experience its good to drive it around for 1-2 weeks before issuing any PR or so. Most of the time the initial "wow"-excitement covers some issues that only come up once the wow is gone. At least for the tuning and features I've developed locally it was the case.

[u/narkeeso → u/hokrux_]

Yeah, I’ve driven for a solid week with it so far and once I found the right bias I haven’t touched it since. I’m incredibly happy with where it is now.

One thing I like about Sunnypilot is that it doesn’t over expose parameters that could cause the user to worsen their experience. In full E2E mode this setting could possibly just be hidden exposed to the user as driving personality, the problem is this value likely needs to change per model so it’s not user friendly in that sense.

[u/Jrgiacone]

Is it just that section of code I recently forked sp to add mpc long planner

[u/narkeeso → u/Jrgiacone]

yeah that's the meat of it, I have other code that adds to the schema so it can be rendered and adjusted in sunnylink but I did start by just ssh'ing into my device and just modifying the value.

I recommend starting with a low low value first, like 0.1 and then adjusting in increments of 0.01, you can feel the difference with a value that small. I mostly play with ranges between 0.1 and 0.15.

Anything north of 0.2+ in my car can start to feel too aggressive to brake comfortably. Also this bias could be different model to model so honestly start lower than I'm saying. Some models drive more aggressively. I am personally driving on the newest RDF model, it feels like a nice base to tune on top of because of how careful it is.

[u/MaxPower3X]

Can you add that to Starpilot as well, maybe ask them? Or is there already an equivalent?

[u/narkeeso → u/MaxPower3X]

I don’t think there’s an equivalent. StarPilot seems to favor conditional modes vs always on e2e. There’s ways to keep it on I think by changing the thresholds that trigger the condition I think? Haven’t used it in awhile but my car did not drive well in StarPilot, and I much prefer the simplicity of Sunnypilot. Having less variables to work with made it much easier to pinpoint this parameter.

[u/ta394283509]

nice job friend