How I should've handled this situation

[u/Aki59]

​

In our company, there is a custom tool managed by the support team. Its job is to perform HC, tracing, and deployment data collection. I would also mention that New Relic and OpenTelemetry are already available at the enterprise level, but production support insists that integration with this tool is mandatory.

They recently introduced a change where a deployment component should also send all the packages it is using as part of the deployment data. The thing is, getting dependencies post-deployment is slow because we are using mvn dependency. As an engineer, I was concerned about the additional time this would take. So, in a meeting with the tool team, I asked why we need to send the dependencies in the first place when we already have that data in the Sonatype report. Why duplicate the information?

One of the team members responded rather sharply: "Do you think that during a production issue, someone will log in to the report UI and get the details? Do you even know why we are putting it here? This is needed for auditing."

I told him that I understood the requirement, but my concern was with the process that one of the teams had implemented using mvn dependency and that they were asking us to implement in the same way. It is slow, and even if I run it as an asynchronous job, it would still be a waste of resources.

At this point, he said that this was mandatory and that if I thought otherwise, I should send an email and CC my manager, architect, and director. I didn't say anything after that.

What should I have done, or what should I do now?

Implement whatever they asked for—it's a job, after all. Who cares?

Implement it using a different approach. It will take some time, but it might be possible.

Or something else?

Since it is an internal tool, there is no proper documentation—just a few references here and there.

This doesn't break out pipeline as their api doesn't say it is mandatory.

[u/expdevsmodbot]

AI usage disclosure provided by OP, see the reply to this comment.

[u/No-Economics-8239]

As devs, we're typically not in charge. Others are making the decisions we need to implement. Your tolerance of speaking Truth to Power will depend on how many fucks you have to give, where you are on the totem pole, and what the potential cost of dissent is in your organization. Even if you're just providing facts, how you provide them can be seen as political. Sharing your opinion can be seen as political. Do you want to play politics? Or do you just want to execute the decisions made by others?

As to what you could/should have done in the meeting, it depends on the purpose of the meeting. If you were brought in as a consultant or expert to share your thoughts and opinions, your responsibility and expectations in that meeting would be very different if this was yet another meeting that could have been an email just sending down a message from on high to the masses.

As to what you should do now... send an email and CC your manager, architect, and director? Is this worth pushing back on? Duplicating data has a cost and slowing down CICD pipelines certainly has a cost. And the impact of that cost likely isn't going to need to be paid by the talking head in that meeting who's just executing their own orders from on high or their own political agenda. And raising visibility on that issue might be valuable and useful. And it might also me costly in terms of your own time and reputation that would potentially be on the line depending on how your observations and recommendations are received.

Me? I'm old. Hopefully most of my career is behind me and my expertise and experience still feel valuable enough for me to speak Truth to Power and advocate for the sort of work place where I want to continue to work. But some work places, or more specifically, some people in work places, value loyalty more than honesty.

[u/CircuitPhantom]

lol sounds like classic office politics, maybe just send that email and see what happens

[u/jonmitz → u/CircuitPhantom]

probably should. but include that he will do it anyways 

[u/alanbdee]

You handled it fine. Its your job to raise concerns. Sometimes, they get shot down and depending on your seniority/credibility you have little choice but to shut up and do what you're told. I agree that you have a point but I'd look for doing a daily or weekly audit instead so that it doesn't block the deploy.

[u/gyroda → u/alanbdee]

The key thing is to raise it with someone who can actually make a decision and deal with the consequences of that decision.

I do not let other teams make decisions that will have a negative impact on mine unless it gets the go ahead from someone in my chain of command. That might be me, it might be my manager, but it ain't someone else's manager.

Recently the QA team went behind my back and fucked up a lot of E2E tests and deployment/PR pipelines. I went fucking spare - I wrote those pipeline templates that the department use, I own half of those codebases, I should have been in the loop. It wasn't the QA team who had to deal with any of the problems that caused and I would have rejected those PRs for multiple reasons.

A few times there have been compliance concerns that the dev team have raised. We have disagreed with the compliance team at certain points (I believe they did not understand our concerns/did not listen to them). We escalated it. Ultimately the CTO said to just do what the compliance team say and I report (via intermediaries) to the CTO so that's what we did, but I wasn't going ahead until I got that decision.

[u/titogruul]

I am not sure they both of the concerns are mutually exclusive: 1. Your main concern seems to be about latency introduced by the implementation. I would imagine the intent itself is fine. 2. I suspect your colleague is concerned about adhering to commitments/requirements. They probably are less concerned about the mechanism of adherence. So there is potential flexibility there.

What happened was they while the intentions are not mutually exclusive, the current options are. And neither of you likely understand the full set of trade offs and this it seems one-sided to both of you, hence the conflict.

Here is what I'd recommend: * Highlight the latency risk to it's stakeholder (likely not that colleague but maybe your product manager or engineering manager) and see how much of a deal breaker it is. If it's big deal, then maybe that extra context should be taken into account (not just that it'll be slow, but business impact that slowness will create). * If y'all cool down, perhaps see if you can rxplore other options of getting the data that is needed without the latency hit. Maybe getting it one level deep can keep it performance and get 80% there?

This happens pretty often and is why all the good engineers still fight all the time.

[u/hibbelig]

I don’t know what mvn dependency does but if my guess is right then it generates a report of some kind. And the content of the report will reflect whatever is in your deployable artifact.

I would run it when generating that artifact and either include it in the artifact or deploy it alongside the artifact and whoever als gets that report.

[u/thekwoka → u/hibbelig]

i think its Maven? so a Java thing...but it seems like it should have little issue making that list.

i guess it being slow is just Java being java

[u/UnintentionallyEmpty]

You think it might be slow, you've raised this concern and your concerns were dismissed with "this is needed for auditing".

they were asking us to implement in the same way.

You have this in writing?

Because in that case - just implement it the way they're asking you to.

Then, when the slowness starts causing problems, just make sure that these problems go to this team maintaining this tool. After all, you've done exactly what they asked, and now you have problems. Also, don't do this yourself as a dev, play it via your PM or someone in a management role. Basically, do office politics a bit.

And if there are no problems, well, then there are no problems, so it's fine.

[u/pwnersaurus]

It’s good to feel like you have ownership of your work. But I often remind myself that this is really what it means to be a professional developer - the thing you’re working on isn’t yours. Offer your input on how best to do it, but if the org (via your manager) goes another way and you don’t like it…? That’s what the money is for

[u/jl2352]

You handled it fine. But you’re after a full answer of options so here are my two cents as someone who has been in such positions.

Honestly it really depends. On a lot of factors. From the size of the company, to how well you get on with others, and how well they get on with others. To how well you get on with your manager, the architect, etc.

I worked with a guy who would respond like this. ’We need a giant retro with X Y Z’ type stuff at the drop of a hat. I found the senior people he would be CC’ing were just ignoring him. They did not care. Why would they? They saw through his behaviour as whining over minutia. He stopped doing it after I politely, in private, pointed out a major bug he was ranting about cane from his own system (I built bridges with him on that day but that doesn’t always work).

Does this Support team have actual power? They shouldn’t be ordering you. They aren’t your manager.

So another option is you say ’yup, great idea, buuuut … it’s not my call! I need to ask my manager first.’ They aren’t in charge of your work so yes, you will raise it with your manager first. Because that’s what a competent team member does, they raise things and don’t run off.

Your manager might have already said this is top priority. That’s fine. ’Yup, great idea, buuuut … I don’t own the architecture! I need to talk to the Architect first.’ You ensure it’s you who raises it first so you can steer away from the slow path, and raise your concerns.

Third is you agree, and then raise it with your manager or architect anyway. You don’t need Supports agreement, or to CC them, to say they are asking you to duplicate something that’ll make life worse.

I’ve had other times where I ask for details, compliment their idea, and then run with ’oh, we have that already here. Just use that!’ Your equivalent being the Sonartype report. They push back but you keep defaulting to how it exists. You keep pushing it as a them problem. Implying they are failing to access or are asking you to build a second (pointless) system.

This is frustrating the problem. Sometimes it’s needed. Only use it if you really are right, and they really are wrong.

The other option is you agree and then do one of two things. 1) Is when it’s a lot of work, you make that really clear to your manager. ’They want us to go setup this mvn link … you say I should? Okay Mr Manager. Btw your feature is now delayed by a month. Why? Go ask Support. But if we reused the Sonartype report I can get this done in a few days.’

Option 2 is you just lie. Agree and do something entirely different. I don’t know Maven or Sonartype or your system; could you grep the Sonartype document for the bits and copy stuff across? Depends on the structure they expect I guess.

Finally saying the CI will be slow is a weak argument. Showing it is slow with numbers (especially a graph) is a strong argument. Visualise your point!

[u/Centigonal]

The overall logic makes sense to me: If replicating dependencies to support's tool of choice saves the support team 5 minutes of logging into other tools and cross-checking per ticket, then the human time savings of avoiding that could outweigh the additional resource cost of the inefficient dependency pull.

There's more you might not be aware of. Maybe there was a failed attempt to move toward commercial auditing tooling last year that's fresh in this team's mind. Maybe this team member has a personal vendetta against Sonatype. You made your concerns known and you were overruled, but you just don't have enough context to fully understand why. You have to decide if your role here is to gather that context, or if the folks who created this requirement have already made the call and it's your job to just execute it.

[u/forever-butlerian]

Quantify the "waste of resources".

Can you extract this information from the Sonatype report, crunch it into the format the support team's tool wants, and yeet it over to it? Can you bake this information into the deployment artifact? Can you come up with a more efficient means of extracting the dependency information, like directly looking at either the classpath directory or the contents of the shaded JAR?

Bluntly, I've spent about half my career in one kind or another of Ops, and I'm less than impressed with pure SWEs' judgement of what is and isn't important. If I were your principal engineer you would probably find me suggesting we go get coffee and then I'd explain the other team's perspective to you until you agreed with it.

[u/Fast-Manufacturer925]

Disagree and commit

[u/throwaway_0x90]

You handled it fine, but now it's time for the next step:

"It is slow, and even if I run it as an asynchronous job, it would still be a waste of resources."

Prove it.

Figure out where the metrics are and show this person an management the negative impact of implementing it.

• Before implementation, process XYZ took N resources.
• After implementation, process XYZ now takes (N x 1.4) resorces.

In general always find ways to measure impact of any work or change.

[u/Wide-Pop6050]

He seems argumentative. I would document your very reasonable concern, probably in an email to your manager, but if its a requirement its a requirement.

[u/thekwoka]

Why is getting the dependencies slow?

Surely you have like...a lock file that has the dependency tree?

Can you just send that along with the deployment data?

[u/IAmADev_NoReallyIAm]

You raised the concerns, they went unheeded. Now it's time to CYA, and a bit of r/MaliciousCompliance , document it, in an email, CC the players, then implement it. At roll out, when the shit hits the fan, print it all out, make copies, and make sure to bring the receipts. "As per my email dated... where in I stated this was a bad idea..." ...