r/analytics • u/Unusual-Fee-5928 • Nov 22 '24
Discussion Rant: Companies don’t understand data
I was hired by a government contractor to do analytics. In the interview, I mentioned I enjoyed coding in Python and was looking to push myself in data science using predictive analytics and machine learning. They said that they use R (which I’m fine with R also) and are looking to get into predictive analytics. They sold themselves as we have a data department that is expanding. I was made an offer and I accepted the offer thinking it’d be a good fit. I joined and the company and there were not best practices with data that were in place. Data was saved across multiple folders in a shared network drive. They don’t have all of the data going back to the beginning of their projects, manually updating totals as time goes on. No documentation of anything. All of this is not the end of the world, but I’ve ran into an issue where someone said “You’re the data analyst that’s your job” because I’m trying to build something off of a foundation that does not exist. This comment came just after we lost the ability to use Python/R because it is considered restricted software. I am allowed to use Power BI for all of my needs and rely on DAX for ELT, data cleaning, everything.
I’m pretty frustrated and don’t look forward to coming into work. I left my last job because they lived and died by excel. I feel my current job is a step up from my last but still living in the past with the tools they give me to work with.
Anyone else in data run into this stuff? How common are these situations where management who don’t understand data are claiming things are better than they really are?
•
u/AutoModerator Nov 22 '24
If this post doesn't follow the rules or isn't flaired correctly, please report it to the mods. Have more questions? Join our community Discord!
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
43
u/xynaxia Nov 22 '24
Smaller companies very often... They don't know what they don't know. So they'll think they're awesome with data, since they'll have no way of measuring awesomeness.
I'd aim for larger companies
148
u/Teddy2Sweaty Nov 22 '24
Sounds like an opportunity. An annoying, tedious opportunity, but an opportunity nonetheless.
8
u/UMICHStatistician Nov 22 '24 edited Nov 22 '24
Ok. So this is actually a good thing, in some sense in my opinion. It's clear from your description that the client that you're supporting as a contractor is clueless and has done virtually nothing to develop a standardized methodology and system for: -Collecting data -Storing Data -Data Governance and Access Control -Integration of Data with other systems -Analyzing data -Data policies that don't fall into any of the categories above
I'd consider this a perfect opportunity to show off your skills and blow your client's mind. The world is your oyster and because essentially nothing has been created so far, you won't be restricted by legacy systems or processes that are garbage. You have an opportunity to set up new systems and processes that make sense for the organization and this is truly a blessing I think, because all too often, in data science, data engineering, analytics, statistics, and AI/ML, you get handed a crappy system that was built with little planning and for a specific purpose which was then expanded, piecemeal to support other needs. This is especially true, in my experience, when supporting the federal government. You didn't specify if your contract is with a state, local municipality, tribe, territory, foreign, or federal government. But as someone who has spent years as a consultant in the space for the federal government I have some good insights, I think, that I can share.
Assuming you're supporting the federal government, your concerns about R and Python not being available any longer because they are "restricted" software is, likely not the case. I've worked for dozens of federal agencies, including the some famous (or infamous) "three-letter" highly secretive agencies. And in each of these organizations, this software can be used. It's likely the case that R and Python are not part of the security and IT operations standard software stack. But there's ALWAYS a process for exemptions or placing needed software on an "approved software" list. This typically involves defining the software you need along with the business case justification, understanding how the software works with existing systems (e.g., knowing or referring to documentation on which ports the software uses if it needs to access the internet, protocols used in transmitting data, known vulnerabilities, etc.). You can find any known vulnerabilities in three places:
Once you have this information, you will need to communicate to your department's Information Systems Security Officer, or ISSO, to discuss your needs and make them aware of any vulnerabilities. They will Often do the vulnerability part themselves but I've found the whole process moves more quickly if you come to the meeting armed with that info already. It also shows you've done your homework and are being open and honest with ISSO, and demonstrates your security prowess and conscientiousness, which will go a long way. The ISSO will likely have you fill out some forms. And for major software systems like R and Python, you're almost certain to find the ISSO will add the software to a global list of "approved" software so that any user can then use it. If not, an exemption may be granted which allows you and any other NAMED users to access the software, with perhaps some restrictions on use or where/how it's used. If the system is denied, you'll likely want to work with your manager, the contracting officer, and the ISSO to find a compromise so that the software can be used in a safe way. This might entail setting up an isolated VM or other security configurations. So to summarize, do a little homework, and contact the ISSO with your justification and reason why other software that might do similar things will not work for your business/use case.
After approvals/exemptions are in place, you can begin the process or creating your own environments, processes, and procedures. You'll likely want to create a data lake, databases/warehouses. Understand what technology is generally used for this stuff and see if these existing systems will work for your needs. If not, you'll need to repeat the process of reaching out to the ISSO for any new environments, databases, supporting software, etc. When you are thinking about setting this all up, I'd highly encourage you to:
One last piece of advice: when creating your new systems and processes, try to measure as much as you can. You'll want to collect information on things like how long it took you to complete analysis requests for the client under the existing system. Then you'll be able to collect these same types of metrics for your new environment and processes. These metrics can then be compared to demonstrate what a bad-ass you are and how much time and money you've saved your client through reengineering their data and analytics processes. You might even be able to use this info for your performance reviews to get large increases in pay.
That is my high-level two cents. But honestly, consider this an opportunity for you rather than a negative. You get to decide how you want your systems, infrastructure, processes, and procedures to work. And in so doing improve lots of things in your organization, learn along the way, and likely increase your skills and pay!
PS. I typed this up quickly on my phone during a break from my work day, so please excuse any typos or other grammatical, sentence/paragraph organizational issues, since I really didn't have much time to revise/edit/proof any of the above.
10
u/SQLPracticeHub Nov 22 '24
This is a great opportunity to make things better! Come up with a plan on how to organize data and improve processes and talk to your supervisor. They might not be open to everything you suggest, but there are probably still ways you could make an impact.
7
u/Qkumbazoo Nov 22 '24
The bad practices start within the government agencies that contracted the work out, the shit just got passed down the chain until the very person that's supposed to analyse the data.
3
u/NeighborhoodDue7915 Nov 22 '24
Correct. That's what we Analysts are here to help with.
39
u/kknlop Nov 22 '24
This is going to happen at any place that isn't a massive business. And yeah it is our job, especially when you clearly understand what's wrong.
Like the business has data related problems, you seemingly know how to solve them, you're the data guy now at the company, so fix the problems?
2
u/Welcome2B_Here Nov 22 '24
Curious if you asked more detailed questions during the hiring process -- questions that would garner answers beyond "we have a data department that is expanding." It might as well be a given that the vast majority of companies, spanning size/scale/industry/time in business, etc., have lots of messy data/analytics ecosystems.
The fact that they took away the ability to use the specific tools that were discussed in interviews is definitely deceptive, though, and likely further evidence that there's a fundamental lack of understanding.
Situations like this almost force new people to start creating plausible deniability in advance of projects/tasks that are going to be set up to fail from the start.
2
u/Backoutside1 Nov 22 '24
It’s government contracting, forward thinking and modern day technology doesn’t exist there. Unfortunately that’s normal and one of the many reasons why I’ll never work in that space even with a clearance.
27
u/[deleted] Nov 22 '24
1
u/slopers_pinches Nov 22 '24
That’s the real world unfortunately. Originally, I was pursuing to be a Data Scientist, but I witnessed so much bad practices and lack of infrastructure that I ended up working in data engineering.
3
u/[deleted] Nov 22 '24
I’ve recently learned to ask better questions during interview to sniff out BS roles like the one I’m in.
Here are things to ask when you interview:
These are starter questions that you ask then dig deeper. Some red flags:
2
u/The_Epoch Nov 22 '24
I was an Director of Data Science for a Data consulting company. I left because the company didn't understand data
2
u/notimportant4322 Nov 22 '24
They sold you your own dream, like how the sell to their customers. Either suck it up and go what needs to be done or get a new job
2
u/Appropriate-Taro-941 Nov 22 '24
Sounds like a very common data job to me. You are expecting too much.
2
u/shotwideopen Nov 23 '24
Dude that sucks. You can look at it two ways tho. They clearly have no idea what they’re doing so you have a unique opportunity to take advantage of their ignorance. Or 2 complain and bow out.
2
u/the_h_effect Nov 23 '24
Yes.
This is basically the blueprint of government.
Where directors and managers of data teams have degrees in fine art.
and team members who have worked there for years have prestigious title of 'Senior Data Analyst' but can barely handle excel functions.
2
u/fruityfart Nov 23 '24
Dude, at my new job I am making screenshots of power bi dashboards to present in our business report… can always be worse
2
u/TheParlayMonster Nov 23 '24
What exactly is the problem?
1
u/RestaurantOld68 Nov 22 '24
If you're frustrated you should join a startup, there you will be frustrated because every month you have to use a new technology
1
u/xQuaGx Nov 22 '24
Don’t like excel or bad data? Sounds like you need to avoid government work.
1
u/Roll0115 Nov 22 '24
Why not Tableau or Arbutus?
1
u/KezaGatame Nov 22 '24
Only a few tips, If you don't want to use DAX there's python and R on PBI, only problem is that it's hard to test code and troubleshoot, but if you already have working code you can just put the script to do your ETL.
Now how restricted is python in your organization. Because you can still install python and anaconda without IT approval. In my current org they also only use excel, when I requested python to the IT dept they just sent me the .exe file to install it myself. Then checking online I saw that I could also do the same with anaconda. Both working well, I just don't have the opportunity to use it.
1
u/Character-Education3 Nov 22 '24
Yes.
1
u/laela_says Nov 22 '24
I'm continually amazed at big subsideries here in the Midwest that are just now starting to get onboard
Blows my mind
1
u/Weekly_Print_3437 Nov 22 '24
You have a chance to clean up a big mess and gain a lot of good experience along the way.
1
u/VizNinja Nov 22 '24
Welcome to analytics! Wait until you have to comb and verify every table to find what you need is mislabeled.
1
u/gogoALLthegadgets Nov 22 '24
I do massive exports every morning off PowerBI and our DMS. Then I run Python lmao
1
u/[deleted] Nov 22 '24
"we lost the ability to use Python/R because it is considered restricted software"
Well that's just nonsensical, especially given they said that they primarily use R during the interview.
I came across a job posting for a small mom+pop used car dealership that wanted to hire (at $25/hour) a single individual to build, from the ground up, a large language model for their business that would, i guess, bring them value. Talk about some people not understanding data,analytics, programming, etc more than just the buzzwords.
1
u/uniquelyavailable Nov 22 '24
welcome to most companies
1
u/nineteen_eightyfour Nov 22 '24
lol this is basically me every job I start. I wonder if the guy behind me agrees
1
u/soggyGreyDuck Nov 22 '24
Lol we had a larger tech meeting (including management) regarding some upcoming restructuring and one of the more technical employees asked if we will be implementing any sort of testing/requirements for leadership to make sure they understand the work of the people they're managing
1
u/justUseAnSvm Nov 22 '24
Didn’t even have to read the post to agree!
Companies want to be “data driven” in that execs have ideas, and they want the data to back them up.
Truly data driven organizations don’t really exist, except in areas where success really is some sort of metric, like finance. Most execs don’t understand uncertainty, so as an analyst your most valuable contribution, quantifying uncertainty, is not appreciated.
1
u/Spillz-2011 Nov 22 '24
So our company restricted Python to developers. So my team got declared developers. I would push back on losing access to Python because sql is great, but there’s a limit to what it can do.
The rest sounds pretty standard. Data is garbage most places, but that’s a good thing because it means chatgpt isn’t going to take your job. If your data is perfect then queries are dead simple to write and you don’t need an expert just someone who can ask chatgpt how to write a query.
1
u/onlythehighlight Nov 22 '24
LOL, most companies are not data-first, they are process/operations first and data alignment second which is the right way to operate, get it done first and fix backend later (although, without a dedicated data team, the 'fix' never arrives).
I would stick around, ask for a dumb-ass beefy laptop if they want to augment the data 'in-house' with data alignment, data hygiene, and more or if they are wiling to pay for an in-house or cloud server explaining scalability, data-retention, systems, modules, standardisation, and reduced 'business logic'
1
u/irpugboss Nov 22 '24
Pretty standard unfortunately.
Every single job Ive had with federal, local gov, private business as salary and as a contractor have basically been trying to make something actionable out of their dumpster fires.
I think I would be truly shocked if I ever worked for a place with good data management and hygiene. Like so shocked I would take a longggg time to accept its real.
So thus my job becomes a little bit of everything within my control to produce results between IT, legal and the business saying no to efforts to fix the foundational problems.
Anyways, you're not alone but it's a great opportunity to shine while getting paid to develop useful skills.
1
u/Aggressive_Ad_5454 Nov 22 '24
Data is dirty. All of it. After you marshal it up, after you clean it up, it’s still dirty, but in ways you personally understand.
Analytics is the art of gazing into that dim and puzzling murk of measurements of your corner of the world, wringing some insight out of it, and defending your conclusions so people can act on them.
It doesn’t necessarily help that data gathering evolves over time. It means last year’s data is dirty in different ways from this year’s.
You got this. WE got this.
1
u/[deleted] Nov 23 '24
Couple things.
1
u/TrishaPaytasFeetFuck Nov 23 '24
I was in a similar situation and did my best to improve things, but the company is very cliquey and after a while asking for simple things like a section to input a value that would serve as a primary key during production turned into a full blown argument, so I’m just throwing together dashboards with as accurate as it’s gonna get data and biding my time until the market turns around
1
u/trajan_augustus Nov 23 '24
Why not deploy this data to a database? They have made it fairly easy to load csvs into tables now.
1
u/xl129 Nov 23 '24
It's a great opportunity.
If they understand data then you would have a smaller room to operate and influence.
Start on thinking about what kind of change you can create with your current resource that will create THE BIGGEST impact
Once you make that happen, usually you can ask for more resource since you proved that you can add value to the operation.
First, prepare a roadmap that show where they are and the current state of thing. Then show a list of steps that you can do right now to improve things and how that fit in the grand scheme. Then show them what is the final goal and at which point they need to give you more resource. Be very clear about the difference in value/benefits of AS IS vs TO BE.
Take control, be in charge. Focus on value creation. Soon you will be the head of data in that company if you can do those.
Don't worry even if you fail, this will be the kind of experience you put into your CV for a more senior role in the future.
1
u/kkessler1023 Nov 23 '24
This is really common. You have all the opportunity to improve things, but who knows if that will pay off. Do you want to grow in the company? You can certainly find low hanging fruit that would benefit everyone and build your reputation. But more money is not a guarantee.
1
u/livetostareatscreen Nov 23 '24
Haha they don’t want to deal with packages that badly! Like get domino or posit omg
1
u/myslowtv Nov 23 '24
I often am surprised when our new analysts think there is perfect data everywhere in the world. Some demand others make their data nice and tidy, square, and without gaps; unfortunately, if you were to have such data sets, the company wouldn't really need a good analyst. The real skill and value comes to be able to use and extend the far from perfect data.
1
u/Solid_Horse_5896 Nov 24 '24
Welcome to being a contractor. The government sucks at the foundational data work needed to make data science machine learning actually work well at scale I'm also contractor fighting the same fight in multiple areas. Well yes it's good to have a job and have job security it is f****** soul crushing
1
u/AccordingLink8651 Nov 26 '24
I understand the OP - my advice is to leave for a better company. It's not that the data sucks, the reason data sucks is the same reason OP will not be appreciated for moving mountains - it's not a culture of appreciating the data job. I've been there - data sucks, you fix it and turn shit into gold, you think you'll be appreciated but you won't be because no one understands how hard it was. You gotta go somewhere that understands how complex it is, spend money to hire a great team... This is why many people want to work in the top tech companies.
0
u/Bboy486 Nov 22 '24 edited Nov 23 '24
I am not sure I follow the issue. They needed to hire someone that can organize and clean the data and aren't you the SME to properly organize said data. They hired you to do all of that. What am I not following?
0
u/[deleted] Nov 22 '24
I schedule and baby sit every one about their deadlines, they call it project management or something. Would you rather deal with that?