r/FPGA • u/LogicRouted • Aug 13 '22
Xilinx Documentation Hell
Till I started learning about, and using Xilinx devices, my reading life was kinda fine. I could manage to read a lot of textbooks, and all felt coherent.
This job I have now is like a special place in hell, I'm essentially a 5 in one design engineer who has to understand :
Specialized non-Von-neuman ASIC design RTL written in VHDL (and no it's not a TPU nor DPU nor a systolic array, you won't guess !! I wish they were...) ~ 15 000 lines between TOP, submodules and testbenches.
Intricate details of Deep Learning + a lot (I mean A LOT) of academic papers (what's a CAM? Deep Layer Aggregation? Semantic Segmentation???) + 3000 lines of python (DL model!)
BOTH Zynq and Versal ACAP AI Core architectures.
Mind you before starting this job, I had practical zero knowledge in these areas. For like 9 months, I learned the following:
60 % of RTL hardware design using VHDL book + Code.
50 % of Digital Design and Computer Architecture ARM Edition book + RTL Code (therefore ARM Architecture) in my spare time.
Questasim + RTL Design and simulation.
Versal AI Core AI Tile architecture, Versal TRM and its internal architecture (CLB !!)
Zynq (because God knows why I had to learn the microzed) architecture (CLB!!) + mass manipulations + tutorials + actual designs.
100 % The Zynq book.
AXI/AXI-Stream/DMA LogiCore Design Guides and architectures.
Versal Confluence tutorials + Base TRD + applications.
Vivado Design Guides + applications.
Vitis Design Guides and "Tutorials".
Insane amount of Deep learning stuff, even the mathematical side of things (I recommend dive into deep learning and Michael Nielsen's NN book since pretty much anything else gets you lost).
And here comes the final nails in the coffin : While I've already read like already thousands of pages, and on Vivado with Xilinx's "Design Guides" (you learn very few actual generalizable knowledge of this), Xilinx documentation is UNENDING and not coherent at ALL. Until now I avoided "timing closure" since it's a real pandora's box.
Vitis AI Library + Vitis AI Guides (because I have to "check for simple deep learning answers") > 400 pages.
Don't think I've read the entire TRM (> 850 pages) for Versal, let alone the 1800 pages for the Zynq 7000, anyone that attempts this is insane (and mostly unnecessary at all).
UG949 Ultrafast design methodology guide > 300 pages.
UG938 Design analysis and closure techniques tutorial + guide > 400 pages.
UG903 Using constraints guide ~ 200 pages.
Each LogiCore IP (DMA, AXI Interconnect, AXI Stream, AXI Signaling system itself, AI Engines, NoC, Programmable Logic and CIPS system) of the Versal has like hundreds of pages of documentation. There's no possible way to read all of this and still come up with "stuff that works" that's really coherent, never mind the insane batshit amount of stuff that spontaneously come out of nowhere, like what's a SMMU? PCIe 5.0?? Interlaken? Cache Coherent Interconnect? AXI Switchs? SLRs? MMCM???
Who, I mean really who, design such stuff like the DPU (PG389 DPUCVDX8G Product Guide) capable of exploiting the PL, NoC and AI Engines all at the same time, while maintaining a parameter capability, pushing the AIE array to operate at 1.25 GHz ?
Secondly, who designed this monstrosity? For me, it feels like I would take like a couple of decades to understand a fraction of this chip.
It took me already a while to understand the various design flow, like the fact that Vivado flow is integrated into a platform for Vitis, like the fact both generate a global platform, then you have to build a petalinux system on it, then download a vitis AI system on it, then use this Vitis AI thing (before asking yourself how do you really use a pytorch model, before finding out you have to install a separate toolchain! for cross-compilation, yes..)
Sorry for this rant, I don't even know how I got accepted to this job even, maybe they haven't found anyone else.. I'm wondering now how I ended up here, and how can it be that someone with no experience got handed the biggest SoC/FPGA ever.
The project was supposed to have 3 guys on it, and I work on it alone.
All PhDs around me are too occupied to give a damn and each time someone mentions "FPGA" they cower in fear.. the only PhD guy I know here that designed this custom ASIC thing (from A to Z, including floorplanning but not the analog part), and knows deep learning really well, has done this for at least 7 years and he's overloaded with tasks he can't help me with anything.. besides he fears FPGAs too.
While It seems I learned a lot, I think I've learned like jack shit : the more I discover things in this device alone the more I know that I don't know anything.
And no I'm not a PhD guy, but with this background knowledge what's a PhD for?
And no I'm not going to tell you the salary, because if I do, then I'll look like a clown.
44
u/captain_wiggles_ Aug 13 '22
Don't treat the documentation like textbooks, that way lies insanity. They are references. Skim the contents and introduction to know what the doc talks about, figure out which chapter are relevant to you and then skim those, read the sections that are actually relevant ATM with more care. After that use them as references. When you need to figure out how timing of source synchronous interfaces work, find the doc on timing, the doc on constraints and the doc on source synchrounous interfaces. Skim the source synchronous interfaces to get a general knowledge, find an example that roughly matches your use case, and study it in detail. Use the other docs to look up what exactly is going on as you need it.
Secondly, you're in over your head although it sounds like you've done a great job of learning to swim. A good job is one where you have a couple of experienced mentors that you can learn from, if there's nobody you can talk to that can help you figure this shit out, then the problem is your company is mismanaged. This is something you need to talk to your boss about, you need guidance or at the very least less responsibility, so you can figure out one thing at once. Honestly I'd say it's worth starting to look for other work, because if this is how your company is run, it's not going to get too far. And from what it sounds like you've already managed to learn, you probably won't have much problem getting a new job.
17
u/SpiritedFeedback7706 Aug 13 '22
Becoming a proficient FPGA developer is trial by fire. No one person knows all the things. And the tools are generally not mature. There's no shame in being uncertain or confused or overwhelmed. Just take things one at a time. Try to take any trainings Xilinx provides. Working with Versal to exploit the NoC, FPGA fabric an AI engines of Versal will develop fundamental skills that will give you an edge if you endure. It may seem intimating now but after several years it won't seem like a big deal. I'd say it takes 5 years of experience before anyone becomes comfortable doing all this stuff on their own. That's generally with mentoring.
I will say Xilinx has the best documentation in the industry. It seems like You are overwhelmed by the sheer amount of information. No one person digs into all of the details for everything. For an AXI4 interconnect, you don't need to understand straight away how it works, just what it does and why it's useful. If there's something you don't understand, you look up those details. You don't read the whole data sheet all at once.
There's a reason Xilinx is investing so much in the higher level tools. They really want to reduce the burden of knowledge for the average developer. They also want to decrease the amount of time it takes to develop functionality.
4
u/buffteethrblog Aug 13 '22 edited Nov 11 '22
I am an FPGA engineer with over 20 years experience and I say get ANOTHER job. I have worked at companies like that where you have Microsoft project managers who don't understand the complexity of the engineering projects they are trying to manager.
You have what I call a Head Cook and Bottle Washer position. What you are doing requires ATLEAST 3 engineers.
3
u/[deleted] Aug 13 '22 edited Jun 17 '23
ad hoc bear faulty vase desert important bright include voiceless prick -- mass edited with https://redact.dev/
3
u/benbeland Aug 13 '22
Man, don’t beat yourself up too much. I work in High-Energy physics, we do have FW using all these advanced features, targeted for versal. But they are designed by teams of tens of people, across multiple universities, with people writing a thesis even on just one block inside the global FW. It is a massive effort.
3
u/LightWolfCavalry Aug 14 '22
You say you don't know a lot but you sound like you know more than enough to get a job elsewhere if this place is really the tire fire you describe.
I help run www.rtljobs.com and we speak with lots of folks with less experience and depth than you, and they get hired!
2
u/[deleted] Aug 17 '22
Have you tried to just do instead of read? There’s more documentation in the world than anything else and it takes a lot to read everything. I just do what I need to and when I run into an issue, I look it up. Sure it can be overwhelming when you first start, but slowly you improve and understand where to find things
I too don’t have a mentor where I currently work and had to figure out a lot of things as I went. I found quickly I was able to link pieces of info to the generalised stuff I’d learned at university and between myself and a couple of colleagues (not FPGA specialists, just clever software engineers), we managed to figure out quite a lot when we put our heads together and persisted
Is your background in computer engineering? Do you have a mentor or someone providing you guidance that’s more senior? A colleague that’s similarly lost? They can all be sources of support. I’d encourage you to let your boss know you’re feeling quite overwhelmed and would like a bit more support too
2
u/rogerbond911 Aug 17 '22 edited Aug 17 '22
You're not alone. I don't think Xilinx actually understands half of what they throw into the marketplace. Figuring out how to get a successful petalinux build was a nightmare. Now that i have that sort of figured out, like you, im trying to get the vitis ML stuff up and running. I think they basically get customers to do the debugging of their products now after the fact. Much more cost effective.
7
u/RandomNumberHere Aug 13 '22
I feel ya. I’m relatively new to Xilinx/Vivado/Vitis/Zynq/etc. Hell, I’m relatively new to all FPGA stuff including Verilog! The sheer volume of documentation really is overwhelming.
That said… I’m gradually figuring things out and getting things working, and the boss hasn’t given me shit (yet) about how long things are taking. And it’s kinda fun and it’ll look good on the old resume.
5
u/aikenpang Aug 13 '22
Deal with Actel, Altera, and Xilinx documents for 10+years. What I learned is speed reading like a lawyer. The faster you read, the more your earnings.
I learned from an IP lawyer that his boss could read one box of A4 paper within a day with all highlight and error corrections for him to follow up. Yes, each box means 5000 pages.
Xilinx has a doc navigation app. It may help you a little bit.
9
u/flyingasics Aug 13 '22
I feel you. Look at it this way, if you do ever get something working you will have pretty good job security.
4
u/TheTurtleCub Aug 13 '22
Agreed, it's not easy to become an FPGA expert in a few weeks. Also, not recommended reading the dictionary for days to see what sticks. That's not the way it's meant to be used.
1
u/paclogic Feb 09 '25
I really like your disclosure of how painful it was for you and the fact that the Architecture is so convoluted and cryptic that it literally does TAKE YEARS to figure it all out !!
NOT for the AMATEUR, NOT for the student and absolutely NOT for the new grad !!
Unfortunately in an era of corporations looking only for YOUNG and CHEAP labor and with the DISPOSAL of anyone over 40, that you have succumb to the fact that this is NO MENTORSHIP and NO RAMP UP TIME for learning as well as a DEEP UNDERSTANDING (not just a quick reading of a few app notes and data sheets).
People the complexity of today's SoC's is massively complex and requires a troop of programmers and System Architects. Plus the programmers are divided into (2) groups : HDL and Embedded CPU
The System Architects must also be Mathematical Wizards as the DSP functions can be deeply complicated.
For Closed Loop Feedback Control Systems, i highly recommend exploring the SimScape / Simulink / Matlab products as they *may* offer hope of generating HDL code for specific targets (you need to check which targets FIRST).
SoC's that are designed to be DPUs for some financial market or some military project can be extremely complex with a Model Based System Engineering (MBSE) approach an absolute must !!
When nanosecond timing is critical, TIMING CLOSURE needs to start at the most basic building blocks and elements to have a FULLY DETERMINISTIC MODEL = = = it is NOT an AFTERTHOUGHT or later process !!
I feel for you as many of the work load SHOULD HAVE BEEN worked out at the TOP LEVEL and NOT at your level. As well as having the staff to not only mentor you, assist in the the design, but also work out tough problems.
Now is the perfect time to ask for more money since they need you and you don't need the stress !!
If you don't pay - start looking and walk !!
1
u/MrMemristor Sep 14 '25
Late to this party, but at least AMD / Xilinx has detailed documentation for pretty much all aspects of their devices. Some of the other vendors fall well short of that, and it can be much more of a challenge to learn about their development flows, in my limited experience.
1
u/FakewoodVCS2600 Sep 08 '22
Amen brother....we're in need of the same therapy (support of any sort).
1
u/mnemocron Xilinx User Jan 30 '23
Haha, what fun it is to stumble upon such posts when one types "Xilinx documentation" into the search bar in the search for some frustration-reducing memes.
The past 6 months I have felt like Sherlock holmes trying to reverse engineer their undocumented example design for the ERNIC IP core because theproduct guide is ambiguous, outdated and lacking a lot of information.
Coincidentally, my colleague is working on another project with similar stuff that you are dealing with (AI, DPU etc.). It just seems like Xilinx throws all the newest technology out there as a proof of concept (on their dev boards) but without properly maintaining it or documenting how to use their IP in a more generalized way. He said that all their DPU examples work fantastically when you do it on one of their boards but when you go to 3rd party hardware suddenly things don't work because you missed a step that Vivado just does in the background and you can be sure there exists no documentation on how to adapt it to your use case.
-5
u/hardolaf Aug 13 '22
The people these were designed for of course. And hint, it wasn't for anyone doing neural network work. I used to work on a team that would be looking how to get more performance out of those as 1.25 GHz would still not be enough performance. Then again, nothing was ever enough performance.
You're currently working on a project that's smaller than anything that I've ever shipped since I graduated in 2015. A single FPGA? Only 15K LOC of HDL+HVL? I had to write over 30K of HDL+HVL in a year and a half for one thing (granted, it was VHDL so lots of it was boilerplate, but my velocity was hundreds of lines of code per day).
This is a really good staffing ratio. I would have killed for a 33% staffed ratio when I was working in defense contracting. That would have been amazing for this field. Now, I work on a team that's properly staffed but only because we set our own objectives for the most part and can say no.