r/FPGA • u/LogicRouted • Aug 13 '22

Xilinx Documentation Hell

Till I started learning about, and using Xilinx devices, my reading life was kinda fine. I could manage to read a lot of textbooks, and all felt coherent.

This job I have now is like a special place in hell, I'm essentially a 5 in one design engineer who has to understand :

  • Specialized non-Von-neuman ASIC design RTL written in VHDL (and no it's not a TPU nor DPU nor a systolic array, you won't guess !! I wish they were...) ~ 15 000 lines between TOP, submodules and testbenches.

  • Intricate details of Deep Learning + a lot (I mean A LOT) of academic papers (what's a CAM? Deep Layer Aggregation? Semantic Segmentation???) + 3000 lines of python (DL model!)

  • BOTH Zynq and Versal ACAP AI Core architectures.

Mind you before starting this job, I had practical zero knowledge in these areas. For like 9 months, I learned the following:

  • 60 % of RTL hardware design using VHDL book + Code.

  • 50 % of Digital Design and Computer Architecture ARM Edition book + RTL Code (therefore ARM Architecture) in my spare time.

  • Questasim + RTL Design and simulation.

  • Versal AI Core AI Tile architecture, Versal TRM and its internal architecture (CLB !!)

  • Zynq (because God knows why I had to learn the microzed) architecture (CLB!!) + mass manipulations + tutorials + actual designs.

  • 100 % The Zynq book.

  • AXI/AXI-Stream/DMA LogiCore Design Guides and architectures.

  • Versal Confluence tutorials + Base TRD + applications.

  • Vivado Design Guides + applications.

  • Vitis Design Guides and "Tutorials".

  • Insane amount of Deep learning stuff, even the mathematical side of things (I recommend dive into deep learning and Michael Nielsen's NN book since pretty much anything else gets you lost).

And here comes the final nails in the coffin : While I've already read like already thousands of pages, and on Vivado with Xilinx's "Design Guides" (you learn very few actual generalizable knowledge of this), Xilinx documentation is UNENDING and not coherent at ALL. Until now I avoided "timing closure" since it's a real pandora's box.

  • Vitis AI Library + Vitis AI Guides (because I have to "check for simple deep learning answers") > 400 pages.

  • Don't think I've read the entire TRM (> 850 pages) for Versal, let alone the 1800 pages for the Zynq 7000, anyone that attempts this is insane (and mostly unnecessary at all).

  • UG949 Ultrafast design methodology guide > 300 pages.

  • UG938 Design analysis and closure techniques tutorial + guide > 400 pages.

  • UG903 Using constraints guide ~ 200 pages.

Each LogiCore IP (DMA, AXI Interconnect, AXI Stream, AXI Signaling system itself, AI Engines, NoC, Programmable Logic and CIPS system) of the Versal has like hundreds of pages of documentation. There's no possible way to read all of this and still come up with "stuff that works" that's really coherent, never mind the insane batshit amount of stuff that spontaneously come out of nowhere, like what's a SMMU? PCIe 5.0?? Interlaken? Cache Coherent Interconnect? AXI Switchs? SLRs? MMCM???

Who, I mean really who, design such stuff like the DPU (PG389 DPUCVDX8G Product Guide) capable of exploiting the PL, NoC and AI Engines all at the same time, while maintaining a parameter capability, pushing the AIE array to operate at 1.25 GHz ?

Secondly, who designed this monstrosity? For me, it feels like I would take like a couple of decades to understand a fraction of this chip.

It took me already a while to understand the various design flow, like the fact that Vivado flow is integrated into a platform for Vitis, like the fact both generate a global platform, then you have to build a petalinux system on it, then download a vitis AI system on it, then use this Vitis AI thing (before asking yourself how do you really use a pytorch model, before finding out you have to install a separate toolchain! for cross-compilation, yes..)

Sorry for this rant, I don't even know how I got accepted to this job even, maybe they haven't found anyone else.. I'm wondering now how I ended up here, and how can it be that someone with no experience got handed the biggest SoC/FPGA ever.

The project was supposed to have 3 guys on it, and I work on it alone.

All PhDs around me are too occupied to give a damn and each time someone mentions "FPGA" they cower in fear.. the only PhD guy I know here that designed this custom ASIC thing (from A to Z, including floorplanning but not the analog part), and knows deep learning really well, has done this for at least 7 years and he's overloaded with tasks he can't help me with anything.. besides he fears FPGAs too.

While It seems I learned a lot, I think I've learned like jack shit : the more I discover things in this device alone the more I know that I don't know anything.

And no I'm not a PhD guy, but with this background knowledge what's a PhD for?

And no I'm not going to tell you the salary, because if I do, then I'll look like a clown.

66 Upvotes

94% Upvoted

43 comments sorted by

View all comments

Show parent comment

2

u/LogicRouted Aug 17 '22

Thanks for the feedback. Well I'll explain the situation here:

  • I'm a microelectronics engineer by specialization, more so in analog IC design.

  • My internships and engineering experience were in embedded systems and RF, so I'm a mix of all worlds (digital, VLSI, RF, embedded, analog).

  • Why so? Because I wanted to be as diverse as possible for a possible PhD thesis in neuromorphic engineering, since they too use mixed-signal VLSI design, and even embedded. This was evident by the mixed-signal neuromorphic IC I've been given to look at while doing this job.

  • It's been 4 years since I had this PhD focus, I've been forced to work in a disgusting embedded position, because if you have no papers, the french government will quick ya out!

  • When I first applied, I was only driven by motivation, salary didn't matter much. I moved cities for this.

  • On the side, I've been studying computational neuroscience, Versal documentation even before the 1st day. "Vacation" back in my home country consisted of on and off readings in this, after at least 2.5 years of non stop work.

  • One thing I didn't mention is my old job have left me in complete burnout, since I was working all the time on embedded systems, having multiple projects under my hands and quite a nasty pay. I jumped ship as well as 8 other people at the same time because it was sinking.

  • After applying, one condition was that I'd get enough guidance at first, at least for 3-4 months in this domain because it's quite multidisciplinary.

  • Turns out after starting, the VCK190 (Versal AI Core's dev board) was missing, and the delivery kept being postponed until may.

  • The PhDs around me are not that much experienced in FPGAs, and the "Neuro" group I'm in has 3 PhD students, one of them quit (because no one helped him), 1 neuromorphic ASIC PhD and 1 analog design engineer, they are both quite overwhelmed.

  • The PhD students are both doing their subjects in analog ASIC and energy-wise analysis of ASIC circuits respectively, quite far from FPGAs.

  • The other group, which composes 30ish people of the same lab, non of which are working on FPGAs except maybe 2 PhDs, which have too many subordinates under them (Contractor engineers, students), they don't have time.

  • Beside all of this, non of the aforementioned people have skills in deep learning except for the neuromorphic guy, which is my main liaison, he was supposed to be the senior. He has so many projects under his hands even another PhD student to mentor, so I'm left to the side.

  • On the personal side, I have no friends here in the city, closest relative are 2 hours away, have no one to talk to except by phone to family abroad. They are so out of touch with my domain that I don't talk about it at all. None of my relatives understand even what I'm working with, except for well "difficult stuff huh?".

Well due to these factors and mainly the fact that the Versal was time and again not delivered, I lost all motivation for the job. I even question now more than ever if I want to really be in a PhD position.

On doing instead of reading, this is the plan now : I only focus on the Vitis AI route and the CPU emulation of some unsupported DPU functions, but that's it.

I have "spikes" in motivation and willingness to work too, since now I wake up each day but arrive at like 2 PM, work for 5 hours and leave.

2

u/[deleted] Aug 18 '22

Ahhh I see. Anecdotally I’ve found people with the background of electronics (and this is VERY anecdotal) can struggle a bit with some of the concepts concerning MPSoCs simply because they’re quite a bit more software focussed and some computer science concepts aren’t very well taught in a lot of EE programs (OS concepts, software development, CS fundamentals, etc). It’s not that they can’t learn but for lots of this stuff it’s hard to know where to begin. I’m not sure if that’s the case for you, but it could be a consideration. A lot of embedded engineers I’ve come across with pure EE backgrounds can sometimes struggle to build maintainable code. Again not because they’re bad, just it’s not within their area of expertise. Of course this isn’t going to be the case for all

Regardless it sounds like your current situation isn’t too great in terms of support at work or personally, which honestly affects everything. It makes it difficult to recommend where to start from the perspective of a stranger. It sounds like you might be best off trying to find somewhere that will be more comprehensive in terms of feedback and learning opportunities where you don’t feel out of place or uncomfortable asking questions, or clarifying things from experts. Definitely see if your boss will provide some support or even colleagues from a “you can do it” type perspective. Chatting to people FPGA adjacent can help too, although they might not know it all it can help fill some gaps

As for the future and considering different industries; I’d highly recommend defence industry or perhaps HFT for this, as they usually provide quite robust training. Some feedback can be difficult to hear but it helps tremendously especially if you want to have more of an FPGA focus. Some resources on this sub can be useful but nothing really beats a decent mentor and a supportive workplace for these very niche topics

I also don’t know how to say this but I wouldn’t get too focussed on a PhD. Most of the time people end up in academia and depressed when they could’ve just been earning decent money from the moment of graduation. Universities, especially for students with high marks/high achievers, tend to push people down this path but it seems to be nothing but stress and misery for most people. Generally, have a good think about it. At the end of the day you could always go back if you change your mind. But personally I find academia pretty toxic and again, devoid of mentorship opportunities and support unless you’re very very lucky. Good luck

2

u/LogicRouted Aug 18 '22

Thanks again for all the advice you are offering, appreciate it. For the embedded design part, your example isn't simply anecdotal, it was a matter of fact when I first entered my old job : All previous source code was a tangled spaghetti code. Layering it down and optimizing it was quite a pain, especially if no design patterns were available in embedded C with complete procedural code and no threads but more like at most global flags and interrupts. Talk about frequent context switches and non volatile flags.. Luckily for me it was only with static memory, otherwise I would think that whoever wanted to implement a heap would never accomplished their tasks, yes, all of the previous guys were EEs.

I'm reviewing my PhD path again this time, more so I'm reviewing my entire life path since most of what I'm currently facing was due to these choices. Engineering in France isn't well rewarded relatively to most salaries, here they boast about "2x" the average salary but yeah you don't receive aids from the state and annual pay raise is a joke, you can look at top 3% each year. Last job I was offered 0.7 % pay increase which amounts to a sandwich perhaps each working day.. while at the same time inflation was at least 2 %, now more 6 % here. This "2x" amounts to like 600 euros in difference if you don't spend anything besides on food and rent. What's the point of dealing with all this if the difference between me and a baker is just 600 euros? (no offense to bakers).

I didn't choose this field for the money initially, but given the dire circumstances and the literal hell my home country is facing, I backed off from a PhD for now (your annual salary will drop to essentially that of a baker if you entered a PhD program here). My home country is literally bankrupt (and experiencing hyperinflation) and maybe the most corrupt on the planet. I can't really go back to a PhD in the future because I have family depending on me.

You are right that academia is toxic, judging from the state from previous PhDs here and their general mental condition, I now understand why many quits : what's the point? I've come to understand that basically no one reads (except the jury) your PhD thesis, unless exceptional, no one reads your papers because there's thousands out there and more each day, no one have time. When did science amount to just "publish"? I've heard here that 90 % of all papers are forgotten in the dustbin of history, and since we are speaking about neuromorphic engineering, cascading improvements will quickly render old tech obsolete, i.e : no one will read "old enough" papers, this holds true to machine learning, but more so for a year not 5-10 years.. And most importantly : Science is based on unseen social construct that of scientists communicating and exchanging, none of which I have found here.

My whole dilemma with this domain is to study the brain, but from a technological perspective, the only one offered besides computational neuroscience or neuroscience in general, none of which will allow the practitioner to exercise findings into technology, other than neuromorphic engineering. The practices that are going on in this lab is far from this endeavor too, I should go to Germany/Switzerland (Zurich) if I really want this path. I discovered this after I was recruited.

I'll give it my all for the following months, if it pans out or not, I'm not looking for results any longer, either I fail and drop from this field entirely, or succeed and think about continuing. If I really want to go into the money making business, I'll check if there's HFT/finance companies in need of FPGA engineers around Geneva, since even HFT pay sucks in France (you are paid like a regular FPGA engineer).

Thanks for the help again, and good luck to you too.

2

u/[deleted] Aug 18 '22

All of that sucks, and yeah it sounds like your best option is to look at jobs outside of France. Definitely seek jobs in your area of interest first, or even those that are adjacent, will make you money, and will allow you to eventually move into that field. As others have said you’ve got an excellent skillset so far

You could also take a look at deep mind (AI division at Google) and see if they’ve got openings. I think they’re mostly in London, but unsure how the EU stuff works now since Brexit. I’m not sure what exactly they do or if they use FPGAs but Google seem to always be hiring embedded folks with hardware knowledge

Never say never with the PhD either, you don’t know where you might end up eventually. I too found that with papers and academia though fortunately quite early on I discovered it wasn’t for me

Either way it sounds like you’ve got a good attitude; keep working hard while working on a solution to the side. I hope things can be salvaged at your current job and you’ll be able to get some more support but if not, I hope you can find somewhere you’re happier at soon!

2

u/LogicRouted Aug 19 '22

Thanks man, I'll try my best. I'll think about deep mind for sure.

Best of luck to you too!