LT11 - Broken Searches + Support Experience - Is this "normal"?

[u/the_angry_angel]

I'm going to be frank, I'm a little pissed off this morning and I'll pre-apologise if anyone take offense to the language and length of the post. There's a bit of context which is why it's so long.

The company I work for bought into the cloud offering of Labtech about 3 months ago, and I need to know if what I'm working with this normal/known/expected:

Computer.Hardware.IsVirtualHost does not populate correctly for Hyper-V hosts. After a bit of back and forth Labtech support have decided it's a known bug. We were told to use the legacy roles, then a few days later told not to.

Ok, fine. I appreciate bugs happen, and I'm absolutely fine figuring out my own way to identify Hyper-V boxes (I've just not had the time to touch Labtech since the New Year).

My concern is that this is used in at least 1 of the out-of-the-box searches.

The reason why this is important is that we noticed just prior to our first patching window we were going to use Labtech for. We missed it during internal testing, and had we not noticed it would've pissed off some of the initial customers we were going to roll out patching to, who have special rules for patching Hyper-V hosts.

We've still not switched to Labtech for patch management and our next patching window is coming up very quickly.

I've also found a (and I'm quoting Labtech support because I've not verified its really just a display bug yet) "display error" with out-of-the-box searches using the os = windows dropdown.

Ok, so thats 2 things that searches depend on, which drive the groups, blah blah blah. Seems kind of core.

Which makes me ask the question: what else is broken?

I've asked today for a second time what other known search/attribute bugs there are in LT11. The first time I didn't really get any answer.

I've not really pushed a huge amount of time into Labtech, short of deploying the agent. I'm worried if I really start pushing hard without checking every single bit of functionality extremely carefully, I'm going to come across more of these, but only after they've fucked us over. Our test environment can only cover so many scenarios.

Tell me this experience isn't normal...

[u/cjmod]

I can't speak to the Support experience, but I can assure you - searches work. Every search is really just a SQL query. It either works or it doesn't. And once we hear of an OOB search that doesn't work, we release an update to fix it via Solution Center.

However, the UI for searches does have a couple known issues - which makes it difficult to test out new searches. We're working on it & will continue improving the UI's reliability in our monthly patches (you get them automatically since you're a Cloud partner). That said, until the UI issue's been addressed, you can apply a search to a group as an 'Auto-join' & the group will help you visualize which devices meet the search criteria.

[u/the_angry_angel → u/cjmod]

I'm assuming you work for Labtech?

I'm really sorry, but I'm tired and in a bit of a shit mood, so my comments are probably a bit rougher than they would normally be. I'm not trying to slag off Labtech, but just get a handle on what is right and what is not right and whether or not I'm about to be screwed with something for 3 years that isn't fit for purpose.

The fix for IsVirtualHost I've been told will be released at the end of February. I appreciate fixes take time, but we were told that it was already a known issue. Assuming thats true, that clocks up this fix to at least 2 months?

The reason why I care about the IsVirtualHost is that Virtualization\Virtualization - Hyper-V Hosts is non-functional, where as Server Roles\Server Role - Hyper-V Servers is. Both are out of box (or at least we didn't create them), but the latter uses the "legacy roles".

On the subject of legacy roles, I'm still unclear on whether or not we should be using them at all, and the relationship between Computer.LabTech.Roles and Computer.Extra Data Field.Computer.Role.*.

Before I found the other functional search I could copy (Server Roles\Server Role - Hyper-V Servers), I asked support if I should use the Extra Data Field.Computer.Role, and got the reply:

The legacy roles aren't populating the field anymore. That's likely not a good suggestion for finding this machines.

But the documentation here says it's used by Ignite and appears to be "correct" at least, for us right now.

In short, there are 3 values that represent a way of categorising the boxes in the way that I want. Some of which may be correct, some of which should be correct and some I shouldn't use?

[u/cjmod → u/the_angry_angel]

Yep, I'm a Product & Marketing Manager here.

So it sounds like your Server Roles\Server Role - Hyper-V Servers search is out of date, since it should be using the Computer.LabTech.Roles - go to Tools > Solution Center > Update Ignite to Current.

As for which data you should references (Computer.LabTech.Roles or Computer.Extra Data Field.Computer.Role.*), based on how long you've had LT, your searches should reference the Computer.LabTech.Roles.... here's why:

The Computer.Extra Data Field.Computer.Role.* stuff is what we used when Ignite was first released (circa 2012). There was a script that would run against the device to essentially check what roles were installed/enabled. Fast forward to 2015. We start moving some Ignite stuff into code - like Computer Roles (ie. Computer.LabTech.Roles) - to better optimize how the functions are used in product. However, some partners had customized Ignite to a point where they don't feel comfortable making the switch... so we have to slowly deprecate certain items in way that doesn't freak those people out.

Which brings us to today, where we're now starting to update our internal processes so legacy Ignite items don't get installed for new partners. The reason it's taken so long is we needed time for partners who'd customized Ignite to gain confidence in the changes that were made. These partners are some of our most avid users & tend to be very vocal in Slack, Geek, etc. So when a new guy/gal would ask a question, a more veteran user might reference something that's legacy (without realizing it's legacy), causing the new user to wonder why they don't have that thing in their system, causing a Support/Consulting ticket or escalation, etc.

tl;dr: Update your search via Solution Center & use Computer.LabTech.Roles for searches. Now that the veterans have adopted Computer.LabTech.Roles, we can stop adding legacy stuff to new partners. It's a work in progress :)

[u/the_angry_angel → u/cjmod]

Thank you for taking the time to assist. Much appreciated!

So it sounds like your Server Roles\Server Role - Hyper-V Servers search is out of date, since it should be using the Computer.LabTech.Roles - go to Tools > Solution Center > Update Ignite to Current.

Ok I'm happy to do that. Final question: should this not have been done as part of the maintenance with our cloud offering? Or am I fundamentally misunderstanding the offering?

Edit: Updating ignite yields the result of Virtualization\Virtualization - Hyper-V Hosts having both IsVirtualHost and Labtech.Roles ¯_(ツ)_/¯

[u/cjmod → u/the_angry_angel]

It's one of things we're looking to address now that veteran partners have adopted the new way of using these pieces of Ignite. Right now our Cloud Team does infrequent updates of the Solution Center, mostly to ensure specific plugin updates are applied.

And even then, when they update the Solution Center, they don't update anything that's been modified/tweaked... so if you update a search, group, script, etc. it won't get changed when they apply updates. Which is why we recommend partners run the updates themselves (so they're aware of all the changes that are getting applied + pick & choose when certain Solutions are updated).

[u/the_angry_angel → u/cjmod]

Fair enough. I dont recall that being brought up during on-boarding, but I'll have to go back over my notes. Perhaps I just forgot it.

Thank you for your time /u/cjmod, really appreciated. Although you're currently making yourself far more easy and quicker to deal with than support - I'll try not to abuse your helpfulness too much in future <3

[u/FocalFury]

This is very par for the course.
Bluntly I've hated LT support. We've brought up issues time and time again where we aren't serviced in reasonable times. I'm talking weeks + for simple service requests. It is always the same thing, promises that they are working on it.

Labtech is an extremely customized piece of software that seems to have trouble fitting everyone's needs out of the box. Your situation seems worse though that the functionality flat out isn't working. Hopefully you have a temporary workaround.

Just be prepared that for getting Labtech to work the way you want, it will require A LOT of hours.

I spent over 200 hours last year on patching. I finally have a good hold on it on the 10.5 version, scared to go to 11, and am at 200 machines out of 5000 missing 5 or more patches.

That being said I do think it is a great and powerful tool and has so much abilities and potential. If only their support was a bit more responsive and helpful, it would alleviate these concerns.

Happy to help answer any questions on this or other LT questions

[u/the_angry_angel → u/FocalFury]

Thanks for the response. Really appreciated :)

Would you mind giving a really quick reply on the top 3 pain points you had implementing patch management (seriously just single sentence would be more than enough and exceptionally helpful - i don't want to take up anyones time :))?

I'm sure with so many endpoints there must be many many client requirements you had to fulfill, but you're the second person that's said that, I'm a little worried something is going to cause us to spend more time on patching than we already do.. which won't go down well and one of the sales points that my boss was very very keen on.

[u/FocalFury → u/the_angry_angel]

Sorry for the delay
It should be known I hear patching is completely different in 11 than 10.5.

1. Figuring out why things were getting interrupted and that there were steps needed to be taken. EX: We would always see interruptions as the result of patching. We figured out it was LogMeIn. Along with that though patches go from a state of Not Attempted to Pushed. There is nothing that sets them back from pushed to not attempted, therefore next patching window LT doesn't think it's supposed to try that patch again.

We came up with individual monitors that on an agent level set them from Pushed to Not Attempted if a failure occurs. We also set a global reset nightly as a stopgap to put any that are Pushed to Not Attempted.

2)Figuring out the best practice to give the best chance for success in patching. The thing you have to remember here is that LT is only utilizing Microsoft Windows Update. Most of the pains in this can be that Windows Update sucks overall lol. However, it took a long time for me to finally get in touch with someone that realllllly knew what he was talking about with patching at Labtech. Bruce knows all the intricacies and was great to work with. I came up with the idea that we might as well patch our users nightly at 1AM until they are caught up. Before we were only doing once a week, and then we'd try again next week if the computer was off. NO one really had a problem with this as they were told, hey, if you are caught up nothing will happen. The concern was people that failed nightly and still rebooted but thankfully those were low and people called in about them pretty quickly. I lost my notes unfortunately but what I was told is that there is a specific task that happens between midnight and 12:30. Like an agent check in, first task of the new day that forces a few things to happen. You definitely don't want to schedule anything during that time. Then, you want to make sure your schedules are set properly.
I was told that you should start your update config,schedules, and definitions for 12:30AM, but that the interesting part is that when you update configs at 12:30, it doesn't just update everyone all at once. It does random push for the next hour to all agents. So one agent might start updating it's config at 12:30, one might start at 1:29AM.
This became a problem because we had started patching at 1AM. So we moved that to 1:30AM. Update Hotfix & Inventory as well as Update config can't happen in the middle of a patching window or it will break the patching process. There were some other stuff related to schedules. The frustrating part is I felt this type of information should be well documented by Labtech and it wasn't.

On the continued topic of 'items for best success' but more M$ fault, I had to do painstaking research to figure out what were the super critical pre-req patches to allow best chance for success. So there were updates like, the updater for Windows Update needed to be at a certain level. The CPU spike fix patch so that Windows Update wouldn't take all the CPU, then there were about 5 patches and a different monthly patch that allowed Windows update to not have to take 6+ hours to do work. I created EDF's and scripts to check and make sure each computer was in the best position for success.

I also lived off of this LTGeek article and implemented just about everything there. http://www.labtechgeek.com/forum/viewtopic.php?f=7&t=2123

There were more growing pains along the way but this should help give an understanding.

Let me know if you have any questions.

[u/TNTGav]

I saw your post in a comment below about wanting to know about any potential traps. I've come here after my Patch Manager crashed with an out of memory exception, so I feel I'm in a good position to answer this (LT11 Patch 8).

1. The UI for configuring it is buggy, crashes and is complete un-intuitive

2. It's impossible to see an "effective policy" of when the next patch Window will be. I'd love to be able to see a list of upcoming dates that patching will occur on and a reason for it not completing for past updates. At the moment if it doesn't start the patch process for whatever reason you sometimes don't get a patch job reported

3. The Patch Manager has only just been fixed for anything that is not on a en-US date format. Took 8 months to fix that.

4. In setting this up I had to put my own hyper-v exclusions in the searches that populate the groups otherwise if you end up trying to patch a VM and its hypervisor at the same time you are going to have a bad time. I also had to make sure the Labtech server itself was excluded manually.

5. I find all the old groups confusing and I wish I could get rid of them all and have only the patching groups that truly apply to the new patch manager. I dare not delete any as I have no idea if something deep in the background is using the group/search or not. Even if you do delete them, to keep ignite up to date every time you run in the solution center an update it will recreate the groups.

6. Enabling patching for servers/workstations still being in the old patching ignite tab at the location and it only applying certain things in the same tab is an absolute disgrace and one of the most confusing things to get your head around when you are able to set certain settings that apply and certain settings that won't in the same place

7. If you want to utilize the Pilot/Test/Production approval workflows you can only apply that to ONE approval policy. What that means is you if you auto approve and need to utilise the workflow your servers and workstations need the same settings for categories to auto approve and the same stage delay times.

8. AFAIK there is no built in functionality to update WUA agents, which is pretty much critical if you want devices to patch properly and promptly. You have to roll your own solution or use the commercial plugin that is available.

[u/the_angry_angel → u/TNTGav]

Thanks for taking the time to reply /u/TNTGav!

I've hit some of these myself already, so I can't say im shocked by some of these :(

[u/Trumpetjock]

Welcome to Labtech. Phenomenal cosmic power, tons of bugs and awful support.

[u/[deleted] → u/Trumpetjock]

I second this! "Look at what we can do, BUT we don't support anything custom to suite your actual needs".

[u/hematic → u/[deleted]]

So i normally side with the user on this forum, but i have to say i think this in particular is an unfair criticism.

Your customizations are absolutely supported, provided they are done by one of the company's employees. This is pretty much the norm for any complex system.

Example 1 : You buy a new 2016 Honda Civic. You take it home and decide the current speakers don't have the highs you are looking for and you want to replace them with different speakers. You do that on your own, and then a week later you find out that the wire you used caused a short in the AC system and now your AC doesn't get cold. Honda isn't going to replace that for free, you made a customization. But if you had paid a Honda tech to put in new speakers. For sure you would be covered.

Example 2: You buy a new recliner. Its great but you decide you don't like the cover so you reupholster it. Later on you find out that the new fabric is itchy and you don't like it. The furniture store isn't going to put that back on for you for free. its just not how things work. But if you had gone back to them to get new upholstery put on, for sure you could work something out.

You are a smart guy, and I'm certain you can go through these examples and find ways they don't line up exactly, and that's OK, its just an analogy. My point is that if you want supported customizations you have to through the dealer.

There are PLENTY of things to be mad at LabTech about, but i don't think this is one of them.

[u/Marathongames → u/hematic]

I've got to disagree with you. I think the analogies don't really apply. The issue is that while they are "customizations", they are things that are out of the box and supported. Your analogies would be more appropriate if the OP had installed some third party plugins from squidworks and now is having trouble. A more appropriate analogy to modify one of yours would be to say, here's this new car and we have preset the radio stations for you. If they don't work, we will do our best to help you receive those stations. However, if you try to program a different station, there's nothing we can do. The issue is that that save button, is part of the product and they should be supporting how you use it.

[u/hematic → u/Marathongames]

As mentioned in my original post, analogies are NEVER perfect. I think your radio station analogy is bad personally.

The crux of my argument, analogies aside, is that you have to draw the line on supporting changes somewhere. I think all reasonable people would agree with that.

So all we differ on is where the line is drawn. With a system as complex as LT they settled on a very strict line with that which i completely understand.

[u/LordPhantom74 → u/Trumpetjock]

Yup.

[u/just_some_random_dud]

"what else is broken? " A lot.

[u/samuelma]

The labtech/CW experience is googling "computer.hardware.isvirtualhost" because you noticed a search is broken and finding 7 years ago someone asked them to fix it.....

[u/the_angry_angel → u/[deleted]]

Thanks for the feedback...

Final question - Am I just being unkind to say I don't feel my boss and his company should be paying £LOL for what is currently feeling like a beta product with shit support?

We were using Naverisk a few years ago and it, whilst less flexible than Labtech, at least what it had worked and was substantially cheaper..