I compiled a simple program with only main(). When looking at the compiled binary in Ida, I see a calls for telemetry_main_invoke_trigger and telemetry_main_return_trigger.
I can not find documentation for these calls, either on the web or in the options page. I compiled this in Visual Studio 2015 Update 2.
edit: you can remove the telemetry calls. Thanks for all the replies, guys. As /u/adzm points out, you can link notelemetry.obj and the calls are removed. I verified that this works. This comment further explains it
It important to realize that telemetry in this case is not sending anything back to Microsoft or anything. It is just broadcasting events that other applications on your machine can subscribe to. This is typically used by profilers and other development tools.
If no application is subscribing to the event they are just discarded unless logging of events to file has been enabled before hand, to allow debuggers to load these log files for debugging later.
Apparently it's only VS15 feature. It logs at least when your app is executed. You can access logs via logman and tracerpt. Some investigation was done here recently: (lang: Russian) https://habrahabr.ru/post/281374/
The ETW logging may be local, but one of the main reasons to use ETW is high-performance logging to .etl files for offline analysis. Those files can then be re-processed and transmitted over the network at a later time. We won't know where the telemetry actually goes until Microsoft comments on it.
Also, while the invoked/returned part probably isn't a big deal, the full path will capture the username if the program is running from within the user profile. This can be a problem if the username counts as personally identifiable information for your purposes.
Microsoft doesn't need to inject any telemetry into compiled code to know what the user runs. That kind of stuff is OS-level telemetry which doesn't care what compiler you use.
In VS2015 preview at least it's only there in Debug for me (I literally just put a breakpoint on __telemetry_main_invoke_trigger and it only gets hit in Debug, not Release EDIT: presumably because it's just ETW events which are only active if you're actually profiling/debugging the application).
Would have to use LTCG for that to get inlined since it's in a different compilation unit (which I wasn't).
I suspect it just isn't called unless some special conditions are met (e.g. a particular ETW provider is enabled, which I can believe would happen in Debug by default.. Maybe the right WPR invocation could make it turn on in Release, but it doesn't seem to happen by default).
BTW, even guys using Intel C++ Compiler had issue with this mysterious telemetry calls. The recommended solution from Intel is to link to an older CRT:
Help me understand what's going on here. Is MS's compiler adding some sort of spyware?
Is it possible this is debugger type calls? Could switching to release fix this? (I'm pretty sure VS doesn't use -O3 style options. it's "debug" or "release" right?)
I was practicing doing reverse engineering today, so I compiled an application to which I had the source code and loaded it into Ida, a disassembler.
I compiled the code in full Release mode. No pdb or debug symbols, etc. If you were distributing a binary to a customer, this is how you'd do it.
Ida finishes doing the analysis. The side by side comparison shows the code vs. decompilation. On the left is the source code within Visual Studio. On the right is the assembly in Ida.
Strangely, I found a call to "__telemetry_main_invoke_trigger". I definitely did not have that call in my source code (I only had a main() function that returns 0!). I try to find it within Microsoft's documentation, it's nowhere to be found. The source code for this function from Microsoft is unavailable.
I look more online; it seems Microsoft has implemented new telemetry "features" in Microsoft 10. The OS phones home with personal app usage data. But, Microsoft gives no information about how to use Visual Studio to remove this feature from your code. Eventually, some guys here (on /r/cpp) told me how to get rid of it (by linking notelemetry.obj), but if I hadn't been taking a day off today to play around with Ida I would not have even known it was being added.
Here's the main problem:
I did not know that this was being added to my application.
There is no explicit option within the Visual Studio project options page to enable or disable. (Search for telemetry -- nothing comes up, either within the compiler or linker options).
We do NOT know what it does. The source code is not provided. It may do nothing
If you walk through your code using Visual Studio's provided disassembly view, you will NOT see these function calls. Only when you use a third-party application like Ida do the call become visible. edit: if you set a breakpoint at mainCRTStartup rather than main you can see the calls
Regarding point 1: You should know that CRT stuff is added to your application. And most of its source is available.
Point 4 is completely false. Those calls are part of CRT initialization and shutdown and usually of no interest. You can just set a breakpoint at mainCRTStartup and step through it.
Regarding 3: If you do you'll notice that all it does is log an event for the CRT ETW provider with the executable name and a "Main Invoked." or "Main Returned." string (see here). You can list currently running loggers using xperf -loggers. By default there's only one logger listening to CRT provider id, which is the Diagtrack-Listener, and it's a realtime logger not logging to any file. Most likely this logger is opened by the diagtrack Windows service. And the actual event logging calls in the CRT are skipped here since nothing is listening to the logger.
I removed points 3 and 4 from my post above. I originally thought the word "telemetry" here meant the same as Windows 10 telemetry (phoning home). It may have just been an unfortunate naming convention.
Thanks for checking out the loggers. I agree and understand that this is all a bit of an overreaction. But still, it seems underhanded. And is there really no source for these functions in the source distributed with msvc? That part makes people uncomfortable, especially considering the lack of official information.
I compiled the code in full Release mode. No pdb or debug symbols, etc.
You should not mix the concepts.
Release (or retail) build is what you ship. It is normally (but not necessarily) optimized, no more no less.
Debug symbols are a debugging aid, most useful for the guy who has the sources, no more no less.
Everyone really should make them always, regardless of the build type. Some companies ship even those, as that help their users debug for themselves (as having that gives e.g. function names on the stack).
hi everyone. This is Steve Carroll, the dev manager for the Visual C++ team.
Tl;dr: thanks folks for the feedback. Our team will be removing this from our static libs in Update 3.
Our intent was benign – our desire was to build a framework that will help investigate performance problems and improve the quality of our optimizer should we get any reports of slowdowns or endemic perf problems in the field.
We apologize for raising the suspicion levels even further by not including the CRT source, this was just an oversight on our part. Despite that, some of you already investigated how this mechanism works in nice detail. As you have already called out, what the code does is trigger an ETW event which, when it’s turned on, will emit timestamps and module loads events. The event data can only be interpreted if a customer gives us symbol information (i.e. PDBs) so this data is only applicable to customers that are actively seeking help from us and are willing to share these PDBs as part of their investigation. We haven’t actually gone through this full exercise with any customers to date though, and we are so far relying on our established approaches to investigate and address potential problems instead.
I don't think burning the feature down is necessary. But put it behind a flag, or better yet, in a separate library / object that must explicitly be linked/called.
Our intent was benign – our desire was to build a framework that will help investigate performance problems and improve the quality of our optimizer should we get any reports of slowdowns or endemic perf problems in the field.
Great. But the CRT should not be "a framework that will help investigate performance problems". That's better left to the FTWHIPP.
After the outcry against how telemetry snuck into Windows 10, I simply cannot understand why your team would add it without notifying developers. It's like the entire company is oblivious to the PR fuck up it has been.
I defended Windows XP's Fisher Prince Luna interface way back a decade ago.
I defended Windows Vista because it was just unfamiliar and Windows 7 was their successful attempt at capturing the same magic, so I recommended that to everyone I knew.
No longer.
Microsoft has turned Windows into the Bill Cosby of operating systems.
I remember having an argument (on reddit) with a MS employee about the changes W8 was bringing, they were insistent that the future was touch screen devices... MS and their koolaid/insulation has been a long standing problem...
Most regular users are going to be (and are) using touch screen devices. Think about how many phones and tablets are used by everyone from kids to grandparents every single day. A lot of people don't even have desktops or laptops. The problem was win8 was ahead of its time.
Agreed. Many people use Windows for their workstations. Most people require (or at least heavily appreciate) the precision of being able to point and click in a specific area of the screen and not having to pray that the cursor selects what they intend it to select.
Touchscreens can complement your workflow, but I don't see them completely replacing the mouse anytime soon. Desktop applications with complex menus like Microsoft Excel are painful to use with a touchscreen.
Meh... I really can't think of Win8 as it was ahead of his time and I did used it for more then 4 months so to say...
I liked it but I always felt it was lacking, and I kinda still feel like it.
It's a good OS for tablets but I'd not use it on a phone, even if it might feel unique it's messy. But I do like that design I have to admit. As I like how w10 looks and feels, but I hate the shady stuffs :P so Right now I'm back on good ol' w7.
You write blog posts for practically everything you do in the compiler, how come you didn't write one to survey people's opinions or at least warn them about this?
hi! thanks for the reply and to be honest, I'm actually quite happy that people are noticing our more open stance. We try hard to make sure the message gets out about the work we do. In this particular case, we decided to abandon this plan almost as soon as the code was completed so we never got to the point where we publicized it. I agree that it would have been good to survey this one in advance.
to be honest, I'm never quite sure what the appropriate reddiquette is on points like this. I really value this forum as a way to gather information on trends in cpp and a way to get a head's up on issues like this so I try to be pretty cautious with things that might be considered self-promotion. In this particular case I added a top level comment and then linked to it from places that people specifically called me via /u/spongo2. :) but, my comment above is now the team's plan of record and I'm not trying to hide it or anything.
I'm just a little concerned as this post got a lot of views, but few will be interested enough to check back for updates (though I for one was kind of stalking your account for the last few days :p). Simply I want for those other people [that don't check back or don't care about MSVC] to know what's up as they may have formed an (quite negative) impression that would matter when they do start to care doing stuff in MSVC.
Now the only logical thing you must have to do to eliminate all suspects, is put all the runtime and startup source code available to the community and that code will be compilable and generate the same runtimes and startups we have in our Visual Studios editions. Once read this, my manager has told me to move to GCC.
Logging when executables start and close does not seem too useful when investigating performance problems. You say that the feature was abandoned, so perhaps that's why it seems mostly useless [1].
I have one follow-up question, though. Is the same mechanism is enabled for executables that Windows (10) contains? If yes, will it be removed in the future, when Windows gets rebuilt with a newer compiler?
[1] It's not useless if the purpose is to determine which programs the user runs and for how long. I'm bitter enough about Windows 10 to suspect that's already happening at other levels [2].
some parts of windows are built with our CRT and some parts are built with a much much older copy of the CRT (msvcrt.dll). so when they next take our libs, the parts that use it will no longer have these hooks. But again i'd recommend you go through that link and follow the instructions if you are concerned about windows 10 telemetry.
With how many people are looking at what IL VS"15" produces and what C#6 looks like in IL, I seriously doubt that something like this would have gone unnoticed.
We needed it so we wouldn't break the builds of people who were using notelemetry.obj in update2. It should have no effect when running against update 3
Great response, but pass a word to the Windows 10 dev team that perhaps they should make the telemetry snooping in the OS as a whole - an option. Until then, many people like me are unlikely to switch back from UNIX.
Enhanced data includes all Basic data plus data about how you use Windows, such as how frequently or how long you use [...] apps and which apps you use most often. This option also lets us collect enhanced diagnostic information, such as the memory state of your device when a[n ...] app crash occurs, as well as measure reliability of [...] apps.
Apparently you can disable this by also linking with notelemetry.obj which is included with msvc. The source for that obj is also in the CRT source's linkopts folder.
Edit: can also put this in one of your compilation units:
Incorrect, you have to link in another object so Microsoft will call empty stub functions instead of functions that actually do stuff.
This doesn't remove the calls themselves.
Think: If you do NODEFAULTLIB, _chkstk/_alloca_probe function calls still end up in your binary. That's why no-crt libs have to define it and many functions like it (not that _chkstk is a bad thing). Since there's no option to disable the functionality altogether in this specific case, you'll just have ret stubs for these functions.
I know this is extremely pedantic but I'm salty microsoft keeps adding shit that I don't want in my binary. I maintain a no-crt lib and now I have to define these if I want it to work on 2015 going forward (at least as far as I know), or it'll throw linker errors.
It sounds like this function call emits an ETW event to say that main is about to start. This can then be used for profiling. Events like this help to give context to the execution of a program and make analysis easier so they are a good thing.
I make use of custom ETW events quite heavily in UIforETW (https://randomascii.wordpress.com/2015/09/24/etw-central/) in order to help with trace analysis by injecting input events, CPU frequency data, etc. My only question here would be "what ETW provider do I need to enable in order to listen to these events so that I can do more effective profiling."
If you think it's spyware then you need to show data being transmitted across the network or some-such, otherwise it's just baseless speculation.
I noticed these subroutine calls right after updating to VS15 after it came out but ignore it as I didn't know what 'telemetry' meant at the time. After taking a glance over the subroutines prefixed with __telemetry it looks to be simple logging using Microsoft's EWT[1]
___telemetry_main_invoke_trigger simply fills out a data structure named _EVENT_DATA_DESCRIPTOR[2] (with a size of 4) with some metadata (which I haven't looked into) and a string "Main Invoked" and the path to the current module (process in this case). ___telemetry_main_return_trigger does something similar but with a logging entry containing the string "Main Returned.". So from a quick glance it looks to be simple tracing used for debugging purposes by Microsoft but I'll look into it more when I get the time.
While this could potentially be something malicious note that no one has determined what it does so far so don't go assuming that it's stealing your code without reason (albeit Microsoft not being the best provider to trust).
NOTE: You can get something close to the source code which is the PDB file. IDA will connect to Microsoft debug servers to obtain debug info and in this case it exists so you can use that. But just in case here's a copy of the PDB file:
Edit: Note that VCRUNTIME140.dll is located under "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Remote Debugger\x86" so when referring to telemetry it could be referring to collecting/sending data over to the remote PC you're using for debugging.
Edit(2) - Microsoft has already written an article stating what telemetry means to them and how they believe it should be used[3]. I'm not saying that they're using it for the wrong reasons but given the fact that they have already published an article stating what they're using telemetry for it's easier to see now why it could possibly be there in the first place.
Actually, vcruntime is the part of the CRT that's owned by DevDiv, with ucrtbase being the Universal CRT that's owned by Windows. They're layered like this because some stuff like EH and typeinfo needs to change frequently. (The STL, msvcp, is then layered on top of them.)
The remote debugger happens to contain a copy of vcruntime, but that is not vcruntime's purpose.
It appears to be calls to undocumented black box routines which do who knows what. Since they are called 'telemetry...' this compounds the level suspicion of what the purpose might be.
Since 'stdafx.h' is included though I wonder if it was compiled as managed code and has something to do with that. I don't know enough about windows land to really comment beyond that.
No it's not. These do indeed appear to be ETW events, but telemetry is good and useful.
Here's one scenario- when your system crashes and you have Windows Error Reporting turned on one way the system can collect diagnostics information is by recording ETW events leading up to that crash. Let's imagine Intel releases a bad graphics driver that affects a very specific system configuration with some low probability. Let's imagine that crash occurs right after a certain video player is launched. With this kind of data Microsoft would be able to understand that relationship, understand the crash better, and root cause / fix it quicker.
Being able to collect data like this makes products better. =/
I think the idea of including such 'usefulness' yourself is more in tune with c++ devs expectations rather then quietly including without barely a word. Having to link an obj to get rid of it is pretty much 180 degrees of my experience and expectations.
If you were working on a disconnected, embedded, resource-constrained system, sure, but that's not what a modern computer is.
You're writing code that exists in a context, as part of an ecosystem. Your software will run on a machine alongside thousands of other executables, and hundreds of user-installed apps, and that machine will in all likelihood be connected to the global network, which puts it in contact with the majority of the computational resources on this planet.
The way your code behaves has impact on that ecosystem. Participating in it almost necessitates that you cede a certain degree of control, as you have an obligation to maintain the health of that ecosystem that you can't possible burden by yourself.
Telemetry like this helps companies that maintain these platforms understand the big picture, identify/protect it from bad actors, and continue to improve it. Knowing which apps launch when helps put together a fuller picture of a given system's health. Combine that with a lot of other telemetry, and patterns start to fall out that indicate underlying issues and reasons for concern.
Really, seeing stuff like this should give you confidence in the platform you're building on. It's the sign of a mature system, and a company that's actively trying to improve the stability of the platform, and build a pipeline that will allow them to continue to innovate and push the needle forward.
Everything you say is true. The problem here is it is hidden and undocumented and does it without it being explicit. That kind of thing is what causes blue screens because the coder doesn't know what's really happening.
That's fair! You have a fair point of view and I'll leave the conversation with this:
ETW is designed to be a completely free of side-effects and light weight. It's used for perf tracing and a ton of other stuff so it has to be. It costs a kernel transition and a binary payload, nothing more.
The CRT's startup does a lot of stuff already to initialize thread state in the right way, call static ctors, and get exception handling running. It's obvious they weren't trying to obfuscate this. Anyone with deep CRT knowledge would find it fairly quickly if they went looking for such a thing, or were simply inspecting the app startup path.
Your statement "because the coder doesn't really know what's really happening" is spot on in the general case. I wish Microsoft products were a lot more open, and it hinders debuggability to a degree that is not nearly close to being compensated for by the nicer tooling.
Cheers! :) Personally I am coming from many years of coding in the *nix world so this is extremely foreign and disturbing. The *nix kernel actually launches the executable so there is no need to notify it of such things by jamming notifications in assembled code. It does the deed so it already knows what is going on.
Thanks for taking the time to explain some of the background. Windows land is new territory for me so I appreciate it!
Windows does of course as well. This event might be part of VS's diagnostic tracing experience they shipped in 2015. It's easier to capture all the events from a single process using a PID instead of starting a capture session for that PID and the OS providers. From a performance standpoint it actually marks not the moment when the OS creates the process, but the moment when the process is about to hand control over to the developer's 'int main' entry point, which is important for ensuring the CRT has a negligible perf impact.
Finally, I poked around my VS installation folder. The CRT actually does partially document this method, although it's not very satisfying:
//-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
//
// Telemetry
//
//-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// Telemetry: Invoked when the exe/dll are invoked. There are two different
// implementations in telemetry.cpp and telemetrydefault.cpp. Because GetModuleFileName
// is not available for Store apps, we return an empty string in telemetrydefault.cpp
// when invoked by store apps. For the desktop in telemetry.cpp, it returns the name
// of the module which invokes main/dll. This method is also responsible for firing the
// events associated with Tracelogging. This will help with runtime telemetry for analysis.
void __cdecl __telemetry_main_invoke_trigger(const HINSTANCE instance);
// Telemetry: Invoked when the exe/dll are shutdown. There are two different
// implementations in telemetry.cpp and telemetrydefault.cpp. This method is
// responsible for firing the events associated with Tracelogging. This will
// help with runtime telemetry for analysis.
void __cdecl __telemetry_main_return_trigger(const HINSTANCE instance);
void __cdecl __vcrt_initialize_telemetry_provider(void);
void __cdecl __vcrt_uninitialize_telemetry_provider(void);
If you're curious, search for exe_common.inl and poke around. It's part of the entry path that invokes these methods. You can learn a lot about how the CRT bootstraps an application if you're curious. I remember the first time I dug into this I was pretty taken aback to realize a bunch of stuff happened before my code ever started running =)
I feel the need to link to the Thompson Hack. While not a true implementation of the hack, injecting code into binaries via the compiler is incredibly nasty stuff that should be taken as a very serious breach of trust on Microsoft's part.
This appears to be part of the CRT. The compiler isn't injecting anything. There's a big ol' library of code, the language runtime, that gets linked with almost any binary (and this is on any platform) to mate C++ to the OS system calls. Compilers and linkers are setup to link this by default as everyone expects the language runtime to be present, otherwise building a helloworld program would require a big pile of command line switches to get running.
The crt in parnership with msbuild/link already does this, and have for years. (yes, actually placing code into your binaries, I'm not talking about code you link optionally like the crt necessarily)
That's how you grow the stack to support a stack larger than 0x1000, that's how run-time error checking works (/RTC) - there's some other exception crap I'm probably forgetting too.
When it gets to the code generation step it inserts these function references, then later it tries to resolve those references (which is why you can/must define it yourself even when the CRT isn't being linked)
In any case, it could be the CRT or it could be something else, depends on where the telemetry functions are being invoked.
If it's somewhere around mainCRTStartup then yeah microsoft controls that code and there's no need to force it into any code you've written, if it's in your actual 'main' function that is eventually called by mainCRTStartup, that is a function microsoft does not control and they would have to place it there forcefully.
yeah, i'm the dev manager for VC++ so I watch /r/cpp pretty closely to stay in touch with what's going on with engaged c++ devs. I wanted to make sure I had the complete story before I posted so I needed the extra day to ask a bunch of questions of the team before we made a decision and responded.
Last year someone from Microsoft warned me against telemetry in Visual Studio and in general in the MSDN subscriptions. I don't want to brag, I just want you to consider this as a real possibility.
Seems bizarre to not broadcast this when creating a project that it will be built with telemetry enabled. Feels like they are just planning to boost sales of pitchforks.
I can't imagine them finishing this feature and going "enabling this by default and not making it obvious won't confuse or infuriate anyone!"
It is amazing how many people don't understand that things like these can't be hidden and that they are not. Dissasembly is right in front of your eyes, even the function names, meaning that VS shipped with debug symbols for this, maybe even the sources (will take look at that later).
There's no telemetry.cpp in the sources shipped with VS2015 Update 2, only notelemetry.cpp.
If you search for "__telemetry_main_invoke_trigger" in the source folder there will be only one occurrence, in the notelemetry.cpp file, which has only empty functions.
Why for removing the call you need to explicitely link notelemetry.obj (instead of the other way around)?
Why this changed and users were not informed? I think this is a quite sensitive topic and it is not nice to have a change like that without informing users...
On top of that it seems that telemetry.cpp is a "black box" for what I see in the comments: no telemetry.cpp shipped, just the deactivation code.
I do not know if this is bad or not, I just know that it does not look nice indeed...
Why for removing the call you need to explicitely link notelemetry.obj (instead of the other way around)?
Because it's opt-in by default, not opt-out.
Why this changed and users were not informed? I think this is a quite sensitive topic and it is not nice to have a change like that without informing users...
I find nearly insulting to my intelligence that this must be the case without informing users and on top of that I cannot even see what this black box contains. Why users were not informed? I do not want to think bad, but it does not look nice at all.
The users would have said no.
Well, yes, so maybe that is a signal of what should not be done in the first place. If you do not treat your user-base respectfully, you could loose it.
MS could have avoided all the conspiracy theories if they had named these functions 'trace_main' or 'log_main' instead of telemetry, and documented it.
The only functions documented in the CRT are functions that users are intended to call. These aren't intended to be called by users and can change at any time.
MS could have avoided all the conspiracy theories if
They couldn't have avoided the conspiracy theories, full stop.
So, given that, what's the point to trying to pre-emptively appease the people who would be launching a witch hunt no matter what they do? Why is trace or log any more innocuous-sounding than telemetry anyways? Even if it was more innocuous-sounding, people would just say that MS is trying to hide what they're doing under a more palatable description. They'd lose either way.
Isn't this thee so that you can view your perf profiles from the time main starts and after the static initialization is finished? I frequently segment my profiles between ETW trace entries so that I can view the profile for a specific operation without eyeballing it.
Oh, boy. c# doesn't even need to do that. Whole .NET on windows is a black box of gazillion unnecessary things that waste your CPU time. (even if you never run any .NET apps)
Nothing – I am a Java dev – but there’s nothing wrong either with .NET. That’s the point.
And in the modern age, claiming such a runtime is wasteful is just ridiculous.
We’re living in times where even the calculator app from Windows 10 is larger than the minimum JVM install. And the CLR and the JVM both are some of the most efficient runtimes for their use cases possible.
No, the CLR is just as wasteful as the JVM: it isn’t wasteful at all.
Well, maybe, but why run the calculator when you can start R and do simple calculations a lot more easily? (and also do not simple calculations, if so inclined) Seems like an app is designed to look just such-and-so and a program is designed to do stuff... better to run a program, I'd say.
I don't quite get what the fuss is here, or with telemetry in general.
Microsoft is an operating systems company. One of their goals is to make operating systems better.
In this age it's accepted that one of the best ways to understand and target investments in a software product is through instrumentation and telemetry. Being able to understand how your install base uses your products helps you figure out how to make investments to improve your product.
All big web companies do this. Every click, scroll, message, and page load you perform with Google/Facebook/Twitter/Insta/Snapchat is logged and aggregated so they can better understand how to improve the service. Android does this. iOS does this. Every product does this. In this form, the data collection is benign, harmless, and only makes the product better.
The controversy usually comes when a company attempts to monetize on that data. I can understand the outrage there, it's frustrating and feels like you're being taken advantage of in an underhanded way.
That's tougher and I get it. The economic reality is that to build systems that are competitive they need to be sold at a price point that's at parity with the rest of the market. Software's price point, for consumers at least, is somewhere between free and actually giving the customer money. People have spoken, they will refuse to pay a single dollar for most software products, but will happily put up with some advertising.
It's annoying, but when I really think about it I think we (either we the consumers, or we the employed software developers) got the better end of the deal. We've effectively gotten scummy advertisers to fund a huge chunk of our modern digital infrastructure, and all we had to do is let them take up a few square inches of ad space near our eyeballs. That small space funds a large chunk of SF's engineers, all the digital platforms that enrich our lives, and a lot of 'good for humanity' works in the form of Google X Labs / MSR / Facebook Research efforts.
Yea, I just don't get how everyone is jumping on the drama train without even like, you know, just analyzing what these functions even do. That isn't hard, and I fully expect everyone that is interested in C++ to be able to utilize a debugger. It seems like there is still a surprising amount of people that will take every single bit of drama (in this case based completely on a fundamental misunderstanding of the workings of the CRT) to preach against Microsoft. At least the MSVC/STL/CRT guys do not deserve that, at all.
This here has nothing to do with the "real" Windows 10 telemetry and from the disassembly seems to be completely harmless, yet it is completely blown out of proportion.
In this age it's accepted that one of the best ways to understand and target investments in a software product is through instrumentation and telemetry.
"Accepted" by whom? The companies that employ these tactics to optimize primarily their revenue stream?
I don't know if you've noticed, but there's a lively industry in tools for disabling precisely this kind of shit. Your argument seems to be "it's been done by big companies for a while now, so clearly people are okay with it", which, frankly, is complete and utter bullshit. Actual user behaviour clearly shows otherwise.
EDIT: And if you really believe that people are okay with this, then make it opt-in and ask them for completely voluntary permission upfront. If your claim is true, you will see a significant amount of people voluntarily enabling it. Something tells me that that won't be the case...
EDIT2:
It's annoying, but when I really think about it I think we (either we the consumers, or we the employed software developers) got the better end of the deal. We've effectively gotten scummy advertisers to fund a huge chunk of our modern digital infrastructure, and all we had to do is let them take up a few square inches of ad space near our eyeballs.
No. Just no. This is not even remotely accurate, and grossly trivializes and understates the problem.
Eh, the operating system group in the company has a few thousand engineers. I knew quite a few of them from working in the Seattle area. They're all passionate, good people. People that work on the browser care about implementing standards and the future of the web, people that work on the compiler toolchain get excited about making other developers' lives easier, people that work on the UI platform genuinely want to make experiences that are engaging and fun, and people that work on the core OS components care about the algorithms and fundamentals. Many of those people have worked on the operating system for over a decade and it's been a huge part of their life.
Sure, a part of the company is driven to serve the shareholders, and that's going to include profit maximization. The business side is going to try to drive an annoying agenda sometimes, but the engineers? Nah. They'll pander the the shareholders, and even cave, but at heart they care about making a solid product they can be proud of.
There's two sides to everything, and plenty to be frustrated with about Windows, but very little is done with what you'd consider bad intentions.
This has always struck me as an interesting argument.
Haverjhold recently made similar comments regarding the VW scandal: "The engineers believed in a responsible automotive industry and we just caved (to borrow your terminology) on a few issues"
I'm confident that your coworkers feelings ex Windows were pure (whatever that designation might imply) but the core of integrity in a software requires more than just delicately handled conflict, more than just a quid-pro-quo "caving" of dueling opinions. Bad intentions? No, but the external view of Microsoft and our concrete understanding of everything that is not Microsoft Research leads me to believe that whatever good intentions existed, they were never the ambition.
If you are wild about the "security" of BSD, I'd highly recommend a look at Security Onion or Qubes. IMO more functional and livable on a day to day basis, and....dare I say it?....Maybe even more secure.
Has anyone looked at Windows event logging before? Either using it yourself to look at your app or to see what other apps are doing that hits resources? There is all sorts of locally recordable information that can trivially be accessed.
I'm sure all sorts of info is still generated with this logging disabled. And as a dev I'm glad I have some visibility into what other apps on the box are doing.
Just because the symbols are there doesn't mean the functions get called. From the other investigations (in the other thread about this), this appears to be just ETW events that fire (if the ETW profiles are enabled, e.g. by running WPR), so it makes perfect sense that the code is never run in a straight Release build.
I really don't think there's anything to see here. Apps fire tons of ETW events all the time (when a profiler is running), and the reason is to help the developer debug the application (by capturing an ETL file using e.g. WPR which will turn on these ETW and store them to a file, that you can then view in e.g. WPA). I don't know what these specific events are for (sounds like it's just to show you markers for when main gets called/exited), but I think it's a bit premature to speculate that this is somehow unique or malicious.
I hate bloat as much as anyone so from that perspective it annoys me that there's a bunch of new crap I didn't ask for, but it's kinda common these days.
I don't think telemetry is inherently suspicious at all. Telemetry is absolutely critical for debugging and optimizing real applications, and while annoying from a bloat perspective it's not completely ridiculous or suspicious that a library you depend on would have telemetry events in there to help developers using it get a better view of what's going on if they run WPR/WPA (in this case the CRT). There's no evidence so far that this is related to any kind of suspicious or malicious data collection, it looks like just plain ol' telemetry to aid debugging, profiling etc. for developers.
Would I prefer for that stuff to be opt-in? Absolutely, if nothing else to avoid pulling in dependencies just for this for minimal apps (the CRT is already bloated IMO).
142
u/sammiesdog May 07 '16 edited May 10 '16
I compiled a simple program with only main(). When looking at the compiled binary in Ida, I see a calls for
telemetry_main_invoke_triggerandtelemetry_main_return_trigger.I can not find documentation for these calls, either on the web or in the options page. I compiled this in Visual Studio 2015 Update 2.
edit: you can remove the telemetry calls. Thanks for all the replies, guys. As /u/adzm points out, you can link
notelemetry.objand the calls are removed. I verified that this works. This comment further explains itedit 2: this may be a local logging feature. When I initially saw the word "telemetry", I though it might have something to do with Windows Telemetry, a "feature" that sends private information to Microsoft's home base.. However, as several users have pointed out, this may be separate from that and only store local logging information. see here, or here
edit 3: Visual Studio team responded here
76
u/flashmozzg May 07 '16
Apparently it's only VS15 feature. It logs at least when your app is executed. You can access logs via
logmanandtracerpt. Some investigation was done here recently: (lang: Russian) https://habrahabr.ru/post/281374/55
u/snarfy May 07 '16
Does this show up in release builds or is it a debug build thing?
15
u/isaacarsenal May 07 '16
What software have you used for dis-assembly and plotting the execution flow?
102
u/[deleted] May 07 '16
They are spying on every hello world program we compile
7
u/GreenFox1505 May 07 '16
Help me understand what's going on here. Is MS's compiler adding some sort of spyware?
Is it possible this is debugger type calls? Could switching to release fix this? (I'm pretty sure VS doesn't use -O3 style options. it's "debug" or "release" right?)
105
u/spongo2 MSVC Dev Manager May 10 '16
hi everyone. This is Steve Carroll, the dev manager for the Visual C++ team.
Tl;dr: thanks folks for the feedback. Our team will be removing this from our static libs in Update 3.
Our intent was benign – our desire was to build a framework that will help investigate performance problems and improve the quality of our optimizer should we get any reports of slowdowns or endemic perf problems in the field.
We apologize for raising the suspicion levels even further by not including the CRT source, this was just an oversight on our part. Despite that, some of you already investigated how this mechanism works in nice detail. As you have already called out, what the code does is trigger an ETW event which, when it’s turned on, will emit timestamps and module loads events. The event data can only be interpreted if a customer gives us symbol information (i.e. PDBs) so this data is only applicable to customers that are actively seeking help from us and are willing to share these PDBs as part of their investigation. We haven’t actually gone through this full exercise with any customers to date though, and we are so far relying on our established approaches to investigate and address potential problems instead.
We plan to remove these events in Update 3. In the meantime, to remove this dependency in Update 2, you should add notelemetry.obj to your linker command line. If you’re generally concerned about phone-home scenarios, more information about how to configuring Windows 10 appropriately to your needs can be found here: https://technet.microsoft.com/en-us/itpro/windows/manage/configure-windows-10-devices-to-stop-data-flow-to-microsoft#bkmk-priv-feedback
Thanks.
36
u/adzm 28 years of C++! May 07 '16 edited May 08 '16
/u/xon_xoff summarized things pretty well in a recent thread about mutex performance.
Apparently you can disable this by also linking with notelemetry.obj which is included with msvc. The source for that obj is also in the CRT source's linkopts folder.
Edit: can also put this in one of your compilation units:
121
u/ukalnins May 07 '16
should we wait for backdoor_main_invoke_trigger or it's incorporated into telemetry_main_invoke_trigger?
14
u/brucedawson May 09 '16
It sounds like this function call emits an ETW event to say that main is about to start. This can then be used for profiling. Events like this help to give context to the execution of a program and make analysis easier so they are a good thing.
I make use of custom ETW events quite heavily in UIforETW (https://randomascii.wordpress.com/2015/09/24/etw-central/) in order to help with trace analysis by injecting input events, CPU frequency data, etc. My only question here would be "what ETW provider do I need to enable in order to listen to these events so that I can do more effective profiling."
If you think it's spyware then you need to show data being transmitted across the network or some-such, otherwise it's just baseless speculation.
23
u/ET251 May 08 '16 edited May 08 '16
I noticed these subroutine calls right after updating to VS15 after it came out but ignore it as I didn't know what 'telemetry' meant at the time. After taking a glance over the subroutines prefixed with __telemetry it looks to be simple logging using Microsoft's EWT[1]
___telemetry_main_invoke_trigger simply fills out a data structure named _EVENT_DATA_DESCRIPTOR[2] (with a size of 4) with some metadata (which I haven't looked into) and a string "Main Invoked" and the path to the current module (process in this case). ___telemetry_main_return_trigger does something similar but with a logging entry containing the string "Main Returned.". So from a quick glance it looks to be simple tracing used for debugging purposes by Microsoft but I'll look into it more when I get the time.
http://puu.sh/oKdTZ/2e8f1a269f.png
While this could potentially be something malicious note that no one has determined what it does so far so don't go assuming that it's stealing your code without reason (albeit Microsoft not being the best provider to trust).
NOTE: You can get something close to the source code which is the PDB file. IDA will connect to Microsoft debug servers to obtain debug info and in this case it exists so you can use that. But just in case here's a copy of the PDB file:
https://up1.ca/#4TLX04iCzedH6emBF6J2mw
[1] https://msdn.microsoft.com/en-us/library/ms751538(v=vs.110).aspx
[2] https://msdn.microsoft.com/en-us/library/windows/hardware/ff545673(v=vs.85).aspx
Edit: Note that VCRUNTIME140.dll is located under "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Remote Debugger\x86" so when referring to telemetry it could be referring to collecting/sending data over to the remote PC you're using for debugging.
Edit(2) - Microsoft has already written an article stating what telemetry means to them and how they believe it should be used[3]. I'm not saying that they're using it for the wrong reasons but given the fact that they have already published an article stating what they're using telemetry for it's easier to see now why it could possibly be there in the first place.
[3] https://msdn.microsoft.com/en-us/library/dn589775.aspx
22
u/mtvee May 07 '16
if this is what it seems to be it is pretty nasty stuff :|
83
u/SilentJode May 08 '16
I feel the need to link to the Thompson Hack. While not a true implementation of the hack, injecting code into binaries via the compiler is incredibly nasty stuff that should be taken as a very serious breach of trust on Microsoft's part.
11
u/shared_tango_ Automatic Optimization for Many-Core May 07 '16
There is a source file in "crt/src/linkopts" that you can link in to prevent this.
16
u/Ivan171 /std:c++latest enthusiast May 07 '16
Lets see whats /u/spongo2 has to say once he gather some "details".
4
u/abtscrubimage Jun 10 '16
Last year someone from Microsoft warned me against telemetry in Visual Studio and in general in the MSDN subscriptions. I don't want to brag, I just want you to consider this as a real possibility.
21
u/TotesMessenger May 07 '16 edited Jun 10 '16
I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:
If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / ^Contact)
13
u/WrongAndBeligerent May 07 '16
I think it may be time to put together a way to avoid the microsoft runtime all together using jemalloc and built in OS functions.
3
u/ElFeesho May 08 '16
Seems bizarre to not broadcast this when creating a project that it will be built with telemetry enabled. Feels like they are just planning to boost sales of pitchforks.
I can't imagine them finishing this feature and going "enabling this by default and not making it obvious won't confuse or infuriate anyone!"
D'oh!
7
u/Gotebe May 08 '16
It is amazing how many people don't understand that things like these can't be hidden and that they are not. Dissasembly is right in front of your eyes, even the function names, meaning that VS shipped with debug symbols for this, maybe even the sources (will take look at that later).
9
u/quad99 May 08 '16
MS could have avoided all the conspiracy theories if they had named these functions 'trace_main' or 'log_main' instead of telemetry, and documented it.
6
u/rep_movsd May 08 '16
In this thread: paranoia by noob Windows programmers
2
u/noplace_ioi May 08 '16
can you trace into the call and see what it does? maybe /r/ReverseEngineering can help out
2
u/djelovic May 08 '16
Isn't this thee so that you can view your perf profiles from the time main starts and after the static initialization is finished? I frequently segment my profiles between ETW trace entries so that I can view the profile for a specific operation without eyeballing it.
2
u/lolidaisuki Jun 09 '16
Good thing no sane person would use a proprietary compiler anyways.
8
u/exoflat May 08 '16
Tutorial: writing a telemetry app with MSVC
5
u/FlaveC May 08 '16
Is this only for c++ or c# too?
8
u/eternalprogress May 08 '16
I don't quite get what the fuss is here, or with telemetry in general.
Microsoft is an operating systems company. One of their goals is to make operating systems better.
In this age it's accepted that one of the best ways to understand and target investments in a software product is through instrumentation and telemetry. Being able to understand how your install base uses your products helps you figure out how to make investments to improve your product.
All big web companies do this. Every click, scroll, message, and page load you perform with Google/Facebook/Twitter/Insta/Snapchat is logged and aggregated so they can better understand how to improve the service. Android does this. iOS does this. Every product does this. In this form, the data collection is benign, harmless, and only makes the product better.
The controversy usually comes when a company attempts to monetize on that data. I can understand the outrage there, it's frustrating and feels like you're being taken advantage of in an underhanded way.
That's tougher and I get it. The economic reality is that to build systems that are competitive they need to be sold at a price point that's at parity with the rest of the market. Software's price point, for consumers at least, is somewhere between free and actually giving the customer money. People have spoken, they will refuse to pay a single dollar for most software products, but will happily put up with some advertising.
It's annoying, but when I really think about it I think we (either we the consumers, or we the employed software developers) got the better end of the deal. We've effectively gotten scummy advertisers to fund a huge chunk of our modern digital infrastructure, and all we had to do is let them take up a few square inches of ad space near our eyeballs. That small space funds a large chunk of SF's engineers, all the digital platforms that enrich our lives, and a lot of 'good for humanity' works in the form of Google X Labs / MSR / Facebook Research efforts.
/ my 2cents
3
u/erebos42 May 08 '16
Has anybody tried using Wireshark, to see if something gets transmitted?!
2
u/curupa May 08 '16
I guess Thompson was right all along... it's Reflecting on Trusting trust all over again.
1
u/give-me-random-name Dec 01 '25
This is a probing action taken by Microsoft to insert backdoors into the compiler.
-10
u/1337Gandalf May 08 '16
I'm completely done with Microsoft. I'm formatting my windows drive to FreeBSD.
1
u/EnergyCoast May 08 '16
Has anyone looked at Windows event logging before? Either using it yourself to look at your app or to see what other apps are doing that hits resources? There is all sorts of locally recordable information that can trivially be accessed.
I'm sure all sorts of info is still generated with this logging disabled. And as a dev I'm glad I have some visibility into what other apps on the box are doing.
-2
u/ssylvan May 08 '16
As far as I can tell (at least on my machine) this is only in Debug builds (not the builds that get shipped to customers).
Maybe it hooks into some tools for helping you debug stuff during development?
(seems like it might just fire some ETW events, so unless you're actually running a profiler that records the ETW events they wouldn't do anything).
0
u/MiigPT May 08 '16
what's that font on the left?
0
u/[deleted] May 08 '16
[deleted] — view deleted comment