MSVC CRT Sneaks in Telemetry by Default? | Hacker News Reader

MSVC CRT Sneaks in Telemetry by Default?

reddit.com

130 pointsfrozenport33 comments

Lots of things inside Windows emit ETW events, which is Windows equivalent of DTrace (basically the entire OS, and .NET), it's super useful for debugging performance related events. It's not "Telemetry" like Google Analytics, it's for _you_ to debug your own programs.

The easiest way to view the output of them is via WPA, you can watch some videos about it at https://msdn.microsoft.com/en-us/library/windows/hardware/hh...

It's not "Telemetry" like Google Analytics, it's for _you_ to debug your own programs.

People have been debugging just fine for the past few decades without extra bloating code in their runtime libraries. A good old-fashioned debugger with tracepoints will accomplish the same thing, and doesn't add overhead when it's not in use.

Tracepoints are very expensive and will definitely make any performance traces incorrect

I don't know what you mean exactly by "expensive", but the time spent in profiling code should obviously not be counted, and it is quite simple to do so.

The real point I'm trying to make is that tracepoints do not require any changes to the original code. They are entirely opt-in and if you do not choose to use them, your application won't have this extra useless code compiled into it.

That said, the few times I've had to use Visual Studio's tracepoints, they were horribly slow. Apparently this is "by design":

https://connect.microsoft.com/VisualStudio/feedback/details/...

Being an RE, I've implemented and used simple tracers that were more than 100x faster than VS' on hardware not quite as fast as that today (around a decade ago). Watching variables in memory and CPU registers changing in near-realtime is something I very much miss when I have to work in higher-level languages...

> I don't know what you mean exactly by "expensive", but the time spent in profiling code should obviously not be counted, and it is quite simple to do so.

Only for the exceedingly trivial case where you exclude profiling CPU time from overall CPU time. Profiling to see when exactly you're CPU bound, and when you're disk bound? Eh. Profiling to see if false sharing is eating your lunch? Eh. Trying to see when you'll fail to make vsync and the framerate will hitch? Eh.

I've had to #ifdef out tracing info that boiled down to "if (happensToBeFalse) {...}" in tight loops because it was throwing off my profiling results too much. Compiler and debugger injected tracing info isn't going to be any cheaper than that.

> The real point I'm trying to make is that tracepoints do not require any changes to the original code. They are entirely opt-in and if you do not choose to use them, your application won't have this extra useless code compiled into it.

Debugger injected tracing is useful, but hardly a panacea. You know this or you wouldn't have written your own tracers, most likely. When you have the source code, writing a separate tracer doesn't make sense - you just embed your stuff directly into your source code and #ifdef it out when necessary (same result: no "extra useless code".) Easier to add data to your events ("what file am I hanging on?") without performance killing deep copies, or reverse engineering structure offsets. No need to subvert platform security on e.g. Xbox One to figure out how to inject your own code at runtime. No need to deal with another process's address space.

Sometimes tracing info is useful to leave in your release builds. Which is cheaper: A few rare if() checks and some dormant code? Or spending hours rebuilding, retesting, redeploying to the production machine that is experiencing some issue you need tracing info to diagnose? As long as you're not shoving it into your inner loops it's fine.

An event on entry/exit of main is about as far away from an inner loop as you can manage. It's fine.

Even easier, you can also use Device Portal [1] in new Windows 10 builds to get a live feed of ETW providers from your device in a browser. Think logcat but with more structure to the events and providers (but without the buffers, sadly)

1. https://msdn.microsoft.com/en-us/windows/uwp/debug-test-perf...

Disclaimer - I work on Device Portal.

A very similar post appeared on reddit. Almost identical.

Man, you're fighting uphill battle. This is a typical case of people being scared of something they don't understand coupled with not willing to learn. Yesterday[1] I read that RFID tags emit radiation because RF means radio frequency, so it's harmful. Today I read that ETW is harmful telemetry and MS injecting code into binaries.

I thought I got over the fact that a lot of people programming on Windows don't understand its internals. But I didn't. Understanding ETW is essential on Windows, just like DTrace is a must on Linux or ktrace on OpenBSD. It's also well documented on MSDN[2]. ETW also typically boils down to a single check for any listener interested in messages. This info is public and has been for ages. I guess I'd understand this attitude if there were no real reasons to bash MS.

But I guess what pisses me the most is the context of that reddit thread. People complain about mutex performance and show code where they mutex vector accesses. This kind of code does not belong in performance critical game code. "Hey, I've got a salad to make, what tools do I need? I've got a hammer but these veggies are not a nail. But I'm sure if I take two hammers to the task my problem suddenly changes to nailing things together!"

For Pete's sake, if a slight performance drop in mutex performance of user mode code hogs your application, you're mutexing too much. Your code is a problem, not a mutex. And if you're using single threaded STL primitives for communication between threads you have no right to complain about CRT performance. None. This lack of intellectual curiosity in people sinks my heart.

Rant out. :S

[1] not on HN though, thankfully [2] well by MSDN standards at least - ETW documentation is hard to grok without referencing sample code in the Windows SDK

Another good example was when Microsost introduced C++/CX, their language extensions for creating WinRT components in C++.

Lots of devs were up in arms that C++ was going to be managed in the new WinRT world, just because C++/CX syntax is based on C++/CLI.

Microsoft had to do lots of sessions to clarify that, and I bet even today there are developers that don't get the difference between C++/CX and C++/CLI.

But it could be telemetry like Google analytics. Given that MS didn't tell us they were doing it, what else are they sneaking in? At the end of the day this sets a really dangerous precedent.

This also might have performance side-effects in highly parallel codes.

A different post showing exactly what the info is and where it goes (hint: nowhere):

https://www.reddit.com/r/cpp/comments/4ibauu/visual_studio_a...

There is no info going in and out, it just adds hooks to the runtime for you to add telemetry if you want it.

you can link notelemetry.obj and the calls are removed

It's still a ridiculous idea to make this the default. Rather, it should be something you opt-in to if you want to debug, more like

you can like telemetry.obj and the calls are inserted

Why is this comment down voted? Is it not a good suggestion? Opt-in to telemetry vs opt-out seems like a pretty reasonable thing to choose? Is that not true?

Is that not true?

Microsoft recently doesn't seem to think so.

Comments from known VC++ devs in the linked Reddit thread[1] suggest there may be more to it than that, but they haven't followed up with confirmation either way yet.

[1] https://www.reddit.com/r/cpp/comments/4hoyzr/msvc_mutex_is_s...

Just to mirror what's in the reddit thread the quick "fix" for this is to simply link to notelemetry.obj just as you would with the other small obj files included in the CRT lib directory[1]. Oddly notelemetry.obj is missing from their list.

[1] https://msdn.microsoft.com/en-us/library/ms235330.aspx

https://gist.github.com/aicpp

I was looking at the C Runtime Library the other day and found the following series of article on how to remove msvcrt.lib:

https://hero.handmade.network/forums/code-discussion/t/94

Basically, the entrypoint for Win32 isn't actually WinMain, but WinMainCRTStartup. To replace it you need to reimplement a few things, including an implementation of memcpy() and memset(), some allocation routines for 32 bit code under 64 bit systems and some floating point routines.

Curious to know what the new universal runtime loads up. I'd love Microsoft to be a bit more open about what their C runtime actually does before a program linked to it loads. Of course, perhaps they are, and I've missed it.

Edit: looks like someone was pissed about this lack of documentation in the beta Vususl Studio beta:

http://levicki.net/articles/2015/12/03/RANT_Microsoft_Visual...

> I'd love Microsoft to be a bit more open about what their C runtime actually does before a program linked to it loads. Of course, perhaps they are, and I've missed it.

The source code to the CRT ships with Visual Studio.

> Basically, the entrypoint for Win32 isn't actually WinMain, but WinMainCRTStartup.

The entry point is whatever appears in the PE header and you can name it whatever you want. It just so happens that the default linker settings point it at CRT, which calls into you. You can change this with linker settings.

Ya, I didn't explain that as well as I might have. That is correct :-)

Note this Reddit comment by what appears to be a couple of MS devs:

(fourbadcats) Indeed. This sure better be an oversight of something not disabled when transitioning from release candidate to RTM. We will lose credibility with our customers if we claim that our application does not collect any usage data when this is going out over the wire. I noticed that notelemetry.obj is not present with VS 2015 Update 1 so this may have been introduced with Update 2. Let's hope it's gone in Update 3.

and this too:

(spongo2) Let me gather some details

However, this is definitely highly disconcerting.

The first comment, from fourbadcats, is apparently from a customer of MS (I don't know who they are), referring to their customers in turn. The second from spongo2 is VC's dev manager, my double boss. (I know very little about this topic as the STL's senior developer; this stuff is in the vcruntime layer below the STL.)

Yes ask your boss.

> We will lose credibility with our customers if we claim that our application does not collect any usage data when this is going out over the wire.

Microsoft wasted Win10 and Office 2016 already (and sneaked it into newer Win7&8 updates). It would have been such a great chance after the misguided/failed Win8, but then MS decided to add invasive telemetry and other phone home crap incl whitelisting IPs and URLs in kernel mode network layer, with no way to deactivate all of it - don't even think MS respects you as a end consumer (you are their sheep and loose control of your PC) and not even as enterprise customer nor with LTSB (except you request a custom build of the network stack).

So, MSFT is injecting code into a binary? This probably won't be a popular opinion here, but this does seem like a pretty good definition of malware to me. It doesn't make it okay that just because it's MSFT they can get away with it.

So every C++ compiler that allows you to build in the run-time library with a flag is malware? The user in this case used the /MT flag which tells the compiler to inject code by design.

The issue here isn't with injecting code, it is with what that code does. I'm sure many would find the lack of informed consent offputting, and I'd agree. This "feature" lacks documentation and or a warning.

You're linking against their C runtime (CRT) standard library, so it's not injecting per se. Standard C libraries often include all kinds of stuff in otherwise "empty" main(){} executables.

That being said, I've never heard of a C runtime including telemetry by default, so malware wouldn't be a totally unfounded opinion.

The point of a linker is sort of to "inject code into a binary", and the point of a library is sort of to be code that gets injected into a binary. How does any of this relate to malware?

I don't know if it's malware, but it's something really close.

Or, to put it another way, it's Microsoft saying, "Hey, you probably want local event tracing in your application, so we enabled it by default." Been a while since I've done Windows dev (and I'll install VS Beta to check this out), but it might be as simple as the default project skeleton including an instrumentation manifest whose presence adds EWT tracing to the build chain. It's basically Windows' syslog equivalent, so unless Win10 is sending back all EWT eventing as a matter of course -- which, Win10 being a privacy carwreck, it might well be! -- I'm not too alarmed.

I don't think it's a reason to be alarmed regardless of the intention, it's just one of those "Huh, isn't that funny?" moments IMO.

Data isn't free for end users; it's capped for a lot of them. While I get that devs can 'fix' this I wonder how many will in practice.

The data stays local at most. It's a hook for devs to add telemetry, not telemetry that goes to the mothership or anything like that.

So much for "developers developers developers developers developers" https://www.youtube.com/watch?v=Vhh_GeBPOhs

Reply on news.ycombinator.com