Tell HN: By default, New Relic will start collecting production log data on 5/3

>Q: What does this mean for my New Relic bill? A: Collecting application logs means that more data will be ingested into the platform, at your standard ingest rate. The APM agent samples logs to ensure optimal agent performance. You can increase or decrease an application’s log volume as desired. Learn more here.

Automatic, opt-out bill increases come 5/3?

They did something similar with distributed tracing. Seems like dirty business tactics are becoming the trend at newrelic.

"We're going to start charging you to grab up your PII unless you explicitly tell us to stop. YOU'RE WELCOME!" sounds like it ought to be illegal somewhere the New Relic developers have to worry about the laws of.

I wonder what they'd do if someone rigged their install to feed their servers junk information. Is that possible? I honestly don't know how the New Relic stuff works.

You can feed their servers as much arbitrary shit as you want - you'll just end up paying for it.

It is illegal. The question is who will be liable, NR or the customer, at first glance it probably will be the customer even though NR is the one causing the violation. Regardless, this kind of move should be punished by customers cancelling en masse.

> I wonder what they'd do if someone rigged their install to feed their servers junk information. Is that possible?

I'd like to think folks would be willing to pay to do it, but unless a large number of people do they probably wouldn't care.

Companies who are collecting data for profit don't really care how accurate any of it is. As long as it looks real enough they can still sell it to data brokers (oh sorry, I mean the "third-party service providers that support our business and operations"). If it doesn't look real enough it's probably easy to filter out.

Ah, so that's how they're paying for the huge number of jobs they seem to be trying to fill all the time. Some days my entire job feed is just New Relic.

Curious where do you look for jobs? I’m in the mortgage industry and might be a good time to start looking.

I have a running LinkedIn search for 'principal software engineer' and 'software engineer' in Portland that has been going for a while. I'm not really looking right now, but I like to see what the trends look like.

Curious what does that mean? "running" it's automated/bought or you just search time to time?

If you do a job search in linkedin, there's an option at the top left to set an alert.

Yep, it's automated. LinkedIn has the concept of saved searches, and these are mine. It sends me an email every day or so with a list of matching jobs in the local area.

I just landed a job after 8 months of applications. Oof. The usefulness probably varies city to city, but for NY tech jobs I got the most mileage out of Built In[1]. Being able to filter with a checkbox for "frontend jobs" was nice. I was also a daily user of Indeed (awful) and LinkedIn (saturated with bad jobs – but their alerts are lovely!).

All three of these sites have a large amount of "promoted" jobs that show up. I'd recommend filtering them out with a script or user style.

I wasn't looking for a start up role, but have heard good things about AngelList[2] for those that are.

[1] https://builtin.com/tech-hubs

[2] https://angel.co/

LinkedIn alerts cover most use cases. For startups: https://topstartups.io/

How else are they going to handle the inflation?

New Relic wants to capture as much data as possible so that they can charge you as much as possible. I run a small environment (~20 small VM's) and we quickly went over the 100GB monthly free plan and that is with just APM Metrics. We found it impossible to figure out how filter out irrelevant metrics and their support/community was not helpful (again, they have no interest in helping limit how much data is collected).

We tried the log ingest but even for 1 VM, it was several GB per day data. 99% of the log data was irrelevant and trying to figure out how to filter it via New Relic was extremely difficult to figure out. We turned it off.

I guarantee that if they turn this on on 5/3, then 90% of their free accounts will shoot past the 100GB free tier. They hope people will convert to paid plans.

If they provided clear and easy to understand examples on how to filter the data before sending it into their environment, it would allow people to decide what data to send and then only have to pay for the data that is relevant instead of EVERYTHING.

Setting the following envvars will disable it:

NEW_RELIC_APPLICATION_LOGGING_ENABLED=false NEW_RELIC_APPLICATION_LOGGING_FORWARDING_ENABLED=false

This is a form of maintenance that really makes me believe to just host a VM I administer is the better option.

NewRelic being predatory again, nothing new to see here. I'll let our account rep know that this makes us hate them even more.

When they changed their billing to be per user our bill went up 20x for no additional value. Unfortunately I haven't found a better APM product that ticks all the boxes yet. I don't care about anything else but they don't understand that. And APM hasn't seen a single new feature in the 5 years I have used it. Stop taking my money to develop features I don't care about.

Yeah, APM is the only reason to use New Relic.

There are other options out there, but surprisingly few open source ones. I know APM is hard, but...

Try this: https://github.com/SigNoz/signoz

I checked the demo videos, it looks like it has the same problems I have with most NewRelic competitors: Too much focus on traces.

In a high-throughput distributed system, traces start being mostly noise due to random latency spikes which are usually not very interesting.

NewRelic is a lot more focused on the average but it breaks it down nicely so that you know where to start optimizing code. I rarely look at the p95 traces to do that.

I use raygun for our .net websites. I found Long outstanding issues that new relic never picked up and queries that were n+1 or just slow.

New relic collects a lot of data but it really is a lot of noise. We dropped new relic because it kept modifying requests in one of apis that caused the result api to be corrupted and new relic ghosted us when unable to fix it.

That's not an APM. It's just distributed tracing.

A real APM cares more about profiling program execution than it does HTTP requests. I need to see function calls, not just HTTP requests.

Sentry's hosted and self-hosted options offer APM, and the source is open with an eventually permissive license.

(disclosure: I work for Sentry)

Sentry is not APM, when I last tested the product it looked like randomized tracing comparable to Datadog.

Edit: I looked at the docs, Sentry still has a sample rate parameter so this product can't give you the same results. With NewRelic I can have 100M transaction per day and then I have a single different transaction once every day and I can be sure that the agent will pick it up - always. With sampling you don't get that. Tracing gets you more depth but I seldom need that, I want APM and I only need tracing for a very very low percentage of transactions.

> I'll let our account rep know that this makes us hate them even more.

“Glad we could help! Just remember to continue making regular payments.”

Their new pricing definitely was higher but was awesome because we could use all their tools. DataDog is a nightmare and every other tool which nickels and dimes you. I love that they charge per GB and admin user.

edit: We had to resort to using APM on a portion of the farm but this allowed us to use it on all of it and finding issues was way faster. Incidents by level 1 were resolved faster as well and that was huge for global teams SLA.

why do you hate them for being, in your own opinion, the best in the world?

The APM product hasn't seen any new features in the 5+ years I have used it. The new UI that they have been working on for years even removed features, luckily you can still switch back to the old UI.

APM is the product which got them off the ground, now they hardly care about it. Yet they have no problem raising prices beyond 10x for certain usage patterns. And its not really negotiable either. I understand that they want to make money, but we are paying hundreds of thousands more, not just a few thousands. And we get exactly the same product we had 5 years ago.

People are equally annoyed about the Datadog pricing model but at least there, I don't pay for what I don't use and I only pay for data ingested. NewRelic has the per user pricing model for one single reason: They want to upsell everything else on their platform because you are already paying for the users. Ingesting logs by default is just another way of costumer unfriendly upselling.

We had the same issue and were so frustrated, that we decided to move to DataDog. Yes, the NewRelic APM UI/UX is better but you get used to DDs pretty fast too. And log management is better on DD side actually.

A fancy APM UI isn’t worth it to tolerate these dark business patterns which they try (or just do) all the time IMO.

We didn’t regret the decision.

We (Livepeer) switched to self-hosted Loki recently for this and couldn't be happier. Completely S3-backed, cheapest option possible, adjusting your retention is as easy as setting an expiration rule on the bucket. Queries are as fast as the hardware you feed it 'cause it's just all your nodes doing a brute-force search.

Query language is a bit of a PITA compared to some of the other options, but you get used to it.

Interesting - didn’t know it could by S3 backed. I shall have to investigate that!

You can get the whole grafana stack to be s3 backed now I think.

Cortex for Prometheus, and tempo also all have s3 storage capabilities iirc

You can also use Thanos (https://thanos.io/) for storing Prometheus metrics into S3. The design is pretty similar to Loki (which Thanos predates), separated into services so you can nicely horizontally scale your ingestion path separately from your read path depending on workloads.

New Relic's logging agent had a bug in it that lost a good portion of our logs for a few months last year. Support were extremely lethargic about diagnosing, and once they found that it was in fact a bug on their end, weren't at all apologetic. We dropped them and moved to DD for logging and it has been generally better so far, although I know by reputation they have price surprises now and again.

Amusingly, despite our account being closed, I received this same email this morning, so somebody there isn't exactly on top of keeping their mail list clean.

IME, they will never stop sending you email.

DD pricing is a f^<king mess for larger organisations. I’ve spent more energy figuring out DD bills than I have AWS. And it doesn’t help that DD Account Managers are just as clueless.

If you're wondering why, the answer is NewRelic ($NEWR) has a market cap of $4B while DataDog ($DDOG) has a market cap of $40B.

Datadog isn’t exactly the best example of friendly pricing as well.

Ugh, I was not happy reading the email this morning. At least you can say they're being transparent with the FAQ, 'yeah, this will increase your bill'.

We already had logging disabled, but changing the default behavior in their favor is shady.

New Relic is already one of the most expensive items in our budget. Last year, I had to tell a producer "yeah, our SaaS that monitors our servers is more expensive than the servers themselves". That changed this year (started ramping up for scale), but it's still in the among the top of services we pay for.

This sounds only slightly less irritating than the Datadog agent which has features on by default that are billed at the on-demand rate.

You will be charged for the increase data usage from New Relic unless you opt-out.

> features on by default that are billed at the on-demand rate.

Can you please elaborate on what these features are? I used to work in the agent team and we were cautious not to have that happen.

Host metrics I believe.

Metrics from all official integrations (including host metrics) are included in the $15/month/host flat pricing. Of course, them being sent will trigger that per-host billing, but that's an expected consequence of installing the agent on a host. The only caveats I know of are:

- some integrations have options to collect more timeseries. If the volume is too high, these will be marked in the documentation as billed as custom metrics

- custom integrations (including custom prometheus/openmetrics instances) are billed as custom metrics, as if they were sent through statsd

- container monitoring (enabled by default in the docker image, but not the other agent installations) is included up to 10 concurrent containers per host ; additional containers are billed on-demand

> but that's an expected consequence of installing the agent on a host

When installing for logging purposes this is not expected or desired. It wasn’t at the point that I encountered it, but I don’t believe the documentation notes this anywhere still.

Interesting, I didn't think about this! Although it can work, using logs or APM without infrastructure monitoring was not a design goal for the agent. For this use case, I'd investigate using https://vector.dev/ instead of the full agent.

Ended up using something similar.

Normally this wouldn’t be an issue and would be more cautious in production, but was using the Agent on some Dev machines to ship logs and was surprised to see Host metrics from them unintentionally along with the bill.

Reminds me of when they intentionally enabled JFR on every single java process. Roughly sextupled our ingest (and associated costs) until we got it shut back off.

A bit strange to have this feature enabled after an upgrade. The following statement is now wrong:

> New Relic services are designed to receive and process telemetry data on the performance of applications, systems, and infrastructure, which typically do not contain any personal data. Customers generally send very little additional personal data to our platform.

https://newrelic.com/blog/how-to-relic/how-demise-of-privacy...

On the topic of privacy and GDPR, I find the previous link a good summary of the arguments used by American companies that cannot respect GDPR. Microsoft or Google say the same things in a lot more words.

If you are logging Personal Data, is that even wise, wrt GDPR? Logs have a funny way of escaping, NEWR notwithstanding. I'm not a Privacy Engineer, but surely everyone understands that you don't log raw passwords. Similarly, if you are bound by GDPR, shouldn't you not log PD?

From my experience most "shops" don't disable IP address logging in their load balancers/web servers, which is personal data under the GDPR and California privacy act.

very good point!

You shouldn’t log personal data but you may do it without planning to do so. For example you may log some error messages that contain personal data.

> If you are logging Personal Data, is that even wise, wrt GDPR?

Has any company ever been driven into bankruptcy because of GDPR violations? Google, Amazon, Vodafone, Yoigo, The Municipality of Bergen, Orange Espagne, Malagatrom, the Bulgarian National Revenue Agency, Enel Energia, Deutsche Wohnen, and Facebook have been fined over and over again, but it hasn't hurt them. If you can make more money abusing people's data than you'd pay in fines, and that's only assuming you're caught, it seems like possible GDPR issues are just another cost of doing business.

I'd like to see someone take this analysis on. Is it valid? If not, why not, assuming American law?

https://web.archive.org/web/20200813235643/http://slawsonand...

> Article 3(2), a new feature of the GDPR, creates extraterritorial jurisdiction over companies that have nothing but an internet presence in the EU and offer goods or services to EU residents[1]. While the GDPR requires these companies[2] to follow its data processing rules, it leaves the question of enforcement unanswered. Regulations that cannot be enforced do little to protect the personal data of EU citizens.

> This article discusses how U.S. law affects the enforcement of Article 3(2). In reality, enforcing the GDPR on U.S. companies may be almost impossible. First, the U.S. prohibits enforcing of foreign-country fines. Thus, the EU enforcement power of fines for noncompliance is negligible. Second, enforcing the GDPR through the designated representative can be easily circumvented. Finally, a private lawsuit brought by in the EU may be impossible to enforce under U.S. law.

[snip]

> Currently, there is a hole in the GDPR wall that protects European Union personal data. Even with extraterritorial jurisdiction over U.S. companies with only an internet presence in the EU, the GDPR gives little in the way of tools to enforce it. Fines from supervisory authorities would be stopped by the prohibition on enforcing foreign fines. The company can evade enforcement through a representative simply by not designating one. Finally, private actions may be stalled on issues of personal jurisdiction. If a U.S. company completely disregards the GDPR while targeting customers in the EU, it can use the personal data of EU citizens without much fear of the consequences. While the extraterritorial jurisdiction created by Article 3(2) may have seemed like a good way to solve the problem of foreign companies who do not have a physical presence in the EU, it turns out to be practically useless.

Currently, a lot of US companies have significant financial presence in Ireland, for tax-loophole reasons. See [0] for the Microsoft example specifically.

I'm not a lawyer, and I know nothing about New Relic specifically. But the worst offenders of the spirit of GDPR really are vulnerable to EU enforcement.

[0] https://www.theguardian.com/world/2021/jun/03/microsoft-iris...

Right, I fully understand that, and it's an important fact.

I'm talking about from the perspective of a small US company with no assets outside the US.

If you don't respect the GDPR, don't have a representative/establishment in EU/EEA... I haven't heard of anything like this happening before, but an EU judge could sanction your company. And Stripe, Paddle, PayPal, or any other payment service provider would abide to.

Likelyhood of that happening? 1/1mil? Who knows. EU GDPR enforcement is only getting better, and the scenario you have asked about is sure to be in the spotlight sooner or later.

> If you don't respect the GDPR, don't have a representative/establishment in EU/EEA... I haven't heard of anything like this happening before, but an EU judge could sanction your company. And Stripe, Paddle, PayPal, or any other payment service provider would abide to.

That's interesting: Sanctioning you by going after business partners. That could be messy, and could spark laws about that here were a European court to actually try it. How much power do we want foreign courts to have over our citizens?

I'm an EU citizen so I want EU countries to have all the power over US citizens! Just joking!

These EU/EEA laws are for consumer protection. To operate in EU is way easier for digital companies than it is for other companies. But as an EU resident you expect uniform application of consumer rights, and you definitely don't want to be surprised along the way.

You have the option of not opening up your services to EU residents, and when the GDPR became effective I've seen websites that did just that. On the other hand GDPR and very similar regulation is going to become more prevalent. You have the CCPA, which is a distilled version of the GDPR, you have the Brazillian LGPD which is a derivation of the GDPR, and other countries are in the works of setting up their own.

In the end, I think that GDPR, thus far, is the most strict of them all, and if you implement universally what the GDPR requires, on a technical level (needs to be emphasized), you'll comply with any other privacy regulation.

I wonder how eager European businesses would be to implement some American law they had no hand in crafting, based on hand-wavy claims of enforceability and "eat your vegetables, it's good for you" type rhetoric.

I think the same goes the other way around. Establishing an US entity is scary in terms of privacy cause then you automatically need to comply to the CLOUD Act (which creates an entirely different legal kerfuffle in regards to GDPR compliance). And even if not established within the US, a global company is still subject to US's export laws (which applies to goods, and services, including software).

The US has a tighter track record on political/economical pressure, even in EU. All the copyright acts and regulation that almost succeed all the time in EU, and happily thus far got rejected, didn't come from within EU, but from the EU-US trade partnership.

My opinion is that it's better to just check of compliance marks on a list, and move on with ones life. Politics are **, either way.

The easiest way is then to simply not track your EU customers. That takes care of 99% of the problem.

I'm interested in enforcement mechanisms a European court would have over a purely American company that did track European customers. People around here seem to assume that Europe has enforcement powers over the entire world, which seems bizarre to me. Do we also collectively assume North Korea can enforce its laws in Silicon Valley?

America seems to believe the exact same thing. The typical way is that they pluck execs out of airport lines.

For instance:

https://en.wikipedia.org/wiki/David_Carruthers

Was a pretty high profile case like that.

You may of course disagree with that but I think that if you intend to do business in a country that you should abide by the laws of that country, even if there is no direct way to enforce them. Sooner or later you may find that nation states have pretty long reach.

This may already be implicit in your question, but that putative company has to be clearly intending to offer goods or services to EU visitors (which isn't specifically defined but might mean something like accepting payment in Euros or advertising towards EU citizens). This seems like a vanishingly small percentage of "small US companies" although I do see your point re: enforcement.

Applying the same principle to other rules would make for weird situations.

For instance, let’s imagine a US business decides to make surprise changes to its T&CO and put penalties on some random behavior, but only applicable to EU users. It would proceed to take exorbitant amounts from EU cards, and refuse chargebacks pointing at the T&CO.

Would it be completely shielded from any fines and lawsuit from EU users ?

My gut instinct would be that even if the US company doesn’t have a EU presence, its service providers do, and a gov. could request any of them to recover legal fines on behalf of the country.

Wow, we handle PHI and this is absolutely not acceptable. Glad we're not using New Relic but I'm terrified of Datadog doing this in the future.

PHI must only be collected on platforms that are certified for PHI. Collection PHI on platforms not certified for PHI is a HIPAA violation and a good way to get sued for all of your dollars x10, as well as criminal proceedings.

Fully-anonymized PHI can be analyzed on anything but the anonymization process must also be certified to have a reasonably low risk of re-identification when combined with other (arbitrary) datasets.

(source: I used to implement and certify PHI data collection platforms.)

> Collection PHI on platforms not certified for PHI is a HIPAA violation

No, it's not. (There are compliance reauirements, both direct and implicit on the HITECH definition of secured PHI, but “certified for PHI” is not one of them, despite marketing myths spread by the private HIPAA certification industry, which pedals certifications without legal meaning.)

> a good way to get sued for all of your dollars x10,

HIPAA penalties are capped at a penalty of $1.5 million per year for violations of any single provision, regardless of number of violations (and $50k per violation).

For the record, I believe extreme violations of HIPAA can result in criminal charges, or so I've read. (Accidentally sending PHI to New Relic almost certainly does not qualify for jail time.)

Most companies that deal with PHI are not covered entities, or their partners, that HIPAA restrictions apply to.

I don't see how this would happen with Datadog. Their agent already has logging capabilities but you have to explicitly enable it

I hope it becomes a legal nightmare for them and become the example of what not to do.

For New Relic's sake I hope they have thought through the consequences under the GDPR because they may well be causing their customers to violate their data processing agreements or even the law and may cause end users' data to be processed without consent. The fines for such tricks are nothing to sneeze at and companies working hard to stay compliant would do well to stay away from a supplier that does not have their best interests at heart.

New Feature: "The delete everything button -- it deletes everything! Great for GDPR requests!"

I think that's called 'close your New Relic account'.

Pretty much. They stink at doing targeted deletions. It required manual deletion of data in batches the last time we needed it.

God these fuckers are shady. They already boned us on the per user billing.

Another nail in the coffin for New Relic. And it might well be the last for me. I've used NR for close to 15 years and have been very disapointed with where they've been moving the last 5 orso.

Removal of development mode & support. The 'improved' price model. Their new UI is buggy as hell and they refuse to fix it. Every month orso you get a new interface that is objectively worse then what it is supposed to replace. Really helpful when shit is on fire and you need to figure out what is going on.

But. It is still very useful too me & finding an alternative is a time investment I haven't been able to make so far. So, for now, we'll continue to take it and deal with their bullshit as it comes. But as soon as I have some time to look for alternatives I'm done

I have a free account and I can disable the automatic log collection on this page: https://one.newrelic.com/data-management-hub just click on "log settings" in the left-hand pane. Doc page with details: https://docs.newrelic.com/docs/logs/logs-context/disable-aut...

I’m sure I will be demolished for asking this dumb Q but… I work at a medium/largish company. We have a dozen tools in addition NR. I can’t help but wonder… is all this shit necessary for services that are so small and simple?

All I do when I use NR to figure out issues or get insight is run some queries by some key name and it shows me a nice graph. I feel like we could have just collected this data into a system we build in a few days and run the analysis ourselves. As an end user I don’t feel like it’s doing anything that fancy besides the graph.

New Relic, the company that tweaked MySQL to ingest more data than anybody could imagine. Cool. But New Relic is very obsolete in 2022 and should not be used.

This sounds interesting, are you suggesting that their metric storage is based on MySQL? From all the public information that I have seen, nothing makes me believe that.

They use Vitess but I think that is limited to information like sharding keys, account configs and other metadata.

They had a blog post about it ages ago. I believe they changed to IndexDB or something else.

Why won't everyone just use fully qualified dates (2022-05-03)! For a second I thought the OP had the date in march and was confused.

FYI: you can have New Relic switch this feature off for your entire company's account(s) globally, and you can later opt-in to it. You can do this before the go-live date. So it won't affect you if you are proactive about it.

Now might be a good time to check out Dynatrace if you are a current New Relic customer. https://www.dynatrace.com/

Do they have a free tier, or are they "enterprise only?"

Or go open-source: https://github.com/SigNoz/signoz

Moved off of Dynatrace years ago, happy to go.

Appmon or the "new one"? Appmon was..a cluster fuck in the latest years for me.

Was really happy to leave it. But a few years later we tested the new Dynatrace and that shit is really shiny. Sadly, cost is expensive unless you are working in enterprice level, so cost is not an issue. But with the massive fuck up of newrelic/DD/Splunk with their skyrocket licenses.. is the cheaper option.

I was going to post "this must be a thing only in the USA, no way they are doing this to European customers due to GDPR", then I went to check my work email and bam, there it is. Gonna set the environment variables to disable this first thing in the morning tomorrow.

That's weird. They don't bill for logs per GB/line, iirc, so from a financial perspective it seems an odd choice - it will cost them lots of money to just pile on lots of logs from users/companies who didn't ask for it ( if they needed them they would have enabled the feature themselves).

How if they are planning to mine and sell data?

I don’t think the will do it.

The reason may be logs can contribute to lot of ingestion volume compared APM, RUM, Infra products. They make lot of money in this case.

Any recent changes to their data processing terms that could explain this?

Just thinking out loud, I'm personally not familiar with this service.

their current pricing works based on ingested data in GB. I suppose logs falls under that.