Stop Wasting New Relic: Avoid These Costly Blunders

Even the most powerful observability platforms, like New Relic, can become liabilities if not used correctly. I’ve seen organizations, large and small, pour significant resources into this technology only to fall short of its promise due to a handful of common, yet entirely avoidable, missteps. We’re going to dissect these errors, reveal their impact, and arm you with the knowledge to truly master your New Relic implementation.

Ignoring Distributed Tracing: The Blind Spot in Microservices

This is, without a doubt, the single biggest mistake I see companies make with New Relic. They’ll deploy agents, get basic APM data flowing, and think they’re covered. But if you’re running a microservices architecture – and let’s be honest, who isn’t these days? – then ignoring distributed tracing is like trying to diagnose a complex electrical problem in a skyscraper by only looking at the lights on one floor. You simply won’t see the full picture.

Distributed tracing, specifically with New Relic Distributed Tracing, stitches together the journey of a single request as it hops between multiple services, databases, message queues, and external APIs. Without it, when a user reports a slow transaction, you might see that Service A is slow, but you won’t know if Service A is slow because it’s waiting on Service B, which is waiting on a database query, or if it’s an internal processing issue. The critical context is lost. I had a client last year, a fintech startup based out of the Atlanta Tech Village, struggling with intermittent transaction timeouts. Their APM showed high response times on their payment processing service. We dug in, enabled full distributed tracing, and immediately saw that the bottleneck wasn’t their payment service itself, but a third-party fraud detection API call that was occasionally spiking to 5-10 seconds. Their internal service was just waiting patiently. Without tracing, they would have spent weeks optimizing the wrong code.

The impact of this oversight is profound. Mean Time To Resolution (MTTR) skyrockets. Developers waste hours, sometimes days, sifting through logs across multiple systems, trying to manually correlate timestamps. This isn’t just inefficient; it’s demoralizing. New Relic’s tracing capabilities, especially with the W3C Trace Context standard, are incredibly powerful. They allow you to visualize these complex interactions, pinpointing exactly where latency is introduced or errors originate. My advice? Make distributed tracing a non-negotiable part of your New Relic rollout, especially for any service that interacts with more than one other component.

Neglecting Custom Instrumentation: Missing the Business Context

Another prevalent error is relying solely on out-of-the-box APM agents. While these agents are fantastic for capturing standard metrics like response time, throughput, and error rates, they often fall short when it comes to understanding the unique business logic of your application. This is where custom instrumentation becomes invaluable. It’s about telling New Relic, “Hey, this specific function call? This database query? This external API integration? That’s really important to my business, and I need to track its performance specifically.”

Think about an e-commerce application. The default agent will tell you the overall response time for the checkout process. But what about the time it takes to validate a coupon code? Or to calculate shipping costs? Or to update inventory after a purchase? These are discrete, critical business steps that, if slow, can directly impact conversion rates. If you’re not explicitly instrumenting these, you’re flying blind on key performance indicators that directly affect revenue. We ran into this exact issue at my previous firm. Our marketing team was complaining about low conversion rates on a specific product page. Standard APM showed the page was fast. Only after we added custom instrumentation to track the pricing engine’s performance did we discover a legacy pricing rule that was adding an extra 800ms for certain product configurations. It was a single, tiny piece of code, but it was costing them sales.

New Relic offers several ways to achieve custom instrumentation:

• Custom Metrics: Using the agent APIs (e.g., recordMetric in Java, record_metric in Python) to report arbitrary numerical values. This is fantastic for tracking things like the number of times a specific feature is used, or the duration of an internal processing step.
• Custom Attributes: Adding extra metadata to transactions or errors. Imagine tagging a transaction with the customer_tier or product_category. This allows for incredibly powerful filtering and analysis in New Relic One.
• Custom Events: For when you need to track discrete occurrences that don’t fit neatly into metrics or attributes. Think about a “UserLoginFailed” event with associated attributes like username and error_code.
• Custom Tracing: Manually creating spans within your code for specific operations that the automatic agent might miss. This works particularly well for asynchronous operations or custom frameworks.

My strong opinion? Every critical business transaction, every third-party API call, and every complex internal computation deserves some level of custom instrumentation. It’s not just about finding errors; it’s about understanding the health of your business processes. Don’t be lazy here; the payoff in actionable insights is enormous.

Alerting Anti-Patterns: The Boy Who Cried Wolf Syndrome

Ah, alerts. The double-edged sword of observability. Too few, and you miss critical outages. Too many, and your on-call engineers become desensitized, leading to alert fatigue and potentially ignored real problems. I’ve seen teams with hundreds of alerts, most of which were informational or low-priority, drowning out the few that truly mattered. This is a common pitfall with New Relic, primarily because it’s so easy to set up basic threshold alerts.

The biggest mistake here is failing to adopt a structured approach to alerting. Simply setting an alert for “CPU usage > 80%” or “Error rate > 5%” across every service is often counterproductive. What’s high CPU for a batch processing service might be normal for a real-time API. What’s a critical error rate for a payment gateway might be acceptable for a non-essential background job.

Instead, I advocate for the Golden Signals approach, popularized by Google’s Site Reliability Engineering (SRE) principles. These four signals are universally applicable to almost any service:

1. Latency: The time it takes to serve a request. Alert on significant increases in average or, more importantly, p99 (99th percentile) latency.
2. Traffic: How much demand is being placed on your system. This could be requests per second, active users, or network I/O. Alert on sudden drops (indicating an outage) or sustained, unexpected spikes (indicating a potential overload).
3. Errors: The rate of requests that fail. This includes explicit errors (HTTP 5xx, exceptions) and implicit errors (e.g., incorrect data returned). Alert on any significant increase.
4. Saturation: How “full” your service is. This is often tied to resource utilization (CPU, memory, disk I/O, network bandwidth). Alert when resources approach critical capacity.

By focusing your primary alerts on these four signals, you cover the vast majority of critical issues. You can then layer on more specific, lower-priority alerts for things like database connection pools or specific business metrics, but these should typically feed into a separate notification channel or be reviewed less urgently. New Relic Alerts provides robust capabilities for defining these conditions, including baseline alerting that adapts to normal behavior, reducing false positives. Don’t just set static thresholds; use baselines where appropriate. It’s a game-changer for reducing noise.

Case Study: The Over-Alerted Retailer

We recently worked with a large retailer in Buckhead, Atlanta, whose e-commerce platform was perpetually “on fire” according to their monitoring. Their on-call engineers were constantly paged for non-critical issues. Their New Relic alert policy had over 200 individual alert conditions, many of which were default settings or copied from other services without context. For example, they had a CPU alert set at 75% for both their front-end web servers and their internal reporting service. The reporting service frequently hit 90%+ CPU during off-peak hours for legitimate data processing, generating constant, unactionable alerts.

Our approach:

1. Audit and Deactivate: We reviewed every alert. Over 150 were either duplicates, misconfigured, or monitoring non-critical metrics. We deactivated them.
2. Implement Golden Signals: We created new alert policies focused on latency (p99), error rates, throughput, and saturation (CPU, memory) for their critical user-facing services. We used New Relic’s baseline alerting for latency and traffic, allowing the system to learn normal patterns.
3. Tiered Notifications: Critical Golden Signal alerts went to PagerDuty. Less critical, but still important, alerts (e.g., specific database connection pool warnings) went to a dedicated Slack channel. Informational alerts were logged.
4. Custom Metric Alerts: For their specific business logic (e.g., “failed payment attempts per minute”), we created custom metrics and set alerts based on historical business impact, not just arbitrary thresholds.

The outcome? Within three months, their weekly PagerDuty incidents dropped by 65%. Engineers reported significantly reduced alert fatigue and a clearer understanding of what constituted a true emergency. Their MTTR for critical issues improved by 25% because the signal-to-noise ratio was finally manageable. This wasn’t just about New Relic; it was about rethinking their entire incident response strategy through the lens of effective observability.

Ignoring Data Retention and Cost Management

New Relic is incredibly powerful, but that power comes with a cost. And one of the biggest mistakes I see organizations make is treating data retention and cost management as an afterthought. They just let everything flow in, year after year, without understanding the implications. This can lead to significant, unnecessary expenses and, ironically, can even make your data harder to analyze effectively.

New Relic offers various data retention periods depending on your subscription level and the type of data (APM, Infrastructure, Logs, Metrics, Traces, etc.). For instance, standard APM metric data might be retained for 90 days, while custom events might have a different default. Many teams simply accept the defaults without question. The problem? Not all data is created equal. Do you really need five years of minute-by-minute CPU utilization for your development environment? Probably not. But do you need detailed transaction traces for your critical production services for at least a few weeks to debug issues? Absolutely.

Here’s what nobody tells you: unmanaged data ingestion is a silent budget killer. I’ve seen companies surprised by their New Relic bill simply because they never bothered to configure data sampling or retention policies. It’s not just the sheer volume of data; it’s the specific types of data. Logs, for example, can be extremely verbose. If you’re ingesting every debug log from every service into New Relic Logs, you’re going to pay for it. A more strategic approach involves:

• Sampling: For high-volume data like transaction traces or log lines, consider intelligent sampling. New Relic allows you to configure agents to sample traces based on various criteria, ensuring you capture enough data for analysis without overwhelming your account.
• Granular Retention Policies: Understand what data needs to be kept for compliance, auditing, or long-term trend analysis, and what can be discarded after a shorter period. Work with your New Relic account team to understand your options for custom retention.
• Excluding Non-Essential Data: Review your agent configurations. Are you sending data from non-production environments that you don’t need to retain long-term? Are there specific log levels (e.g., DEBUG) that can be filtered out at the source or during ingestion?
• Tagging for Cost Allocation: Use New Relic tags to categorize your data by team, environment, project, or cost center. This allows you to understand where your observability spend is going and justify it.

My recommendation is to conduct a quarterly review of your New Relic data ingestion and retention strategy. It’s not a one-time setup; it needs to evolve with your application landscape. This proactive management can save you tens of thousands of dollars annually and ensure you’re getting the most value from your investment.

Lack of Consistent Naming and Tagging Conventions

This might seem like a minor organizational detail, but believe me, it’s a monumental headache if ignored. When you have dozens, hundreds, or even thousands of services and hosts reporting to New Relic, a lack of consistent naming conventions and tagging will quickly turn your powerful observability platform into an unmanageable mess. You won’t be able to find anything, filter effectively, or create meaningful dashboards and alerts.

Imagine trying to find all services associated with your “Customer Portal” application, running in your “Production” environment, managed by the “Team Phoenix” engineering group. If your services are named inconsistently (e.g., “customer-portal-prod-web,” “CustPortal_Prod_App,” “web_customer_prod”), and you haven’t applied tags for environment or team, you’re in for a painful manual search. This cripples your ability to perform efficient cross-service analysis, understand dependencies, or even assign ownership for incidents. It’s particularly frustrating when trying to compare performance metrics across different versions of the same service, or across different deployment regions (e.g., US-East vs. EU-West).

Here’s how to avoid this:

1. Standardized Naming: Enforce a clear, documented naming convention for all applications, services, and hosts. Something like [ApplicationName]-[ServiceType]-[Environment]-[Region] (e.g., OrderService-API-Prod-USEast). Consistency is key.
2. Mandatory Tagging: Leverage New Relic tags religiously. Define a set of mandatory tags for every entity reporting data:
   • environment (e.g., prod, staging, dev)
   • team (e.g., Team Phoenix, Billing Squad)
   • application (the logical business application it belongs to)
   • owner (the individual or group responsible)
   • cost_center (for financial tracking)

   These tags can be applied automatically via agent configuration, infrastructure automation tools (like Terraform), or directly through the New Relic UI/API.

3. Automate Where Possible: Integrate tagging into your CI/CD pipelines and infrastructure-as-code definitions. If a new service is deployed, ensure it’s automatically provisioned with the correct New Relic agent configuration and mandatory tags. This prevents human error and ensures compliance.
4. Regular Audits: Periodically review your New Relic inventory to identify untagged or inconsistently named entities. I recommend a monthly audit, especially in dynamic cloud environments where new resources pop up frequently.

A well-tagged and consistently named New Relic environment is a joy to work with. It allows for effortless filtering in New Relic One dashboards, targeted alerting, and rapid incident response. Without it, you’re essentially looking for a needle in a haystack, and that’s a losing proposition in the fast-paced world of modern technology.

Mastering New Relic isn’t just about deploying agents; it’s about strategic implementation, continuous refinement, and a deep understanding of your own architecture and business needs. Avoid these common missteps, and you’ll transform your observability platform from a mere data collector into a powerful engine for operational excellence and informed decision-making.