Stop Wasting Money on New Relic: 5 Fixes

New Relic, a powerful observability platform, promises deep insights into your application and infrastructure performance. However, I’ve witnessed countless teams, even seasoned ones, stumble when implementing and maintaining it. Getting it right can transform your operational efficiency, but missteps can lead to noisy alerts, missed critical issues, and a significant drain on your budget. So, how do you avoid these common pitfalls and truly master your New Relic deployment?

1. Over-Sampling or Under-Sampling APM Data

One of the most frequent mistakes I encounter is teams either collecting too much data, leading to exorbitant costs, or too little, rendering the insights useless. It’s a delicate balance, particularly with application performance monitoring (APM) agents.

The Fix: Configure Transaction Sampling and Custom Instrumentation Wisely

You need to be intentional about what data New Relic’s APM agent collects. By default, agents often sample a percentage of transactions to reduce overhead. For critical services, you might want a higher sample rate, but for less impactful background jobs, you can dial it back significantly.

Here’s how we typically approach this for Java applications, though the principles apply broadly:

1. Adjust Transaction Sampling: Open your newrelic.yml file (for Java agents, this is usually in the agent directory). Locate the transaction_tracer: section.
2. Set transaction_tracer.transaction_threshold: This defines the minimum response time (in seconds) for a transaction to be considered a “slow transaction” and captured by the tracer. I often set this conservatively, say to 2.0 seconds initially, then fine-tune it based on baseline performance.
3. Set transaction_tracer.enabled: Ensure this is true.
4. Configure transaction_tracer.sampling_priority_mode: Set this to "rate".
5. Define transaction_tracer.max_samples_at_rate: This is the big one. It controls how many transaction traces are captured per minute. For high-volume, non-critical endpoints, I usually advise starting with 10 or 20. For core business transactions, we might go up to 50 or 100. This isn’t a percentage, but an absolute cap.

Screenshot Description: A screenshot of a newrelic.yml file open in a code editor, highlighting the transaction_tracer section with transaction_threshold: 2.0, sampling_priority_mode: "rate", and max_samples_at_rate: 20 clearly visible.

2. Neglecting Synthetic Monitoring for User Journeys

Application uptime monitoring is foundational, but simply checking if your server responds isn’t enough. Your application can be “up” but functionally broken, leading to frustrated users and lost revenue. I’ve seen this play out too many times, especially with complex multi-step forms or e-commerce flows.

The Fix: Implement Browser and API Synthetics for Critical Flows

New Relic Synthetics allows you to simulate user interactions and monitor API endpoints from various global locations. This proactive approach catches issues before your users do.

1. Identify Critical User Journeys: Think about what a user absolutely must be able to do. For an e-commerce site, this is typically:
   • Loading the homepage
   • Searching for a product
   • Adding to cart
   • Proceeding to checkout
   • Completing a purchase

   For a SaaS application, it might be:

   • Logging in
   • Creating a new record
   • Viewing a dashboard
2. Create Browser Monitors: For complex, multi-step journeys, use Browser monitors (specifically “Scripted Browser” monitors). These allow you to write JavaScript (using the Selenium WebDriver API) to navigate pages, click elements, fill forms, and assert content.

   Example Script Snippet (for a login flow):

   
   var assert = require('assert');
   
   // Load website
   $browser.get('https://your-app.com/login').then(function(){
     // Assert title
     return $browser.waitForAndFindElement(By.id('username'), 5000);
   }).then(function(usernameField){
     // Enter username
     return usernameField.sendKeys('testuser');
   }).then(function(){
     // Find password field
     return $browser.findElement(By.id('password'));
   }).then(function(passwordField){
     // Enter password
     return passwordField.sendKeys('testpass');
   }).then(function(){
     // Click login button
     return $browser.findElement(By.id('loginButton')).click();
   }).then(function(){
     // Wait for dashboard to load
     return $browser.waitForAndFindElement(By.css('.dashboard-header'), 10000);
   }).then(function(dashboardHeader){
     // Assert successful login
     return dashboardHeader.getText().then(function(text){
       assert.ok(text.includes('Welcome'), 'Dashboard header not found or incorrect');
     });
   });
   

3. Create API Test Monitors: For checking specific API endpoints or microservices, use API Test monitors. These are faster and cheaper than browser monitors. You can make GET, POST, PUT, DELETE requests and assert response codes, headers, and body content.

   Screenshot Description: A screenshot of the New Relic Synthetics UI, showing a “Scripted Browser” monitor configuration with the JavaScript editor open, displaying the login flow script. Another screenshot shows an “API Test” monitor configured to hit a /health endpoint with assertions for a 200 status code.

4. Configure Locations and Frequency: Run these monitors from at least 3-5 geographically diverse locations (e.g., US East, EU Central, APAC Southeast) every 5-10 minutes. This gives you a true picture of global user experience.

3. Alert Fatigue and Misconfigured Notifications

Nothing saps trust in your monitoring system faster than constant, irrelevant alerts. I once worked with a team in Atlanta whose Slack channels were a perpetual waterfall of New Relic notifications, masking genuine issues and leading everyone to ignore them. When a real outage hit, it took far too long to notice because of the noise.

The Fix: Thoughtful Alert Policies, Incident Preferences, and Muted Conditions

New Relic’s alerting system is powerful, but it requires careful configuration.

1. Define Clear Alert Policies: Instead of one giant policy, create policies for different levels of criticality or different teams. For example, a “Critical Production” policy might page SREs, while a “Non-Critical Dev” policy might just send a Slack notification.
2. Set Incident Preferences: This is absolutely critical. Navigate to Alerts & AI > Policies. When creating or editing a policy, under “Incident preferences”, choose “By policy”. This ensures that if multiple conditions within the same policy are violated for the same entity, it creates a single incident. Choosing “By condition” or “By condition and entity” leads to a deluge of individual incidents for related problems.

Screenshot Description: A screenshot of the New Relic Alerts & AI interface, specifically the “Incident preferences” section within an alert policy, showing “By policy” selected and highlighted.

3. Configure Notification Channels: Link your policies to the appropriate channels. For critical alerts, PagerDuty or Opsgenie integrations are essential for on-call rotation. For less urgent issues, Slack or Microsoft Teams integrations work well.
4. Use Baselines for Dynamic Thresholds: For metrics like CPU utilization or response time, fixed thresholds can be problematic. A 70% CPU might be normal during peak hours but an anomaly at 3 AM. New Relic allows you to use baseline alerts, which dynamically adjust thresholds based on historical patterns.

   Example Baseline Alert (NRQL):

   
   FROM Metric SELECT average(apm.service.response.time) WHERE appName = 'MyWebApp'
   

   Then, when setting the threshold, select “Baseline” and configure your desired “Standard Deviations” (e.g., 3 standard deviations above the baseline for a critical alert).

5. Mute Conditions for Maintenance: When you know you’ll be deploying or performing maintenance, proactively mute the relevant alert conditions. This prevents false positives and maintains trust in the system.

4. Ignoring Infrastructure Monitoring Integration

Many teams treat APM and infrastructure monitoring as separate entities. They’ll have New Relic APM for their application and perhaps AWS CloudWatch or a different tool for their EC2 instances or Kubernetes clusters. This siloed view makes root cause analysis a nightmare. Is the application slow because of a code issue, or is the underlying database server experiencing I/O bottlenecks?

The Fix: Unify Your Observability Stack

New Relic is designed to be an end-to-end observability platform. You should be leveraging its infrastructure agents and integrations.

1. Install Infrastructure Agents: Deploy the New Relic Infrastructure agent on your hosts (EC2 instances, VMs). This agent collects critical metrics like CPU, memory, disk I/O, network traffic, and process-level data. It automatically links to your APM data, allowing you to see the host’s health alongside your application’s performance.

Screenshot Description: A screenshot of the New Relic UI showing a service map for an application, with interconnected nodes representing services and underlying infrastructure elements (like a specific EC2 instance or Kubernetes pod) clearly visible and linked, displaying health status.

2. Integrate Cloud Platforms: For public cloud users (AWS, Azure, GCP), configure the respective New Relic integrations. This pulls in metrics from services like S3, RDS, Lambda, Azure App Services, and Google Cloud Functions directly into New Relic.

   For example, for AWS, go to Infrastructure > AWS > Add an AWS account and follow the prompts to grant read-only access via IAM roles. This allows New Relic to pull CloudWatch metrics, configuration data, and more.

3. Kubernetes Integration: If you’re on Kubernetes, the New Relic Kubernetes integration is non-negotiable. It provides visibility into pods, deployments, nodes, and namespaces, linking container performance to your application and host metrics. I often advise clients in the Georgia Technology Center to deploy the New Relic Kubernetes operator via Helm charts, which automates the deployment of agents and integrations.

5. Failing to Regularly Prune and Review

New Relic isn’t a “set it and forget it” tool. Over time, applications evolve, services are deprecated, and monitoring needs change. Failing to keep your New Relic configuration clean leads to clutter, irrelevant data, and wasted spend.

The Fix: Schedule Quarterly Review Cycles for Dashboards, Alerts, and Agents

Treat your observability platform like any other critical system – it needs maintenance.

1. Dashboard Audit: Every quarter, review your dashboards.
   • Are all the widgets still relevant?
   • Are there duplicate dashboards?
   • Are dashboards linked to services that no longer exist?
   • Is anyone actually looking at this dashboard? (New Relic Insights can sometimes show dashboard view counts.)

   Delete or archive unused dashboards. Less clutter means faster insights.

2. Alert Condition Review: Go through each alert policy and condition.
   • Is the threshold still appropriate?
   • Is the notification channel correct?
   • Are there alerts for services that have been decommissioned?
   • Are there “flapping” alerts that trigger constantly without a real issue? Adjust thresholds or mute them if they’re not actionable.

   We ran into this exact issue at my previous firm, where an alert on a legacy service continued to page our on-call team for months after the service was sunset, costing us valuable sleep and attention.

3. Agent Cleanup: Check your APM and Infrastructure agents regularly.
   • Are there agents reporting for applications that have been shut down?
   • Are there duplicate agents for the same application?
   • Are all agents on the latest stable version? (Outdated agents can miss new features or have performance issues.)

   Remove or update these agents. This directly impacts your data ingestion costs.

4. Cost Analysis: Use New Relic’s own cost management tools. Navigate to Account > Usage. This dashboard shows you your data ingestion by type (APM, Infrastructure, Logs, Synthetics) and entity. Identify the biggest data consumers and investigate if that level of granularity is truly necessary. Sometimes, simply adjusting APM sampling (as discussed in Step 1) or reducing log verbosity can lead to significant savings.

   Screenshot Description: A screenshot of the New Relic “Usage” page, showing a breakdown of data ingestion by product category (APM, Infrastructure, Logs) and the associated costs over a monthly period, with options to filter by account or entity.

Mastering New Relic isn’t about avoiding every single minor hiccup, but about sidestepping the major pitfalls that lead to wasted resources and missed opportunities. By focusing on smart data collection, proactive monitoring, intelligent alerting, unified observability, and continuous refinement, you can transform New Relic from a cost center into an indispensable operational asset. For more strategies to boost tech performance, explore other articles on our site. Also, if you’re looking to slash cloud costs, FinOps practices can complement your New Relic optimization efforts. Finally, understanding the cost of unreliable tech further emphasizes the importance of a well-maintained monitoring solution.