Building a Server-Side Event Tracking  Application

The percentage of web users browsing with built-in browser tracking protection enabled is rapidly growing, and as browsers begin to ship with these features enable by default, like Apple’s ITP, the client-side attribution techniques that digital advertisers have relied on for years will become progressively less effective.

I do some work with a software company who’s primary audience is younger tech-savvy users. Based on what we’ve tracked over the past 6 months, over 30% of their audience has tracking protection enabled, and an even larger percentage are on mobile devices. In a scenario like this, server-side tracking is the only accurate and effective approach to connect paid ad hits to web conversions attribution, but the options available aren’t so great.

Server-side Event Tracking Software

• Segment can do this well, but it’s usually very expensive for companies with a decent amount of volume and there are certain circumstances where you’ll end up coding your own cloud functions to get what you need
• Google Tag Manager server-side tagging provides a close to free alternative, but it’s difficult to set up, has a lot of limitations and I have a feeling it will be detected and blocked by tracking protection in the near future
• mParticle is a good alternative to Segment, but it has the same drawbacks
• ClickHouse is a great option for high volume data, but it only handles the collection of events.

Options for Server-side Event Tracking

After reviewing the lackluster options, we decided to build our own system using the Laravel web application framework. This has many distinct advantages:

• No limitations to what we can do: we can support every advertiser that can receive server side events
• Stay under the radar with a less detectable tracking patterns
• Ability to rapidly adapt with the changing landscape

Custom Server-side Event Tracking Web Application

While we did build our own web application to do this, we didn’t do it from scratch. Laravel provides many built-in capabilities that were ideal for us to work with, including an API ORM and API routing.

From a business perspective, the web application is handling the following for us:

1. Marketing websites track pageviews and events by making an API request to a subdomain
2. When the API’s receive an event or pageview it makes outgoing API requests to advertising platforms like TikTok, Facebook and Google Ads

To meet the strict rules of various tracking protection mechanisms, we’re using a server-side 1st-party cookie to track the user session. This cookie stores the click ID and other details about the first visit, and creates an anonymous identifier that persists in that user’s browser.

Cross-device Attribution Tracking

On the marketing website, a web based application process collects and verifies their phone number. The phone number is verified using an OTP text message, where they’re provided with a unique code they have to enter to confirm it is in fact their phone. Once that phone number is verified, a unique user_id is provided to that user, and we can associate that device with the individual anonymously.

This approach allows us to handle cross device attribution seamlessly

If that person starts the application process on their laptop but doesn’t finish, they can continue later on their phone by re-verifying their phone again. This allows us to accurately identify the same person from their laptop and their phone.

Technical Details

To handle all this fancy stuff for a reasonably large volume of users, we’ve used the following technical approach. Under the hood it’s not all that complex, though we do go beyond what’s mentioned here to provide additional models for reporting and audience generation. These are the exact types of things that are difficult to do with off the shelf systems.

This system is effectively a micro, DIY OLAP (online analytical processing) application. We use it to collect and store data from multiple data sources, including websites, applications and internal systems.

Incoming API Requests from Marketing Website(s)

How it works

1. API routes receive incoming POST requests with JSON payloads.
2. Payloads are validated with custom rules and return appropriate HTTP responses.
3. Valid incoming requests create a new Event model, storing the request, response, event name and user details into a MySQL database table with JSON type columns.

Outgoing API Requests to Advertising Platforms

How it works

1. A job is pushed into a Redis powered queued for background processing by the server. This job contains methods for each advertising platform that we’ll make an API request to.
2. The server processes jobs from the queue, creating a new Track model record for each outgoing API request made. Each record contains an incoming_id column that creates a many-to-one relationship with the incoming event that triggered it.

Cloud Hosting

We’re hosting the application with Vapor using AWS resources under the hood. This allows us to comply with strict security protocols by keeping the application in a private cloud. The URL used is a subdomain of the main corporate website where paid advertising is driven to, allowing us to use a 1st-party cookie on both the API server and the marketing website receiving paid ad hits.

Benefits

The things that make this worth the effort.

Performance

This custom application is actively running as we speak, and based on what we’ve seen so far, it’s an absolute beast. It has met all of our needs for tracking server-side conversion events and pageviews reliably, and has also been configured to handle custom audience generation and reporting too.

The redis queue handles spikes in volume really well, processing items one by one no matter how many are received. The cloud host has been configured with auto-scaling, and it will scale up automatically to handle traffic bursts, and scale down when volumes are low.

Smart Errors & Replays

We’ve configured CloudWatch logging to notify us when exceptions or errors do occur, which allows us to react quickly to solve them.

Errors are bound to happen, especially in a system that connects to outside services which can change without notice.

Most of these errors will occur with the outgoing API requests, as those are the aspects that will change most frequently without us knowing (thanks Facebook). By using the queue we’re able to record these errors and all associated data that was sent in to the job that triggered it. When we do fix the issue we can replay those events that previously failed.

Most if not all server-side side event tracking API’s typically receive an event datetime that represents when it occurred. We use this pattern to our advantage, and when the events are replayed the datetime of the error is used to retroactively backfill the requests.

The ability to replay outgoing requests that previously error’d out provides us with a way to track every single valid event we’ve received, no matter what happens.

Fast Adaptation to Changing Landscape

Limitations and server-side tracking APIs are changing all the time, and this system gives us the ability to quickly change with them. We can move much faster than if we were dependent on a third-party software provider.

To keep up with these rapid changes I highly recommend Cookie Status. It’s a great resource for understanding the specific technical limitations imposed by various tracking protection mechanisms implemented by the major browsers and browser engines.

Authors Note

If you’re looking for a dependable analytics consultant and/or web app developer to build something similar for you, please let me know I’m available for hire. I have a lot of real-world experience building systems like this, which improve the ROI of digital advertising spend by reducing customer acquisition costs.