The Inception of a Wrapper, Neutrality and a Look Ahead

By the summer of 2015, Index Exchange (along with our peers in the marketplace) was actively, albeit manually, integrating header bidding into publisher sites at great scale while realizing the advantages it could bring to the industry at large. While header bidding has since delivered on the promise of greater transparency, competition and more revenue for media companies, our publishers came to us at this time to expose a pain point: arduous implementation. After talking with several of our media company partners, it became apparent that while the promise of this new technology was bright, someone needed to solve the engineering challenge that all publishers otherwise exclusively bore, integrating partners into the header. Thus, the publishing community itself (not ad tech) gave birth to the Header Tag Wrapper.

Origin

The Wrapper came about as a result of real publisher needs. Header bidding was changing their business in an unexpected but exciting way, driving higher CPMs and programmatic revenues. However, header bidder implementations were also taking up precious engineering and ad ops time within each publisher who had adopted. To Index, the solution was obvious – take these taxing integrations off of our publishers’ plate and have Index productize the problem and bear the cost of making adoption seamless. And even more importantly, allow publishers the benefit of additional demand competing in a parallel auction with low operational complexity. By reducing the barriers to entry and the friction to integrate, we saw the potential for a great acceleration in the adoption of header technologies.

As a product-focused company and with two thirds of our total headcount residing in our engineering department, we looked no further than our group of dedicated and responsible engineers to name this new offering (and build it too!). Because the earliest incarnation essentially and quite basically meant bringing different blocks of code (external header bidding libraries) into the publishers’ environment and wrapping them with latency control, our engineers thought of a simple, yet practical name for this new product, the Wrapper.  

We quickly found the most difficult challenge would be gaining the trust and support of our peers in the header, many of whom traditionally would have been viewed as competitive to Index. The entire effort was underpinned by our founding principles of transparency, neutrality and collaboration – no matter the cost. We knew that by working more closely with all bidders, not just our own, we could deliver more benefit for our publishers. So after many meetings, calls, drinks, lunches, and more calls, we struck understandings, deals, and integrations with publishers, and many of the top sources of appreciable demand. The Wrapper was a go.

Development and Fees

The question we face everyday is how can we continue to ensure we are providing publishers with the most value possible without them having to think about creeping take rates, bias, or obfuscation which has unfortunately become the ad tech norm.

In the age of the header, we now wear two hats. As a participant in the Wrapper where Index is a bidder (we also participate in other wrappers too, such as prebid.js), the first step has been to explicitly disclose our fee structures to our publishers. Our rates are set, and we can guarantee every media dollar that comes into our exchange goes to the publisher outside of one transparent fee. We also believe all bid data that we process belongs to our publishers, and make raw transactional logs available to our clients.

When providing the Wrapper solution itself, we elected to make this offering complimentary, absent of any fees. We also decided to have the Wrapper live in the browser, rather than server-side, so that every piece of code that defines it and drives it can be inspected by anyone (“View Source” in your web browser is all you need to inspect 100% of the Wrapper code base). It’s one thing to claim a product to be transparent and fair, it’s another thing to subject it to a perpetual audit by virtue of its inherent inspectability. In addition, we opt to host the Wrapper code on infrastructure external to Index – specifically on Akamai’s giant edge network, ensuring high speed availability, and also external activation from Index infrastructure. When the Wrapper, hosted on Akamai, calls the Index Bidder, it’s from the same distance to Index as it is to any other bidder.

The Wrapper has quickly developed the reputation as being a ruthless Enforcer. It requires each participant to conform to the transparent, neutral and collaborative marketplace publishers have set out to build. As the Enforcer, the Wrapper can’t have allegiances and must treat all bidders equally. Unfortunately the marketplace continues to be plagued by alarming disparities among header bidding participants. These include: ad library size (a client side browser bandwidth “cost” especially impactful in mobile, bloated libraries slow down pages), infrastructure (global data center footprints or the lack thereof), speed (bidding fast is easier said than done), and architecture (something we’ll cover extensively below). Latency issues which were quietly hidden before the Wrapper was playing the role of the Enforcer, were quickly coming to the forefront. It’s one thing to agree to the rules, it’s another thing to be bound by them.

Publishers began exercising control over timeouts, resulting in better user experience, and a better operational header strategy, but doing so has presented challenges for certain bidders. There have been suggestions the Wrapper, a code base that is 100% inspectable, is in some way favoring the Index bidder, given the speed with which we’re capable of responding. This is not what is happening. In fact, the Wrapper is doing exactly what publishers designed it to do – it is enforcing their decisions. However, not every platform is built for success in this new paradigm. Not surprisingly I suppose, in an environment where re-architecting one’s platform is viewed as a monumental effort, it’s a lot easier to blame the Wrapper.

Since the beginning, we have referred to the Wrapper as a publisher directed product. The idea was invented by publishers. Its feature set continues to grow by the day through great feedback and collaboration. Most of the features in the Wrapper are designed around user experience, specifically ensuring that the ad delivery process is impacted as little as possible by the process of fetching demand. Publishers want better user experiences, not jarring ones.

In an effort to address any claims of impropriety surrounding the Wrapper, its function or operation, we have created a table of Wrapper elements that have implications on bidders. In fact, these are requirements that have been brought forth by publishers over the last year and a half. We follow these best practices, as part of the architecture of the Index Bidder, and we welcome all external bidders to gain any perceived advantage by adopting them as well.

If a bidder times out, or misses out on an ad slot, it is entirely because of the participant’s bidder. That participant has a choice: they can ask the publisher to relax the Enforcement (increase timeouts, remove functionality like prefetching, randomize bidders or sequence them a certain way, etc.), or they can conform to the requirements of the publisher. Our hope is that over time, and by being vocal, and communicative, we can encourage all bidders to accelerate the development, investment, and maturity of their technologies. This is to ensure that every one of their bids makes it to the publisher, no revenue is ever lost, and participants maintain unfettered confidence in the integrity of the Wrapper.

Please find the detailed Wrapper Elements table below.

Looking Ahead

The Wrapper is not a static product, and it cannot be built without partners. In order for header bidding to be sustainable and to continue to drive control into the hands of publishers, we need to work together to ensure every participating bidder matches the speed of publisher evolution. Beyond the changing needs of publishers, we need to go one step further and meet the expectations of the consumer, who in the age of ad blocking is expecting a better, faster, and more seamless experience. We will continue to work tirelessly to improve both our Bidder and the Header Tag Wrapper to maintain the privilege we have received in working directly with the esteemed group of media companies who have opted to use our technology.

Just as many routinely change cell phones on an annual basis, we cannot assume the Wrapper will be a static codebase. It is going to evolve with the needs of tomorrow, and Bidders have to continue to keep pace.

Over the past year, next to our publishers’ partners, our partnerships with our header bidding peers have been instrumental to the success of the Wrapper. They hold us to the highest standard of neutrality and transparency, and our engineers work to live up to that everyday with near constant improvements. We have also certified many bidders, and consulted with our peers on architectural improvements to their platforms which will improve the overall efficiency for all. We are incredibly proud to have implemented the Wrapper across some of the largest global media companies in the world, who run up and down the comScore 100. And that list continues to grow.

We are excited to continue to lead this evolution and extend a helping hand to any integrated partner.

Wrapper element

Definition

How it helps (or does not help) publishers

What advantage (or disadvantage) could it give a bidder

Why don’t all bidders support it?

Bidder ordering (static order) All bidders are called asynchronously at the same time by the wrapper. But, due to a browser’s internal queue, each bidder will actually be called in a static order milliseconds after each other, and not at precisely the same time. Publishers can choose what order bidders are called in, based on their own preferences. The impact from this is very minimal as the delay between calls is generally 1-2 milliseconds or less, and typically the wrapper timeout is between 500-1000ms. N/A
Bidder ordering (randomization) The order of bidders is randomized rather than a static order. Rather than choosing the bidder order, Publishers leave the decision to randomization, which prevents any preference. Same as above N/A
Latency controls & measurement All bidders are allowed the same publisher controlled time period to complete their responses before the wrapper calls the ad server. All bidders are measured using the same method from when they are called by the wrapper to when they respond. Publishers can control the trade off between latency, revenue and user experience by reducing, increasingly and ultimately experimenting with latency. Consistent latency measurement ensures that all bidders are treated equally. Bidders that have invested in low latent, rapid response times will be able to participate even if a publisher reduces latency to improve user experience. Having a timeout system in place allows the wrapper to gracefully handle potential race conditions while coordinating asynchronous events increasing participation from all bidders in the final signal to the ad server. Building a lightning fast, lightweight bidder takes a large engineering investment and lots of experience. A well engineered bidder also requires a well scaled, global data center footprint which requires considerable infrastructure investments.
Light weight bidder libraries Bidders vary a great deal in size and some require the loading of large JavaScript libraries. The larger the library the more code that a user has to load into their browser which can hurt the user experience especially on mobile due bandwidth limitations. A lightweight bidder library minimizes how much data is consumed by the end user in order to activate the bidder. Reducing the overall data load for a page improves the user experience and improves total page speed. Some bidders ask publishers to preload a library outside of the wrapper as the time required to load their library can lead to their bidder not being able to meet the publisher specified latency timeout. Moving a library outside of the wrapper creates additional work for the publisher’s engineering team and introduces an unknown amount of latency. It can often be easier to write a lot of code than to optimize a library down to a small amount of elegant code. In some cases bidders were developed over time instead of through a single concerted development effort. With investments and time, library sizes will continue to shrink.
Prefetch Making bid requests to a header bidding API prior to knowing how many ad slots are on a given web page, or will be on a given web page, through scrolling by an end user. Initiating bid requests as early as possible means more bids make it back into the wrapper before the publisher defined timeout, which increases publisher revenue. Additionally, this allows responsive ads to render immediately when they are scrolled into view, rather than stopping ad load when a new ad is scrolled into view to wait for the wrapper to initiate a new auction. Bidder’s able to support prefetch will timeout less, compete with other demand sources more, and ultimately participate in rendering ads faster for publishers and the end user. Prefetch requires significant technical infrastructure, engineering and ensuing costs to support. This is because the early initiation from the header will be inclusive of all potential ad slots a page may render, regardless of whether the end user actually scrolls every slot into view (meaning infrastructure cost may be incurred for ad slots that never come into view).
Single request architecture (SRA) The bidder makes one single network call through the wrapper that spans N number of ad slots, and sends multiple bids back to the wrapper. If there are 8 slots on a page, there will only ever be 1 network call. Network calls can consume resources within a browser’s queue that can block other critical elements on a page from loading. As such, publishers prefer to keep the number of network calls down, which this dramatically helps with. By only initiating one single request, the bidder is more likely to be prioritized by the browser’s internal queue of work, ensuring the bidder participates in all slots available on any given page. It can be an engineering intensive endeavor to move a bidder from MRA to SRA. The SRA architecture also requires more infrastructure competency as each request is more expensive to process than a single MRA request.
Multi request architecture (MRA) The bidder makes a network call through the wrapper for each ad slot on the page. If there are 8 slots on a page, there will be 8 network calls. Publisher pages generally have multiple slots, and with multi request architecture, each slot is a network call, which can create congestion within the web browser. And if multiple bidders use MRA, this negative effect multiplies dramatically. Based on the way a browser prioritizes work and the speed of the MRA bidder, there may be instances where the bidder is not eligible for every slot. Chrome for instance only processes 6 TCP requests at any given time per domain. This is a common bidder architecture originating from earlier iterations of header bidding architecture.
Mediation The wrapper effectively runs an auction, selects one winner, and passes that one bid into the ad server. Publishers require less line items to manage all of their price competition based header bidding within their ad server, but this comes at the expense of not having a full record of all participants bids (win or lose) within the ad server. The main disadvantage is a bidder that wants to work with a publisher to prioritize its bids for reasons other than price (ex: guaranteed budgeted deals, pacing issues) must have custom JavaScript implemented in the Wrapper. N/A
Non-Mediation The wrapper acts as a gateway to the ad server, pushing all bids that arrive within the publisher defined timeout directly to the ad server. Publishers can control the winning bid selection using the ad server, a tool they are familiar with. There is also generally highly granular reporting available in the ad server since all the bids are passed into it. All bids are treated the same by the wrapper, and the ad server makes the ultimate decision. Simplifies reporting and optimization for bidders through a third party ad server driving decisions. By passing all key/values into the ad server each bidder can reconcile precisely what deal/bid information was sent as opposed to relying on an intermediation layer. N/A

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s