Building Scalable Recommendation Systems at Flippa

Building truly scalable and practical recommendation systems has long been a challenge in modern data science and it wasn’t until the introduction of embedding driven neural networks that these problems actually became tractable. However, scale is just part of the problem.

At Flippa, using cutting edge machine-learning, we have developed a system that matches our buyers with quality listings picked specifically for them by leveraging the latent(or hidden) intent of their actions and behaviour which informs their preference.

The Single-Purchase-Single-User Problem

Flippa boasts a whopping 2M+ active buyers and trying to learn each individual preference becomes especially challenging when considering that once a buyer and listing are successfully matched, (usually) they are both removed from the pool.

Most traditional methodologies assume that either:

a) all users will consume multiple products during their lifecycle

or

b) the same product is consumed by multiple users

The prototypical example of the traditional recommender system is Netflix. As a user continues to spend their time consuming movies, an algorithm can learn the commonality across the watch history and find similar unwatched movies from the catalogue (content-based filtering).

Another common approach is collaborative-filtering. Using the collective experience of multiple users, inference can be made about how different groups like/dislike different movies.

E.g.:

Enjoying **The Shawshank Redemption** and **The Dark Knight** infers that the user will probably enjoy **Fight Club** because most users who enjoy the **The Shawshank Redemption** and **The Dark Knight** also enjoy **Fight Club**

However – given the unique nature of the Flippa marketplace, preference would have to be learnt in a more flexible way.

flippa.com enables users to express their interest across many different dimensions. A user can view, watch, discuss and search for listings. So, the problem then shifts to determining which of these dimensions can be used as a proxy for consumption or understanding which action actually infers intent/preference. The problem would need to be simplified or reduced.

Or would it?

Graphs

Rather than reduce the problem space, Graph Neural Networks (GNNs) can help solve recommendations by allowing for input in a less constraining form. By identifying all the nodes (users, listings, search terms, etc) and connecting them via edges (viewed, watched, searched-for, purchased), a graph can be constructed which outlines the highly dimensional journey of a user.

Utilising the GNN architecture, we are no longer required to reduce the problem or make assumptions on which actions we deem important and which not. Given enough data, the GNN learns the complex relationship between all the nodes in the graph and can then answer questions such as which user is most likely to purchase listing X?

The end result? Introducing Flippa’s AI recommendation engine.