Recommendation Engines That Actually Pay Off

At ARTime, I built a recommendation engine that generated $47,000 in additional monthly revenue across our client base. That's not a vanity metric — it was measurable, attributable lift in average order value, tracked across 100,000+ monthly active users. Here's how it worked, what I'd do differently, and the lessons that apply to any product considering recommendations. Most recommendation engine projects start with "let's add AI." That's why most fail. The right starting question is: what user behavior, if nudged, would create business value? For us, the answer was specific. Our e-commerce platform served MENA hospitality and retail. Users were browsing, finding what they came for, and checking out. The gap was in discovery — they weren't seeing products they'd want but didn't know to search for. The business metric was average order value (AOV). If we could surface relevant add-ons and alternatives, AOV would go up. That framing — "increase AOV by surfacing relevant products users aren't finding on their own" — determined everything downstream. The algorithm, the UI placement, the success metric, the rollback plan. There are three common approaches to recommendations, and the choice matters more than people think.

Approach	Best For	Weakness
Content-based	Rich product metadata	Limited to "more like this" — no serendipity
Collaborative filtering	Lots of user-item interactions	Cold start problem (new items/users)
Hybrid	When you can afford the complexity	Engineering overhead

We had 2 million+ user-item interactions and growing. Products had metadata, but it was inconsistent across categories and languages. Collaborative filtering was the right call: let the behavior of similar users guide recommendations, without depending on clean metadata. I used Python with scikit-learn. The core was item-based collaborative filtering — finding items that were frequently co-purchased or co-viewed.

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
from scipy.sparse import csr_matrix

# Build the user-item interaction matrix
# Rows: users, Columns: items, Values: interaction strength (views=1, purchases=3)
def build_interaction_matrix(events):
    """Convert event stream to sparse user-item matrix."""
    user_ids = {u: i for i, u in enumerate(events['user_id'].unique())}
    item_ids = {it: j for j, it in enumerate(events['item_id'].unique())}

    rows = events['user_id'].map(user_ids)
    cols = events['item_id'].map(item_ids)
    values = events['interaction_type'].map({'view': 1, 'purchase': 3, 'wishlist': 2})

    return csr_matrix(
        (values, (rows, cols)),
        shape=(len(user_ids), len(item_ids))
    ), user_ids, item_ids

# Compute item-item similarity
def compute_item_similarity(interaction_matrix):
    """Items that are interacted with by similar users are similar."""
    # Transpose so items are rows
    item_matrix = interaction_matrix.T
    # Normalize to reduce popularity bias
    normalized = normalize(item_matrix)
    return cosine_similarity(normalized, dense_output=False)

def recommend(user_interactions, item_similarity, item_ids, n=5):
    """Given a user's history, recommend items they haven't interacted with."""
    scores = np.zeros(len(item_ids))

    for item_id, weight in user_interactions.items():
        if item_id in item_ids:
            idx = item_ids[item_id]
            # Add similarity scores from this item
            similar = item_similarity[idx].toarray().flatten()
            scores += similar * weight

    # Zero out items the user already interacted with
    for item_id in user_interactions:
        if item_id in item_ids:
            scores[item_ids[item_id]] = 0

    # Return top N
    top_indices = np.argsort(scores)[-n:][::-1]
    return [(list(item_ids.keys())[i], scores[i]) for i in top_indices if scores[i] > 0]

This is intentionally simple. No deep learning, no neural collaborative filtering, no transformer-based embeddings. Just TF-IDF weighted interactions and cosine similarity. The simplicity was the point — it was explainable, debuggable, and fast enough to run nightly batch jobs. Collaborative filtering has one fatal flaw: it can't recommend items with no interaction history, and it can't personalize for users with no history. We handled this in layers: New users: Show popularity-based recommendations (most purchased in their category/region) until we had enough interactions to personalize. Not as good as personalized, but better than nothing. New items: Give them a visibility boost for the first two weeks — surface them in recommendations regardless of the algorithm's output. This seeded them with interactions so they could enter the collaborative filtering pool. The fallback: If the algorithm had low confidence (few similar users, sparse data), fall back to popularity. Never show empty recommendation slots. A relevant popular item is better than a confident bad recommendation. The algorithm is half the work. The other half is UI placement — and this is where most teams underperform. We A/B tested four placements:

1. Product page sidebar ("You might also like") — +3% AOV
2. Cart page ("Frequently bought together") — +8% AOV
3. Post-purchase ("Customers who bought this also bought") — +2% AOV (low impact on current order, decent for retention)
4. Search results (re-ranked with personalization) — +5% AOV

The clear winner was the cart page. Users had already committed to purchasing; relevant add-ons felt like a convenience, not a sales push. The same recommendations on the product page felt like advertising and converted worse. Lesson: the algorithm determines relevance. Placement determines whether relevance translates to revenue. The $47K figure was real, but getting to it required discipline.

• Holdout group: 10% of users saw no recommendations for the entire test period. Their AOV was the baseline.
• Time window: We measured over 90 days to account for novelty effect (the initial spike when something is new).
• Segmentation: The lift wasn't uniform. Power users (5+ orders/month) saw +24% AOV. Casual users saw +9%. This told us where recommendations created the most value.

The mistake I see teams make is measuring recommendation success by click-through rate. CTR is easy to game — show sensational recommendations and clicks go up. But if those clicks don't convert to purchases, you've just added friction. Measure the business metric, not the engagement metric. Two things, in hindsight. I'd start simpler. The first version should have been "users who bought X also bought Y" — a SQL query on co-purchase history, no ML. It would have validated the concept in a week instead of a month. The ML version was better, but the marginal improvement didn't justify the upfront complexity for the first release. I'd build the feedback loop earlier. We tracked what was recommended and what was purchased, but we didn't close the loop fast enough — the model retrained weekly. Moving to a daily retrain with streaming data would have caught trend shifts faster, especially during seasonal events like Ramadan when purchasing patterns change dramatically. Recommendation engines are one of the highest-ROI AI features you can build — when they're grounded in a real business question and measured honestly. The technology is not the hard part. scikit-learn and cosine similarity will get you most of the way there. The hard part is the product thinking: where to place recommendations, how to handle cold starts, how to measure success without fooling yourself, and how to resist the urge to over-engineer before you've validated the concept. Start with a SQL query. Ship it. Measure. Then decide if you need ML.