Section I: Problem Statement

CVS Health is continuously exploring ways to improve its e-commerce platform, cvs.com. One potential enhancement is the implementation of a complementary product bundle recommendation feature on its product description pages (PDPs). For instance, when a customer browses for a toothbrush, they could also see recommendations for related products like toothpaste, dental floss, mouthwash, or teeth whitening kits. A basic version of this is already available on the site through the “Frequently Bought Together” (FBT) section.

Traditionally, techniques such as association rule mining or market basket analysis have been used to identify frequently purchased products. While effective, CVS aims to go further by leveraging advanced recommendation system techniques, including Graph Neural Networks (GNN) and generative AI, to create more meaningful and synergistic product bundles.

This exploration focuses on expanding the existing FBT feature into FBT Bundles. Unlike the regular FBT, FBT Bundles would offer smaller, highly complementary recommendations (a bundle includes the source product plus two other items). This system would algorithmically create high-quality bundles, such as:

• Self-care Package (scented candle + bath salt + face mask)
• Travel Kit (neck pillow + travel adapter + toiletries)

This strategy has the potential to enhance both sales and customer satisfaction, fostering greater loyalty. While CVS does not yet have the FBT Bundles feature in production, it is developing a Minimum Viable Product (MVP) to explore this concept.

Section II: High-Level Approach

The core of this solution is a Graph Neural Network (GNN) architecture. Based on the work of Yan et al. (2022), CVS adapted this GNN framework to its specific needs, incorporating several modifications.

The implementation consists of three main components:

1. Product embeddings using a GNN
2. User embeddings through a transformer
3. A re-ranking scheme to personalize recommendations

Section III: In-Depth Methodology

Part 1: Product Embeddings

Module A: Discovering Product Segment Complementarity Relations Using GPT-4

Embedding plays a critical role in this approach, converting text (like product names) into numerical vectors to help machine learning models understand relationships. CVS uses a GNN to generate embeddings for each product, ensuring that relevant and complementary products are grouped closely in the embedding space.

To train this GNN, a product-relation graph is needed. While some methods rely on user interaction data, CVS found that transaction data alone was not sufficient, as customers often purchase unrelated products in the same session. For example:

• In a sample of 1 million transactions, 85% of items came from different product categories.
• In a manual inspection of 500 transactions, 81% contained at least one unrelated item.

Instead, CVS utilized GPT-4 to identify complementary products at a higher level in the product hierarchy, specifically at the segment level. With approximately 600 distinct product segments, GPT-4 was used to identify the top 10 most complementary segments, streamlining the process.

Module B: Evaluating GPT-4 Output

To ensure accuracy, CVS implemented a rigorous evaluation process:

1. Curated Dataset: A sample of 10% of product segments was selected for manual review to create a ground truth dataset.
2. Full GPT-4 Run: GPT-4 was used to map complementary segments for the remaining products.
3. Metric Calculation: Precision @ K, Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (NDCG), and Hit Rate were used to evaluate the accuracy of the results.

These results confirmed strong performance in identifying complementary relationships.

Module C: Learning Product Embeddings

With complementary relationships identified at the segment level, a product-relation graph was built at the SKU level. The GNN was trained to prioritize pairs of products with high co-purchase counts, sales volume, and low price, producing an embedding space where relevant products are closer together. This allowed for initial, non-personalized product recommendations.

Part 2: User Embeddings

To personalize recommendations, CVS developed user embeddings. The process involves:

1. Retrieving the user’s 10 most recent purchases.
2. Encoding the textual features (e.g., titles, descriptions) of these products.
3. Passing these encoded features through a transformer to generate a user context vector, representing the user’s preferences.

This framework is currently based on recent purchases, but future enhancements will include demographic and other factors.

Part 3: Re-Ranking Scheme

To personalize recommendations, CVS introduced a re-ranking step:

1. Generate non-personalized recommendations using product embeddings.
2. Compute the Hadamard product (element-wise multiplication) of user and product embeddings.
3. Average the resulting values to obtain a re-ranking score.
4. Re-rank the recommendations to match the user’s preferences.

Section IV: Evaluation of Recommender Output

Given that CVS trained the model using unlabeled data, traditional metrics like accuracy were not feasible. Instead, GPT-4 was used to evaluate recommendation bundles, scoring them on:

• Relevance: How well each recommendation complements the source product.
• Synergy: How well all items in the bundle complement each other.

The results showed that the model effectively generated high-quality, complementary product bundles.

Section V: Use Cases

1. Enhancing Sales and Customer Experience: Complementary product recommendations can drive increased sales across CVS’s retail and pharmacy segments by encouraging customers to purchase related items.
2. Advanced Analytics for Decision-Making: CVS’s data analytics team can use insights from the recommendation system to optimize inventory and marketing strategies.

Section VI: Future Work

Future plans include:

• Integrating additional data sources (e.g., clickstream data)
• Conducting extensive A/B testing
• Refining model parameters to improve performance