Skip to content

Commit 6ad28e8

Browse files
whongyiwhongyi
committed
first commit
0 parents  commit 6ad28e8

12 files changed

+521
-0
lines changed

‎README.md

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Post-click Feeback for Recommender Systems
2+
This is the code repo for our RecSys 2019 paper: [Leveraging Post-click Feedback for Content Recommendations](https://cornell-nyc-sdl-postclick-recsys.s3.amazonaws.com/paper.pdf). In this paper, we leverage post-click feedback, e.g. skips and completions, to improve the training and evaluation of content recommenders. Check our paper for more details.
3+
4+
# Install
5+
We used [OpenRec](https://github.com/ylongqi/openrec) to build our recommendation algorithms. It is built on the [TensorFlow](https://github.com/tensorflow/tensorflow) framework.
6+
7+
To install the depencies needed for this repo:
8+
```
9+
$ ./scripts/install.sh
10+
```
11+
12+
# Data
13+
We randomly sampled data from two public available datasets for experiments. For preprocessing, please refer to our paper.
14+
- [ByteDance](https://biendata.com/competition/icmechallenge2019/). Contains user interactions with short videos (average 10 seconds in length), including whether or not each video was completed.
15+
- [Spotify](https://www.aicrowd.com/challenges/spotify-sequential-skip-prediction-challenge). Contains music listening sessions across Spotify users. A user may skip or complete listening to each song.
16+
17+
After processing data into train, validaion, test sets, put them under a "dataset" folder. Refer to `dataloader.py` for data format.
18+
19+
20+
# Train and Evaluate
21+
To train a BPR-NR model on the Bytedance dataset under the post-click-aware evaluation metric (click-complete as positive observation, click-skip as negative observation, non-click as missing observation):
22+
```
23+
$ python3 bpr_postclick.py --dataset=bytedance --l2_reg=0.01 --p_n_ratio=0.4 --eval_explicit
24+
```
25+
where `p_n_ratio` corresponds to the hyperparameter $\lambda_p,n$ in the paper. It controls the weights put on each types of signals, and can be any float between 0 to 1.
26+
27+
If you want to see the separated performance on click-skip and click-complete items, add the `eval_rank` flag:
28+
```
29+
$ python3 bpr_postclick.py --dataset=bytedance --l2_reg=0.01 --p_n_ratio=0.4 --eval_explicit --eval_rank
30+
```
31+
32+
If you want to evaluate on the click-only metric (click as positive, non-click as negative), remove the `eval_explicit` flag:
33+
```
34+
$ python3 bpr_postclick.py --dataset=bytedance --l2_reg=0.01 --p_n_ratio=0.4
35+
```
36+
37+
Similarly, to train a WRMF-NR model on the Spotify dataset, we use two hyperparameters to control the weights on positive and negative samples:
38+
```
39+
$ python3 wrmf_postclick.py --dataset=spotify --l2_reg=0.001 --pos_ratio=0.6 --neg_ratio=0.2 --eval_explicit
40+
```
41+
42+
# Experiments
43+
To replicate the experiments in the paper, refer to `./scripts/{bpr,wrmf}_exp.sh`. Assign dataset to be one of {bytedance, spotify} when you run the script. You can also add your own experiments following the instructions in those scripts:
44+
```
45+
$ ./scripts/wrmf_exp.sh bytedance
46+
```
47+
48+
# Citation
49+
To cite our paper:
50+
```
51+
Hongyi Wen, Longqi Yang, and Deborah Estrin. 2019. Leveraging Post-click
52+
Feedback for Content Recommendations. In Thirteenth ACM Conference on
53+
Recommender Systems (RecSys ’19), September 16–20, 2019, Copenhagen, Denmark.
54+
ACM, New York, NY, USA, 9 pages.
55+
```
56+
57+
```
58+
@inproceedings{wen2019leveraging,
59+
title={Leveraging Post-click Feedback for Content Recommendations},
60+
author={Wen, Hongyi and Yang, Longqi and Estrin, Deborah},
61+
booktitle={Proceedings of the 13th ACM Conference on Recommender Systems},
62+
year={2019},
63+
organization={ACM}
64+
}
65+
```
66+
67+
# Contact
68+
If you have questions related to this repo, feel free to raise an issue, or contact us via:
69+
- Email: [email protected]
70+
- Twitter: [@hongyi_wen](https://twitter.com/hongyi_wen)

‎bpr_postclick.py

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
import numpy as np
2+
import sys, os
3+
import argparse
4+
from openrec import ModelTrainer
5+
from openrec.recommenders import BPR
6+
from openrec.utils import Dataset
7+
from openrec.utils.evaluators import AUC
8+
from openrec.utils.samplers import RandomPairwiseSampler, EvaluationSampler
9+
from stratified_pairwise_sampler import StratifiedPairwiseSampler
10+
from dataloader import *
11+
12+
13+
### training parameter ###
14+
total_iter = 10000 # iterations for training
15+
batch_size = 1000 # training batch size
16+
eval_iter = 1000 # iteration of evaluation
17+
save_iter = eval_iter # iteration of saving model
18+
19+
### embeding ###
20+
dim_user_embed = 100 # dimension of user embedding
21+
dim_item_embed = 100 # dimension of item embedding
22+
23+
24+
def exp(dataset, l2_reg, p_n_ratio, eval_explicit, save_log, eval_rank):
25+
26+
if dataset == 'spotify':
27+
data = loadSpotify()
28+
29+
elif dataset == 'bytedance':
30+
data = loadByteDance()
31+
32+
else:
33+
print ("Unsupported dataset...")
34+
return
35+
36+
# save logging and model
37+
log_dir = "validation_logs/{}_{}_{}_{}_{}/".format(dataset, l2_reg, p_n_ratio, eval_explicit, eval_rank)
38+
os.popen("mkdir -p %s" % log_dir).read()
39+
if save_log:
40+
log = open(log_dir + "validation.log", "w")
41+
sys.stdout = log
42+
43+
44+
# prepare train, val, test sets
45+
train_dataset = Dataset(data['train'], data['total_users'], data['total_items'], name='Train')
46+
if p_n_ratio is None:
47+
train_sampler = RandomPairwiseSampler(batch_size=batch_size, dataset=train_dataset, num_process=5)
48+
else:
49+
train_sampler = StratifiedPairwiseSampler(batch_size=batch_size, dataset=train_dataset, p_n_ratio=p_n_ratio, num_process=5)
50+
if p_n_ratio > 0.0:
51+
print ("Re-weighting implicit negative feedback")
52+
else:
53+
print ("Corrected negative feedback labels but not re-weighting")
54+
55+
eval_num_neg = None if eval_explicit else 500 # num of negative samples for evaluation
56+
if eval_rank:
57+
# show evaluation metrics for click-complete and click-skip items separately
58+
pos_dataset = Dataset(data['pos_test'], data['total_users'], data['total_items'],
59+
implicit_negative=not eval_explicit, name='Pos_Test', num_negatives=eval_num_neg)
60+
neg_dataset = Dataset(data['neg_test'], data['total_users'], data['total_items'],
61+
implicit_negative=not eval_explicit, name='Neg_Test', num_negatives=eval_num_neg)
62+
pos_sampler = EvaluationSampler(batch_size=batch_size, dataset=pos_dataset)
63+
neg_sampler = EvaluationSampler(batch_size=batch_size, dataset=neg_dataset)
64+
eval_samplers = [pos_sampler, neg_sampler]
65+
else:
66+
val_dataset = Dataset(data['val'], data['total_users'], data['total_items'],
67+
implicit_negative=not eval_explicit, name='Val', num_negatives=eval_num_neg)
68+
test_dataset = Dataset(data['test'], data['total_users'], data['total_items'],
69+
implicit_negative=not eval_explicit, name='Test', num_negatives=eval_num_neg)
70+
val_sampler = EvaluationSampler(batch_size=batch_size, dataset=val_dataset)
71+
test_sampler = EvaluationSampler(batch_size=batch_size, dataset=test_dataset)
72+
eval_samplers = [val_sampler, test_sampler]
73+
74+
# set evaluators
75+
auc_evaluator = AUC()
76+
evaluators = [auc_evaluator]
77+
78+
79+
# set model parameters
80+
model = BPR(l2_reg=l2_reg,
81+
batch_size=batch_size,
82+
total_users=train_dataset.total_users(),
83+
total_items=train_dataset.total_items(),
84+
dim_user_embed=dim_user_embed,
85+
dim_item_embed=dim_item_embed,
86+
save_model_dir=log_dir,
87+
train=True,
88+
serve=True)
89+
90+
91+
# set model trainer
92+
model_trainer = ModelTrainer(model=model)
93+
model_trainer.train(total_iter=total_iter,
94+
eval_iter=eval_iter,
95+
save_iter=save_iter,
96+
train_sampler=train_sampler,
97+
eval_samplers=eval_samplers,
98+
evaluators=evaluators)
99+
100+
101+
102+
if __name__ == '__main__':
103+
104+
parser = argparse.ArgumentParser(description='Parse parameters')
105+
parser.add_argument('--dataset', type=str, default='bytedance', help='dataset to use')
106+
parser.add_argument('--l2_reg', type=float, default=0.01, help='l2 regularization of latent factor')
107+
parser.add_argument('--p_n_ratio', type=float, default=None, help='pos-neg pair ratio during sampling')
108+
parser.add_argument('--eval_explicit', action='store_true', help='turn on to use labels to evaluate, by default treat click as positive and non-click as negative')
109+
parser.add_argument('--eval_rank', action='store_true', help='show ranking accuracy for pos and neg samples')
110+
parser.add_argument('--log', action='store_true', help='turn on for logging results to file, by default will print on screen')
111+
args = parser.parse_args()
112+
print (args)
113+
114+
# run experiments
115+
exp(dataset=args.dataset, l2_reg=args.l2_reg, p_n_ratio=args.p_n_ratio, eval_explicit=args.eval_explicit, save_log=args.log, eval_rank=args.eval_rank)

‎dataloader.py

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
import numpy as np
2+
3+
4+
def loadSpotify():
5+
data = {}
6+
data['train'] = np.load('./dataset/spotify/train.npy')
7+
data['val'] = np.load('./dataset/spotify/val.npy')
8+
data['test'] = np.load('./dataset/spotify/test.npy')
9+
data['pos_test'] = np.load('./dataset/spotify/pos_test.npy')
10+
data['neg_test'] = np.load('./dataset/spotify/neg_test.npy')
11+
data['total_users'] = 229792
12+
data['total_items'] = 100586
13+
return data
14+
15+
16+
def loadByteDance():
17+
data = {}
18+
data['train'] = np.load('./dataset/bytedance/train.npy')
19+
data['val'] = np.load('./dataset/bytedance/val.npy')
20+
data['test'] = np.load('./dataset/bytedance/test.npy')
21+
data['pos_test'] = np.load('./dataset/bytedance/pos_test.npy')
22+
data['neg_test'] = np.load('./dataset/bytedance/neg_test.npy')
23+
data['total_users'] = 37043
24+
data['total_items'] = 271259
25+
return data

‎negative_pointwise_sampler.py

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
import numpy as np
2+
import random
3+
from openrec.utils.samplers import Sampler
4+
5+
6+
def NegativePointwiseSampler(batch_size, dataset, pos_ratio=0.5, neg_ratio=0.3, num_process=5, seed=100):
7+
8+
random.seed(seed)
9+
def batch(dataset=dataset, batch_size=batch_size, seed=seed):
10+
11+
num_pos = int(batch_size * pos_ratio)
12+
num_neg = int(batch_size * neg_ratio)
13+
14+
while True:
15+
16+
input_npy = np.zeros(batch_size, dtype=[('user_id', np.int32),
17+
('item_id', np.int32),
18+
('label', np.float32)])
19+
20+
pos_ind = 0
21+
neg_ind = 0
22+
ind = 0
23+
while pos_ind + neg_ind < num_pos + num_neg:
24+
entry = dataset.next_random_record()
25+
if entry['neg_implicit'] == False and pos_ind < num_pos:
26+
input_npy[ind] = (entry['user_id'], entry['item_id'], 1.0)
27+
pos_ind += 1
28+
ind += 1
29+
30+
if entry['neg_implicit'] == True and neg_ind < num_neg:
31+
input_npy[ind] = (entry['user_id'], entry['item_id'], 0.0)
32+
neg_ind += 1
33+
ind += 1
34+
35+
for ind in range(batch_size - num_pos - num_neg):
36+
user_id = random.randint(0, dataset.total_users()-1)
37+
item_id = random.randint(0, dataset.total_items()-1)
38+
while dataset.is_positive(user_id, item_id):
39+
user_id = random.randint(0, dataset.total_users()-1)
40+
item_id = random.randint(0, dataset.total_items()-1)
41+
input_npy[ind] = (user_id, item_id, 0.0)
42+
43+
yield input_npy
44+
45+
46+
s = Sampler(dataset=dataset, generate_batch=batch, num_process=num_process)
47+
48+
return s

‎scripts/.ipynb_checkpoints/bpr_exp-checkpoint.sh

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
#!/bin/sh
2+
trap "exit" INT
3+
4+
for l2_reg in 0.1 0.01 0.001; do
5+
for p_n_ratio in 0.0 0.2 0.4 0.6 0.8 1.0; do
6+
python3 bpr_implicit.py --dataset=$1 --l2_reg=$l2_reg --p_n_ratio=$p_n_ratio --log=True
7+
done
8+
done

‎scripts/.ipynb_checkpoints/install-checkpoint.sh

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# install dependencies
2+
sudo apt-get update -y
3+
sudo apt-get install python3-pip
4+
5+
# pip install
6+
pip3 install tensorflow
7+
pip3 install numpy
8+
pip3 install termcolor
9+
pip3 install tqdm
10+
11+
# install openrec
12+
git clone https://github.com/whongyi/openrec.git
13+
cd openrec
14+
git checkout logging
15+
sudo python3 setup.py install

‎scripts/.ipynb_checkpoints/pmf_exp-checkpoint.sh

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#!/bin/sh
2+
trap "exit" INT
3+
4+
for l2_reg in 0.1 0.01 0.001; do
5+
for pos_ratio in 0.0 0.2 0.4 0.6 0.8 1.0; do
6+
for neg_ratio in 0.0 0.2 0.4 0.6 0.8 1.0; do
7+
python3 pmf_implicit.py --dataset=$1 --l2_reg=$l2_reg --pos_ratio=$pos_ratio --neg_ratio=$neg_ratio --log=True
8+
done
9+
done
10+
done

‎scripts/bpr_exp.sh

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/bin/sh
2+
3+
# Experiments settings for BPR-NR models with different hyperparameters
4+
# Note that BPR-BL is a special case of BPR-NR if we fix `p_n_ratio` to 0
5+
for l2_reg in 0.1 0.01 0.001 0.0001; do
6+
for p_n_ratio in 0.0 0.2 0.4 0.6 0.8 1.0; do
7+
python3 bpr_postclick.py --dataset=$1 --l2_reg=$l2_reg --p_n_ratio=$p_n_ratio --log --eval_explicit
8+
done
9+
done
10+
11+
12+
# Experiment settings for BPR, note that here `p_n_ratio` is set to `None` so that the model will use standard pairwise sampling during training
13+
for l2_reg in 0.1 0.01 0.001 0.0001; do
14+
python3 bpr_postclick.py --dataset=$1 --l2_reg=$l2_reg --p_n_ratio=None --log --eval_explicit
15+
done
16+
17+
# To evaluate the performance on click-only data, i.e. the conventional implicit feedback evaluation setting, remove the `eval_explicit` flag. For example:
18+
# python3 bpr_postclick.py --dataset=bytedance --l2_reg=0.01 --p_n_ratio=0.4 --log
19+

‎scripts/install.sh

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
pip3 install tensorflow
2+
pip3 install numpy
3+
pip3 install termcolor
4+
pip3 install tqdm
5+
6+
# install a forked version of openrec, with support to some customized functions
7+
git clone https://github.com/whongyi/openrec.git
8+
cd openrec
9+
sudo python3 setup.py install

‎scripts/wrmf_exp.sh

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/bin/sh
2+
3+
# Experiments settings for WRMF-NR model with different hyperparameters
4+
# Note that WRMF-BL is a special case of WRMF-NR if we fix `neg_ratio` to 0
5+
for l2_reg in 0.1 0.01 0.001 0.0001; do
6+
for pos_ratio in 0.0 0.2 0.4 0.6 0.8 1.0; do
7+
for neg_ratio in 0.0 0.2 0.4 0.6 0.8 1.0; do
8+
python3 wrmf_postclick.py --dataset=$1 --l2_reg=$l2_reg --pos_ratio=$pos_ratio --neg_ratio=$neg_ratio --log --eval_explicit
9+
done
10+
done
11+
done
12+
13+
# Experiemnt setting for the standard WRMF model
14+
# `neg_ratio` is set to `None` to use stratified pointwise sampleing during training
15+
for l2_reg in 0.1 0.01 0.001 0.0001; do
16+
for pos_ratio in 0.0 0.2 0.4 0.6 0.8 1.0; do
17+
python3 wrmf_postclick.py --dataset=$1 --l2_reg=$l2_reg --pos_ratio=$pos_ratio --neg_ratio=None --log --eval_explicit
18+
done
19+
done
20+
21+
# To evaluate the performance on click-only data, i.e. the conventional implicit feedback evaluation setting, remove the `eval_explicit` flag. For example:
22+
# python3 wrmf_postclick.py --dataset=bytedance --l2_reg=0.01 --pos_ratio=0.4 --neg_ratio=0.2 --log

0 commit comments

Comments
 (0)