User Conditional Hashtag Prediction for Images

• DOI: 10.1145/2783258.2788576
• Publication Date: 2015-08-07T15:38:27Z
• Authors: Emily Denton; Jason Weston; Manohar Paluri; Lubomir Bourdev; Rob Fergus
• Keywords: Leverage (statistics); Concatenation (mathematics)

Understanding the content of user's image posts is a particularly interesting problem in social networks and web settings. Current machine learning techniques focus mostly on curated training sets image-label pairs, perform classification given pixels within image. In this work we instead leverage wealth information available from users: firstly, employ user hashtags to capture description content; secondly, make use valuable contextual about user. We show how metadata (age, gender, etc.) combined with features derived convolutional neural network can be used hashtag prediction. explore two ways combining these heterogeneous into framework: (i) simple concatenation; (ii) 3-way multiplicative gating, where model conditioned metadata. apply models large dataset de-identified Facebook demonstrate that modeling significantly improve tag prediction quality over current state-of-the-art methods.