The goal of this research is to develop an A/B testing method to automatically compare the user experience (UX) of alternative designs for a web application in a real context with a large number of users. The challenge that it poses is to find mechanisms to predict the UX with machine learning techniques. This submission outlines the motivation, research goal, current status and remaining work.