Jinyoung N Kim

20170046431, Feb 16, 2017

Aug 11, 2015

14/822905

- Redmond WA, US
Jinyoung Kim - Bellevue WA, US
Hyun Joon Jung - Cupertino CA, US
Ahmed Awadallah - Redmond WA, US

G06F 17/30
G06N 99/00

Techniques for evaluating the quality of results obtained by a search engine. In an aspect, an evaluation platform utilizes task-level formulation to increase the accuracy of search result quality evaluation. Furthermore, initial queries may be reformulated until search results are deemed to satisfy the task description. Side-by-side comparison of results from multiple search engines is further provided to enhance the sensitivity of evaluation. Alternative aspects provide for collection of behavioral signals for training a classifier to classify the quality of an evaluator's feedback, as may be applied in, e.g., a crowd-sourcing context.

20160147754, May 26, 2016

Nov 21, 2014

14/550640

- Redmond WA, US
Jinyoung Kim - Bellevue WA, US
Imed Zitouni - Bellevue WA, US

G06F 17/30
H04L 29/08

The claimed subject matter includes techniques for offline evaluation of ranking functions. An example system includes a first module configured to receive production log data, the first module to pre-process the production log data to generate an exploration data set. The example system also includes a second module configured to perform offline estimation of online metrics for ranking functions using the exploration data set. The example system also includes a third module to evaluate a proposed ranking function by comparing the estimated online metrics to a set of baseline metrics of a baseline ranking function and detecting that the estimated online metrics of the proposed ranking function exceed, are lower than, or are within a predetermined range of the baseline metrics.

20150356489, Dec 10, 2015

Jun 5, 2014

14/297619

- Redmond WA, US
Imed Zitouni - Bellevue WA, US
Steven Shelford - Vancouver, CA
Jinyoung Kim - Bellevue WA, US

G06Q 10/06

Results, generated by human workers in response to HITs assigned to them, are evaluated based upon the behavior of the human workers in generating such results. Workers receive, together with an intelligence task to be performed, a behavior logger by which the worker's behavior is monitored while the worker performs the intelligence task. Machine learning is utilized to identify behavioral factors upon which the evaluation can be based and then to learn how to utilize such behavioral factors to evaluate the HIT results generated by workers, as well as the workers themselves. The identification of behavioral factors, and the subsequent utilization thereof, is informed by the behavior of, and corresponding results generated by, a trusted set of workers. Results evaluated to have been improperly generated can be discarded or simply downweighted. Workers evaluated to be operating improperly can be removed or retrained.