Drill-down - Interactive Retrieval of Complex Scenes using Natural Language Queries

Fuwen Tan
Department of Computer Science
University of Virginia

  • Date: Friday October 4th, 2019
  • Time: 12:00PM
  • Location: Rice 242

Abstract This work explores the task of interactive image retrieval using natural language queries, where a user progressively provides input queries to refine a set of retrieval results. Moreover, our work explores this problem in the context of complex image scenes containing multiple objects. We propose Drill-down, an effective framework for encoding multiple queries with an efficient compact state representation that significantly extends current methods for single-round image retrieval. We show that using multiple rounds of natural language queries as input can be surprisingly effective to find arbitrarily specific images of complex scenes. Furthermore, we find that existing image datasets with textual captions can provide a surprisingly effective form of weak supervision for this task. We compare our method with existing sequential encoding and embedding networks, demonstrating superior performance on two proposed benchmarks: automatic image retrieval on a simulated scenario that uses region captions as queries, and interactive image retrieval using real queries from human evaluators.

An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation

Wanyu Du
Department of Computer Science
University of Virginia

  • Date: Friday September 27th, 2019
  • Time: 12:00PM
  • Location: Rice 242

Abstract Paraphrase generation is a fundamental research problem that can benefit many other downstream NLP tasks, such as machine translation, text generation, document summarization, and question answering. Previous methods mainly use supervised learning to learn a paraphrase decoder. Given a source sentence, a decoder will decode the target words step by step from a large vocabulary. However, this type of learning and decoding method always suffers from the exposure bias: the current prediction is conditioned on the ground-truth during training but on previous predictions during decoding, and it may lead to propagating and accumulating errors when decoding the text. To deal with this challenge, both reinforcement learning (RL) and imitation learning (IL) have been widely studied, but the lack of direct comparison leads to only a partial image on their benefits. In this work, we present an empirical study on how RL and IL can help boost the performance of generating paraphrases, with the pointer-generator as a base model. And our experiments on the benchmark datasets show that: (1) imitation learning can constantly perform better than reinforcement learning, and (2) the pointer-generator model with imitation learning can outperform the state-of-the-art methods with a large margin.

No seminar on September 20th, 2019

Evaluating Differentially Private Machine Learning in Practice

Bargav Jayaraman
Department of Computer Science
University of Virginia

  • Date: Friday September 13th, 2019
  • Time: 12:00PM
  • Location: Rice 242

Abstract Differential privacy is a strong notion for privacy that can be used to prove formal guarantees, in terms of a privacy budget, ε, about how much information is leaked by a mechanism. However, implementations of privacy-preserving machine learning often select large values of ε in order to get acceptable utility of the model, with little understanding of the impact of such choices on meaningful privacy. Moreover, in scenarios where iterative learning procedures are used, differential privacy variants that offer tighter analyses are used which appear to reduce the needed privacy budget but present poorly understood trade-offs between privacy and utility. We quantify the impact of these choices on privacy in experiments with logistic regression and neural network models. Our main finding is that there is a huge gap between the upper bounds on privacy loss that can be guaranteed, even with advanced mechanisms, and the effective privacy loss that can be measured using current inference attacks. Current mechanisms for differentially private machine learning rarely offer acceptable utility-privacy trade-offs with guarantees for complex learning tasks: settings that provide limited accuracy loss provide meaningless privacy guarantees, and settings that provide strong privacy guarantees result in useless models.

Strange Geometry in High Dimensions and its Implication for Machine Learning

Tom Fletcher
Department of Electrical and Computer Engineering, Department of Computer Science
University of Virginia

  • Date: Friday September 6th, 2019
  • Time: 12:00PM
  • Location: Rice 242

Abstract Modern data lives in high dimensions, e.g., the number of pixels in an image, the number of words in a document, etc. In this talk, I will present some of the geometric oddities of random samples in high-dimensional Euclidean space. Our intuition about distances, angles, and volumes, which we acquire from 2D and 3D reasoning, doesn’t serve us well in higher dimensions. This has important implications for machine learning. One of the most famous is the existence of adversarial examples, which are data that can be slightly perturbed to change a correct classification into an incorrect one. I will outline a couple of existing conjectures for how high-dimensional geometry leads to adversarial examples, but also argue that these explanations are not fully satisfactory. Finally, I will present some recent work on how to detect vulnerability to adversarial attacks using nonlinear manifold geometry.