Ishan Misra on Self-Supervised Learning and Vision

Misra contrasts supervised, semi-supervised and self-supervised learning, explaining how labels are used and why human annotation does not scale. He describes the data itself as the source of supervision, with examples including masked-word prediction, predicting video frames and relating different crops of an image.

The speakers discuss self-supervised learning as a possible route to learning common sense from observation and interaction. They consider the difficulty of defining a complete taxonomy, the compositional nature of categories, and how similarity and dissimilarity can support understanding without naming every object.

Misra recalls the effort required to annotate objects with bounding boxes. Fridman raises questions about what counts as an object and how human vision interprets scenes, including pose, gravity and social cues. They discuss what self-supervised learning can contribute and why systems communicating with people still need interpretable outputs.

Misra explains the distributional hypothesis: words appearing in similar contexts can acquire related representations. The speakers discuss masked-word prediction, its use in language models and how scaling has extended its applications. They also consider whether prediction from driving videos can provide useful self-supervision.

Misra describes transformers and self-attention as a way for elements to use information from a wider context than a local convolutional window. They compare language and vision: visual signals are more complex to predict and affected by capture conditions, while language has a finite vocabulary and structured tokens. Misra says computer vision is harder and discusses the limits of current performance.

Misra explains contrastive learning through positive and negative examples and describes energy-based models as a framework for relating different learning methods. They discuss how self-supervised systems obtain related and unrelated examples from context, image crops and video frames, and introduce data augmentation.

Misra outlines image transformations such as cropping, changing brightness and contrast, and rotation. The speakers discuss how augmentations encourage similar features for altered views of an image, whether this resembles human perception, and the possibility of more imaginative but physically consistent transformations.

Misra says augmentation is fundamental to visual self-supervised learning, while noting that current transformations encode human assumptions and can produce unrealistic examples. They consider learning image-aware augmentations, the usefulness of tags, and whether human activity such as driving or playing video games can supply learning signals.

Misra explains why contrastive methods can depend on large numbers of useful negative examples. He describes clustering and self-distillation as alternatives, and explains representation collapse. He then presents Suave as an online clustering method that assigns image crops to clusters while using an equal-partition constraint to prevent collapse.

Misra explains SEER’s aim of training on a large collection of uncurated internet images rather than relying on ImageNet’s curated categories and framing biases. He describes the RegNet architecture and the importance of memory efficiency, and discusses data and augmentation as key ingredients, along with the communication demands of distributed training.

Misra describes VISSL as a PyTorch-based framework containing visual self-supervised methods and evaluation benchmarks. It is intended for researchers training or comparing models and developing new techniques. They discuss the difficulty of creating small-scale experiments whose results reliably transfer to larger datasets.

Misra describes a method that learns shared representations from paired audio and video without labels. Matching the modalities can help recognize actions and sounds or locate a sound source in a video. They discuss how a second modality can clarify ambiguous visual events and whether other senses matter for intelligence.

The speakers discuss active learning, including Misra’s work on an agent that asks questions about images. They consider how a system might identify what it does not know and choose useful examples for labeling, and compare this with collecting edge cases from deployed systems.

Misra discusses predictive learning from driving videos and using uncertainty to find surprising cases. He estimates that autonomous driving may be at least five to ten years away, with important variation by driving context. The speakers consider sensors, edge cases, human behavior, driver supervision and the social and regulatory demands of deploying autonomous vehicles.

Misra identifies data efficiency and generalization as major challenges and discusses the difficulty of defining correctness for machine-learning systems. He distinguishes fast recognition from reasoning about unfamiliar combinations, and talks about continual learning, catastrophic forgetting and the challenge of interpreting what a model has learned.

The speakers consider what human-level or general intelligence might require, including emotion, communication and interaction with the physical world. Misra says some form of embodiment may be needed to understand the world and discusses self-awareness as a system understanding its role and limitations. Fridman considers how people might relate to systems that display human-like qualities.

Misra describes object boundaries emerging in visual representations despite not being explicitly taught. They discuss whether object permanence, rotation, symmetry and counting might also emerge from data, and whether such learned concepts could help with machine reasoning tests.

Misra questions the general usefulness of simulated training environments, citing the cost of modeling visual detail and changing real-world behavior. They discuss the difficulty of simulating human driving behavior and consider whether immersive virtual worlds could become appealing places to spend time.

Misra advises researchers to choose problems that are both interesting and feasible, and to pursue work they genuinely care about. For papers, he recommends focusing on one idea and beginning to write early, so that writing can clarify the argument and reveal gaps in the work.

Misra recommends Python for people beginning machine learning. He recounts using Caffe, TensorFlow and Lua Torch before moving to PyTorch, and says he finds PyTorch easier to debug. The speakers discuss the trade-offs of competing frameworks and the role of open-source code.

Misra encourages beginners to investigate why a model or algorithm is not working rather than immediately seeking a quick answer. He speaks about persistence through research failures and urges young people to pursue what they want, while accepting that failure is part of the process.

Fridman asks whether Misra has found the meaning of life and whether AI could help. Misra describes the search as ongoing and says people’s changing goals and perspectives are part of what makes humans interesting. The conversation ends with thanks and the host’s closing quotation from Arthur C. Clarke.