Skip to main content

The three AI pioeers and their research

 


https://www.linkedin.com/posts/helloashar_ai-deeplearning-generativeai-activity-7418944581091606528-XzoC?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAE-1ZoBN6fDf4c4-AG6JRdo4nKXNrYV99U

1. Barret Zoph: Automating AI and RLHF

Zoph’s work moved the industry from hand-designed models to AI-designed architectures and led the "alignment" phase for ChatGPT.

  • Neural Architecture Search with Reinforcement Learning (2016) – The foundational paper for AutoML.

  • Learning Transferable Architectures for Scalable Image Recognition (2017) – Introduced NASNet.

  • Searching for Activation Functions (2017) – Discovery of the Swish activation function.

  • Efficient Neural Architecture Search via Parameter Sharing (ENAS) (2018) – Made NAS computationally feasible.

  • AutoAugment: Learning Augmentation Policies from Data (2018) – Automated data processing.

  • SpecAugment: A Simple Data Augmentation Method for ASR (2019) – Revolutionized speech recognition.

  • Switch Transformers: Scaling to Trillion Parameter Models (2021) – A landmark paper on Mixture of Experts (MoE).

  • Scaling Instruction-Finetuned Language Models (Flan) (2022) – Key work on instruction tuning.

  • GPT-4 Technical Report (2023) – (Co-author) Lead for the post-training/alignment sections.

2. Luke Metz: Learned Optimization and Generative Models

Metz’s work focuses on the "meta" level—training models to understand how they should learn and solve complex logic.

  • Unsupervised Representation Learning with DCGANs (2015) – One of the most famous papers in generative AI history.

  • Unrolled Generative Adversarial Networks (2016) – Solved the "mode collapse" problem in GANs.

  • Meta-Learning Update Rules for Unsupervised Representation Learning (2018) – Early work on "learning to learn."

  • Understanding and Correcting Pathologies in the Training of Learned Optimizers (2019) – Identified why AI optimizers fail.

  • VeLO: Training Versatile Learned Optimizers by Scaling Up (2022) – Introduced a general-purpose AI optimizer that outperforms human-coded ones.

  • Gradients are Not All You Need (2022) – Explored alternative ways to update model weights.

  • Beyond the Imitation Game: Quantifying Capabilities (BIG-bench) (2023) – A massive benchmark for testing LLMs.

  • ChatGPT System Card / GPT-4o (2024) – Technical work on model safety and system behavior.

3. Samuel Schoenholz: The Physics of Deep Learning

Schoenholz provides the mathematical "blueprints" that allow us to build massive models without them breaking.

  • Deep Information Propagation (2016) – Defined how signals travel through deep neural networks.

  • Deep Neural Networks as Gaussian Processes (2017) – Connected deep learning to classical statistics.

  • Resurrecting the Sigmoid in Deep Learning (2017) – Used "Dynamical Isometry" to train massive 10,000-layer networks.

  • Wide Neural Networks Evolve as Linear Models (NTK) (2019) – Proved the Neural Tangent Kernel (NTK) theory.

  • JAX MD: A Framework for Differentiable Physics (2020) – Built the library for AI-powered physics simulations.

  • Neural Tangents: Fast and Easy Infinite Neural Networks (2019) – A library for studying theoretical AI limits.

  • Tensor Programs V: Tuning Large Neural Networks ($\mu$P) (2022) – Developed $\mu$Transfer, the method used to scale GPT-4 efficiently.

  • Scaling Deep Learning for Materials Discovery (GNoME) (2023) – Published in Nature, detailing the discovery of 2.2 million new crystals using AI.


Comments

Post a Comment

Popular posts from this blog

Telecom OSS and BSS: A Comprehensive Guide

  Telecom OSS and BSS: A Comprehensive Guide Table of Contents Part I: Foundations of Telecom Operations Chapter 1: Introduction to Telecommunications Networks A Brief History of Telecommunications Network Architectures: From PSTN to 5G Key Network Elements and Protocols Chapter 2: Understanding OSS and BSS Defining OSS and BSS The Role of OSS in Network Management The Role of BSS in Business Operations The Interdependence of OSS and BSS Chapter 3: The Telecom Business Landscape Service Providers and Their Business Models The Evolving Customer Experience Regulatory and Compliance Considerations The Impact of Digital Transformation Part II: Operations Support Systems (OSS) Chapter 4: Network Inventory Management (NIM) The Importance of Accurate Inventory NIM Systems and Their Functionality Data Modeling and Management Automation and Reconciliation Chapter 5: Fault Management (FM) Detecting and Isolating Network Faults FM Systems and Alerting Mecha...
1 comment
Read more

"Depth-Guard" – 3D Spatial Occupancy monitor Challenge -2

  Project Title: "Depth-Guard" – 3D Spatial Occupancy Monitor 1. The Problem In a smart warehouse, a robot needs to know if a loading zone is clear or occupied. A 2D camera alone can’t tell the difference between a "flat picture of a box" on the floor and an "actual 3D box." The Goal: Build a Python-based system that uses Computer Vision and Depth Perception (AI 3D) to identify objects and determine their 3D volume (Size) and Distance from the camera. 2. Intern Tasks Object Detection: Use a pre-trained model (like YOLOv8) to draw 2D boxes around objects. Depth Mapping: Use a depth estimation model (like MiDaS or a simulated Stereo-depth feed) to calculate how far each object is. Occupancy Logic: If an object is closer than 1 meter and larger than a specific volume, mark the zone as "BLOCKED." Alert System: Print a warning if the 3D space is too crowded. 3. Sample Datasets (Simulation) Since interns may not have 3D cameras (LiDAR/RGB-D), pr...
Post a Comment
Read more

Simple Virtual Waiting Room -Challenge 1

   Simple Virtual Waiting Room (VWR) 1. The Problem Our website can only handle 10 users per minute . If more than 10 people try to access it at once, the server will crash. We need a system that: Counts incoming users. Redirects "overflow" users to a waiting page. Admits them back to the main site one by one as space becomes available. 2. Intern Tasks Create a Gateway: A simple script that checks: if (active_users < 10) { allow } else { send to queue } . Build the Queue: Use a simple list (FIFO) to store user IDs. The Wait Page: A basic HTML page that says: "You are number X in line. Estimated wait: Y minutes." Admission Logic: Every 30 seconds, pull the next user from the queue and "admit" them. 3. Sample Datasets (Simulation) Provide these two datasets to the interns. They should write a script to "read" these files and simulate how their system reacts. Dataset A: The Traffic Surge (Input) This file simulates users arriving at the ...
Post a Comment
Read more