Check out the paper: https://arxiv.org/pdf/2502.14499

More on Roberta: https://rraileanu.github.io/

More on Deepak: https://dnathani.net/

Paper Link: https://arxiv.org/abs/2405.03547

Richard's website: https://xingyousong.github.io/

Check out Nyckel at: https://www.nyckel.com/

• The original AutoGluon Paper: "AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data": https://arxiv.org/abs/2003.06505
• AutoML Fall School 2022 AutoGluon presentation, a good way to understand the philosophy behind AutoGluon: https://www.youtube.com/watch?v=VAAITEds-28
• AutoGluon multi-modal paper: https://dl.acm.org/doi/abs/10.1145/3534678.3542616

The paper he published presents his framework, HAMLET, which stands for Human-centric AutoML via Logic and Argumentation. It allows a user to iteratively specify constraints in a formal manner and, once defined, those constraints become logical premises. Those premises, when combined together, can produce conflicts with one another, thereby reducing the search space and providing deeper intuition back to the user.

To learn more about HAMLET, see the paper here: https://ceur-ws.org/Vol-3135/dataplat_short2.pdf and the repo here: https://github.com/QueueInc/HAMLET

To follow Joseph on LinkedIn, see his profile here: https://www.linkedin.com/in/joseph-giovanelli/

They’ll talk about self-supervision, issues of scale, the motivation behind the terminology, the origins of the Research for Foundation Models Institute at Stanford, outcome homogenization, emergence and phase transitions, and some of the social consequences to look out for.

Thank you both for this conversation. As the world is coming to terms with GPT-4, this will be increasingly relevant.

Paper: On the Opportunities and Risks of Foundation Models.
Rishi's twitter: @RishiBommasani
Center for Research on Foundation Models (CRFM)

Raymond is a prolific writer on LinkedIn. You should follow him here: https://www.linkedin.com/in/raymondpeck/.

In the episode, we spend some time discussing posterior predictive probabilities before discussing how exactly they’ve pre-fitted their network, how they got their training data, what the network looks like, and how the system is performing.

To give you a taste of it, on datasets up to 1,000 training instances and 100 features, it takes less than a second to train and predict a classifier!

Read their paper here: https://arxiv.org/pdf/2207.01848.pdf

Follow Samuel on Twitter, here: https://twitter.com/SamuelMullr

Follow Noah on Twitter, here: https://twitter.com/noahholl

A Framework for Machine Learning of Model Error in Dynamical Systems - https://arxiv.org/abs/2107.06658

Related works
Autodifferentiable Ensemble Kalman Filters - https://epubs.siam.org/doi/abs/10.1137/21M1434477

Universal Differential Equations for Scientific Machine Learning - https://arxiv.org/abs/2001.04385

Continuous-time nonlinear signal processing: a neural network based approach for gray box identification - https://ieeexplore.ieee.org/document/366006

A generalised approach to process state estimation using hybrid artificial neural network/mechanistic models - https://www.sciencedirect.com/science/article/abs/pii/S0098135496003365

Does AutoML work for diverse tasks? - https://blog.ml.cmu.edu/2022/07/07/automl-for-diverse-tasks

Follow Junhong @JunhongShen1

As you’ll hear, Marius is full of ideas for where AutoML systems can and should go. These ideas are crystallized in a blog-post, published here: https://www.automl.org/rethinking-automl-advancing-from-a-machine-centered-to-human-centered-paradigm/

If you’re searching for research directions, this conversation left me with dozens of ideas. Marius and his team are doing phenomenal work to make AutoML systems more trustworthy and more human-centric.

We will be reviewing content from the following papers:

• Bayesian Optimization with a Prior for the Optimum - https://arxiv.org/abs/2006.14608
• Explaining Hyperparameter Optimization via Partial Dependence Plots - https://arxiv.org/abs/2111.04820
• Enhancing Explainability of Hyperparameter Optimization via Bayesian Algorithm Execution - https://arxiv.org/abs/2206.05447
• πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization - https://arxiv.org/abs/2204.11051

Follow Marius on Twitter here - https://twitter.com/LindauerMarius, and AutoML.org on Twitter here - https://twitter.com/AutoML_org.

To help Marius complete the survey he mentioned, please visit the link here - https://www.soscisurvey.de/hpo-method-validation/.

To learn more about AutoML, visit AutoML.org, here - https://www.automl.org/

While talking to many of you, it became clear that this origin story is one that a lot of people have some vague sense about, but not a very concrete knowledge of so hopefully this episode can help to flesh out the narrative with greater clarity.

We'll be discussing his survey paper from 2016, "Algorithm selection for combinatorial search problems: A survey", to be found here:

https://scholar.google.com/citations?view_op=view_citation&hl=en&user=P37OkUUAAAAJ&citation_for_view=P37OkUUAAAAJ:W7OEmFMy1HYC

As well as the original SAT-zilla paper, to be found here:
https://www.aaai.org/Papers/JAIR/Vol32/JAIR-3214.pdf

Follow Lars' work on Twitter, @larskotthoff.

Cedric is a PhD student at Karlsruhe Institute of Technology, in Germany, where was defending his thesis 2 days after we spoke!

You can find his paper here: https://hal.inria.fr/hal-03667231/document

We have on the show Vadim Borisov, a research fellow at the University of Tubingen as well as Kathrin Sessler, a PhD student from the same university.

This episode will be led by Ankush Garg, exploring Vadim and Kathrin's recent paper “Deep Neural Networks and Tabular Data: A Survey”.

Paper: https://arxiv.org/pdf/2110.01889.pdf
SAINT paper: https://arxiv.org/abs/2106.01342
Deep learning architectures and the hippocampus paper: https://arxiv.org/abs/2112.04035

We will also do a brief form of house keeping about the show and some of the upcoming episodes.

As always, if you have ideas, questions, feedback, or would like to be interviewed on the show: please reach out at hello@automlpodcast.com.

Jeff is a Professor of Computer Science at the University of British Columbia. He was previously a Senior Research Scientist at Uber AI, and more recently a Research Team Leader at OpenAI. He's currently a Faculty Member at the Vector Institute.

We’re going to be talking about whether it is even possible that humans never create AGI, the enormous potential upside of AGI, the risks of AGI (including existential risks), Jeff’s fascinating transition into the field of AI, whether academia remains receptive to similar academic transitions, how long it might take to get to AGI and whether we have the right playbook for it, the prevailing status quo track (he mentions a couple): The first being a large language model approach, that he calls “standing on the shoulders of giant human datasets”, some of the limitations of this approach, the main track, that he calls “the manual path to AGI”, his new contribution, the AI Generating algorithms, which is essentially how we get out of the way and let the AI do much more, whether we already have an existence proof for how this can work, the three pillars of the AI Generating approach, AI Generating environments and capability oriented evaluation, the production of interesting or useful behaviors, the hidden Turing award to the person who would figure out AI Generating environment, learning pathologies, the circuitous paths of innovation, the seeming miracle of Darwinian evolution and the challenges of replicating it, the increasingly complex environments Darwinian evolution produces, the moral dimension to toying around with increasingly intelligent agents, AI and vegetarianism, and the extent to which we should be dismissive of the subjective claims of increasingly intelligent models.

His paper can be found here: https://arxiv.org/abs/1905.10985
Follow him on Twitter here: https://twitter.com/jeffclune
Learn more on his website here: http://jeffclune.com/

Link to book - https://link.springer.com/book/10.1007/978-3-030-67024-5
Link to OpenML - https://www.openml.org/

We’ll be covering such topics as how NAS research is done both at Samsung and at Microsoft,  the relationship between NAS and product teams, devices and the various types of constraints they expose, how to featurize hardware contexts, layer-wise latency calculations, surrogate models and the kinds of hardware-aware data they require, the current limitations of NAS, reinforcement learning and NAS, multi-objective optimization in the context of reinforcement learning, reward sparsity, reward shaping and shaping functions, primary and secondary rewards, their concept of co-regulated shaping, Q functions and the effects of potentials, AUTOCOMET, the future of NAS and other topics.

Thank you for tuning in!

To learn more about AUTOCOMET, find the paper here: https://arxiv.org/pdf/2203.15408.pdf

We'll be covering an enormous amount of ground surrounding intelligence and its evaluation.

We’ll touch on topics such as operating conditions in ML, agent characteristic curves,, the challenge with average performance scores, task-oriented evaluation, capability-oriented evaluation - in humans and other animals, animal-AI Olympics, meta-data annotation, performance robustness as a function of latent attributes, the limitations of aggregating behavioral performance into a handful of metrics, whether intelligence is meaningless or all-explanatory, the concept of generality in intelligence, AGI and the problem of distributions of tasks, calibration of task difficulties, whether capability metrics are incommensurable, complexity and compressibility, compression-based algorithmic information theory, the limits of intelligence, the intelligence of hybrid systems, whether we’re ready to talk about consciousness given our limited understanding of intelligence, whether rights should be tethered to pain or to cognition, animal rights, and what the AutoML community can do to help with this line of work.

If you're interested in this line of work, here is a recent paper by José and his team: https://ryanburnell.com/wp-content/uploads/Burnell-et-al-2022-Not-a-Number.pdf

His book, The Measure of All Minds, can be found here: https://www.amazon.com/Measure-All-Minds-Evaluating-Intelligence/dp/1107153018

We’ll be covering a lot of different topics, such as the distributional hypothesis in machine learning, evaluation criteria, the reductive nature of current evaluation methods, task systems, the desiderata of assessor models, how to build assessor models, when to use them, what happens when our models become increasingly complex, how assessor models can help with AI explainability, whether they can help against adversarial scenarios, the challenges of scoring, bias in human evaluators, how assessor models relate to AI alignment, and other topics.

If you'd like to learn more about assessor models, visit the following pages.

José Hernández-Orallo (Wout's advisor): https://scholar.google.com/citations?user=n9AWbcAAAAAJ&hl=en

Training on the Test Set: Mapping the System-Problem Space in AI paper: https://www.aaai.org/AAAI22Papers/SMT-00432-Hernandez-OralloJ.pdf

Karthi is a research staff member in IBM Research at the Watson Research Center.

He studies the relationship between humans, machines, data and the societal implications of machine learning.

He was involved in the initial development of the open source AI Fairness 360 toolkit, where he’s still an active contributor.

His papers won various best paper awards like the 2015 IEEE International Conference on Data Science and Advanced Analytics.

He is an associate editor of Digital Signal Processing and a member of the IEEE and he holds a PhD in electrical engineering from Arizona State University.

We will be discussing a recent paper that he published about explainability methods for similarity learners. So we’ll go into detail about what similarity learning, or metric learning, is.

You can find the paper here: https://arxiv.org/pdf/2202.01153v1.pdf

To learn more about metric learning, please see this survey paper: https://people.bu.edu/bkulis/pubs/ftml_metric_learning.pdf.

We’ll be discussing the motivation behind NAS, the current state of its deployment, the biggest use-cases today, the three components that make up NAS, the drawbacks to the current NAS paradigm, search spaces and how to design them, search strategies and how to choose them, graph representations of neural architectures, evaluation strategies and zero-cost approximations, bias in search space design, risks, the future of hand-designed architectures, and other topics.

Vasco is a PhD student at NOVA School of Science and Technology in Portugal and a co-founder of a Computer Vision startup called DeepNeuronic.

Find his paper, Towards Less Constrained Macro-Neural Architecture Search, here: https://arxiv.org/pdf/2203.05508.pdf

We discuss whether or not there exists a gap between deep learning and gradient boosted decision trees, what the future of a gap might look like, and the extent to which the embedding of numerical features can give deep learning architectures a necessary boost in performance.

Two of his recent papers are useful in this discussion:

• On Embeddings for Numerical Features in Tabular Deep Learning - https://arxiv.org/abs/2203.05556
• Revisiting Deep Learning Models for Tabular Data - https://arxiv.org/abs/2106.11959

You can find Yury in the following places:

• GitHub - https://github.com/Yura52
• Twitter - https://twitter.com/YuraFiftyTwo

Enjoy!

Julian has a background in math, and studied law at Harvard Law School and he speaks about the unique challenges of adopting machine learning in the legal world due to the various mechanisms of missingness in confidential documents.

If you're interested in learning more about missing data and statistical inference, here are several resources to help you get started:

• GOV2001 - Harvard: https://projects.iq.harvard.edu/gov2001/home
• One hour course by Dr. Gary Keng on missing data on YouTube: https://www.youtube.com/watch?v=qlPs8Ioa56Y
• Julian’s paper: https://arxiv.org/abs/2112.09275v4
• Statistical Analysis with Missing Data book: https://www.amazon.com/Statistical-Analysis-Missing-Roderick-Little/dp/0471183865?

One of the theses of Torsten’s work is that in order to understand the impact of our AI on our existing systems, it’s important to have a deeper understanding of those systems to begin with. His simulation is a first step in that direction: an attempt to model the interaction of various agents according to information theoretic principles and study the consequent emergent phenomena.

There are two versions of the paper.

The first is a shorter summary of the work, light on the math, and can be found here: https://psyarxiv.com/wqcmb/.

The longer version of the paper can be found here: https://arxiv.org/pdf/2106.05414.pdf.

Enjoy!

Laurent works at OVHCloud, most recently as a data scientist but previously in various software engineering roles. He published his thesis on AutoML at OVHCloud, and had previously released a paper titled TPOT-SH: A Faster Optimization Algorithm to Solve the AutoML Problem on Large Datasets.

The conversation centers around two classic papers in AutoML:

• Multi-Objective Automatic Machine Learning with AutoxgboostMC by Pfisterer et al: https://arxiv.org/abs/1908.10796
  
  
• An ADMM Based Framework for AutoML Pipeline Configuration by Sijia Liu, Parikshit Yam et al: https://arxiv.org/abs/1905.00424

Enjoy!

Jessica is a Senior Research Scientist at Google Research working on machine learning for healthcare. Before joining Google in 2019, she was a postdoctoral fellow at University College London (UK) and Stanford University (USA), developing machine learning techniques for neuroscience discovery and clinical predictions.

Throughout her career, her interests have lied not only in the technical advancement of machine learning methods, but also in critical aspects of their deployment such as their credibility, fairness, robustness or interpretability.

As you’ll hear, we are orbiting Jessica’s latest paper titled “Best of both worlds: local and global explanations with human-understandable concepts”. You will find lots more reference material in the show notes on the website.

Jessica, thank you for joining us!

Iman is interested in Artificial General Intelligence and so he’s researching systems that can over time develop increasingly more complex skills and a richer body of knowledge.

Iman received his Bachelor’s degree in Computer Engineering from the University of Tehran and his Master’s degree from Washington State University and that’s where he’s currently a research assistant.

Georgios studied electrical and computer engineering at Democritus University of Thrace at Xanthi, in Greece. He specialized in information and electronics.

He’s also worked as a research engineer at Athena Research and Innovation Center - studying ML systems in production. For his PhD, he’s researching Reliable Machine Learning pipelines for mature MLOps systems at International Hellenic University in Greece.