Research Is Tree Pruning

Transcribed conversation with Sam Layton - a Stanford researcher and his opinion on being a a good researcher. TLDR; Apply the scientific method. Don’t trick yourself into thinking that you’re doing the scientific method by just doing science. Write down the hypothesis and seriously consider all your options. Frequently take a step back to look at your tree and the branch you’re currently in. This is what separates good researchers from timeless researchers.

When I first started doing research, I thought, Wow, I'm so smart. I come up with such creative ideas. And I was an idiot, right?

Genuinely, the ideas you come up with are pretty much entropy. The best way I like to think of it is: if you come up with good ideas, that means you have high entropy. High entropy is good. But the thing that actually gets you good discoveries is discipline.

That was shocking to me. I thought the smartest, best people, the people who made the best discoveries, were the ones who just came up with the most brilliant ideas. And no. Yes, you need high-entropy, brilliant ideas. But discoveries are only achieved when you also have high discipline.

So what does research discipline mean?

The most important thing you can do is come up with your hypotheses and then ask:

For example, we had a theoretical method that worked. The next question was how to turn it into something practical. We knew one approach from the theory, and we had another idea involving a convex least-squares solve. I knew a convex solve could give us the best possible answer under its objective, but how well would that generalize compared with using the regular convolution formula?

So I asked: How can I test this across a bunch of different functions?

I set up the experiments. Rather than saying, "I'm going to come up with this crazy solution," I just asked the questions.

Research is not about being creative enough to imagine the perfect solution. It is about having the discipline to ask the right questions. You come up with a hypothesis that makes sense, but that is only the part that gets glamorized. Real research begins when you figure out what experiment will uncover the most information, regardless of what the answer turns out to be.

Then you run it. You look at the result and say, Okay, this is really surprising. This is really shocking. This did not go how I thought it would. And then you drill down again, relentlessly.

When I do research, I literally write out my hypothesis beforehand. Then I ask, What experiment will work?

And it becomes a tree.

You start with one hypothesis and ask a question. You run an experiment and get a result. If the result is surprising, it might spawn three more hypotheses. Now you have to decide which of those hypotheses matters most.

Once you choose one, you might have four different experiments that could test it. Again, you have to decide which experiment will tell you the most.

That is the work.

Here is a concrete example, even if it is a little silly. Say you are building a business that sells AI automation, and your hypothesis is:

Maybe an independent dermatology or dental practice is easier because you do not have to go through layers of upper management, and it probably does not already have the technology you can offer.

Fine. Write the hypothesis down: small practices = easy sale.

Now, what does "easy" even mean? You need a way to measure it.

One experiment is straightforward: walk into small practices, get past the receptionist, speak to the owner, explain problems you have solved for similar practices, and try to sell. See how it goes.

But that is only one branch. Another experiment is to investigate which companies are actually adopting AI products right now. Who is selling AI, and who is buying it? List the products and industries. Maybe you discover that adoption is concentrated in coding, law, and insurance. That is interesting. Why?

Your original hypothesis was just that small practices would be easier to sell to. But this result gives you a new question. Maybe what sells well is not "AI for small companies." Maybe it is AI for highly specialized work that requires a lot of focus.

Now you can form another hypothesis and test other industries with those properties. If a candidate industry does not buy AI products well, cross that hypothesis off the list.

Honestly, research is literally just tree pruning. That is what research is.

You do need to come up with good hypotheses. That is the little spark of genius people talk about. But research gets glamorized because, at the end of the day, the paper reports only the path that reached the final result.

Readers do not see the full tree. They do not see all the hypotheses, experiments, dead ends, and branches you pruned away. They just see one clean path:

They came up with this, discovered that, and then found the answer. Wow, what a stroke of genius.

Every once in a while, at every branch, you do have to be creative enough to form a new hypothesis. That is valuable. But it is almost irrelevant compared with the ability to prune the research tree.

There are infinitely many rabbit holes. Especially when you are younger, it is easy to choose one and go all the way down. You do not realize that you could take a step back, look at the branch next door, and find that it would be far more fruitful.

There is a story about a mathematician who spent around thirty years working on one problem, only for a graduate student who had worked on it for about a year and a half to solve it two months earlier. The mathematician had essentially solved it too, but someone else got there first. That is my worst nightmare.

The point is not that persistence is bad. The point is that research requires the discipline to keep asking whether this is still the right branch.

You need to take a step back and ask, What actually matters here?

Any time you come across something that does not make sense, that is gold. That is your signal down the tree. Any time you see something and think, Huh. That doesn't make sense. That's fascinating, you need to drill down into it.

Research is not about being a genius and having strokes of inspiration. It is about following paper trails, being relentless, and being extremely curious.

If what you are researching does not spark that feeling of I just cannot stop thinking about this, it is not going to be useful. You might have a little stroke of genius in the shower here and there, but that is not enough.

I used to think, I can conceive this entire machine-learning architecture. It makes sense in my brain. Then I would disappear down one rabbit hole and spend so much time thinking about it because I was not doing experiments.

The scientific method is experiments. Experiments mean forming hypotheses and probing the spokes efficiently.

Here is what I have found with AI: AI is really good at implementing experiments.

If you tell it the exact experiment to run and exactly what you are doing, it will implement it so well. If you have already decided, These are the three experiments I should run to test this hypothesis, and you give it precise instructions, it is fantastic.

It is terrible at pruning the tree.

It is so bad at it. I genuinely do not know how quickly it will get better, because models improve by seeing examples, and they do not get many examples of what it looks like to prune a scientific tree. Papers show the final path, not the actual pruning process.

I believe in inspiration in research. I believe in revelation. But every time I offload the cognitive work of deciding, Is this a good idea or not? to AI, my progress halts.

It is such a temptation. The other day, I just wanted AI to figure a problem out, so I told it, essentially, "Make this work. Try to figure it out." I bashed my head against it for two days and made no progress. The time was completely wasted. If I had taken an extra thirty minutes to think it through and understand what was going on, it would have worked.

Here is how I use AI instead.

First, I write down my hypothesis. I set it in stone long enough to test it. Then I think really hard:

That is your revelation.

I might come up with four or five possible experiments, but I still need the discretion to prune away the crap. Most of my ideas are crap too, right?

Once I choose a route, I give that route to AI. I tell it the figure I want to see and what I expect to see. I specify the exact axes, how many subplots I want, which lines belong on each plot, which parameter values to test, and even the colors if that helps. I try to be specific enough that I can already picture the graph that would expose the hypothesis.

Then I let AI implement it. It is so good at that.

The key is that you choose the path down the tree. Choosing the graph is part of choosing what evidence would expose the truth of the hypothesis.

Here is a real example. You do not need to understand all the technical details.

In one theoretical construction, the weights in the first layer, (W_1), were supposed to be exactly uniform. Imagine a vector in which every entry is about (31.24). I compared that with a nonuniform vector containing values of very different sizes, then normalized it so that its average absolute magnitude was the same.

The nonuniform vector performed roughly the same, maybe even a little better. At first I thought, Okay, great. That basically follows the theory.

Then I asked: What happens if I interpolate between the random vector and the totally uniform vector? What happens as I nudge the random weights toward uniformity?

I expected a smooth curve. Instead, the slightest nudge toward uniformity completely ruined the performance.

I was like, What? That is so strange. Why is it that way?

I could not get off of it.

My first hypothesis was that some strange change in the distribution occurred as the vector moved toward uniformity. I had AI create an animated histogram of the roughly two hundred weight values. I expected to see a distribution shift, a spike around zero, or something else that could explain why the performance jumped after the tiniest nudge.

Nothing interesting happened.

On the ordinary linear histogram, the distributions looked almost the same. That was not what I expected, but it told me a ton. I had walked down one spoke of the tree and learned that my first explanation was wrong.

So I took a step back and formed another hypothesis.

Some weights were extremely small, maybe (0.01). When I nudged every weight toward (31.24), the update to one of those tiny values was enormous relative to its starting size, even when the interpolation step looked tiny on a linear scale. Maybe what mattered was not the overall linear distribution. Maybe what mattered was the presence of these small ratios.

I tested that. I moved the weights toward (31.24) again, but this time I protected the entries whose magnitudes were less than one. Instead of the performance curve immediately getting worse, it stayed flat while the unprotected version rose.

That was it. The model needed those small numbers in order to do well. The tiniest linear nudge was erasing them.

If I had plotted the distribution on a log scale, the important change would have been obvious from the beginning. I had been interpreting the system linearly when what mattered was log-linear. What really mattered was the small ratios.

That uncovered a new mechanism we had not seen before. It helped us understand why these models generalized better, explained several other phenomena through the same lens, and helped us figure out how to apply the method to real-world data.

It did not come from sitting in the shower and imagining the answer. It came from writing down a hypothesis, running the experiment, noticing that the result did not make sense, pruning the wrong explanation, and forming the next hypothesis.

Your intuition is worth so much less than the information you get by actually drilling in and having discipline.

Knowing when to stop is probably what separates good researchers from great researchers. It is the same problem as pruning the tree: you need a pulse on which hypotheses are most worth pursuing.

I like to think about research almost like Thompson sampling. You pull the levers on different hypotheses. You may return to a branch later, but for now you run down the node that is giving you the most progress.

If you keep getting stuck, if you are no longer making progress, or if your experiments stop producing rich new hypotheses, look at the unexplored nodes you generated along the way. Once another leaf becomes valuable, once it becomes "hot" enough, switch to that leaf.

You are not necessarily declaring the old branch dead forever. You are allocating limited attention across an infinitely large tree.

One other thing I am learning right now is that research and engineering often overlap, but they are very different.

Sometimes you need research in order to do engineering, and you use engineering in order to do research. But the two require different lenses.

Research is about hypotheses, information, experiments, surprises, and pruning the tree.

Engineering is about requirements, designs, and specifications. Given a set of requirements, how can I make the simplest possible thing that satisfies all of them? Engineering means learning to manage requirements almost like currency.

Right now I am trying to build an optimizer that uses facts we discovered about these networks. I want a variant of the Adam optimizer that takes advantage of those facts to find these solutions in the wild. That means I have a list of requirements, and I keep messing up when I treat it like an open-ended research problem instead of an engineering problem.

Previously, I made the opposite mistake. Before I learned how to do research, I applied an engineering lens to research, and that messed me up too.

So you need to recognize: Am I engineering a solution right now, or am I actually researching? They are different skill sets. If you use one in place of the other, it is going to mess you up.

This is what I wish I had learned much sooner:

Write the hypothesis down. Ask the question. Think really hard about which experiment is actually valuable. Choose the experiment that will give you the maximal amount of information. Run it. Pay attention to whatever does not make sense. Form the next layer of hypotheses. Prune away what matters less. Repeat.

You only have so much time, and the tree of possible ideas is infinitely large. Without discipline, you will waste so much of that time wandering through it.

Research is not mainly about coming up with crazy ideas. It is not mainly about being a genius or waiting for a stroke of inspiration.

Research is literally just tree pruning.