The hardest problems in computer science are those that run in superpolynomial time, doubling the number of steps required each time the input size increases—or more than doubling! Computers may be able to solve those problems for very small input sizes, but they very soon get to the point where it can take months or years of computing time to solve them.
Computer scientists use a different approach for solving those hard problems: instead of trying to find the perfect solution, they try to find an approximate solution. A good answer is better than none at all.
One way to come up with approximate answers to a problem is to use a heuristic, a technique that guides an algorithm to find good choices. When an algorithm uses a heuristic, it no longer needs to exhaustively search every possible solution, so it can find approximate solutions more quickly. A heuristic is a shortcut that sacrifices accuracy and completeness.
To better understand heuristics, let's walk through one of the most famous hard problems in computer science.
Traveling Salesperson Problem
The traveling salesperson problem (TSP) asks the following question:
"Given a list of cities and the distances between each pair of cities, what is the shortest possible route that visits each city and returns to the origin city?"
For example, this diagram shows the shortest tour between 46 German cities:
46 dots labeled as German cities. A line goes through the dots, going through each of them exactly once.
The TSP problem was originally faced by traveling salesmen, but is actually relevant to many areas: routing data around a network, manufacturing microchips, observing locations with a telescope, and even sequencing DNA. In all those cases, we want a solution that will find an efficient path between multiple locations.
The brute force approach
TSP is a combinatorial problem, and that's what makes it so hard. The only way a computer can find the optimal solution is the "brute force approach": try every possible path between cities, measure the distance of each path, and pick the path with the shortest distance.
The visualization below generates four random cities, computes the possible paths, and highlights an optimal path. Click on the paths to see the possibilities it comes up with, or start over to try it with a new set of cities.
Do you notice that it always generates two optimal paths? The paths look the same, but they're in reverse order. Technically, those are both optimal solutions, and a salesperson might choose one shortest path instead of the other for non-distance reasons.
It doesn't take the computer very long to generate the possible paths for 4 cities—less than a tenth of a millisecond on my machine. It's fairly quick even for 5 cities, 6 cities, and 7 cities; you can try those yourself if you trust your computer. But the run time grows exponentially, so a few milliseconds soon becomes a few seconds.
Here's a table of run times for 4 to 11 cities on my machine:
Notice the jump from 520 to 5770 at the end there? That's when I decided to take pity on my poor laptop and not go any higher.
At that rate, the computer would need about milliseconds to compute the paths for the 46 German cities from the earlier diagram. That's about years, way longer than the universe has even been in existence. Of course, there are computers that are more powerful than my laptop, but even a supercomputer that is 200,000 times more powerful would require years.
So how was that path of 46 cities computed above? With a heuristic, of course!
Developing a heuristic
One way to come up with a heuristic is to consider how you yourself would approach the problem. Humans don't like wasting unnecessary effort on solving problems, so we often find shortcuts that get us to a good enough solution.
The visualization below shows 5 random cities. Propose a shortest path by clicking the cities, starting with the next city to visit from A. When you're done, find out how your path compares to the optimal path.
How did your path compare to the optimal path? What heuristics did you use to decide the order to visit the cities? Could the computer use that same heuristic?
Not all heuristics are equal, but they can all give us ideas for different ways to guide the computer to find a good solution more quickly.
The nearest-neighbor heuristic
A common heuristic for TSP is "nearest neighbor": the computer always picks the nearest unvisited city as the next city on the path.
Try it out in the visualization below. Now that we're using a heuristic, you can try it out with many more cities than before. Even 46 cities takes just a few milliseconds on my machine.
Impressive, right? Without a heuristic, we might have waited until the heat death of the universe to find the perfect solution. Now, thanks to a simple heuristic, we see results nearly instantly.
But, remember, a heuristic only gives an approximate solution. For this particular heuristic, the solution will be, on average, about 25% longer than the shortest possible path. There are even times where the nearest neighbor heuristic will give the worst route.
Computer scientists have come up with dozens of other heuristics for TSP, including "ant colony optimization", a heuristic inspired by the way that ants deposit pheromones along a path. For each heuristic, they analyze how close the heuristic gets to finding the perfect solution, how much time it takes, and how often the worst case happens.
There are many problems besides TSP that benefit from heuristics.
Here are a few:
Knapsack problem: You have a knapsack and a set of item types, each with a weight and value. How many of each item can you pack so that the total weight doesn't exceed the limit and the total value is maximized? This type of problem shows up in many forms across different industries, like financial advisors selecting investments for a portfolio or factories figuring out the optimal way to cut up raw materials. One heuristic is to sort by value/weight ratio when selecting the next item to pack.
A graphic with a knapsack and 4 boxes. The knapsack is labeled as 15 kg. Box 1 is labeled as 12 kg, 2. Box 3 is labeled as 4 kg, 1.
Game-playing: For a computer to beat a human at a game (or at least lose respectably), it must pick the move with the greatest chance of success. To really predict success, the computer needs to calculate the entire game tree from current move to final move, anticipating all the moves the human could make in response. For the simple game of Tic-Tac-Toe, thats possible situations, but for a more sophisticated game like Chess, that's situations! Fortunately, heuristics like "minimax" help prune down the tree of possibilities.
A tree diagram with three levels. The first level shows an empty Tic-Tac-Toe board. The second level shows three Tic-Tac-Toe boards, with an X in a different spot in each one. The third level shows 12 Tic-Tac-Toe boards, with an O in a spot in each one as well.
Facility location: Imagine a supermarket chain that has 20 locations in a state and wants to build 5 warehouses to deliver food to the supermarkets. Their goal is to minimize costs and maximize delivery speed. The facility location problem is finding the optimal location for those 5 warehouses. This same problem is faced by web apps that want to distribute their backend servers to respond to user requests quickly. Heuristics like "local search" help narrow down the array of possible locations.
A two-part diagram. The first part shows 20 blue dots distributed around a rectangular space. The second part shows 5 pink dots interspersed in those dots, with arrows connecting each pink dot to a blue dot.
All of these are combinatorial problems, where a computer would need to search an exponentially growing number of combinations to find the optimal answer. Combinatorial problems are quite common in the real world, so both companies and computer scientists alike care deeply about finding heuristics that will give the best approximate solutions.
Want to join the conversation?
- Great question!
We actually do have an infinite loop checker in our programming environment at Khan Academy. That's necessary since we evaluate the code in real time, and it is very easy to write an infinite loop on your way to writing a finite loop. An infinite loop typically freezes the browser tab, so we need to prevent that. Our loop checker wraps every loop in a timer. If the timer goes off before the loop is done, then we pop up the error. We have to stay on the safe side, so sometimes we might prevent code from finishing that would have with a few more seconds.
(The problem of knowing whether a program will ever stop is an "undecidable problem", described in the next article!)
You can try running the code on a webpage locally (save the code as tsp.html and open in a browser) and see how long it takes. You may need to restart your browser at some point, however!(18 votes)
- Thank you for a good explanation of what is heuristic. If this method was applied to finding a vaccine, which I am assuming it had to have been, since vaccines take from 5 to 15 years to find, and they came up with this in five months, how far away from an optimal solution is it? Is there a way to approximate this?(2 votes)
- Why is the TSP problem combinatorial (mentioned in the introduction of the problem), rather than permutatorial? Order matters in permutations, meaning that A-B-C-D would be different than D-C-B-A or A-C-B-D, right? Isn't this true in this problem?(1 vote)
- You're right. TSP depends on ordered combinations (permutations). The term "combinatorial" (I believe) comes from the branch of mathematics called "combinatorics", which deals with combinations and permutations. This is why you'll generally see "combinatorial" in the context of Computer Science problems(1 vote)