The intersection of two current events have born fruit:

- I’m reading Christopher Bishop’s legendary book: “Pattern Recognition and Machine Learning”, (freely available as PDF from the book’s website!)
- I recently discovered Shiny for R, which allows building interactive web apps in R.

To try out Shiny, I created an interactive visualization for Kullback-Leibler divergence (or KL Divergence). Right now, it only supports two univariate Gaussians, which should be sufficient to build some intuition.

If you like it, let me know! If it turns out to be popular, I might add more features, or create similar visualizations for other concepts!

Consider an unknown probability distribution , which we’re trying to approximate with probability distribution , then

can informally be interpreted as the amount of information being lost by approximating with . As you might imagine, this has several applications in Machine Learning. A recurring pattern is to fit parameters to a model by minimizing an approximation of (ie, making “as similar” to as possible). This blog post elaborates in a fun and informative way. If you have never heard about KL divergence before, Bishop provides a more formal (but still easy to understand) introduction in Section 1.6 of PRML.

Using the visualization tool, find out (or verify) the answer to the following questions:

- Is ? Always? Never?
- When is ?
- Let and . Which is larger: ? Why?
- Is ever negative? When, or why not?

In this post, we will prove that the decision version of the set-covering problem is NP-complete, using a reduction from the vertex covering problem (which is NP-complete). This is Exercise 35.3-2 in CLRS3[1]. We will follow the template given in an earlier post.

We will use the definition for a set cover given on page 1118 in CLRS3[1], also given on Wikipedia. Since we’re concerned with the decision version, we seek to answer whether a set cover exists for (the set containing the elements to be covered), **containing a maximum of** **sets**

Let be the certificate. We assume that each uniquely covers at least one , so:

(1)

This gives us that we can verify that and that covers in , which proves the problem is in NP.

We will show that the decision version of the set-covering problem is NP-hard by showing:

(2)

The vertex cover of an undirected graph is a subset , such that each is adjacent to at least one [1]. Its decision version is to determine whether a vertex cover of at most size exists for .

We can reduce any instance of decision-vertex-cover for to an instance of decision-set-cover by letting and letting the family of sets , such that contains all edges in adjacent to . We can perform this reduction in , which is in polynomial time.

The example image above shows a vertex cover of size. Making the reduction for this example gives:

(3)

We will now prove for any vertex cover for and a set cover for sets and constructed in accordance with the reduction above:

(4)

Since , and we know is a vertex cover for , we can find for by selecting from all sets corresponding to .

Similarily, if exists, by the way we defined the reduction, we know it covers . This means We can find by taking the vertices corresponding to .

Due to the 1:1 correspondence of vertices in and sets in :

- [1]T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein,
*Introduction to Algorithms*. MIT Press, 2009.

In this post, we will prove that the set-partition problem is NP-complete using a reduction from the subset sum problem (which is NP-complete[1]). This is Exercise 34.5-5 in CLRS3[1]. We will follow the template given in an earlier post.

Given a set of numbers, determine whether can be partitioned into and , such that:

(1)

Let be the certificate. We can verify the certificate in linear time by summing the elements in each set, and comparing it

We will show that the set-partition problem is NP-hard by showing:

(2)

We will prove that the existence of with sum can be partitioned into as described by the set-partition problem ( being the sum of ). We note that can be constructed in by first summing the elements in and then adding to S, which proves our reduction is performed in polynomial time.

We will now prove both directions of the biconditional separately.

Suppose exists. Let be , i.e the “out”-elements. Let be the sum of . Let and be the sum of . We can now make the following observations:

(3)

This proves:

(4)

Suppose an equal-sum partitioning of exists. The sum of each partition must then be . We consider the partition containing the element to be . Let . We can observe . The sum of is . In other words: is a subset of with sum , i.e. a witness to

- [1]T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein,
*Introduction to Algorithms*. MIT Press, 2009.

In this post, we will prove that 0-1 integer programming is NP-complete using a reduction from 3-CNF-SAT (which is NP-complete[1]). We will follow the template given in an earlier post.

The decision problem for 0-1 integer programming is formulated as follows[2]:

Given an integer matrix and an integer -vector , determine whether there exists an integer -vector with elements in , such that:

(1)

Note: For vectors **all** elements in are element-wise less than or equal to all elements in .

Let the certificate be vector . can be computed in . Finally, elementwise comparison of and can be performed in to determine if . This proves a certificate for the problem can be verified in polynomial time.

We will show that the problem is NP-hard by showing:

(2)

For 3-CNF formula , containing variables and clauses, construct matrix such that:

(3)

Also, construct -vector such that:

(4)

In other words: the number of negated literals in clause .

We observe that and can be constructed in , proving that the reduction can be done in polynomial time.

We will now prove that the existence of -vector is satisfiable. Consider vector to represent an assignment of the variables in , where:

(5)

Let . From Equations 1, 3 and 4:

(6)

From Equation 6 and the definition of the operator

(7)

(8)

Our reduced instance looks as follows (zeroes omitted):

(9)

We can see that satisfies and that

(10)

- [1]R. M. Karp, “Reducibility among Combinatorial Problems,” in
*Complexity of Computer Computations*, Springer US, 1972, pp. 85–103. - [2]T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein,
*Introduction to Algorithms*. MIT Press, 2009.

This is the first post in a series of posts where I will attempt to give visual, easy to understand, proofs^{*} of NP-completeness for a selection of decision problems. In this post, I will give a “template” which can be used (and will be used for the proofs I post).

I will assume the reader is familiar with decision problems and the complexity classes P, NP and NP-hard. If not, please see the Wikipedia article on NP-completeness, or better yet, take a look in Chapter 34 of [1].

For decision problem :

(1)

In other words, we will need to show that is in NP, and in NP-hard.

For a decision problem to be in NP, it is sufficient to show that a **certificate** (an encoding of a solution to the problem) can be **verified** to represent a solution to the decision problem in polynomial time.

- Certificate for instance of can be verified in polynomial time.

For a decision problem to be NP-hard, it is sufficient to show that any instance of another decision problem , which is **already known to be NP-hard**, can be reduced to an instance of , such that the reduction of into can be performed in polynomial time. We express the existence of such a reduction as:

(2)

In other words: is at least as hard as

- Any instance of can be reduced to an instance of in polynomial time; and
- Decision is “Yes” is “Yes”

Using knowledge about other problems already proven to be NP-complete, and some creativity, we are now ready to attempt proving NP-completeness.

^{*}The series is a spin-off of some work I did in order to prepare for an exam. I will attempt to be as rigorous and correct as I can, within reason. The main purpose of the series is to give intuition, which the reader can then use to develop rigor, if desired.

- [1]T. H. Cormen, C. E. Leiserson, R. L. Rivest, and S. Clifford,
*Introduction to Algorithms*. MIT Press, 2009.