Let’s pick a paperback and a hardback There are 4 x 6 possible choices Now pick any three of the remaining 8 books 8C3 = 56 This means there are 4 x 6 x 56 = 1344 selections of 5 books that will contain at least one paperback and one hardback
8C3 means picking 3 from 8 without worrying about order 8C3 = 8 x 7 x 6 / (3 x 2 x 1) = 56