The code shown below computes an approximation algorithm, greedy heuristic, for the 0-1 knapsack problem in Apache Spark. Having worked with parallel dynamic programming algorithms a good amount, wanted to see what this would look like in Spark.
The Github code repo. for the
Knapsack approximation algorithms is here, and it includes a Scala solution. The work on a Java version is in progress at time of this writing.
Below we have the code that computes the solution that fits within the knapsack W for a set of items each with it's own weight and profit value. We look to maximize the final sum of selected items profits while not exceeding the total possible weight, W.
First we import some spark libraries into Python.
# Knapsack 0-1 function weights, values and size-capacity.
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit
from pyspark.sql.functions import col
from pyspark.sql.functions import sum
Now define the function, which will take a Spark Dataframe with all the items, each with a name, weight and profit, and the global total weight.
def knapsackApprox(knapsackDF, W):
'''
A simple greedy parallel implementation of 0-1 Knapsack algorithm.
Parameters
----------
knapsackDF : Spark Dataframe with knapsack data
sqlContext.createDataFrame(knapsackData, ['item', 'weights', 'values'])
W : float
Total weight allowed for knapsack.
Returns
-------
Dataframe
Dataframe with results.
'''
Create a DataFrame for each item, that is the ratio of profit over weight. Disregard all items weights that are larger than the global knapsack Weight W since they will not fit regardless. Sort, in Spark, all item rows by the ratio value, high to low.
# Add ratio of values / weights column.
ratioDF = (knapsackDF.withColumn("ratio", lit(knapsackDF.values / knapsackDF.weights))
.filter(col("weights") <= W)
.sort(col("ratio").desc())
)
Get a current Spark Session to feed to the sql command.
# Get the current Spark Session.
sc = SparkSession.builder.getOrCreate()
Now we take the sorted list of filtered items, with profit ratio, and compute a partial sums of weight. This adds each element's weight to the prior sum of weight, which will be how to select the items that are less than W. There are other ways to do this, used the sql with sum() below.
# An sql method to calculate the partial sums of the ratios.
ratioDF.registerTempTable("tempTable")
partialSumWeightsDF = sc.sql("""
SELECT
item,
weights,
values,
ratio,
sum(weights) OVER (ORDER BY ratio desc) as partSumWeights
FROM
tempTable
""")
Now that we have the partial sums of weights when the DataFrame of elements is sorted by profit weight ratio, we get only the items that have a partial sum of weights less than the knapsack size W, and this is the final solution set.
# Get the max number of items, less than or equal to W in Spark.
partialSumWeightsFilteredDF = (
partialSumWeightsDF.sort(col("ratio").desc())
.filter(col("partSumWeights") <= W)
)
Return the partialSumWeightsFilteredDF as the solution set.
# Return the solution elements with total values, weights and count.
return partialSumWeightsFilteredDF
For a non-approximate, optimal, solution's algorithm would need to be much differently, and Bulk Synchronous Parallel (BSP) solutions would be a good place to start looking.
Testing:
Now to use this function, below is some code to plug in values and test the knapsack approximation solution.
# --------------------------------------------
# Test the Approximate Knapsack function test
# --------------------------------------------
# Pull in the knapsack library.
import random
from pyspark.sql import SparkSession
from knapsack import knapsack
# Create the SparkContext.
sc = SparkSession \
.builder \
.appName("Knapsack Approximation Algorithm Test") \
.getOrCreate()
Set the problem size to 10.
# Knapsack problem size.
N = 10
Setup a Python list with some uniform random data for N items in
# Setup sample data for knapsack.
knapsackData = [('item_' + str(k), random.uniform(1.0, 10.0), random.uniform(1.0, 10.0)) for k in range(N)]
Now create the knapsack items, with column names, item, weights and values, using the list KnapsackData.
# Make a Dataframe with item(s), weight(s), and value(s) for the knapsack.
knapsackData = sc.createDataFrame(knapsackData, ['item', 'weights', 'values'])
# Display the original data
print "Original Data:"
print knapsackData.show()
print "\n"
The weight is random based on a range of the number of items N, and we could set it in a number of ways.
# Create a random maximum weight
W = random.uniform(N * 1.3, N * 1.6)
# Show the weight.
print "W: "
print W
print "\n"
Now we just call the knapsack approximation method passing the knapsackData and total knapsack size W.
# Call the knapsack greedy approximation function, with data and size 5.
k = knapsack.knapsackApprox(knapsackData, W)
That's it, so we show the results of the selected elements:
# Show the results Dataframe.
print "Selected Elements:"
print k.show()
print "\n"
And add up the totals via RDD lambda function map() method to get the sum of values and weights. The sum of values is the total knapsack profit.
# Show totals for selected elements of knapsack.
sumValues = k.rdd.map(lambda x: x["values"]).reduce(lambda x, y: x+y)
sumWeights = k.rdd.map(lambda x: x["weights"]).reduce(lambda x, y: x+y)
numResults = k.count()
print "Totals:"
print "Sum Values: ", sumValues
print "Sum Weights: ", sumWeights
print numResults
print "\n"
# ------------------------------------------
# End of Approximate Knapsack function test
# ------------------------------------------
That is all. The Scala version is very similar to this, and I did not yet test this for very large N to see how long it takes. The sorts may have performance implications as well as the method to take the partial sums. Tests showed it to work correctly.
The code here can also be found at:
Spark.Packages.Org for the knapsack approximation.