### Apache Spark Knapsack Approximation Algorithm in Python

The code shown below computes an approximation algorithm, greedy heuristic, for the 0-1 knapsack problem in Apache Spark. Having worked with parallel dynamic programming algorithms a good amount, wanted to see what this would look like in Spark.

The Github code repo. for the Knapsack approximation algorithms is here, and it includes a Scala solution. The work on a Java version is in progress at time of this writing.

Below we have the code that computes the solution that fits within the knapsack W for a set of items each with it's own weight and profit value. We look to maximize the final sum of selected items profits while not exceeding the total possible weight, W.

First we import some spark libraries into Python.

```# Knapsack 0-1 function weights, values and size-capacity.
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit
from pyspark.sql.functions import col
from pyspark.sql.functions import sum
```

Now define the function, which will take a Spark Dataframe with all the items, each with a name, weight and profit, and the global total weight.

```def knapsackApprox(knapsackDF, W):
'''
A simple greedy parallel implementation of 0-1 Knapsack algorithm.

Parameters
----------
knapsackDF : Spark Dataframe with knapsack data
sqlContext.createDataFrame(knapsackData, ['item', 'weights', 'values'])

W : float
Total weight allowed for knapsack.

Returns
-------
Dataframe
Dataframe with results.
'''
```

Create a DataFrame for each item, that is the ratio of profit over weight. Disregard all items weights that are larger than the global knapsack Weight W since they will not fit regardless. Sort, in Spark, all item rows by the ratio value, high to low.

```    # Add ratio of values / weights column.
ratioDF = (knapsackDF.withColumn("ratio", lit(knapsackDF.values / knapsackDF.weights))
.filter(col("weights") <= W)
.sort(col("ratio").desc())
)
```

Get a current Spark Session to feed to the sql command.

```    # Get the current Spark Session.
sc = SparkSession.builder.getOrCreate()
```

Now we take the sorted list of filtered items, with profit ratio, and compute a partial sums of weight. This adds each element's weight to the prior sum of weight, which will be how to select the items that are less than W. There are other ways to do this, used the sql with sum() below.

```    # An sql method to calculate the partial sums of the ratios.
ratioDF.registerTempTable("tempTable")
partialSumWeightsDF = sc.sql("""
SELECT
item,
weights,
values,
ratio,
sum(weights) OVER (ORDER BY ratio desc) as partSumWeights
FROM
tempTable
""")
```

Now that we have the partial sums of weights when the DataFrame of elements is sorted by profit weight ratio, we get only the items that have a partial sum of weights less than the knapsack size W, and this is the final solution set.

```    # Get the max number of items, less than or equal to W in Spark.
partialSumWeightsFilteredDF = (
partialSumWeightsDF.sort(col("ratio").desc())
.filter(col("partSumWeights") <= W)
)
```

Return the partialSumWeightsFilteredDF as the solution set.

```    # Return the solution elements with total values, weights and count.
return partialSumWeightsFilteredDF
```

For a non-approximate, optimal, solution's algorithm would need to be much differently, and Bulk Synchronous Parallel (BSP) solutions would be a good place to start looking.

Testing:

Now to use this function, below is some code to plug in values and test the knapsack approximation solution.

```# --------------------------------------------
# Test the Approximate Knapsack function test
# --------------------------------------------

# Pull in the knapsack library.

import random
from pyspark.sql import SparkSession

from knapsack import knapsack

# Create the SparkContext.
sc = SparkSession \
.builder \
.appName("Knapsack Approximation Algorithm Test") \
.getOrCreate()
```

Set the problem size to 10.

```# Knapsack problem size.
N = 10
```

Setup a Python list with some uniform random data for N items in

```# Setup sample data for knapsack.
knapsackData = [('item_' + str(k), random.uniform(1.0, 10.0), random.uniform(1.0, 10.0)) for k in range(N)]
```

Now create the knapsack items, with column names, item, weights and values, using the list KnapsackData.

```# Make a Dataframe with item(s), weight(s), and value(s) for the knapsack.
knapsackData = sc.createDataFrame(knapsackData, ['item', 'weights', 'values'])

# Display the original data
print "Original Data:"
print knapsackData.show()
print "\n"
```

The weight is random based on a range of the number of items N, and we could set it in a number of ways.

```# Create a random maximum weight
W = random.uniform(N * 1.3, N * 1.6)

# Show the weight.
print "W: "
print W
print "\n"
```

Now we just call the knapsack approximation method passing the knapsackData and total knapsack size W.

```# Call the knapsack greedy approximation function, with data and size 5.
k = knapsack.knapsackApprox(knapsackData, W)
```

That's it, so we show the results of the selected elements:

```# Show the results Dataframe.
print "Selected Elements:"
print k.show()
print "\n"
```

And add up the totals via RDD lambda function map() method to get the sum of values and weights. The sum of values is the total knapsack profit.

```# Show totals for selected elements of knapsack.
sumValues = k.rdd.map(lambda x: x["values"]).reduce(lambda x, y: x+y)
sumWeights = k.rdd.map(lambda x: x["weights"]).reduce(lambda x, y: x+y)
numResults = k.count()
print "Totals:"
print "Sum Values: ",  sumValues
print "Sum Weights: ",  sumWeights
print numResults
print "\n"

# ------------------------------------------
# End of Approximate Knapsack function test
# ------------------------------------------

```

That is all. The Scala version is very similar to this, and I did not yet test this for very large N to see how long it takes. The sorts may have performance implications as well as the method to take the partial sums. Tests showed it to work correctly.

The code here can also be found at: Spark.Packages.Org for the knapsack approximation.

### A way to Merge Columns of DataFrames in Spark with no Common Column Key

Made post at Databricks forum, thinking about how to take two DataFrames of the same number of rows and combine, merge, all columns into one DataFrame. This is straightforward, as we can use the  monotonically_increasing_id() function to assign unique IDs to each of the rows, the same for each Dataframe. It would be ideal to add extra rows which are null to the Dataframe with fewer rows so they match, although the code below does not do this. Once the IDs are added, a DataFrame join will merge all the columns into one Dataframe. # For two Dataframes that have the same number of rows, merge all columns, row by row. # Get the function monotonically_increasing_id so we can assign ids to each row, when the # Dataframes have the same number of rows. from pyspark.sql.functions import monotonically_increasing_id #Create some test data with 3 and 4 columns. df1 = sqlContext.createDataFrame([("foo", "bar","too","aaa"), ("bar&qu

### Stream PRAM: Research: Darrell Ulm @ Microsoft Research

Stream Pram is a paper co-written by Darrell Ulm, cat be accessed at Darrell Ulm Stream Pram Research Paper This is a paper about a multiple instruction stream style model of Parallel Random Access Memory (PRAM) parallel computation. The paper deals mostly with theoretical parallel computation as compared to applied parallel computing. Other links about the Stream Pram. Profile . Wordpress , Tumblr