Started using Kaggle a while back, profile link for Darrell Ulm

 Started using Kaggle a while back, profile link for Darrell Ulm : https://www.kaggle.com/darrellulm 

I’m someone who enjoys learning new things and working through interesting technical challenges. My background is in Computer Science, and over time I’ve worked with several programming languages such as Python, C, C++, and PHP/MySQL. I like exploring data driven problems, parallel computing, and the kinds of algorithms that make systems more efficient.

I have experience using tools like Apache Spark for large scale data processing, and I’ve been spending more time studying different areas of Artificial Intelligence, including neural networks and modern language models. I’m always trying to understand how these ideas work in practice and how they can be applied in useful ways.

I also have a long history with web development and content management systems. Drupal has been a major part of that work, and I’ve used versions from 4 through 10 on a variety of projects.

Overall, I’m here to keep improving my skills, learn from others, and contribute wherever I can.

Threads profile for Darrell Ulm

I've recently taken the step of joining 

Threads, Computer Science, Computing: Darrell Ulm (https://www.threads.com/@darrell_r_ulm ),  

as I embark on a journey to relearn and expand my existing knowledge in areas like artificial intelligence. My current focus involves delving deeper into the intricacies of AI, particularly exploring the fascinating world of Large Language Models (LLMs) and understanding how these sophisticated models are developed and utilized. I'm also revisiting the fundamentals of Neural Networks, the core building blocks that enable AI systems to learn and make predictions. Given the computational demands of these fields, I'm also keen on extending the principles and applications I previously learned in parallel processing, which plays a crucial role in efficiently handling the complex computations involved in AI.

Darrell R. Ulm

Python for Data Science

Looking at more resources online for Python for Data Science.

There are many good resources available.

Of course the main tools are: Numpy, Pandas, MathPlotLib, SkiKit-Learn has some amazing tools.

Kaggle for instance has Data Science contents, but good to install a local system like the Jupyter Notebook to speed things up as the Kaggle editor can lag and take some time to run on small data-sets.

The newer DataCamp has some neat tutorials on it and simple App to do daily exercises on your mobile device.

Here is the Python DataScience Handbook. Really useful.

A short tutorial: Learn Python for Data Science, a fun read.

A list of cool DataSci tutorials is here, and another how to get started with Python for DS.

Will add more later.

A way to Merge Columns of DataFrames in Spark with no Common Column Key

Made post at Databricks forum, thinking about how to take two DataFrames of the same number of rows and combine, merge, all columns into one DataFrame. This is straightforward, as we can use the monotonically_increasing_id() function to assign unique IDs to each of the rows, the same for each Dataframe. It would be ideal to add extra rows which are null to the Dataframe with fewer rows so they match, although the code below does not do this.

Once the IDs are added, a DataFrame join will merge all the columns into one Dataframe.

# For two Dataframes that have the same number of rows, merge all columns, row by row.

# Get the function monotonically_increasing_id so we can assign ids to each row, when the

# Dataframes have the same number of rows.

from pyspark.sql.functions import monotonically_increasing_id

#Create some test data with 3 and 4 columns.

df1 = sqlContext.createDataFrame([("foo", "bar","too","aaa"), ("bar", "bar","aaa","foo"), ("aaa", "bbb","ccc","ddd")], ("k", "K" ,"v" ,"V"))

df2 = sqlContext.createDataFrame([("aaa", "bbb","ddd"), ("www", "eee","rrr"), ("jjj", "rrr","www")], ("m", "M" ,"n"))

# Add increasing Ids, and they should be the same.

df1 = df1.withColumn("id", monotonically_increasing_id())

df2 = df2.withColumn("id", monotonically_increasing_id())

# Perform a join on the ids.

df3 = df2.join(df1, "id", "outer").drop("id")

df3.show()

Started a GitHub repository as look at code snippets for Apache Spark.