Week 3: ReLUctant to begin

Mar 18, 2019

The 3rd week was a little uneventful, as it was largely reading without any visible data coming out of it.

For the week, I continued to learn more about the different parts of a machine learning algorithm. The first day was spent tidying up my overfitting and underfitting algorithms from last week. The other 2 days I was able to spend this week was spent investigating the neural network side, something which I was a little more unfamiliar with. This included the types of neural network layers like ReLu and SoftMax.

ReLU (Rectified Linear Unit) is a fairly simple idea. Take all the weights which end up negative, and turn them into 0. Okay…that seems a little…simple. The role of this is to introduce non-linearity to something that’s up to this point, largely been dealing with linear only functions. As I’ve said in a previous blog post, data is rarely given in a straight linear function. At some point or another, the non-linear data must be met with a non-linear network layer to eventually turn into a linear output. ReLU just happens to be the most efficient way to begin this transition (the reason being beyond the scope of my application focused project). ReLU is what is called an activation function.

Softmax is another type of activation function, which takes all the data it receives from previous layers and gives out a final output which assigns probabilities to each of its options. In other words, it says “this item has a 30% chance of being a bag, 15% chance of being a blanket, 2% chance of being a table, etc.”

That just about sums up what I learned this week. Next week is likely to transition from learning to working on/with neural networks. Hope you also learned a bit today

2 Replies to “Week 3: ReLUctant to begin”

  1. Alex Y. says:

    Very interesting stuff. It seems like you are focusing more on the algorithmic side of machine learning. I’ve done some machine learning on robotics over the summer and it’s really interesting discovering more efficient algorithms. Looking forward to more posts and learn about your amazing project.

  2. Aarushi N. says:

    I met someone who went to UIUC and he is familiar with neural networking and machine learning, if you want I could give you his information and you could talk to him about it?

Leave a Reply

Your email address will not be published.