Deep Reinforcement Learning

How can we understand something that we have never seen before?

What impels us to think and act?

How do we learn?

Once upon a time, these questions were purely philosophical endeavors — and seemingly hopeless ones at that. Questions too far from reasonable application to be considered with any meaningful consequence.

However, amidst the ever-rising tide of technological innovation, these questions have reemerged from the depths of philosophy to be examined in closer detail by new fields, seemingly far removed from their philosophical origins.

Now, these questions are no longer things merely to ponder, for they have now grown to immense consequence in the young yet highly promising field of Deep Reinforcement Learning.

First, some background: the field of Deep Reinforcement Learning is a natural extension of Deep Learning, which is a natural extension of machine learning, which is a natural extension of statistical inference.

All clear?

If not, then don’t worry… because you are about to be a lot more confused anyways.

But before we get confused by Reinforcement Learning, let’s start with its older brother – Deep Learning.

Deep Learning is a pretty name for the type of machine learning that statistical theory hasn’t yet been able to catch up to. Made possible by a progressive increase in general computational power and public dataset size, Deep Learning is essentially the application of basic machine learning principles to more complicated functions – often called Neural Nets. The complexity of these functions allows them to map the data that they take in to a higher dimensional space that accounts for relationships between elements of the incoming data. During computation, various tricks are used to achieve useful properties, such as the nonlinearity between hidden layers that allows for the approximation of more complex functions, or dropout and entropy reduction in training to approach desirable statistical properties. This mess of engineering tricks fastened onto the underlying structure provided by traditional machine learning has allowed Neural Nets to solve problems that have previously seemed too difficult for traditional statistics.

For example, by seeking to encode the underlying information in a dataset of images and paintings, combinations of Neural Networks called GAN’s can create mesmerizing, psychedelic artwork.

In Deep Learning, the goal is often a simple form of classification: mapping a set of data to what it means. In essence, we want to be able to say that an image of a duck is a duck and that an image of a cat is a cat. From a human perspective, this task seems trivial, yet we can hardly begin to describe a solution. What about this arrangement of data causes us to say that the image it represents truly is a cat?

If, for instance, you laid out the list of pixels that represent the image, you could go through them one by one and verify that every single pixel does not necessarily make this picture a picture of a cat – and the same thing can be done with every set of two pixels. Nevertheless, and however indescribable by mere language, Deep Learning is able to capture underlying patterns in images, and match the image to its meaning with shocking, sometimes superhuman accuracy. Every day, new applications are imagined for this technology, weather it be in the realm of art, medicine, or business.

Reinforcement learning is a fascinating extension to Deep Learning, relying essentially on modification to traditional classification that is simple in premise: instead of seeking the proper name for an image, we seek proper action for an image – as defined by the propensity of the action to maximize total reward. This is most simply interpreted in the context of games, then extrapolated into more consequential scenarios, such as the development of self-driving cars.

The goal in both levels of this problem, in games and in the real world, involves recognizing that a certain action is likely to achieve a desired outcome given an observed state. In essence, this is a variant of the classic classification problem, but clearly, more complexities are introduced by the ability to alter environmental state with action. Nevertheless, much of the existing Deep Learning architecture can remain, and we can simply change the loss function to seek to maximize total reward gained over a task, rather than minimize error in prediction of the proper image label.

As researchers seek to improve performance in their Reinforcement Learning algorithms, they are, whether they know it or not, seeking answers to the philosophical questions we started with. And as we are able to create algorithms that are able to learn more efficiently, we are offered a glimpse at the answers to these questions and insight into the inner workings of the mind in the process.

How can we understand something that we have never seen before?

What impels us to think and act?

How do we learn?

1. https://adepratt.weebly.com/student-work.html

2. https://experiments.withgoogle.com/collection/ai

3. https://sigmoidal.io/dl-computer-vision-beyond-classification/

EUREKA  Posted by 25 Jul 2019 / 0 comments

The Budding Life of a Researcher

Growing up, I never had a real affinity towards plants. I had no green thumb whatsoever, and neither did anyone in my family. Any attempts made in growing plants whether it be in school projects or for myself usually resulted with a unwatered withered corpse. It was with this mindset that I approached my now mentor Jason Johns in the Hodges Lab. At the time, I saw this as an opportunity to reconnect with nature. With trepidation, I now stepped forward into a project that would have me help to grow ~350 plants known as Aquilegia Jonesii. In hopes of finding the genetic basis of alpine adapted dwarfism in the species. The purpose of this is to one day avoid the problem of lodging. Lodging is something that occurs when a plant grows too tall and tips over itself. As a result, the plant dies and a bunch of resources have now been wasted in helping the plant grow to adulthood.

Now, the big hurdle is to extract the DNA from the plants and to sequence every single one. To do this, I take some leaves from the plant, preferably younger, since it is easier to break down plant tissue in younger plants. Afterwards, I place the leaves in a tube along with some metal balls and freeze them using liquid nitrogen. I break down the tissue with a machine called a bead beater. Then, I add a solution and centrifuge it to place all the plant material at the bottom and suck up the liquid goodies left up top. Unfortunately, this includes DNA, proteins, and others. So, to isolate DNA I add magnetic beads and proceed to hit the solution with a mix of chemical washes in hopes of isolating the DNA. Finally, I add water along with the magnetic beads, which have the negatively charged DNA attached. The water having a higher charge than the magnetic beads, pulls the DNA. Finally, we have our isolated DNA. Afterwards, I take a small amount of DNA and i quantify the amount of DNA in ng per mL of solution. To do this I add a solution to the DNA and take it to a machine which spits out the concentration. This has mainly consisted of my lab work thus far. However, a majority of my time has been spent caring for my plants. I have had to clean them up, re-pot them, and tag them. While this may not seem like a lto this has easily been what I have spent most of my time on. I have washed around ~300 pots and spent hours carefully transporting each plant into bigger pots. I make it a habit to go to the green house once a day to check on them and just to admire their beauty. If I am being completely honest this has been one of the most satisfying parts of my research.

Caring for these plants is truly something special and I would never give it up. Recently, one of my plants have flowered. Knowing that I have aided in their development also makes me feel almost accountable for them. I have this connection to every single one of them, that brings me a peculiar warmth seeing them grow larger everyday. One day, I hope to bloom as beautifully as this plant, and become a full fledged researcher. Until then, I continue to happily work in the Hodges’ lab.

EUREKA  Posted by 18 Jul 2019 / 0 comments

Wait a minute… Don’t Lasers Heat Things Up? A Laser Cooling Primer for the Uninitiated

Classical mechanics is the study of macroscopic objects and how they react to forces, and it works well. Really well. But when it comes to small particles, the same rules don’t apply. Quantum mechanics is the underlying theory that all particles can behave like waves, and vice versa.

We can’t see quantum effects in our day to day lives because things are too hot and heavy. Even though all particles behave likes waves, the amalgamation of the sheer number of them that it takes to make anything macroscopic cancels out any ambiguity. To see the effects of quantum theory in the lab, we have to cool lithium atoms down to just microKelvin away from absolute zero and pack them all together, where they will begin to show quantum interactions.

Unfortunately, I can’t buy ultra-cold lithium at Costco (you need a gold membership), so how do we make it ourselves? Cue the lasers. Laser cooling takes advantage of the interaction between atoms and light to slow them down. Since temperature is a measure of average kinetic energy, they are now ‘colder’.

“I hate to break it to you Max, but you’ve really lost it this time. We’re talking about lasers here. LASERS. Lasers heat things up. I saw this video of a laser blowing up a balloon.”

Ok fine. You got me. I saw that video too and it’s cool. Lasers usually heat things up. They are good at doing this because of how tightly packed the energy in a laser beam is. To put it in perspective, the average person probably puts out about 250 Watts while running. A horsepower is almost 800 Watts, and engines can put out many hundreds of horsepower. So, it may be a bit of a shock that even a 0.2 Watt laser beam is actually pretty powerful, sometimes even enough to permanently blind you. Yikes. Even with less than 1/1000th of the power that you can put out just by running, a laser can do some serious damage.

This is partially due to the fact that laser light is coherent, effectively meaning it is all the same color. However, this combination of homogeneity and power ‘density’ actually also makes it perfect to cool atoms down. But how do you do it?

The answer is very clever: using the doppler shift. Most people have experienced an ambulance siren dramatically changing pitch as soon as it passes you. This is because the motion of the car changes the spacing of the sound waves travelling towards you, and so they hit your ear more or less frequently if the car is moving towards or away from you, respectively. We interpret this increased or decreased frequency as a change in pitch. Atoms ‘see’ light in exactly the same way. Light also has wavelike properties, and the motion of the atom will change the perceived frequency of the light depending on its velocity (see picture).

Another important piece of the puzzle that we need to use is the fact that a given atom can absorb or emit light only at specific frequencies. This is another quantum phenomenon, and although it is strange, it is true. A frequency of light corresponds to a color, so think of these specific frequencies as a specific color of light. For lithium, it is 671 nm, which is a deep red.

We’ve got the ingredients. How do we get the cool? Well, throw out your Ray-Bans. Imagine sending out light that has a frequency that is a little less than the one we need for the transition. If we send this beam into a cloud of gaseous atoms, then only the atoms that are moving towards the beam will see an increase in frequency, and therefore the right color light for the transition. The other atoms will see frequencies that are too low.

Even though photons do not have any mass, they do have momentum. When the atom moving towards the photon absorbs it, it gets a kick back in the opposite direction, and it is now slower! Over many cycles, we kick more and more of the atoms until they are cool enough to confine.

Although this is only the first step of many to get temperatures that are low enough to explore quantum interactions, it is amazing that massless light can cool atoms with real mass. Under the right conditions, lasers aren’t just a way to pop balloons and remove tattoos that you thought you wouldn’t regret.

EUREKA  Posted by 25 Oct 2018 / 0 comments

Talking to a Professor 101

Whenever I tell someone, whether it be a friend of mine, my family, or even a faculty member, that I am researching methamphetamine addiction in mice, one of the first questions I get asked is usually, “where do they get the meth from?” This question kind of bothers me. I rarely get asked specific questions about my research, and the opportunity to share the knowledge I have gained during my internship is frustratingly uncommon. To be quite honest, I don’t even have a good answer to this question. I understand that meth is a scary and almost alien substance to many people that I interact with, so the fascination with the drug itself is not unexpected. However, having the frustrating experience of people focusing on the wrong things when I am trying to convey interesting things to them has really opened my eyes to how to ask questions about research. I have begun to realize that asking research questions that can be answered with a single sentence reveal almost nothing in terms of how much other interesting information they can share. So, I have compiled a list of tips from my own experience that can help you ask deep and thought-provoking questions, as I believe that asking good questions is a major part of being a good researcher.

Step 0: Research the Research

Oftentimes, you will know beforehand if you are going to talk to a professor about their research. I have found that their websites and published article lists are often outdated; however, gathering any information you can about their area of expertise can be a tremendous help in asking great questions and getting interested in their research.

Step 1: Be confident

Professors can be quite intimidating. They are incredibly knowledgeable, experienced, and intelligent people, and meeting a professor whose work you admire can be quite daunting. I remember my first time meeting a professor one-on-one at UCSB, I was extremely nervous. I walked into his office, sweaty and totally unprepared. The first question he asked me was, “So do you have any questions for me?” It took me about fifteen seconds to stammer out my first words, and by the time we were done with the interview I thought I had completely blown it. I ended up being offered a position as a research assistant, and it made me realize that professors are people too. Professors do not always need to hear deep, thoughtful questions about their research, and they will understand that you will make mistakes. Just focus on showing your curiosity and personality, and the knowledge and critical questions will come later.

Step 2: Make connections

As a wise man once said, “Your network is your net worth.” Don’t be afraid to ask personal questions, and always remember to send a thank you email after you talk with a professor. Actions such as these will make them remember you, and there are countless opportunities out there that you will not find on your own. Having a good network of people can open many doors and help you achieve things you could never have done on your own. Beyond research, professors are amazing mentors and advice-givers, and having someone as knowledgeable as a professor to ask questions to can literally change your life.

Step 3: Have fun!

Getting a position in a research lab is never life or death. In fact, you probably won’t even really enjoy your first research experience. However, making mistakes and learning from them is a huge part of life in general, and research is no different. Don’t be afraid to try something scary! Email that professor, ask the “dumb” questions, and most of all, have fun. Ask interesting questions you want to know the answers to, and have a good time doing it, because your energy will be contagious!

Hopefully, this short guide will help you the next time you talk to a professor, and know that your research experience will teach you an incredible amount about yourself and what you enjoy. As a final piece of advice, try to make the most of every moment, because that is what life is all about.

EUREKA  Posted by 23 Oct 2018 / 0 comments

A lab meeting is less of an audition, and more of a casual conversation

When entering the world of research for the first time, there are aspects that might seem scary at first, and interacting and working in your first lab meeting is no exception. As I share my story, hopefully it can leave you more confident to handle your meeting with poise.

During a seemingly regular Sunday noon, I had sent a nerve-wracking email to a faculty advisor that I had wished to work with at the time (and blessed to continue working with today). About an hour later, he sent a response that would send me bursting out of my chair, while numbing my body at the same time due to the fear that was building up in my body. At the time I was both excited, yet nervous of the potential embarrassment that I might showcase during the meeting.

Fast forward to the next day, and after hours of not being able to sleep, practicing my slurred speech, and trying to calm my hand’s involuntary shaking, I was right at the front door of the lab trailer and decided to knock with mild hesitation. After a second or so, I was greeted by this kind and amazing lady who I would soon realize to be a senior scientist in the lab. She would greet me and ask who I was, and in response, my lips would rumble, barely uttering my name. Recognizing my name, she told me that the lab was expecting me and showed me to the meeting.

There it was: A room full of postdocs, graduate students, a few undergrads, and of course, the faculty advisory himself, all sat around this huge oval table filled with laptops and data from their own respective projects. As I overheard each person talk about their own complex work, my brain would sink, doubting myself for a second, and questioning whether I deserve to be in this room in the first place.  I was afraid to make a fool of myself by uttering some information that would have conveyed my lack of expertise in their research, let alone physics itself.

As the meeting started, I was told to introduce myself. From this moment on, I will give a couple pieces of advice that come from my experience after introducing myself:

  1. These scientists are not what they seem on paper and on the big screen: Throughout the course of the meeting, these scientists– and especially the faculty advisor– would constantly make jokes that were very funny and, of course, very science-y. I thought it was hilarious and a bit caught off guard as I thought these scientists to be very stern and always serious, but it was always informal and very chill. It was also very funny to note that that senior scientist had “warned” me that they would be making jokes all the time as that is just personality of the atmosphere during their lab meetings. Of course, not all labs are like this, but the point here is that these researchers are very human and not the robots that they are stereotyped to be.
  2. It is ok to not know, and it is also ok to not explain using the most complex vocabulary: When going into this meeting the one question that I am certain all lab meetings will ask on the first day is, “is there anything specific that you would like to do in our research?” I am also certain that they will not ask you that until the end of the meeting. The reason why I say this is because they understand that you are not an expert in what they are doing, nor are you obligated to be an expert on the first day. They try to ask you in the end because they want you to listen to what they are doing and get comfortable with their research. Also, they would encourage to explain who you are and what you want as simply as possible. As I said, they are human too, not a walking dictionary, and they also want to understand how you will work in their team as well and talking to them in a complicated way might leave a bad imprint on how you work with others.

 As I left the meeting, many of the stigmas that would make these researchers intimating had been torn down, and I would look forward to the next meeting that would come. They welcomed me with open arms, as I am sure will also happen to you in your first meeting. Like I said, not all labs are the same, But I hope this story will give you a positive outlook in your experience. you may be anxious at first, but at the end, know that it is all worth it.

EUREKA  Posted by 06 Sep 2018 / 0 comments