Georgia Tech’s Deep Learning Course

I was already signed up for Computer Vision when word came that there would be a new Deep Learning class next semester. I quickly got onto the waitlist and was near the front of the line to get into the 150 person class. But I’m so glad I was successful because Deep Learning was the highlight of my OMSCS program.

I am specializing in Machine Learning for my OMSCS degree and this was my final class that I needed to complete my specialization. By the time Deep Learning became available, I’d already completed Machine Learning for Trading, Machine Learning, and Reinforcement Learning. So I knew machine learning fairly well by the time I started. This turned out to not help as much as I would have thought, though I’m glad I was at least already strongly familiar with the necessary vocabulary.

How Tough Is the Class?

This is a tough class in the sense that it is a lot of work and quite a bit of hard math. But because the grades come mostly through projects that you can turn in multiple times until you get full credit, getting an A in this class is more a matter of choosing to work hard than passing crazy tests. Not that this class doesn’t have crazy tests, or rather quizzes. The grade breakdown is as follows:

  • 60% Programming Assignments
  • 20% Final Project
  • 15% Quizzes
  • 5% Participation

I ended the class with a whopping 96.7%. The highest score I’ve ever had in a class other than AI4R where all the points came from projects and so I got a 100%. An A is a standard 90% or above, by the way.

I also worked my tail off. In terms of amount of work, this was the single hardest class I’d ever taken, including compared to Graduate Algorithms. But unlike Graduate Algorithms, I never worried that I’d get a bad grade or fail.

Calculus Required

I’ve had other classes with pre-requisites of calculus. The Machine Learning class stated it was a requirement. The fact that I didn’t actually remember any Calculus at all (its been over 30 years since I last had a math class!) didn’t slow me down at all in the Machine Learning class.

Deep Learning is a very different beast. You will need to not only know Calculus but Multivariate Linear Calculus to even get through your first project. The second project has a substantial calculus requirement as well, though not as bad as the first project. Calculus also shows up on the first few quizzes and dominates the first quiz.

Oh, by the way, Multivariate Linear Calculus is pretty much not taught in schools anywhere. So you won’t be alone in not knowing it in advance.

This part is terrifying. I had to learn Calculus again in 2 weeks. But I managed to pick it up and run with it and I got through this part. I am really glad I did, because I know how to do back propagation manually now. It’s an amazing algorithm that I’d never really fully understand without having to have learned to do it by hand first.

After the first two projects, the calculus requirements vanish as you start to use PyTorch which does the math for you. After that, the class gets less terrifying but not necessarily less work.

Many people feel like they just can’t do math. This just isn’t true. I’m not all that particularly good at math. Most of us aren’t. But you probably can learn enough calculus to get through this class. It might make sense, however, to brush up before the class starts over the summer of something rather than trying to learn it quickly like I did.

Python and PyTorch

This is a class where you probably don’t want to be learning Python once you start the class. However, there are so many python classes in OMSCS, odds are you’ve already learned it by now anyhow, right? You don’t need to know PyTorch before starting the class, though it wouldn’t hurt. Start with this book as a beginner’s guide and it will boost your confidence.

Should You Take Machine Learning First?

Should you take the Machine Learning class first? Probably. But I’d not say it’s a hard requirement. As previously mentioned, it helps a lot with the vocabulary. But Deep Learning is significantly different from other forms of Machine Learning and has its own unique vocabulary to boot. So I’d say this falls into the ‘good idea but not required’ category. If you dislike Machine Learning or find it overwhelming, then this class is probably a bad idea.

Oh, and if you plan to take multiple Machine Learning courses first, I’d highly recommend the exact order I took them: Machine Learning for Trading -> Machine Learning -> Reinforcement Learning->Deep Learning. You can swap RL and ML if you want. But ML4T is a great introduction to the Machine Learning course that even has the same textbook. Or at least that was true back when I took it.

The Dreaded Quizzes

There is a weekly quiz in this class and they are harsh and unforgiving. I scored at 81.57% on the quizzes, but I had in the 60% range most of the way through until the end when they finally dropped my two lowest scores. You’ll probably get to drop only one lowest score. I got lucky due to Covid. But don’t count on it.

The first quiz was particularly bad – see criticisms section. And there were a few quizzes I actually did well on, where ‘well’ means like in the 80s. I think I was fairly average when it came to quizzes. This despite studying hours for each one every single week while also worrying about the current project.

Let me be blunt, the quizzes in this class are like mini-midterms straight out of the Machine Learning course except you have one per week. They are trying to trick you and you feel like you’re playing ‘stump the monkey’ where you are the monkey. I think the instructors got the hint at some point that this was kind of silly and they started to give stronger hints on what to study for the quiz. This helped a lot. But sometimes they literally lied on their hints – I don’t think on purpose – and this made the whole thing sort of frustrating at times. It was pretty common to hear people say “Ok, I’ve got to go fail the quiz, I’ll be right back.”

Here is the good news: at only 15% of your score, the quizzes just don’t matter that much overall. They are good ways to learn the material, however, so I recommend studying them and taking them seriously enough. But don’t sweat them too much. And certainly don’t let them scare you away from the class.

Participation Scores

The “participation” score comes primarily from reading some really cool Deep Learning papers and doing a write-up. Some of these papers are seminal and must-reads. You then make comments on other student’s write-ups. This is not particularly hard and it was easy to get full credit. It did take quite a bit of time to read the long technical paper, but I suspect you could always skim it instead of reading it and still do well. But I was so enthralled by the subject matter I read every single one in-depth.

The Assignments

The assignments are where this class really shines. You get to learn some pretty amazing things in this class. For the first assignment, we built a simple Feed Forward Deep Learning algorithm using only Numpy and doing back prop by hand. It was scary hard at first until your brain started to figure it out. Then it wasn’t so bad. Though I didn’t realize that until days before I had to turn it in. I came away feeling like I really understood back prop – probably the single most important algorithm in machine learning today.

For the second assignment, you implement a Convolutional Neural Network (CNN) both by hand and with PyTorch. CNNs are amazing when used with graphics. This was another great assignment.

The third assignment was the single best one. You get to do stuff like create adversarial examples, implement a way to see what features mattered, and reverse engineer a network to generate images. Exciting stuff.

The fourth assignment, unfortunately, was bad. Unlike the other 3 assignments, this was being introduced for the first time and it had a lot of bugs. So this might not be so bad for you like it was for us.  But the assignment was using PyTorch to implement Recurrent Neural Networks (RNNs) including Transformers – the network type underlying everything cool imaginable in Deep Learning that isn’t graphics. GPT-3 uses Transformers, for example.

The assignments cover all the basic network types you need to know to be proficient in Deep Learning. The quizzes cover a lot broader material, so again, study for the quizzes even if you don’t really need the points.

The Group Project

The group project consists of picking something you want to do, finding 2 or 3 others that agree, and then working together on that assignment. It’s pretty open-ended and there are tons of suggestions and even Facebook projects (complete with real-life Facebook engineer tutors) to work on. We picked a supposed ‘easy’ project where we took images of food and converted them to a recipe. We failed miserably but still got a good grade because you get graded on writing up what you tried to do, not on actually getting a good model.

Unfortunately, everything you hate about group projects applies here. I had a hard time getting a team together and then at least one member of the team was just not that interested in participating much.  Even those of us that did participate had a tough time finding times we could do work together and it was hard to split up assignments in a fair way. But we got through it and it was even fun at times.

You might want to make sure someone in the group has a GPU.


As I mentioned in my first OMSCS post, I am an educational hobbyist. So I hate anything that gets in the way of the pure joy of learning and tend to be highly critical of it. For this class, my biggest criticism by far was the initial attitude of the instructors over the calculus requirement.

On the first test, they casually mentioned there would be some calculus on the first quiz and that it was a 30-minute timed test. They then gave us a huge list of things to study, including two math chapters that didn’t include any calculus. The quiz was like 80% of the points on the calculus that they barely mentioned at all. And for someone that hadn’t had calculus in 30 years, there was no way I could do the large number of problems fast enough to get through each in 2 minutes. So I got a 30% on my first quiz and was terrified the whole class would be like this – really tough math with zero warning of what to expect.

I complained to the instructor that I needed a more realistic warning of what to study and was told that I should drop the class. This whole attitude is a poor fit for OMSCS in my opinion. But to the instructor’s credit, I think he changed his attitude quite a bit within the first couple of weeks once he started to realize how OMSCS students are different than on-campus students. This was his first time doing OMSCS, so I’m prepared to give him a pass this time. I doubt it will be as big a problem in future classes.

For us older OMSCS students – particularly educational hobbyists like myself – we are willing to put in the necessary time to come up to speed. And we know how to study well. It simply did not matter that I didn’t know calculus – yet. If there had been a warning as simple as saying ‘80% of the points on the first quiz will be calculus’ I’d have done much better on it. I know this because I retook the quiz offline (using my scratch paper) and I scored much better once I wasn’t being timed. I just needed more practice. And this was without any significant prep due to the misleading instructions on what to study for. But with so much material to cover — chapters and chapters of which were not on the actual quiz — I simply hadn’t spent enough time prepping for what actually mattered.

To the instructor’s credit, he was much more clear on what to study after that and even made a post stating that if you did well on the quiz offline (as I had done – I had told him as much so he was likely indirectly responding to me) that I’d probably do fine in the class.

I get why instructors on regular campuses expect their students to have the prerequisites in advance and use a standard approach of telling a troubled student to drop if they aren’t ready. But this is the wrong thing to say to an OMSCS student, in my opinion, and my high final score proves it. I only had one class and I was ready to spend lots of time on it.

What would be even better is if the instructors actually gave good prep material and collected good links to help students rusty on their math to brush up quickly. This isn’t that hard and there is just no reason not to do this with OMSCS students.

But until the instructors figure this out, my advice is to not panic if you don’t do well on the calculus at first. That and study up in advance if you can.

My second biggest criticism is the quizzes themselves. I feel like they are just trying too hard to be tricky. I never understood the point of doing this. I think they could make the quizzes less about stumping us and more about testing our knowledge.

My final criticism is that I felt the lectures were just okay. Not terrible or anything, but pretty high level and didn’t do a great job of preparing you for the quizzes or projects on their own. But then again, that’s pretty common in all OMSCS courses. Lectures are often just a starting point.

Praise for the Class

Let me just say that – my one criticism above aside — the instructors in this course were amazing. They were incredibly responsive and they really cared about making sure you completed each assignment successfully. If you get stumped, just go to office hours and they will literally pull up your code with you and go over it with you. A few times I stumped the instructors and they couldn’t figure out what was wrong with my code. But even then they just had me send my code into Piazza (on a private channel) and even looked up my code that I had submitted previously and came up with ways to help me figure out what I was doing wrong and get me moving again.

This same ‘we care about you’ attitude carried over to how well the instructors answered piazza posts and how they were responsive to concerns (like when I was worried about how to handle the fact that one of our team members on the final project had yet to even start and we were nearly done.)

Also, this really is just the best class I had in OMSCS. You really need to take this class. This is doubly true if you are in the Machine Learning specialization. What is the point of having a machine learning specialization if you don’t know Deep Learning well?

Final Thoughts

This is the perfect final course for your machine learning specialization, though you can also just take it for fun somewhere. Be prepared to work hard, but don’t panic as much as I did. It’s not that hard to get a good grade if you really want it. The instructors are awesome and the one big problem I had with them – being told to drop the course – is likely to not be a repeat problem. Study calculus in advance if you can and take at least one machine learning course before this one if you can. But I got through the class without knowing to study calculus in advance and I did great.

Leave a Reply

Your email address will not be published. Required fields are marked *