Top products from r/ControlProblem

Are you talking about what to study this summer?

I haven't read through that guide too much but here's a few points:

The first three bullets under "background" are probably very useful in a wide range of contexts and problems. Definitely you can't go wrong starting there, and you probably should. And when you've mastered that, honestly, you can just go further into AI and ML methods (by the time that you finish, you should have a really good idea of what else you are interested in and what else there is to read about, and you'll be asking "where can I learn more about the theory behind nonlinear SVMs" in r/machinelearning rather than asking us for a general syllabus). You really won't go wrong if you spend 90% of your time just learning how to do these things really well, IMO. And practice implementation of these things so you have practical skills you can demonstrate and so that you remember it all really well. Now maybe you won't be on top of everything in the AI safety field the way it's currently going, if you just do this, but you'll be flexible to work on lots of things, and you can get up to speed on those things later on.

Also there's a lot of stuff there, so you can specialize. It's okay if you decide that you really don't understand game theory and just want to learn about programming and statistics, for instance. But you should at least start to explore every subject just to see what it's about.

Then if you want to go into value learning in particular, you'll want to look at preference inference and reward engineering in that syllabus. I'd recommend getting to at least the majority of the priority-2 stuff in the first 3 categories before turning to this stuff, though it's okay to explore it earlier. Also there is some contemporary work on machine ethics, much of it listed here, but I'm honestly not too impressed by it (unless I'm wrong or there are some modern papers which I haven't seen, it seems to be very much stuck in technical methods which are not cutting edge in the least), and I don't think it's important to learn, but you can take a look at some of it when you like. It's probably a good idea to look at Wallach and Allen's book at some point, or at the very least to look up what people mean when they talk about top-down vs bottom-up morality and the arguments for each. I think Allen has a paper summarizing it which you can probably find.

The moral theory section in that syllabus is pretty poor. At least, it's not an actual overview of the relevant aspects of moral theory; it's just a collection of odd papers and ideas which AI safety folks have happened to find interesting and useful, and some of it is other areas of philosophy besides moral theory. If you are new to philosophy, I worry it will give a skewed picture of what it's really about, and you won't be able to interpret them correctly if you don't have some experience understanding and discussing the bodies of philosophical work where there is already lots of secondary material and experts who can readily check your ideas and understanding (e.g.: you read Reasons and Persons, then since lots of people have talked about Reasons and Persons and there's lots of reviews and discussions about it on the Internet, you'll be able to talk about it and lots of people can tell you if you have a good understanding of philosophical writing and theory. Compared to starting with something like Ord's paper on moral trade, where you'll immediately know more about moral trade than 90% of philosophers do, which is great for your confidence but not so great for your education in philosophy). And on a higher level, there should be more diversity in philosophical views than just the things that these people are currently interested in. This is more important if you want to research value learning and alignment, and less important otherwise.

If you want to really understand moral theory well enough to talk about and design systems which learn it, I recommend this collection (have not looked at it, aside from a few of its readings, but I researched and asked opinions to determine that it's probably the best for this purpose): https://www.amazon.com/Ethical-Theory-Anthology-Russ-Shafer-Landau/dp/0470671602/ref=pd_cart_vw_2_2?_encoding=UTF8&psc=1&refRID=BE9N6J1887PCY9HH5PE4 If you want something shorter and/or free, you can go to plato.stanford.edu and read about whatever moral theories you feel interested in ("consequentialism", "virtue ethics", "deontological ethics", etc). Reasons and Persons is good too (haven't read it either but I'm familiar with many of its ideas). Now the papers in the Berkeley reading list are still worth looking at, if you read other stuff too.

But unlike /u/TheConstipatedPepsi, with respect, I don't recommend starting with moral theory rather than the technical stuff, because it really won't teach you how to do math and computer science, it will just delay the point at which you finally learn those mathematical and computational foundations. (Though some of those papers do contain bits of math, and in that respect they can get you a little accustomed to how math is applied and notated in research papers, which is a decent skill to develop early on, so that might be a decent exercise to spend ~5% of your time on.)

Also make sure you're on top of your scheduling and habits etc so you don't get off track. Check out Thomas Frank's channel on Youtube for advice on that. Not specific to AI at all, but still. It's good to know about. Hope that helps.

Absolutely. This topic is a favorite of mine and I've invested quite a bit of my free time in independent study of it. The place to start, for sure, is Li&Vitanyi. This is a graduate text and it assumes you already have a general familiarity with undergrad topics in CS. The key is to understand the prefix-free complexity measure. Once you get that down, the rest of it is a matter of careful applications of this key idea to other areas. It's not conceptually difficult but the proof techniques used in computability are a bit alien relative to proof techniques used in most other fields of CS/math. One of the coolest developments to come out of AIT is the incompressibility method, which is a new, general-purpose proof technique which is applicable to virtually any domain of mathematics.

Implications include artificial general-purpose intelligence (AGI), universal AI, philosophy and physics and many more. I consider AIT to be the under-appreciated diamond-mine discovered by 20th century mathematicians that outshines all other, admittedly beautiful, areas of modern mathematics by many orders of magnitude.

https://peerj.com/DustinJuliano/

The author isn't affiliated with any organization. He doesn't show up on Google Scholar or any other academic search.

https://www.amazon.com/AI-Security-Dustin-Juliano/dp/1535119004

The book was published by "CreateSpace Independent Publishing Platform". So it doesn't look like it was peer reviewed or anything. Based on the few articles I looked up, it seems like good references are used.

I wouldn't spend time reading it.

Edit: Just looked at Amazon's 'About the Author'-

> He formulated a mathematical conjecture for generalizing intelligence based on the requirement for digital sentience, which he has termed the new strong AI hypothesis. . . .
>He founded the AI Security movement (http://aisecurity.org)

Since he didn't appear in Google Scholar or my universities search engine, this is very dubious. I would not accept this as a reference for a paper.