Reddit reviews The Garbage Collection Handbook: The Art of Automatic Memory Management (Chapman & Hall/CRC Applied Algorithms and Data Structures series)

If you want more than an overview:

https://www.amazon.com/Garbage-Collection-Handbook-Management-Algorithms/dp/1420082795

Or, if you're on a budget, the previous edition can be had for under $20 used:

https://www.amazon.com/Garbage-Collection-Algorithms-Automatic-Management/dp/0471941484

What can be even cheaper is to pick a recent GC paper (e.g. https://www.cs.utexas.edu/~speedway/fp031-kermany.pdf) and then follow the citations back in time. Often if you google the author and the name of the paper, you'll find a .edu hit on the first page of google that has the paper as a PDF for free (either the author(s) themselves host it, or a professor teaching a class posted it for their students to read).

For basic background, read:

https://en.wikipedia.org/wiki/Cheney's_algorithm

and

https://en.wikipedia.org/wiki/Tracing_garbage_collection#Basic_algorithm

As Cheney's algorithm and tri-color marking are two of the basic building blocks for many other GC algorithms.

Unlike some other topics in CS, garbage-collection papers tend to not invent their own mathematical notations for describing their algorithms, and they tend to be more empirically focused (since worst-case bounds on most novel GC algorithms are not particularly interesting). Those two things make the papers quite approachable.

It’s gonna be hard to give you much without picking a specific OS—details can vary widely even within a single OS family line—but an OS book is probably a good place to start. Tanenbaum’s book is the go-to.

Alternatively, there are books on specific techniques like garbage collection, or books more generally on programming for UNIX/POSIX/Linux or Windows (via Win16/32/64 API, not the six-mile-high shitheap piled atop), which would tell you some about how memory management works in down at least to whatever abstract hardware interface the kernel uses.

This is not a good approach to garbage collection, but it's the best you can do, sans optimization, for a language like C. It's not unsafe if done correctly since you err on the side of caution and assume anything that could be a pointer is a pointer. It is quite slow and will leak some memory over time (due to misclassifying integers as pointers), which is why languages that are designed for garbage collection use more efficient designs requiring cooperation from the language.

If you're interested in the stuff, I highly recommend The Garbage Collection Handbook. It's loaded with information and algorithms but also includes approachable, pragmatic discussion of the characteristics and tradeoffs of automatic memory management systems.

I would start with this survey by Wilson. It introduces a number of simple approaches and refers to the papers that first came up with them. There's also The Garbage Collection Handbook which goes into significantly more depth.

My advice for your language is to start with a simple mark and sweep collector. Make sure you can configure how often collections are run and can make dumps of your heap whenever you want to -- you'll be in debugging hell otherwise. Once you're done with that, switching to a copy-collector and then implementing incremental collection via a read-barrier or a write-barrier is a logical next step.

Finally, don't worry about microoptimisations (yet). Garbage collectors are a terrible pain to debug even when your code is clear; if your code is clever, you'll never figure out what's going wrong.

Book: https://www.amazon.com/Garbage-Collection-Handbook-Management-Algorithms/dp/1420082795

Article that helped me a lot: https://journal.stuffwithstuff.com/2013/12/08/babys-first-garbage-collector/