Reddit reviews Guitar Scales in Context: A practical encyclopaedia and playing guide to musically learn scales on guitar

One thing that has helped me to is take the CAGED chord shape arppregios for each chord and then limit myself to practicing them in a certain part of the neck but choosing the same part of the neck for the chords in the progression.

So, in this example, you'd do it with the two chords in Shakedown Street. I'd take the arpreggio diagrams for F major and E minor, in each of the CAGED positions, and match them up where the positions most closely overlap so you limit yourself to 4 or so frets at a time. There are endless references for this but a straightforward one I like is this book. Since your progression is only two chords, do it with those two.

You could do it with almost every other song, though. One I have found very helpful is Rample On Rose since there are so many chords in the progression and you'll cycle through major, dominant and minor all over the neck. And you could do the same on Deal if you're looking for diminished practice, and so on.

After you're comfortable hitting the arpreggio chord tones, then you can add in the other notes from the scales for more color just as you said with the lydian sound for the major 7th chord, etc.

That's still all playing "vertically" (improvising harmonies over a chord), though, and not quite "horizontally" (weaving the harmonies from all the chords in the progression into melodic concepts). I don't have a trick for you there as I'm still working on it myself.

I will say, however, that ability to comfortably hit all the right chord tones over the neck (what you'd get from the CAGED arpreggio exercies in every position) really helps since your brain will want to do things melodically over the neck and being comfortable throughout goes a very long way.

Hope it helps!