Habitat plays a huge factor, deep water close to shore, good kelp structure, lots of stuff to eat, etc. But, it's also that La Jolla is where everyone fishes. LJ is convenient, has a good launch and has great fishing close to shore.
The fish are always moving up and down the coast, but they encounter the most fishermen in LJ.
In case you want more info:
Thesis on yellowtail in the Southern California Bight
Southern California Yellowtail Tagging Program, unfortunately no longer running.