I have that happen to me on occasion at Dana Point -
A whole great big ocean to fish, one or two other kayakers show up later in my area and they want to fish right on top of me, and I got to my spot first.
I don't know why - it's almost like parking your car way out in left field in a parking lot all by it's lonesome, come back an hour later and someone is parked right next to you.