Self-Ownership

Hello yall! I don’t consider myself a libertarian but I am very attracted to it(which is also why I’m very excited to join the sub ≧◡≦). This question/post is less concrete and more philosophical in nature. It concerns the concept of Self-Ownership, the general and common-sense idea that we have sovereignty over ourselves and that we can basically do what we want as long as we don’t directly hurt eachother.

I encountered a critique of this principle formulated by philosopher Eric Mack. I originally encountered the argument in a blog post by Catholic Philosopher Edward Feser(who I follow a fair bit). Instead of trying to reformulate the argument, I will just insert a quote from Feser here. This is the link to the original blog post(which I recommend reading if you have the time) and here is the quote:

Suppose, for example, that you and I are castaways and wash up on some tiny island upon which no human beings have ever trod. You immediately pass out on the beach, while I get to work constructing a bamboo fence whose perimeter happens entirely to enclose your body. Upon waking, you accuse me of imprisoning you and thereby violating your self-ownership rights, and demand to be released. Suppose I then respond as follows: “I have not imprisoned you at all! I’ve simply homesteaded all the land around you — which you had no right to, since it was virgin territory — and I’ve built a fence around it, to make sure you don’t come onto my land and take any of the resources I’ve justly acquired. True, you’ve got nothing in the way of resources in the seven-foot by four-foot plot of sand I’ve left you, but that’s not my fault. That’s just your bad luck, sorry. I suppose it would be nice of me to give you some of mine, but at most I’d be unkind rather than unjust if I decide not to do so. And I was very careful not to touch you as I built my fence. I do respect your right of self-ownership, after all!”

How may a libertarian respond to the argument? If it is to be accepted, then what are the limits that should be placed on Self-Ownership? Should it even be accepted? Is it misunderstanding something? Thanks and God bless you!