There are all kinds of legitimate reasons for keeping secrets, which is why you should be using every tool at your disposal to protect your privacy from the forces of surveillance capitalism.
So as you might have guessed, there's all kinds of shenanigans you can get up to if you steal someone's phone number. Mobile phones are frequently used for authentication now so there's all sorts of accounts you'd be able to access and transactions you'd be able to authorize. Brian Krebs wrote last year about some ways you can help to protect yourself from this sort of thing.