The service account does not have permission to call Directory APi for your G Suite instance. What it does have access to do is act as a user in your domain within the context of the Directory API scope(s) you granted it access to.
When getting service account credentials you need to add the sub=admin@yourdomain.com parameter so that you are acting as your domain admin, not the service account. See:
https://developers.google.com/identity/protocols/OAuth2ServiceAccount
And some code examples at:
https://developers.google.com/admin-sdk/directory/v1/guides/delegation