Troubleshoot common issues

This page lists various issues that you might encounter when configuring VPC Service Controls.

Unexpected scoped policy behavior

You might notice some unexpected VPC Service Controls violations that your scoped policy should allow. It is a known issue that if you don't have an organization-level access policy, then you might experience some unexpected issues with your scoped access policies.

To overcome this issue, create an access policy at the organization level using the following command:

gcloud access-context-manager policies create --organization <var>ORGANIZATION_ID</var> --title <var>POLICY_TITLE</var>

Replace the following:

ORGANIZATION_ID: The organization ID.
POLICY_TITLE: A human-readable title for your access policy.

For more information, see Create an access policy.

Shared VPC

When using Shared VPC, a service perimeter that includes projects that belong to a Shared VPC network must also include the project that hosts the network. When projects that belong to a Shared VPC network are not in the same perimeter as the host project, services might not work as expected or might be blocked entirely.

Ensure that the Shared VPC network host is in the same service perimeter as the projects connected to the network.

Unable to add a VPC network

The following error can occur when you try to add a VPC network to a service perimeter:

ERROR: (gcloud.access-context-manager.perimeters.update) PERMISSION_DENIED: Permission 'compute.networks.get' denied on resource '//compute.googleapis.com/projects/PROJECT_NAME/global/networks/VPC_NETWORK_NAME' (or it may not exist)

This error occurs due to one of the following reasons:

The VPC network doesn't exist.
The VPC network exists, but doesn't have a subnet.
The caller doesn't have the required permission.

To resolve this issue, complete the following steps:

Verify if the VPC network specified in the error message exists by viewing the networks in your project.
- Before verifying the VPC network, make sure that the Compute Engine API is enabled in the project associated with the API call by completing the following steps:
  1. In the Google Cloud console, go to the APIs & Services page.
    Go to APIs & Services
  2. On the APIs & Services page, verify if the Compute Engine API is listed.
  3. If the Compute Engine API is missing, enable the API.
    Enable the API
Verify if at least one subnet exists in the VPC network by viewing the subnets. If there are no subnets, add a subnet to the VPC network.
Check if the caller has the following permission on the host project of the VPC network: compute.networks.get. This permission lets you view VPC networks in a project.
- Ask the administrator of the organization that owns the host project of the VPC network to grant the caller an IAM role with the compute.networks.get permission on the host project. For example, the Compute Network Viewer role.
  
  For more information about granting roles, see Manage access.

Ensure that you read the limitations associated with using VPC networks in service perimeters.

Requests between perimeters

Normally, access levels are used to allow requests from outside a service perimeter for protected resources inside a perimeter.

However, a request from a project in a perimeter for a protected resource in another perimeter is denied, even if an access level normally allows the request.

For example, assume Project A in Perimeter 1 requests a resource from Project B. The resource in Project B is protected by Perimeter 2. Because Project A is in a perimeter, even if an access level for Perimeter 2 normally permits the request for the protected resource, the request is denied.

Use one of the following approaches to facilitate requests between perimeters:

Use egress policy and ingress policy. To allow requests from another perimeter to protected resources in your perimeter, the other perimeter must use an egress policy and you must set an ingress policy in your perimeter.
Use perimeter bridges. Bridges allow two or more projects in different perimeters to make requests to any services in those projects. These requests are permitted even if the services are protected by the respective perimeters.
Ensure that both the requesting service and the target resource are unprotected by the perimeters. In this scenario, the operation succeeds because the services are not protected.

The email address is invalid or non-existent

When updating a perimeter that contains a deleted principal, you might encounter the The email address is invalid or non-existent error.

To fix this issue, you must remove the invalid email address from all perimeters:

Export all your perimeters. The following sample command exports a list of service perimeters in JSON format:

gcloud access-context-manager perimeters list \
    --policy=POLICY_NAME \
    --format="json(name,title,description,perimeterType,status,\
spec,useExplicitDryRunSpec)" \
    > my-perimeters.json

Remove the invalid email address from the my-perimeters.json file and save it as my-perimeters-updated.json.
Bulk replace all your perimeters.

Ingress and egress rules violations

The audit log contains information about the ingress and egress rules violations that help you understand perimeter violations.

Ingress rule violation

An ingress rule violation indicates that an API client outside of the perimeter tried to access a resource inside the perimeter. The service perimeter rejects the request as there are no matching ingress rules or access levels.

An ingress rule violation in the audit log contains the following details:

The name of the perimeter in which the ingress rule violation occurred.
The resource inside the perimeter that the API client outside the perimeter tried to access.

In the following ingress rule violation example, an API client outside the perimeter tries to access the Cloud Storage bucket prod-protected-storage-bucket inside the perimeter prod-perimeter.

ingressViolations: [
  0: {
    targetResource: "projects/1234/buckets/prod-protected-storage-bucket"
    servicePerimeter: "accessPolicies/123456789/servicePerimeters/prod-perimeter"
  }
]

To resolve this issue, create an ingress rule for your perimeter. For more information about ingress rules, see Ingress rules reference.

Egress rule violation

An egress rule violation in the audit log indicates one of the following events:

An API client inside the perimeter tried to access a resource outside the perimeter.
An API request that involves a resource inside the perimeter and a resource outside the perimeter. For example, a Cloud Storage client that calls a copy command where one bucket is within the perimeter and the other bucket is outside the perimeter.

The service perimeter rejects the request because there are no matching egress rules. An egress rule violation in the audit log includes the following details:

The source type such as network or resource.
The source, which is a resource or network, whose perimeter encountered an egress violation.
The perimeter that encountered an egress violation.
The target resource outside the perimeter that the request tried to access.

In the following egress rule violation example, the API request includes a resource from projects/5678, which is inside the perimeter prod-perimeter, and an object from the Cloud Storage bucket external-storage-bucket, which is outside the perimeter.

egressViolations: [
  0: {
    sourceType: "Resource"
    source: "projects/5678"
    targetResource: "projects/4321/buckets/external-storage-bucket/objects/corp-resources.json"
    servicePerimeter: "accessPolicies/123456789/servicePerimeters/prod-perimeter"
  }
]

To resolve this issue, create an egress rule for your perimeter. For more information about egress rules, see Egress rules reference.

Debugging requests blocked by VPC Service Controls

The VPC Service Controls audit log is the primary tool for debugging a request blocked by VPC Service Controls.

When access has been blocked unexpectedly, consult the audit logs in the project protected by the service perimeter. These logs contain significant data about the requested resources and the reason why the request was denied. For information about diagnosing the audit logs, see Access the violation analyzer.

If you use VPC Service Controls service patterns to manage access to unsupported Google services and encounter unexpected denials, the reason for this issue might be the following:

The violation analyzer doesn't support these violations.
Audit logs for these violations might have limited metadata. Check the protoPayload.request.url field in the log entry.

Ensure your YAML configuration in the service perimeter accurately reflects the intended URL patterns.

The following sections list the violationReason values that you might encounter when using VPC Service Controls.

NETWORK_NOT_IN_SAME_SERVICE_PERIMETER

The reason for this issue might be one of the following:

A client in a VPC network within a service perimeter tries to access a project that is not in the same perimeter. This request results in an egress violation. Create an egress rule to fix this issue.
A client in a VPC network that is outside a service perimeter tries to access a project that is protected by the service perimeter. This request results in an ingress violation. Create an ingress rule to fix this issue.

The client might send the request from a Compute Engine or Google Kubernetes Engine VM, or from an on-premises network through Cloud Interconnect or a VPN configured using a VPC network.

The following diagram shows that an egress violation occurs when a client in a VPC network within a service perimeter tries to access a project outside the perimeter:

An egress violation due to NETWORK_NOT_IN_SAME_SERVICE_PERIMETER.

Here is an example of an egress violation:

egressViolations: [
{
  servicePerimeter: "accessPolicies/<POLICY_NAME>/servicePerimeters/<PERIMETER_NAME>"
  source: "projects/<NETWORK_PROJECT_NUMBER>"
  sourceType: "Network"
  targetResource: "projects/<RESOURCE_PROJECT_NUMBER>"
}
]

Where:

<POLICY_NAME> is the numeric name of your access policy.
<PERIMETER_NAME> is the name of the service perimeter.
<NETWORK_PROJECT_NUMBER> is the project number of the Google Cloud project that holds your VPC network.
<RESOURCE_PROJECT_NUMBER> is the project number of the Google Cloud project that contains the resource.

The following diagram shows that an ingress violation occurs when a client outside the perimeter tries to access a project inside the perimeter:

An ingress violation due to NETWORK_NOT_IN_SAME_SERVICE_PERIMETER.

Here is an example of an ingress violation:

ingressViolations: [
{
          targetResource: "projects/<RESOURCE_PROJECT_NUMBER>",
      source: "projects/<NETWORK_PROJECT_NUMBER>"
          servicePerimeter: "accessPolicies/<POLICY_NAME>/servicePerimeters/<PERIMETER_NAME>"
}
]

Where:

<RESOURCE_PROJECT_NUMBER> is the project number of the Google Cloud project that contains the resource.
<NETWORK_PROJECT_NUMBER> is the project number of the Google Cloud project that holds your VPC network.
<POLICY_NAME> is the numeric name of your access policy.
<PERIMETER_NAME> is the name of the service perimeter.

Resolution

To resolve this error, create an ingress or egress rule for your perimeter.

RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER

This issue occurs when a single request is accessing multiple resources but the resources are not in the same service perimeter. This issue occurs irrespective of where the client is located and whether the client has access to the resources.

The following diagram shows a client accessing resources from a project outside the perimeter and from a project within the service perimeter:

An egress violation due to a client accessing resources from a project outside the
perimeter.

The following diagram shows a client accessing resources from projects which are in two different service perimeters but the perimeters do not communicate with each other:

An egress violation due to a client accessing resources from projects which are
in two different service perimeters.

Here is an example of an egress violation:

egressViolations: [
{
  servicePerimeter: "accessPolicies/<POLICY_NAME>/servicePerimeters/<PERIMETER_NAME>"
  source: "projects/<RESOURCE_PROJECT_INSIDE_THIS_PERIMETER>"
  sourceType: "Resource"
  targetResource: "projects/<RESOURCE_PROJECT_OUTSIDE_THIS-PERIMETER>"
}
]

Where:

<POLICY_NAME> is the numeric name of your access policy.
<PERIMETER_NAME> is the name of the service perimeter.
<RESOURCE_PROJECT_INSIDE_THIS_PERIMETER> is the project number of the Google Cloud project that is within the perimeter.
<RESOURCE_PROJECT_OUTSIDE_THIS_PERIMETER> is the project number of the Google Cloud project that is outside the perimeter.

Resolution

To resolve this issue, create an egress rule for your perimeter.

NO_MATCHING_ACCESS_LEVEL

This issue occurs when the IP address, device requirement, or user identity doesn't match any ingress rules or access levels assigned to the perimeter. This means that a client that isn't a part of the Google Cloud network tries to access Google Cloud network resources from outside the perimeter. For example, the IP address corresponding to the callerIp field of the audit record doesn't match any CIDR ranges defined in the access levels for the service perimeter.

If the caller IP address is missing or appears as an internal IP address, then this violation might be due to a Google Cloud service that isn't integrated with VPC Service Controls. The reason could be that the Google Cloud service tries to access a protected service and fails, as expected.

To fix this issue, we recommend creating an ingress rule instead of an access level because an ingress rule provides granular access control.

The following diagram shows a client trying to access resources from outside the perimeter:

An ingress violation due to NO_MATCHING_ACCESS_LEVEL.

Here is an example of an ingress violation:

authenticationInfo: {
  principalEmail: "EMAIL"
}
requestMetadata: {
callerIp: "<PUBLIC_IP_ADDRESS>"
deviceState: "Cross Organization"
}
ingressViolations: [
        {
          targetResource: "projects/<RESOURCE_PROJECT_NUMBER>",
          servicePerimeter: "accessPolicies/<POLICY_NAME>/servicePerimeters/<PERIMETER-NAME>"
        }
  ]

Where:

<EMAIL> is the email address of the service account or authenticated user.

If you use a Google Cloud service that VPC Service Controls doesn't support, email addresses that belong to the domain google.com are redacted and replaced with google-internal. google-internal refers to internal Google-owned identities.
<PUBLIC_IP_ADDRESS> is the IP address of the caller. For a caller from the internet, this will be the public IPv4 or IPv6 address.
<RESOURCE_PROJECT_NUMBER> is the project number of the Google Cloud project that contains the resource.
<POLICY_NAME> is the numeric name of your access policy.
<PERIMETER_NAME> is the name of the service perimeter.

Note that in this case, metadata.accessLevels could still be present since these access levels might not be specified in the violated perimeter.

SERVICE_NOT_ALLOWED_FROM_VPC

This issue occurs when a client tries to access Google Cloud resources from a VPC network. The client might send the request from a Compute Engine or Google Kubernetes Engine VM, or from an on-premises network through Cloud Interconnect or a VPN configured using a VPC network.

To fix this issue, ensure that the service being called is allowed by the VPC accessible services configuration of the service perimeter.

Example scenarios

The following examples cover issues that you might encounter while using VPC Service Controls.

Cloud Storage access from on-premises
BigQuery access from VM outside of project
Cross-project BigQuery query
Move Cloud Storage file inside perimeter
Move Cloud Storage file outside perimeter
BigQuery dataset copy from VM inside perimeter
Managed Service for Apache Spark job reading from project
Unsupported service with restricted VIP
Log export to project outside perimeter
BigQuery extract to Cloud Storage

Cloud Storage access from on-premises

In this example, VPC Service Controls blocks a request from an employee workstation (identified by callerIp) to a Cloud Storage bucket in project corp-storage.

The request generates the following audit log record:

{
 insertId:  "222lvajc6f7"
 logName:  "projects/corp-storage/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "someone@google.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/_"
   ]
   violationReason:  "NO_MATCHING_ACCESS_LEVEL"
  }
  methodName:  "google.storage.NoBillingOk"
  requestMetadata: {
   callerIp:  "b1d5:d26d:5b17:43fe:d358:586b:db59:9617"
   destinationAttributes:

Troubleshoot common issues Stay organized with collections Save and categorize content based on your preferences.

Unexpected scoped policy behavior

Shared VPC

Unable to add a VPC network

Requests between perimeters

The email address is invalid or non-existent

Ingress and egress rules violations

Ingress rule violation

Egress rule violation

Debugging requests blocked by VPC Service Controls

NETWORK_NOT_IN_SAME_SERVICE_PERIMETER

Resolution

RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER

Resolution

NO_MATCHING_ACCESS_LEVEL

SERVICE_NOT_ALLOWED_FROM_VPC

Example scenarios

Cloud Storage access from on-premises

Troubleshoot common issues