Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Global rate limit not working #50591

Open
2 tasks done
NamanGajjar26 opened this issue Apr 22, 2024 · 4 comments
Open
2 tasks done

Global rate limit not working #50591

NamanGajjar26 opened this issue Apr 22, 2024 · 4 comments

Comments

@NamanGajjar26
Copy link

NamanGajjar26 commented Apr 22, 2024 •
edited by istio-policy-bot

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

i try to set Global rate limit on api path
i try to make like this

  `apiVersion: v1
  kind: ConfigMap
  metadata:
    name: ratelimit-config
    namespace: istioapi
  data:
    config.yaml: |
      domain: api-ratelimit
      descriptors:
        - key: ratelimitheader
          descriptors:
            - key: PATH
              value: "/api/CustomerControllers"
              rate_limit:
                unit: minute
                requests_per_unit: 1
  ---
  apiVersion: networking.istio.io/v1alpha3
  kind: EnvoyFilter
  metadata:
    name: filter-global-ratelimit
    namespace: istioapi
  spec:
    workloadSelector:
      labels:
        app: demoapidock
    configPatches:
      - applyTo: HTTP_FILTER
        match:
          context: SIDECAR_INBOUND
          listener:
            filterChain:
              filter:
                name: "envoy.filters.network.http_connection_manager"
                subFilter:
                  name: "envoy.filters.http.router"
        patch:
          operation: INSERT_BEFORE
          value:
            name: envoy.filters.http.ratelimit
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
              domain: api-ratelimit
              failure_mode_deny: true
              timeout: 10s
              rate_limit_service:
                grpc_service:
                  envoy_grpc:
                    cluster_name: outbound|8081||ratelimit.istioapi.svc.cluster.local
                    authority: ratelimit.istioapi.svc.cluster.local
                transport_api_version: V3
  ---
  apiVersion: networking.istio.io/v1alpha3
  kind: EnvoyFilter
  metadata:
    name: filter-ratelimit-svc
    namespace: istioapi
  spec:
    workloadSelector:
      labels:
        app: demoapidock
    configPatches:
      - applyTo: VIRTUAL_HOST
        match:
          context: SIDECAR_INBOUND
          routeConfiguration:
            vhost:
              name: ""
              route:
                action: ANY
        patch:
          operation: MERGE
          value:
            rate_limits:
              - actions:
                  - request_headers:
                      header_name: "x-rate-limit-please"
                      descriptor_key: "ratelimitheader"
                      skip_if_absent: true
                  - request_headers:
                      header_name: ":path"
                      descriptor_key: "PATH"
  `

this below log i get

time="2024-04-22T08:39:49Z" level=info msg="Stats initialized for stdout"
time="2024-04-22T08:39:49Z" level=info msg="Stats flush interval: 10s"
time="2024-04-22T08:39:49Z" level=info msg="Tracing disabled"
time="2024-04-22T08:39:49Z" level=warning msg="connecting to redis on redis:6379 with pool size 10"
time="2024-04-22T08:39:49Z" level=debug msg="Implicit pipelining enabled: false"
time="2024-04-22T08:39:49Z" level=debug msg="loading domain: api-ratelimit"
time="2024-04-22T08:39:49Z" level=debug msg="loading descriptor: key=api-ratelimit.ratelimitheader"
time="2024-04-22T08:39:49Z" level=debug msg="Creating stats for key: 'api-ratelimit.ratelimitheader.PATH_/api/CustomerControllers'"
time="2024-04-22T08:39:49Z" level=debug msg="loading descriptor: key=api-ratelimit.ratelimitheader.PATH_/api/CustomerControllers ratelimit={requests_per_unit=1, unit=MINUTE, unlimited=false, shadow_mode=false}"
time="2024-04-22T08:39:49Z" level=info msg="Waiting for initial ratelimit config update event"
time="2024-04-22T08:39:49Z" level=debug msg="waiting for runtime update"
time="2024-04-22T08:39:49Z" level=info msg="Successfully loaded new configuration"
time="2024-04-22T08:39:49Z" level=info msg="Successfully loaded the initial ratelimit configs"
time="2024-04-22T08:39:49Z" level=warning msg="Listening for HTTP on '[::]:8080'"
time="2024-04-22T08:39:49Z" level=debug msg="Waiting for config update event"
time="2024-04-22T08:39:49Z" level=warning msg="Listening for debug on '0.0.0.0:6070'"
time="2024-04-22T08:39:49Z" level=warning msg="Listening for gRPC on '[::]:8081'"
{"level":"info","ts":1713775199.264663,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.go.mallocs","type":"counter","value":"12370.000000"}}
{"level":"info","ts":1713775199.264796,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.go.totalAlloc","type":"counter","value":"2036448.000000"}}
{"level":"info","ts":1713775199.264804,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.redis_pool.cx_total","type":"counter","value":"10.000000"}}
{"level":"info","ts":1713775199.264810,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.go.frees","type":"counter","value":"1843.000000"}}
{"level":"info","ts":1713775199.264818,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"runtime.load_attempts","type":"counter","value":"1.000000"}}
{"level":"info","ts":1713775199.264823,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.service.config_load_success","type":"counter","value":"1.000000"}}
{"level":"info","ts":1713775199.264828,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapObjects","type":"gauge","value":"10527.000000"}}
{"level":"info","ts":1713775199.264833,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.alloc","type":"gauge","value":"2036448.000000"}}
{"level":"info","ts":1713775199.264837,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.sys","type":"gauge","value":"0.000000"}}
{"level":"info","ts":1713775199.264843,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapSys","type":"gauge","value":"7471104.000000"}}
{"level":"info","ts":1713775199.264848,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapIdle","type":"gauge","value":"4055040.000000"}}
{"level":"info","ts":1713775199.264852,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"runtime.num_values","type":"gauge","value":"1.000000"}}
{"level":"info","ts":1713775199.264856,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapReleased","type":"gauge","value":"4022272.000000"}}
{"level":"info","ts":1713775199.264859,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.nextGC","type":"gauge","value":"4194304.000000"}}
{"level":"info","ts":1713775199.264863,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.lastGC","type":"gauge","value":"0.000000"}}
{"level":"info","ts":1713775199.264868,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.numGoroutine","type":"gauge","value":"16.000000"}}
{"level":"info","ts":1713775199.264873,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapAlloc","type":"gauge","value":"2036448.000000"}}
{"level":"info","ts":1713775199.264876,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.redis_pool.cx_active","type":"gauge","value":"10.000000"}}
{"level":"info","ts":1713775199.264880,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapInuse","type":"gauge","value":"3416064.000000"}}
{"level":"info","ts":1713775199.264884,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.gcCPUPercent","type":"gauge","value":"0.000000"}}
{"level":"info","ts":1713775199.264889,"logger":"gostats.loggingsink","msg":"flushing all stats","json":{"type":"all stats","value":"0.000000"}}

Version

istio version 1.21.1
minikube version: v1.32.0
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

Additional Information

No response
@zirain
Copy link
Member

zirain commented Apr 22, 2024

enable debug log for ratelimit, then check the logs

           - name: LOG_LEVEL
              value: debug

@NamanGajjar26
Copy link
Author

NamanGajjar26 commented Apr 22, 2024 •
edited

already add this
-name: LOG_LEVEL
value: debug
than i get log

apiVersion: apps/v1
kind: Deployment
metadata:
name: ratelimit
namespace: istioapi
spec:
replicas: 1
selector:
matchLabels:
app: ratelimit
strategy:
type: Recreate
template:
metadata:
labels:
app: ratelimit
spec:
containers:
- image: envoyproxy/ratelimit:8b6a44b0 # 2022/08/16
imagePullPolicy: Always
name: ratelimit
command: ["/bin/ratelimit"]
env:
- name: LOG_LEVEL
value: debug
- name: REDIS_SOCKET_TYPE
value: tcp
- name: REDIS_URL
value: redis:6379
- name: USE_STATSD
value: "false"
- name: RUNTIME_ROOT
value: /data
- name: RUNTIME_SUBDIRECTORY
value: ratelimit
- name: RUNTIME_WATCH_ROOT
value: "false"
- name: RUNTIME_IGNOREDOTFILES
value: "true"
- name: HOST
value: "::"
- name: GRPC_HOST
value: "::"
ports:
- containerPort: 8080
- containerPort: 8081
- containerPort: 6070
volumeMounts:
- name: config-volume
mountPath: /data/ratelimit/config
volumes:
- name: config-volume
configMap:
name: ratelimit-config

@NamanGajjar26
Copy link
Author

NamanGajjar26 commented Apr 22, 2024 •
edited

this is log and pods
minikube kubectl -- get all -n istioapi

`NAME                                          READY   STATUS    RESTARTS       AGE    
pod/demoapidock-deployment-576d455d5c-bcv5k   2/2     Running   2 (95s ago)    55m
pod/ratelimit-588556ff5-njqxx                 2/2     Running   4 (37s ago)    54m
pod/redis-6d4886fcb4-5sl7d                    2/2     Running   2 (95s ago)    54m

NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE   
service/demoapidock-service   NodePort    10.103.149.237   <none>        5028:31773/TCP               55m
service/ratelimit             ClusterIP   10.102.230.212   <none>        8080/TCP,8081/TCP,6070/TCP   54m
service/redis                 ClusterIP   10.111.47.118    <none>        6379/TCP                     54m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE   
deployment.apps/demoapidock-deployment   1/1     1            1           55m
deployment.apps/ratelimit                1/1     1            1           54m
deployment.apps/redis                    1/1     1            1           54m

NAME                                                DESIRED   CURRENT   READY   AGE  
replicaset.apps/demoapidock-deployment-576d455d5c   1         1         1       55m
replicaset.apps/ratelimit-588556ff5                 1         1         1       54m
replicaset.apps/redis-6d4886fcb4                    1         1         1       54m

`
minikube kubectl -- logs -l app=ratelimit --tail=-1 -n istioapi

`time="2024-04-22T09:33:36Z" level=info msg="Stats initialized for stdout"
time="2024-04-22T09:33:36Z" level=info msg="Stats flush interval: 10s"
time="2024-04-22T09:33:36Z" level=info msg="Tracing disabled"
time="2024-04-22T09:33:36Z" level=warning msg="connecting to redis on redis:6379 with pool size 10"
time="2024-04-22T09:33:36Z" level=debug msg="Implicit pipelining enabled: false"
time="2024-04-22T09:33:36Z" level=debug msg="loading domain: api-ratelimit"
time="2024-04-22T09:33:36Z" level=debug msg="loading descriptor: key=api-ratelimit.ratelimitheader"
time="2024-04-22T09:33:36Z" level=debug msg="Creating stats for key: 'api-ratelimit.ratelimitheader.PATH_/api/CustomerControllers'"
time="2024-04-22T09:33:36Z" level=debug msg="loading descriptor: key=api-ratelimit.ratelimitheader.PATH_/api/CustomerControllers ratelimit={requests_per_unit=1, unit=MINUTE, unlimited=false, shadow_mode=false}"
time="2024-04-22T09:33:36Z" level=info msg="Waiting for initial ratelimit config update event"
time="2024-04-22T09:33:36Z" level=info msg="Successfully loaded new configuration"
time="2024-04-22T09:33:36Z" level=info msg="Successfully loaded the initial ratelimit configs"
time="2024-04-22T09:33:36Z" level=warning msg="Listening for HTTP on '[::]:8080'"
time="2024-04-22T09:33:36Z" level=debug msg="waiting for runtime update"
time="2024-04-22T09:33:36Z" level=debug msg="Waiting for config update event"
time="2024-04-22T09:33:36Z" level=warning msg="Listening for debug on '0.0.0.0:6070'"
time="2024-04-22T09:33:36Z" level=warning msg="Listening for gRPC on '[::]:8081'"
{"level":"info","ts":1713778426.634342,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.go.totalAlloc","type":"counter","value":"2037168.000000"}}
{"level":"info","ts":1713778426.636543,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.go.mallocs","type":"counter","value":"12377.000000"}}
{"level":"info","ts":1713778426.636551,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.service.config_load_success","type":"counter","value":"1.000000"}}
{"level":"info","ts":1713778426.636555,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"runtime.load_attempts","type":"counter","value":"1.000000"}}
{"level":"info","ts":1713778426.636559,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.go.frees","type":"counter","value":"1845.000000"}}
{"level":"info","ts":1713778426.636562,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.redis_pool.cx_total","type":"counter","value":"10.000000"}}
{"level":"info","ts":1713778426.636571,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.gcCPUPercent","type":"gauge","value":"0.000000"}}
{"level":"info","ts":1713778426.636672,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.numGoroutine","type":"gauge","value":"16.000000"}}
{"level":"info","ts":1713778426.636680,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.sys","type":"gauge","value":"0.000000"}}
{"level":"info","ts":1713778426.636684,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapInuse","type":"gauge","value":"3670016.000000"}}
{"level":"info","ts":1713778426.636689,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapReleased","type":"gauge","value":"3735552.000000"}}
{"level":"info","ts":1713778426.636694,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapObjects","type":"gauge","value":"10532.000000"}}
{"level":"info","ts":1713778426.636699,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.alloc","type":"gauge","value":"2037168.000000"}}
{"level":"info","ts":1713778426.636706,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapAlloc","type":"gauge","value":"2037168.000000"}}
{"level":"info","ts":1713778426.636714,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.lastGC","type":"gauge","value":"0.000000"}}
{"level":"info","ts":1713778426.636719,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"runtime.num_values","type":"gauge","value":"1.000000"}}
{"level":"info","ts":1713778426.636723,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.redis_pool.cx_active","type":"gauge","value":"10.000000"}}
{"level":"info","ts":1713778426.636730,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapSys","type":"gauge","value":"7438336.000000"}}
{"level":"info","ts":1713778426.636734,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapIdle","type":"gauge","value":"3768320.000000"}}
{"level":"info","ts":1713778426.636739,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.nextGC","type":"gauge","value":"4194304.000000"}}
{"level":"info","ts":1713778426.636745,"logger":"gostats.loggingsink","msg":"flushing all stats","json":{"type":"all stats","value":"0.000000"}}
{"level":"info","ts":1713778436.633376,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.go.frees","type":"counter","value":"30.000000"}}
{"level":"info","ts":1713778436.633422,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.go.totalAlloc","type":"counter","value":"26128.000000"}}
{"level":"info","ts":1713778436.633429,"logger":"gostats.loggingsink","msg":"flushing counter","json":{"name":"ratelimit.go.mallocs","type":"counter","value":"536.000000"}}
{"level":"info","ts":1713778436.633435,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapObjects","type":"gauge","value":"11038.000000"}}
{"level":"info","ts":1713778436.633440,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapAlloc","type":"gauge","value":"2063296.000000"}}
{"level":"info","ts":1713778436.633444,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.alloc","type":"gauge","value":"2063296.000000"}}
{"level":"info","ts":1713778436.633448,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapIdle","type":"gauge","value":"3768320.000000"}}
{"level":"info","ts":1713778436.633452,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.nextGC","type":"gauge","value":"4194304.000000"}}
{"level":"info","ts":1713778436.633456,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.lastGC","type":"gauge","value":"0.000000"}}
{"level":"info","ts":1713778436.633460,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"runtime.num_values","type":"gauge","value":"1.000000"}}
{"level":"info","ts":1713778436.633464,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.redis_pool.cx_active","type":"gauge","value":"10.000000"}}
{"level":"info","ts":1713778436.633468,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapSys","type":"gauge","value":"7438336.000000"}}
{"level":"info","ts":1713778436.633472,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapInuse","type":"gauge","value":"3670016.000000"}}
{"level":"info","ts":1713778436.633476,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.heapReleased","type":"gauge","value":"3735552.000000"}}
{"level":"info","ts":1713778436.633481,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.gcCPUPercent","type":"gauge","value":"0.000000"}}
{"level":"info","ts":1713778436.633485,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.numGoroutine","type":"gauge","value":"16.000000"}}
{"level":"info","ts":1713778436.633489,"logger":"gostats.loggingsink","msg":"flushing gauge","json":{"name":"ratelimit.go.sys","type":"gauge","value":"0.000000"}}
{"level":"info","ts":1713778436.633493,"logger":"gostats.loggingsink","msg":"flushing all stats","json":{"type":"all stats","value":"0.000000"}}`

@NamanGajjar26
Copy link
Author

any update!
please help me solved this error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zirain @NamanGajjar26