Istio Traffic Connector

Traffic Connectors
1

Hey I was trying to set up the Istio Traffic connector, but the sidecar container is getting an error

2024-06-05T05:20:16.096234Z     info    Opening status port 15020                                                                                                                                                                                                                 
2024-06-05T05:20:16.686188Z     info    ads     All caches have been synced up in 1.141395556s, marking server ready                                                                                                                                                              
2024-06-05T05:20:16.695969Z     info    xdsproxy        Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"                                                                                                                               
2024-06-05T05:20:16.698012Z     info    Pilot SAN: [istiod.istio-system.svc]                                                                                                                                                                                                      
2024-06-05T05:20:16.787888Z     info    Starting proxy agent                                                                                                                                                                                                                      
2024-06-05T05:20:16.774013Z     info    sds     Starting SDS grpc server                                                                                                                                                                                                          
2024-06-05T05:20:16.774169Z     info    starting Http service at 127.0.0.1:15004                                                                                                                                                                                                  
2024-06-05T05:20:16.929341Z     info    starting                                                                                                                                                                                                                                  
2024-06-05T05:20:16.929534Z     info    Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields --log-format %Y-%m-%dT%T.
%fZ     %l      envoy %n %g:%#  %v      thread=%t -l warning --component-log-level misc:error --concurrency 2]                                                                                                                                                                    
2024-06-05T05:20:17.398684Z     info    cache   generated new workload certificate      latency=690.613183ms ttl=23h59m59.601322354s                                                                                                                                              
2024-06-05T05:20:17.398892Z     info    cache   Root cert has changed, start rotating root cert                                                                                                                                                                                   
2024-06-05T05:20:17.398961Z     info    ads     XDS: Incremental Pushing ConnectedEndpoints:0 Version:                                                                                                                                                                            
2024-06-05T05:20:17.399050Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.600951394s                                                                                                                                                           
2024-06-05T05:20:19.146729Z     info    xdsproxy        connected to upstream XDS server: istiod.istio-system.svc:15012                                                                                                                                                           
2024-06-05T05:20:19.583701Z     info    ads     ADS: new connection for node:productpage-v1-58ddb45974-dhmkm.default-1                                                                                                                                                            
2024-06-05T05:20:19.583949Z     info    cache   returned workload certificate from cache        ttl=23h59m57.416055199s                                                                                                                                                           
2024-06-05T05:20:19.584437Z     info    ads     SDS: PUSH request for node:productpage-v1-58ddb45974-dhmkm.default resources:1 size:4.0kB resource:default                                                                                                                        
2024-06-05T05:20:19.585355Z     info    ads     ADS: new connection for node:productpage-v1-58ddb45974-dhmkm.default-2                                                                                                                                                            
2024-06-05T05:20:19.585994Z     info    cache   returned workload trust anchor from cache       ttl=23h59m57.414008904s                                                                                                                                                           
2024-06-05T05:20:19.586248Z     info    ads     SDS: PUSH request for node:productpage-v1-58ddb45974-dhmkm.default resources:1 size:1.1kB resource:ROOTCA                                                                                                                         
2024-06-05T05:20:19.967584Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load NullVM plugin     thread=14                                                                                                         
2024-06-05T05:20:19.967655Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load Wasm code thread=14                                                                                                                 
2024-06-05T05:20:19.967832Z     critical        envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:475     Plugin configured to fail closed failed to load thread=14                                                                                                 
2024-06-05T05:20:20.285950Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load NullVM plugin     thread=14                                                                                                         
2024-06-05T05:20:20.294307Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load Wasm code thread=14                                                                                                                 
2024-06-05T05:20:20.294487Z     critical        envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:475     Plugin configured to fail closed failed to load thread=14                                                                                                 
2024-06-05T05:20:20.295879Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load NullVM plugin     thread=14                                                                                                         
2024-06-05T05:20:20.295889Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load Wasm code thread=14                                                                                                                 
2024-06-05T05:20:20.296008Z     critical        envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:475     Plugin configured to fail closed failed to load thread=14                                                                                                 
2024-06-05T05:20:20.297024Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load NullVM plugin     thread=14                                                                                                         
2024-06-05T05:20:20.297033Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load Wasm code thread=14                                                                                                                 
2024-06-05T05:20:20.297152Z     critical        envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:475     Plugin configured to fail closed failed to load thread=14                                                                                                 
2024-06-05T05:20:20.306128Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load NullVM plugin     thread=14                                                                                                         
2024-06-05T05:20:20.306142Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load Wasm code thread=14                                                                                                                 
2024-06-05T05:20:20.306270Z     critical        envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:475     Plugin configured to fail closed failed to load thread=14                                                                                                 
2024-06-05T05:20:20.307246Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load NullVM plugin     thread=14                                                                                                         
2024-06-05T05:20:20.307255Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load Wasm code thread=14                                                                                                                 
2024-06-05T05:20:20.307378Z     critical        envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:475     Plugin configured to fail closed failed to load thread=14                                                                                                 
2024-06-05T05:20:20.308365Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load NullVM plugin     thread=14                                                                                                         
2024-06-05T05:20:20.308374Z     error   envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:109     Wasm VM failed Failed to load Wasm code thread=14                                                                                                                 
2024-06-05T05:20:20.308492Z     critical        envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:475     Plugin configured to fail closed failed to load thread=14                                                                                                 
2024-06-05T05:20:20.359520Z     warning envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138    gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 10.152.183.10_91
53: Unable to create Wasm HTTP filter                                                                                                                                                                                                                                             
0.0.0.0_15014: Unable to create Wasm HTTP filter                                                                                                                                                                                                                                  
0.0.0.0_80: Unable to create Wasm HTTP filter                                                                                                                                                                                                                                     
10.152.183.235_15021: Unable to create Wasm HTTP filter                                                                                                                                                                                                                           
0.0.0.0_9080: Unable to create Wasm HTTP filter                                                                                                                                                                                                                                   
0.0.0.0_15010: Unable to create Wasm HTTP filter                                                                                                                                                                                                                                  
10.152.183.249_5000: Unable to create Wasm HTTP filter                                                                                                                                                                                                                            
virtualInbound: Didn't find a registered implementation for 'istio_authn' with type URL: 'io.istio.network.authn.Config'                                                                                                                                                          
        thread=14                                                                                                                                                                                                                                                                 
2024-06-05T05:20:20.384909Z     warn    Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected                                                                                 
2024-06-05T05:20:22.364327Z     warn    Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
2024-06-05T05:20:24.365125Z     warn    Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
2024-06-05T05:20:26.365725Z     warn    Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
2024-06-05T05:20:28.364799Z     warn    Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
2024-06-05T05:20:30.375507Z     warn    Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
2024-06-05T05:20:32.363656Z     warn    Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
2024-06-05T05:20:34.364960Z     warn    Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected

I had updated the base image of the istio-filter to 1.22.0 since my cluster is running that istio version.

Please suggest me a way to get it working.

2

Hi, I’m Shivansh from the Akto team, let me check this error and get back to you.

1 Like
3

Alright shivansh, I hope your traffic connector works in any Istio version. Please get back to me soon.

4

Is there a solution to this error?

5

Hi @rootshell348

It seems you are trying to run bookinfo sample application that uses istio. I have been successfully able to install my istio version 1.22.0 on bookinfo and get it running. Here are the steps I followed -Now I proceeded with the following steps -

1. Create an EC2 instance on AWS (Amazon Linux 2023) and installed docker, kubectl, minikube
2. minikube start --force
3. yum install -y git
4. git clone https://github.com/akto-api-security/istio-filter.git
5. cd istio-filter/
6. vi Dockerfile (changed version to 1.22.0 here)
7. docker buildx build --platform linux/amd64 -t aktosecurity/istio-proxy:latest .
8. docker push aktosecurity/istio-proxy:latest
9. curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.22.0 TARGET_ARCH=x86_64 sh -
10. cd istio-1.22.0
11. export PATH=$PWD/bin:$PATH
12. ls
13. istioctl install --set profile=demo -y
14. kubectl label namespace default istio-injection=enabled
15. kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
16. vi samples/bookinfo/platform/kube/bookinfo.yaml and add (istio-proxy sidecar in `productpage` pod)
17. kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

Here are my logs for istio-proxy container:

[root@ip-172-31-22-109 istio-1.22.0]# kubectl logs productpage-v1-5ff9b44b76-l2ltg -c istio-proxy
2024-06-06T07:17:37.896594Z	info	FLAG: --concurrency="0"
2024-06-06T07:17:37.896615Z	info	FLAG: --domain="default.svc.cluster.local"
2024-06-06T07:17:37.896621Z	info	FLAG: --help="false"
2024-06-06T07:17:37.896624Z	info	FLAG: --log_as_json="false"
2024-06-06T07:17:37.896627Z	info	FLAG: --log_caller=""
2024-06-06T07:17:37.896630Z	info	FLAG: --log_output_level="default:info"
2024-06-06T07:17:37.896633Z	info	FLAG: --log_rotate=""
2024-06-06T07:17:37.896636Z	info	FLAG: --log_rotate_max_age="30"
2024-06-06T07:17:37.896702Z	info	FLAG: --log_rotate_max_backups="1000"
2024-06-06T07:17:37.896723Z	info	FLAG: --log_rotate_max_size="104857600"
2024-06-06T07:17:37.896743Z	info	FLAG: --log_stacktrace_level="default:none"
2024-06-06T07:17:37.896770Z	info	FLAG: --log_target="[stdout]"
2024-06-06T07:17:37.896788Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2024-06-06T07:17:37.896805Z	info	FLAG: --outlierLogPath=""
2024-06-06T07:17:37.896824Z	info	FLAG: --profiling="true"
2024-06-06T07:17:37.896842Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2024-06-06T07:17:37.896861Z	info	FLAG: --proxyLogLevel="warning"
2024-06-06T07:17:37.896878Z	info	FLAG: --serviceCluster="istio-proxy"
2024-06-06T07:17:37.896897Z	info	FLAG: --stsPort="0"
2024-06-06T07:17:37.896915Z	info	FLAG: --templateFile=""
2024-06-06T07:17:37.896934Z	info	FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2024-06-06T07:17:37.896955Z	info	FLAG: --vklog="0"
2024-06-06T07:17:37.896981Z	info	Version 1.22.0-aaf597fbfae607adf4bb4e77538a7ea98995328a-Clean
2024-06-06T07:17:37.897001Z	info	Set max file descriptors (ulimit -n) to: 1048576
2024-06-06T07:17:37.897221Z	info	Proxy role	ips=[10.244.0.13] type=sidecar id=productpage-v1-5ff9b44b76-l2ltg.default domain=default.svc.cluster.local
2024-06-06T07:17:37.897332Z	info	Apply proxy config from env {}
2024-06-06T07:17:37.898451Z	info	cpu limit detected as 2, setting concurrency
2024-06-06T07:17:37.898686Z	info	Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
proxyAdminPort: 15000
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
2024-06-06T07:17:37.898724Z	info	JWT policy is third-party-jwt
2024-06-06T07:17:37.898744Z	info	using credential fetcher of JWT type in cluster.local trust domain
2024-06-06T07:17:38.001721Z	info	Prometheus scraping configuration: {true /metrics 9080}
2024-06-06T07:17:38.001799Z	info	Workload SDS socket not found. Starting Istio SDS Server
2024-06-06T07:17:38.001829Z	info	CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2024-06-06T07:17:38.001863Z	info	Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2024-06-06T07:17:38.002273Z	info	Opening status port 15020
2024-06-06T07:17:38.020423Z	info	ads	All caches have been synced up in 124.096479ms, marking server ready
2024-06-06T07:17:38.020641Z	info	xdsproxy	Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2024-06-06T07:17:38.022261Z	info	Pilot SAN: [istiod.istio-system.svc]
2024-06-06T07:17:38.023427Z	info	Starting proxy agent
2024-06-06T07:17:38.023469Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --concurrency 2]
2024-06-06T07:17:38.028155Z	info	sds	Starting SDS grpc server
2024-06-06T07:17:38.028382Z	info	starting Http service at 127.0.0.1:15004
2024-06-06T07:17:38.093012Z	warning	envoy main external/envoy/source/server/server.cc:835	Usage of the deprecated runtime key overload.global_downstream_max_connections, consider switching to `envoy.resource_monitors.downstream_connections` instead.This runtime key will be removed in future.	thread=12
2024-06-06T07:17:38.093688Z	warning	envoy main external/envoy/source/server/server.cc:928	There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.downstream_connections` resource monitor.	thread=12
2024-06-06T07:17:38.100681Z	info	xdsproxy	connected to delta upstream XDS server: istiod.istio-system.svc:15012	id=1
2024-06-06T07:17:38.125528Z	info	ads	ADS: new connection for node:productpage-v1-5ff9b44b76-l2ltg.default-1
2024-06-06T07:17:38.126725Z	info	ads	ADS: new connection for node:productpage-v1-5ff9b44b76-l2ltg.default-2
2024-06-06T07:17:38.228773Z	info	cache	generated new workload certificate	latency=207.923079ms ttl=23h59m59.77123147s
2024-06-06T07:17:38.228812Z	info	cache	Root cert has changed, start rotating root cert
2024-06-06T07:17:38.228830Z	info	ads	XDS: Incremental Pushing ConnectedEndpoints:2 Version:
2024-06-06T07:17:38.228881Z	info	cache	returned workload trust anchor from cache	ttl=23h59m59.771120578s
2024-06-06T07:17:38.228937Z	info	cache	returned workload certificate from cache	ttl=23h59m59.771064027s
2024-06-06T07:17:38.229179Z	info	cache	returned workload trust anchor from cache	ttl=23h59m59.770823193s
2024-06-06T07:17:38.229434Z	info	ads	SDS: PUSH request for node:productpage-v1-5ff9b44b76-l2ltg.default resources:1 size:1.1kB resource:ROOTCA
2024-06-06T07:17:38.229612Z	info	cache	returned workload trust anchor from cache	ttl=23h59m59.770389635s
2024-06-06T07:17:38.229288Z	info	ads	SDS: PUSH request for node:productpage-v1-5ff9b44b76-l2ltg.default resources:1 size:4.0kB resource:default
2024-06-06T07:17:39.253366Z	info	Readiness succeeded in 1.3625734s
2024-06-06T07:17:39.253777Z	info	Envoy proxy is ready

Can you try with these steps? Or even better - try with aktosecurity/istio-proxy:latest image that we just created? This will help understand if the problem is with the underlying OS or in the steps above.

6

Hi @shivansh, I think it seems to be working now. I was using microk8s earlier to test. Switching to minikube makes it work.

I have one more query. Does this proxy filter work in all istio versions?

7

The envoy filter uses lua v3 API and HTTP_FILTER for the implementation. Istio versions which support the same, should also work with the filter.

1 Like
8

Hey, I have a quick doubt. Can I use the k8s traffic connection daemonset even in an Istio cluster in order to collect traffic data?

Also I see in the lua script that the “host” field always returns 0.0.0.0 . How can I identify which pod or service the request has hit?

9

Hi @rootshell348

You can use the k8s daemonset with the istio cluster as well. Make sure that the istio cluster is not on mTLS and the TLS termination is happening before the pods, not at them, since the k8s daemonset does not support TLS directly.

For the other qn, whenever a service/pod is hit, the host header is populated for the same and that should be read by the istio filter in the “request headers”. The “0.0.0.0” in the istio filter is for a different purpose and is not used for determining the host of the service/pod.