Hello everyone,
I have been fighting with this for 7 days now and am getting nowhere (only frustrated). I really hope someone can help me. Please keep in mind that I am no network expert, as I believe the problem lies here.
The problem:
Attempts to register a gitlab-runner results in this error: screenshot of the error
The setup
Everything is installed on a single server in my home network. So it's
- Laptop accessing the server
Internet - Router (FritzBox) 192.168.1.1 - Server 192.168.1.100
- Other
The server runs
- Ubuntu 18.04.4 LTS
- Docker version 19.03.8, build afacb8b7f0
I got my gitlab and gitlab-runner working a few months ago without https (I figured being a one-man team inside my own network I don't need https). I used docker-compose to run gitlab, postgresql and redis and 'normal' docker to run a gitlab-runner. That too was a struggle for me and it took me a while to figure out that I had to use url = "http://192.168.1.100:30080/" to register the runner.
But then I decided to upgrade to https using a self-signed certificate. I did this because I wanted to use the gitlab-buildin docker registry to speed up my builds, and as I understand that requires https.
I succeeded with gitlab. I can view my repositores, push changed, create issues and whatnot. But, as the title says, I am unable to register a gitlab-runner over https.
docker-compose.yml
Lets start with the docker-compose, which starts postgres, redis, gitlab and now also the gitlab-runner:
version: '3.7'
services:
postgresql:
restart: always
image: postgres:12-alpine
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "10"
environment:
- POSTGRES_USER=xxxxxxxxxxx
- POSTGRES_PASSWORD=xxxxxxxxxxx
- POSTGRES_DB=xxxxxxxxxxx
volumes:
- /opt/postgresql:/var/lib/postgresql:rw
redis:
restart: always
image: redis:5-alpine
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "10"
gitlab:
image: 'gitlab/gitlab-ce'
restart: always
hostname: 'treffer-technologies.home-webserver.de'
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "10"
links:
- postgresql:postgresql
- redis:redis
environment:
GITLAB_OMNIBUS_CONFIG: |
# postgres
postgresql['enable'] = false
gitlab_rails['db_username'] = "xxxxxxxxxxx"
gitlab_rails['db_password'] = "xxxxxxxxxxx"
gitlab_rails['db_host'] = "postgresql"
gitlab_rails['db_port'] = "5432"
gitlab_rails['db_database'] = "xxxxxxxxxxx"
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'utf8'
# redis
redis['enable'] = false
gitlab_rails['redis_host'] = 'redis'
gitlab_rails['redis_port'] = '6379'
# nginx
nginx['redirect_http_to_https'] = true
registry_nginx['redirect_http_to_https'] = true
# email
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.gmail.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "xxxxxxxxxxx"
gitlab_rails['smtp_password'] = "xxxxxxxxxxx"
gitlab_rails['smtp_domain'] = "xxxxxxxxxxx"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
# other
gitlab_rails['gitlab_shell_ssh_port'] = 30022
# https://docs.gitlab.com/omnibus/settings/ssl.html#lets-encrypt-integration
external_url 'https://treffer-technologies.home-webserver.de:30443'
# registry
registry_external_url 'https://treffer-technologies.home-webserver.de:30090'
ports:
# host:container
# both ports must match the port from external_url above
- "30080:30080"
# the mapped port must match ssh_port specified above.
- "30022:22"
# https
- "30443:30443"
# registry
- "30090:30090"
volumes:
- /opt/gitlab/config:/etc/gitlab:rw
- /opt/gitlab/log:/var/log/gitlab:rw
- /opt/gitlab/data:/var/opt/gitlab:rw
depends_on:
- postgresql
- redis
runner:
image: 'gitlab/gitlab-runner:alpine'
restart: always
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "10"
volumes:
- /opt/gitlab-runner/config:/etc/gitlab-runner
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- gitlab
As you can see, the url of my gitlab is https://treffer-technologies.home-webserver.de:30443.
gitlab-runner register
And here is the registration code:
docker run --rm -t -i -v /opt/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner:alpine --debug register \
--non-interactive \
--executor "docker" \
--docker-image alpine:3 \
--url "https://treffer-technologies.home-webserver.de:30443" \
--registration-token "xxxxxxxxxxx" \
--description "gitlab-runner-docker" \
--tag-list "build,test,deploy" \
--locked="false"
which, wenn executed, results in this error:
Runtime platform arch=amd64 os=linux pid=6 revision=4c96e5ad
version=12.9.0
Checking runtime mode GOOS=linux uid=0
Running in system-mode.
Trying to load /etc/gitlab-runner/certs/treffer-technologies.home-webserver.de.crt ...
Dialing: tcp treffer-technologies.home-webserver.de:30443 ...
ERROR: Registering runner... failed runner=xxxxxxxx status=couldn't execute
POST against https://treffer-technologies.home-webserver.de:30443/api/v4/runners:
Post https://treffer-technologies.home-webserver.de:30443/api/v4/runners:
dial tcp [2001:16b8:a582:1800:314f:5277:9434:77ad]:30443:
connect: cannot assign requested address
PANIC: Failed to register this runner. Perhaps you are having network problems
According to Supported options for self-signed certificates I copied the same certificate I created and use for my gitlab to /opt/gitlab-runner/config/certs/treffer-technologies.home-webserver.de.crt. The content beginns with -----BEGIN, so I think it is encoded in PEM.
Firewall
ufw is inactive until this problem is resolved.
Logs
As far as I can tell, the registration process is not reaching my gitlab, since I can find no signs of a request in the gitlab logs. This is why I believe I have a network problem.
Probing gitlab-runner container
Using docker-compose exec runner /bin/sh I found out that:
ping gitlab
PING gitlab (172.22.0.5): 56 data bytes
64 bytes from 172.22.0.5: seq=0 ttl=64 time=0.055 ms
64 bytes from 172.22.0.5: seq=1 ttl=64 time=0.105 ms
64 bytes from 172.22.0.5: seq=2 ttl=64 time=0.150 ms
64 bytes from 172.22.0.5: seq=3 ttl=64 time=0.154 ms
64 bytes from 172.22.0.5: seq=4 ttl=64 time=0.151 ms
^C
--- gitlab ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.055/0.123/0.154 ms
172.22.0.5 is exactly the IP of the docker-container gitlab, as expected. However, using register against https://gitlab:30443 results in
Dialing: tcp gitlab:30443 ...
ERROR: Registering runner... failed runner=xxxxxx
status=couldn't execute POST against https://gitlab:30443/api/v4/runners: Post https://gitlab:30443/api/v4/runners: dial tcp: lookup gitlab on 8.8.8.8:53: no such host
PANIC: Failed to register this runner. Perhaps you are having network problems
ping treffer-technologies.home-webserver.de
PING treffer-technologies.home-webserver.de (2001:16b8:a582:1800:314f:5277:9434:77ad): 56 data bytes
ping: sendto: Address not available
Adding the line
172.22.0.5 treffer-technologies.home-webserver.de
to the hosts of the gitlab-runner-container makes the ping work, but the register still results in
Trying to load /etc/gitlab-runner/certs/treffer-technologies.home-webserver.de.crt ...
Dialing: tcp treffer-technologies.home-webserver.de:30443 ...
ERROR: Registering runner... failed runner=xxxxxxxx status=couldn't execute POST against https://treffer-technologies.home-webserver.de:30443/api/v4/runners: Post https://treffer-technologies.home-webserver.de:30443/api/v4/runners: dial tcp [2001:16b8:a582:1800:314f:5277:9434:77ad]:30443: connect: cannot assign requested address
PANIC: Failed to register this runner. Perhaps you are having network problems
/etc/hosts
of gitlab-runner docker container
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.22.0.4 0181ad73e31f
# manually added to make ping work
# 172.22.0.5: gitlab-container
172.22.0.5 treffer-technologies.home-webserver.de
of host / the server
127.0.0.1 localhost
127.0.1.1 HP-ProDesk-400-G5-Desktop-Mini
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
The self-signed certificate
I used this guide. Basically, I did openssl genrsa and used treffer-technologies.home-webserver.de and FQDN. The content starts with ----- BEGIN.
More information
Thank you for reading all of this. If you want to help and need more information I will provide them as fast as I can. Thanks :)
edit: entered image description, typos, grammar (I am german), removed statement that this is my first question (not true, it is my second), added probing gitlab-container, changed ips to reflect the current state after my tinkering
