Opensource networking -Building SONiC NOS images using gitlab-ci

A few months ago I began looking into what options are there for opensource or "flexible" switching needs. I found about these keywords:

1. ONIE - https://opencomputeproject.github.io/onie/
2. SONiC - https://sonicfoundation.dev/
3. DentOS - https://dent.dev/

The first keyword leads us to the "Open Network Install Environment" - the bootloader that is widely adopted by many vendors which "enables" the world of switching to be more flexible.

The second and third keywords are the current "Alive" and prominent opensource NOS (Network Operating Systems) that seem to have a future. They both approach at the problem of creating a NOS a little differently, but you can search for that on your own. Of the two - SONiC is much more mature in 2024, fulfills more needs and supports a lot more hardware in the current state.

How do I get SONiC?

After dealing with finding out which hardware supports SONiC you usually run into an issue - the public images provided do not have enough builtin features. That's where the versatility of SONiC comes into play. You can download and build your own binaries rather easily. Of course there are some caveats - sonic is built very WILDLY to put it easily.

You can check out the build repository here, this is the most important piece of the puzzle for us -> https://github.com/sonic-net/sonic-buildimage

Because I have a working Gitlab installation at hand I decided to automate the building process, instead of having to write make file commands via CLI, i have gitlab-ci overseeing the build process.

Requirements for building your own SONiC with gitlab-ci:

1. Ubuntu virtual machine or a bare metal server for building.
2. Gitlab Runner running on the VM or on the bare metal server. (https://docs.gitlab.com/runner/)
   1. NB! I tried to use a DinD runner using the docker executor, which is the recommended way by Gitlab, but I ran into issues that stem from the way SONiC is built. Maybe I could get it working, but for now I am using the shell executor. (It does have some caveats regarding having a "clean" build environment each runthrough.
   2. I used this ansible role to set up the runner: https://github.com/riemers/ansible-gitlab-runner
3. A lot of RAM and DISK storage (look at the recommended specs from the sonic-buildimage docs.)

Some more caveats...

Because of the shell-executor that gitlab-runner provides and the use of some sudo commands from sonic-buildimage. I have created an sudoers entry for the gitlab-runner user, so the CI can activate the commands without using the password.

gitlab-runner ALL=(ALL) NOPASSWD: /usr/bin/chown -R gitlab-runner ./sonic-buildimage/fsroot-vs
gitlab-runner ALL=(ALL) NOPASSWD: /usr/bin/chown -R gitlab-runner ./sonic-buildimage/fsroot-broadcom
gitlab-runner ALL=(ALL) NOPASSWD: /usr/bin/chown -R gitlab-runner ./sonic-buildimage/fsroot-broadcom-dnx
gitlab-runner ALL=(ALL) NOPASSWD: /usr/bin/chown -R gitlab-runner ./sonic-buildimage/fsroot.docker.bullseye
gitlab-runner ALL=(ALL) NOPASSWD: /usr/bin/rm -rf fsroot*

Of course you can also use this sudoers config so you don't have to whitelist every single command...

gitlab-runner ALL=(ALL) NOPASSWD:ALL

The actual CI file

The actual .gitlab-ci.yml file looks like this:

---
stages:
  - build
  - upload

variables:
  UNATTENDED: 1

### BUILD STAGE ###
.build_sonic:
  stage: build
  timeout: 12h
  before_script:
    - docker info
    - export PATH="~/.local/bin:$PATH"
    - pip3 install --user j2cli --quiet
    - if [[ ! -d "$CI_PROJECT_DIR/sonic-buildimage" || ! -d "$CI_PROJECT_DIR/sonic-buildimage/.git" ]]; then git clone --recurse-submodules https://github.com/sonic-net/sonic-buildimage.git && cd sonic-buildimage && make reset; fi
    - docker system prune -a -f
    - git checkout "$CI_COMMIT_REF_NAME"
    - sed -i 's/SONIC_CONFIG_BUILD_JOBS = 1/SONIC_CONFIG_BUILD_JOBS = $(shell nproc)/' rules/config
    - sed -i 's/INCLUDE_ICCPD.*\sn/INCLUDE_ICCPD = y/' rules/config
    - sed -i 's/INCLUDE_RESTAPI.*\sn/INCLUDE_RESTAPI = y/' rules/config
    - sed -i 's/.*ENABLE_TRANSLIB_WRITE\s=\s.*/ENABLE_TRANSLIB_WRITE = y/' rules/config
    - sed -i 's/INCLUDE_FIPS.*\sy/INCLUDE_FIPS = n/' rules/config
    - sed -i 's/.*ENABLE_ZTP\s=\s.*/ENABLE_ZTP = y/' rules/config
    #- echo "SONICYANG_IMPORTS += sonic-vlan.yang" >> src/sonic-mgmt-common/models/yang/sonic/import.mk
    #- echo "SONICYANG_IMPORTS += sonic-portchannel.yang" >> src/sonic-mgmt-common/models/yang/sonic/import.mk
    - sed -i '541 i sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install "requests<2.32.0"' ./build_debian.sh
  script:
    - make init
    - make configure PLATFORM=$PLATFORM_TO_BUILD
    - make SONIC_BUILD_JOBS=$(nproc) target/$TARGET_TO_BUILD
  after_script:
    - if [ -d "$CI_PROJECT_DIR/sonic-buildimage/fsroot-vs" ]; then echo "fsroot-vs directory exists. Change ownership." && sudo chown -R gitlab-runner ./sonic-buildimage/fsroot-vs; fi
    - if [ -d "$CI_PROJECT_DIR/sonic-buildimage/fsroot-broadcom" ]; then echo "fsroot-broadcom exists. Changing ownership." && sudo chown -R gitlab-runner ./sonic-buildimage/fsroot-broadcom; fi
    - if [ -d "$CI_PROJECT_DIR/sonic-buildimage/fsroot-broadcom-dnx" ]; then echo "fsroot-broadcom-dnx exists. Changing ownership." && sudo chown -R gitlab-runner ./sonic-buildimage/fsroot-broadcom-dnx; fi
    - if [ -d "$CI_PROJECT_DIR/sonic-buildimage/fsroot.docker.bullseye" ]; then echo "fsroot.docker.bullseye exists. Changing ownership." && sudo chown -R gitlab-runner ./sonic-buildimage/fsroot.docker.bullseye; fi
  artifacts:
    name: $TARGET_TO_BUILD
    paths:
      - sonic-buildimage/target/$TARGET_TO_BUILD
    expire_in: 3 days
  when: manual

build_sonic_vs_x64:
  extends: .build_sonic
  variables:
    PLATFORM_TO_BUILD: vs
    PLATFORM_ARCH_TO_BUILD: amd64
    TARGET_TO_BUILD: sonic-vs.img.gz
  when: manual

build_sonic_broadcom_x64:
  extends: .build_sonic
  variables:
    PLATFORM_TO_BUILD: broadcom
    PLATFORM_ARCH_TO_BUILD: amd64
    TARGET_TO_BUILD: sonic-broadcom.bin
  when: manual
### END BUILD STAGE ###

### UPLOAD STAGE ###
.upload_sonic:
  stage: upload
  variables:
    PACKAGE_REGISTRY_URL: '${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/sonic-${PLATFORM_TO_BUILD}-${PLATFORM_ARCH_TO_BUILD}/sonic-${CI_COMMIT_BRANCH}-build-${CI_JOB_ID}/${TARGET_TO_BUILD}'
  script:
    - |
      curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file sonic-buildimage/target/${TARGET_TO_BUILD} "${PACKAGE_REGISTRY_URL}"

upload_sonic_vs_x64:
  extends: .upload_sonic
  variables:
    PLATFORM_TO_BUILD: vs
    PLATFORM_ARCH_TO_BUILD: amd64
    TARGET_TO_BUILD: sonic-vs.img.gz
  needs: ['build_sonic_vs_x64']
  dependencies: ['build_sonic_vs_x64']

upload_sonic_broadcom_x64:
  extends: .upload_sonic
  variables:
    PLATFORM_TO_BUILD: broadcom
    PLATFORM_ARCH_TO_BUILD: amd64
    TARGET_TO_BUILD: sonic-broadcom.bin
  needs: ['build_sonic_broadcom_x64']
  dependencies: ['build_sonic_broadcom_x64']
### END UPLOAD STAGE ###

All of the jobs are triggered manually, as it is a rather lengthy process to build them. Also the Gitlab shell executor is not very isolation friendly for parallel processes.

As you can see from the code I only currently build for VS and Broadcom, but adding new other variants is simple.

Adding custom "patches" and fixes is another thing that is easily done via gitlab-ci. As you can see I am modifying build parameters as well via common Linux utils like sed. So I don't have to make my own fork of sonic-buildimage repository.

The Pipeline view looks something like this:

The upload jobs sends the built artifacts as a package to the internal generic package registry.

Conclusion

I hope this helps people build their own versions of SONiC more easily. As the build process is rather complicated from the official source and it takes a lot of time, so for it to run in the background with an UI to control all of it, makes life just a little bit easier 😄