Performance

info

The English user guide is currently in beta preview. Most of the documents have been automatically translated from the Japanese version. Should you find any inaccuracies, please reach out to Flatt Security.

For the vast majority of use cases, Takumi Runner delivers performance on par with or better than GitHub-hosted runners. This is a very important point for us, because we do not want users to endure a slower CI/CD execution time to use Takumi Runner.

Benchmark Results

The table below shows execution times for several CI workflows. It compares Takumi Runner to the GitHub-hosted runner.

Workflow	GitHub-hosted runner	Takumi Runner	Difference
Frontend CI	13m 26s	12m 58s	-28s
Rust CI	39m 37s	29m 16s	-10m 21s
Proto CI	51s	1m 56s	+1m 5s

The workflows used above contain typical CI flows such as linting, building, and running tests, with multiple jobs executed both in series and in parallel. Larger jobs, especially CPU-intensive ones, tend to show noticeable improvements, while small jobs sometimes end up slightly slower.

On Takumi Runner, job queue times tends to be slightly slower, usually by about 5-10s. This means a workflow with 3 jobs in series will slow the total execution time by about 15-30s. In most jobs though, the faster processing time will bring the total execution time to a faster time than the GitHub-hosted runner's execution time.

Frontend CI content

The Frontend CI checks for formatting, typescript typing, and runs jest tests, all in parallel.

jobs:
    check-formatting:
        name: Check formatting
        runs-on: <ubuntu-latest | takumi-runner>
        steps: <checkout repo, install neccesary packages, check lint for js, markdown, css, etc>

    check-typing:
        name: Check typing
        runs-on: <ubuntu-latest | takumi-runner>
        steps: <checkout repo, install neccesary packages, run pnpm typescript checks>

    jest:
        runs-on: <ubuntu-latest | takumi-runner>
        name: Jest test (${{ matrix.segment }} - ${{ matrix.chunk }})

        strategy:
            fail-fast: false
            matrix:
                segment: ['FOSS', 'EE']
                chunk: [1, 2, 3]

        steps: <checkout repo, install neccesary packages, test with jest>

    all-checks-pass:
        needs: [jest, check-formatting, check-typing]
        name: All checks pass
        runs-on: <ubuntu-latest | takumi-runner>
        steps: <check if all other jobs passed>

Rust CI content

The Rust CI builds, lints, and tests the application.

jobs:
    check-changes:
        runs-on: <ubuntu-latest | takumi-runner>
        name: Check for changes
        steps: <checkout repo, check for changes>

    build:
        name: Build services
        needs: check-changes
        runs-on: <ubuntu-latest | takumi-runner>
        steps: <checkout repo, install rust, build rust services>

    check-linting:
        name: Run linting
        needs: check-changes
        runs-on: <ubuntu-latest | takumi-runner>
        steps: <checkout repo, install neccesary packages, run linting>

    test:
        name: Run tests
        needs: check-changes
        runs-on: <ubuntu-latest | takumi-runner>
        steps: <checkout repo, install dependencies and packages, run tests>

    all-checks-pass:
        needs: [build, test, check-linting]
        name: All checks pass
        runs-on: <ubuntu-latest | takumi-runner>
        steps: <check if all other jobs passed>

Proto CI content

The Proto CI checks linting and for breaking changes.

jobs:
    check-changes:
        runs-on: <ubuntu-latest | takumi-runner>
        name: Check for changes
        outputs:
            proto: ${{ steps.filter.outputs.proto || 'true' }}
        steps: <checkout repo, check for changes>

    check-linting:
        name: Check linting
        needs: check-changes
        runs-on: <ubuntu-latest | takumi-runner>

        steps: <checkout repo, install buf, lint proto files>

    check-breaking-changes:
        name: Check for breaking changes
        needs: check-changes
        runs-on: <ubuntu-latest | takumi-runner>

        steps: <checkout repo, install buf, check for breaking changes>

    all-checks-pass:
        needs: [check-changes, check-linting, check-breaking-changes]
        name: All checks pass
        runs-on: <ubuntu-latest | takumi-runner>
        steps: <check if all other jobs passed>

Hardware Benchmarks

The following benchmarks compare the CPU, memory, and disk I/O performance of Takumi Runner against the GitHub-hosted runner. Both runners use 2 vCPUs and 7.8 GiB of RAM.

Metric	GitHub-hosted runner	Takumi Runner
CPU throughput	4,098 events/s	5,457 events/s
Memory bandwidth	8,493 MiB/s	3,433 MiB/s
Disk write	1,559 MiB/s	416 MiB/s
Disk read	117 MiB/s	1,588 MiB/s

sysbench and fio are used for these measurements. CPU performance has the largest impact on most CI/CD runtimes, because linting, building, and unit testing are primarily CPU-bound. Our top priority is therefore to ensure Takumi Runner CPUs outperform those on GitHub-hosted runners.