CI/CD Pipeline

Our CI/CD scripts reside in .github/workflows because we are using GitHub actions to build our services. Refer to those files (just read the comments) for the SSOT. Here, we present a short description.

Main build script

The main build script is .github/workflows/build_main.yml. It has some jobs, some of which calls the other build_xxx workflow files.

In general, the build process looks like the following diagram.

1. The init job

   • Checkouts the code and installs CI workspace dependencies. (We cache CI dependencies on S3 to make it a little bit faster. Also note that CI workspace dependencies are not the same as the main package.json dependencies.)
   • Generate build info: based on what files are changed, we determine (1) what services to build and (2) which jobs/pipelines to run.
   • Generate app version: we use the <branchOrPrNumber>-<commitNumber>-<commitHash> pattern, which may look like master-26708-be8e5f1a31 or pr1234-26708-abcdf1234.

2. The install job

   • This is the step where we install the main NPM dependiencies for our www app (root package.json)

3. The main build jobs

   • Some build process happens simultaneously during this phase
   • tvlk-xxx workflow matrix builds individual TVLK5 services (see .github/workflows/build_tvlk5.yml or the next section for more details)
   • unittest-xxx runs unit test per unit-test groups (which group it belongs to depends on the testPipelineName property inside a package's package.json)
   • Other service build pipelines (e.g. those for webapx and webstd)
   • Other testing pipeline (e.g. lint and typecheck)

4. The finish job

   • This basically checks the results of the preceding jobs/pipelines to ensure they are either successful or skipped (a job/pipeline might be skipped if we don't need to run it on the changed files)

TVLK5 build script

The following explanation pertains to the .github/workflows/build_tvlk5.yml workflow file.

A TVLK5 service build pipeline has 2 main jobs: "build" and "perftest"

1. The build phase

   • First we download the code and dependencies from S3.
   • Then we prepare the CI: download staging secrets (from AWS) and download next build cache (so the build can run faster if the cache from a previous build exists).
   • Then we run the api-mock service (so we can resolve all API calls during prerender locally instead of relying from the real API).
   • Then we build the service's Next.js application.
   • Then we do some post-build operations: validate the build result, standardize next build ID (so each services have the same build ID wven though they are built in separate environments), and remove SSG cache (forces Next to render the page again when it's first visited, this time with the real API rather than the mock API).
   • Then we upload the cache and the assets to S3.
   • Then we build the Docker image and uploads it to ECR.
   • Then we tell our staging (ASEN) that it should now use the new image version.