2 posts tagged with "intro"

When we first developed TVLK5, we include an express server to handle both route prefix (e.g: id-id) extraction from the URL and also to handle dynamic route requirement from product teams. The reason we use express for dynamic route was because we can't support next.js dynamic route feature since Next 9 originally launched.

When next.js is updated in 9.2.1, we finally able to integrate dynamic route into our framework.

If you're not interested in the detail you can skip to migration guide

Brief History​

When we start working on the framework last year, we have next.js 9 as the base to build upon. The problem was our URLs for customer facing site have this locale prefix to determine which country and language that the user is in. We added custom middleware in the express server to remove the route prefix by matching it with the regex and forward the prefix (if any) via headers. This headers then get picked up by our next.js application. From there, it knows the current route prefix and can validate entirely inside getInitialProps.

Why express middleware? Because we have requirement where user can create dynamic route such as /activities/:country/detail/:nameId without having to bother with route prefix when matching the url so our middleware get applied to all URLs, and the next handler can simply match the URL without prefix.

We didn't use next dynamic route from the start because we want to have first-party support for client-side navigation. In practice, using dynamic route is not possible because next.js was expecting consistent href and as which is impossible as our href which maps to file path doesn't have route prefix while as that displayed in the address bar has a route prefix. It doesn't scale to create route prefix directory for all products. Imagine if everytime you create a page you'll need to create N files, each in different route prefix directory. It's a nightmare.

Shortly after, Custom Route RFC is opened and it seems like we can finally use this to replace our route prefix logic in express. Only after this PR landed that allows mismatching href and as we can finally integrate seamlessly with next.js dynamic routes.

Going forward, this is the recommended way to create dynamic route.

Migration Guide​

If you have dynamic route in express, you've added both express middleware and 1 page handler prefixed by _ in its filename. To migrate to next dynamic route, you'll need to migrate all your dynamic routes in one commit because it's not currently possible to handle both at the same time.

First, you'll need to remove legacyRoutePrefixMiddleware in your express server. This middleware is used to make sure that you can still add dynamic route in your express server.

Next, rename all your dynamic route file (prefixed with _) into next.js dynamic route file format. You can use this guide when renaming your route file:

• Express uses :param while next.js uses [param]. For example if you have /activities/:country/detail/:nameId you can rename your route file to activities/[country]/detail/[nameId].tsx
• Express uses * for wildcard match, while next uses [...param]. For example if you have /explore/* you can rename your route file to explore/[...path].tsx

Generally you can use any string to name your params, except 2 reserved words: routePrefix and actualPath. This is because we use this in our next rewrites to handle route prefix extraction.

If you've renamed all your dynamic route files, the next step is removing all your dynamic route definition in your server logic. And you're done!

Make sure you do these before May 1st because we'll start throwing error when you use custom express server that relies on route prefix removal logic. In the meantime, you'll get warning log when you still use express dynamic route method.

By now you have to hear about this project called TVLK5. This post is an attempt to explain briefly what it is, how does it differ from how we used to develop products, what's changed since initial introduction and what's next.

TVLK5 is an umbrella project to improve web development in Traveloka customer facing site, specifically how we author our code. It's technically our 2nd big change (but who's counting right), with the first one being Blocks and TvPage.

It's easy to call this a framework but we don't actually hide any backing implementation this time. It consists of express+nextjs boilerplate and few modules that you can integrate with. If you need to modify server middleware, you still working with an express app. The same can be said with nextjs, you follow their convention, with the entry inside pages directory. Want to do something fancy with nextjs? Just head to their documentation and it should work the same.

This is the main difference with our legacy system where you're kinda sucked in to how the framework was built, with routes being defined in specific file not even using express convention, middleware replaced with Filters and so on.

To understand how much is changed compared to our previous framework you can read some of these comparison below

Hello World​

In TVLK5, to create new page you only need to touch single file (pages/route.tsx following next.js convention), compared to 5+ files you need to modify in previous framework (up to 2 routes, server module, client module, component, and webpack entry point).

If you need access to some properties, you might also need to configure your redux store. Ouch

Bonus point: you're not blocked by web-infra anymore for code review.

Build speed​

In CI, entire TVLK5 workflow (JS build, docker+push, and performance tests) is faster than JS build of our monolith app. With faster feedback loop means you wait less time for your PR to be merged just because there's minor adjustment that you have to make.

Locally, you get the benefit of HMR where you can see your change in an instant. In our testing, adding log statements to the code will take > 20 seconds in previous framework, while the new one only took ~2 seconds. That's 10x improvement for a single feedback loop.

Content Resource & friends​

In previous framework, fetching translation and any resource data are tied strongly to react-diode implementation which makes it difficult to optimize because CR/IR/IS have unique needs. In TVLK5 the syntax is simpler and still familiar, yet it have benefits like:

• Code split friendly. We leverage React.Suspense to make sure that you can place any resource query anywhere in the tree and it will be fetched only if your component renders it. You don't need to add children manually, it will automatically inferred based on request.

// Before:
Diode.createContainer(Component, {
  children: [Child],
  queries: Diode.createQuery(ContentResourceQuery, {
    GeneralLayout: {
      logIn: null,
    },
  }),
});

// After:
useContentResource({
  GeneralLayout: {
    logIn: '',
  },
});

• Dead fields elimination. When you fetch image resource / image sliders, there's a lot of fields returned by backend API where most of them aren't even used. In TVLK5, those fields will be automatically removed if you don't use it, similar to how GraphQL works.

// Before:
Diode.createContainer(Component, {
  children: [Child],
  queries: Diode.createQuery(ImageSliderQuery, {
    // All fields will be hydrated in JSON even if it's not used
    PaymentPartnersDark: {},
  }),
});

// After:
useImageSlider({
  PaymentPartnersDark: {
    // only link will be hydrated
    link: '',
  },
});

Mobile redirection​

This is common usage yet previously there's a few boilerplate when adding this functionality. In TVLK5, this can be as simple as single line change

// Before:
function pageRedirect(req, res) {
  const url = req.router.url('PAGE_ROUTE_KEY');
  return res.redirect(url);
}

// After:
Page.mobileRedirection = true;

There's an advanced redirection logic that you can do, but most of the time you don't need that anyway. Why bother writing to much code?

Confusing Lifecycle​

Previously we have requestReceived, beforeFetch and afterFetch. There's no clear way regarding what should be done inside those lifecycle, you only know what happen.

In TVLK5, by leveraging next.js we have getInitialProps that you can do to parse request data and fetch from backend API, or routeProtection that you can use to block access to specific route.

// Before
class Page extends TvPage {
  requestReceived(req, res) {
    // wait is there req.user?
    if (req.user) {
      res.status(403);
      return ResponseHandler.endPageLifecycle();
    }
  }
  beforeFetch(req, res) {
    super.beforeFetch(req, res);
    req.props.key = 'value';
  }
  afterFetch(req, res) {
    // ????
    super.afterFetch(req, res);
  }
}

// After
Page.getInitialProps = async (ctx, appCtx) => {
  return { key: 'value' };
};

Page.routeProtection = ctx => {
  if (ctx.user.loggedIn) {
    return { status: 200 };
  }

  return { status: 403 };
};

Parallel server-side API call​

Using Filter in previous framework means each filter is executed serially, this means that any API call happen inside different filter will be called sequentially.

In TVLK5, we optimize to fetch as much data as possible to reduce server latency. We even provide you with Server Timing to see how each server-side call performs.

Production debugging​

Due to modular approach, it's now feasible to test production build locally without breaking your laptop. You can build and run production build using either staging or production API.

# Before:
cd packages/nodejs-web
NODE_ENV=production pnpm build
## if it doesn't crash, then
NODE_ENV=production node lib/index.js

# After:
pnpm --filter @traveloka/webxpe-desktop build
pnpm start

Client-side routing​

Previously, to use client-side routing, you have to change a lot of things, starting from changing your base page, your route registration, your component render method, and more. This is assuming you don't want to code split because it's a different case altogether (thanks to Diode). It's not a pleasant experience.

In TVLK5, client-side routing get first class support, which means it works by default without you having to reconfigure anything. In next.js every page is different bundle so it's code-split by default.

// Before:
// duplicate path declaration with registration?
<Link to="/path/:key/another" />
// or "server-side" navigation
window.location.assign(
  // Don't forget to add this in RouteResourceGroups
  router.url('SOME_KEY_THAT_YOU_NEED_TO_FIND_IN_REGISTRATION_FILE')
)
// and another ceremonies


// After:
<Link route={ROUTE_PATH} params={{ key: 'value' }} />
// or programmatic client-side navigation
const route = useLocalizedRouter();
const [href, as] = route(ROUTE_PATH, {
  params: {
    key: 'value',
  }
});
Router.push(href, as)

Those are non-exhaustive list of what's new in TVLK5. If you're curious, read our getting started docs.

Migration Requirement​

If you want to migrate your services to use TVLK5 these are few things you should know:

• TVLK5 only officially supports @traveloka/web-components as component primitive. This is probably one of the major blocker / where you'll need to spend most effort in. We'd suggest you refactor your component to use web-components before performing any migration.
• There's no more dependency to nodejs-web. Each services can only depend to @traveloka/core, @traveloka/app-desktop, @traveloka/app-mobile, and other micro packages (such as @traveloka/user or @traveloka/merchandising). You can extract your utility modules to your own package, and make nodejs-web depend to that package, but not the other way around
• The new services will only handle language-country routing, e.g: /id-id prefix. This means your service have to be compliant with URL standardization effort. In the future nodejs-web will only be used as redirection for naked locale and legacy /en prefix.

Updates​

At this time of writing there's 3 product in production using TVLK5: Experience, DLP, and Explore. Thanks to experience team for becoming pilot for this adoption 🎉!

Since XPE team released their servies to production, we have made some improvement to the framework:

• There's now 3 experimental features that you can try to improve your page performance: SSR cache, progressive hydration, and static subtree
• We have completed migration of shared desktop and mobile components (header, footer, sidebar) available in @traveloka/app-desktop and @traveloka/app-mobile. If your page uses this component and you already use web-components for all your components, you'll have easier time migrating your services.

We also received contribution from various engineers in feature parity, bug fixes and other enhancements:

• Thanks to Agal, we now have better OTP integration and external login capability
• Thanks to Yusei we now have frictionless setup using Yeoman generator.
• Thanks to Ferdinand, Raibima, Billy, Paula for bug fixes and DX improvements.
• Thanks to Dzulham and Ferdinand we have external tracking manager setup with automatic segment pageview tracking.
• Thanks to Ryan, we have easy performance tracking with mock recorder. This means adding performance test is as easy as visiting your page in dev and modify 1 JSON file.
• Thanks to Abi and Asendia we have migrated to Jenkins with the ability to do partial build. This means in average new TVLK5 services can finish building in under 10 minutes, including performance tests (the production build itself should finish in about a minute).

What’s Next?​

While most of the groundwork has already finished, there's more that we haven't got to it yet:

• Adoption of nextjs advanced features (dynamic route, module/nomodule, rewrites). Using dynamic route allow us to create route with params without touching express code, while module/nomodule will improve our page performance in newer browsers due to different transpilation method
• Staging & Release DX improvement. This is only slightly related to TVLK5 itself but it's an issue. Currently there's so many thing to touch when you want to create new services and release it to production. We're planning to simplify this process so you can migrate your screen 1 by 1.
• Dev DX improvement (especially logging and hanging process). We noticed there's a strange bug where some process are not closed even after you stop the main proxy. This can cause issues in performarnce and battery life. If possible, we also like to remove nohoist limitation when you setup your packages. This could hopefully improve installation time
• Docs completion, most of the docs are already completed, but the advanced topics sections (especially Architecture) are still missing. We plan to finish the remaining docs in H2.

If you have any suggestion on what to improve in the framework, don't hesitate to contact us in frontend-club channel.