Skip to content

nakamasato/monolith

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

137 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Monolith

What is it?

Monolith is a deep learning framework for large scale recommendation modeling. It introduces two important features which are crucial for advanced recommendation system:

  • collisionless embedding tables guarantees unique represeantion for different id features
  • real time training captures the latest hotspots and help users to discover new intersts rapidly

Monolith is built on the top of TensorFlow and supports batch/real-time training and serving.

Quick start

Prerequisite

  • Python 3.8.6
  • Bazel Version 4.0.0
  • Java: 1.8.0
  • Kafka: 2.13-2.8.1

Python setup

asdf local python 3.8.6 python -m venv .venv . .venv/bin/activate pip install -r requirements.txt 

Bazel setup

export BAZEL_VERSION=4.0.0 curl -fLO "https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-installer-darwin-x86_64.sh" chmod +x "bazel-$BAZEL_VERSION-installer-darwin-x86_64.sh" ./bazel-$BAZEL_VERSION-installer-darwin-x86_64.sh --user export PATH="$PATH:$HOME/bin" 

Examples

  1. native_training:demo: bazel run //monolith/native_training:demo --output_filter=IGNORE_LOGS ⚠️ (segmentation fault)
  2. batch: bazel run //markdown/demo:demo_local_runner -- --training_type=batch ⚠️ (segmentation fault)
  3. streaming: ❌ static assertion failed due to requirement
  4. Kubernetes:
    1. EKS: todo
    2. GKE: todo

Demo and tutorials

There are a tutorial in markdown/demo on how to run distributed async training, and few guides on how to use the MonolithModel API here.

Tips

When you encounter a problem and want to clean up the bazel cache, you can use the following command:

bazel clean --expunge 

About

ByteDance's Recommendation System

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages

  • Python 56.7%
  • C++ 38.7%
  • Starlark 3.2%
  • Go 0.8%
  • Shell 0.3%
  • Dockerfile 0.2%
  • Makefile 0.1%