Unified at|docker_shell script

[luarss]

Fixes https://github.com/The-OpenROAD-Project-private/AutoTuner/issues/126

Diffs

• Replace ad-hoc at_shell/docker_shell scripts with a single etc/docker_shell supporting --image, --user, --orfs-flow, --port, --dry-run, and --help flags.

---

Mapping pwd to /OpenROAD-flow-scripts/flow (read-only)

• etc/docker_shell defaults to mounting pwd at /work, which is writable.

To use the AT flow:

etc/docker_shell --image autotuner:1.0 -- <command>

For the Rapidus/ORFS use case where pwd should overlay /OpenROAD-flow-scripts/flow, pass --orfs-flow:

etc/docker_shell --orfs-flow -- openroad -version 2>&1)

Sourcing of ../env.sh

• env.sh is sourced automatically only in --orfs-flow mode, where /OpenROAD-flow-scripts/flow is the working directory and OR tools need to be added to PATH.

Image selection priority

1. --image IMAGE flag
2. $OR_IMAGE environment variable
3. openroad/orfs:latest (built-in default)

@jeffng-or FYI

[luarss]

@jeffng-or FYI

[jeffng-or]

Doesn't quite seem to work correctly:

1. support make DESIGN_CONFIG=... - nor working

jeffng@dev-secure-jeffng-experiments:/workspace/bitdeerAT$ ./docker_shell.jack --image openroad/orfs-autotuner:26Q1-636-gb9f42fead make DESIGN_CONFIG=designs/asap7/gcd/config.mk metadata + DEFAULT_IMAGE=openroad/orfs:latest + MOUNT_TARGET=/work + USE_USER=false + DRY_RUN=false + PORTS=() + IMAGE= + CMD_ARGS=() + [[ 5 -gt 0 ]] + case "$1" in + IMAGE=openroad/orfs-autotuner:26Q1-636-gb9f42fead + shift 2 + [[ 3 -gt 0 ]] + case "$1" in + echo 'Unknown option: make' Unknown option: make + exit 1 jeffng@dev-secure-jeffng-experiments:/workspace/bitdeerAT$ ./docker_shell.jack --image openroad/orfs-autotuner:26Q1-636-gb9f42fead bash + DEFAULT_IMAGE=openroad/orfs:latest + MOUNT_TARGET=/work + USE_USER=false + DRY_RUN=false + PORTS=() + IMAGE= + CMD_ARGS=() + [[ 3 -gt 0 ]] + case "$1" in + IMAGE=openroad/orfs-autotuner:26Q1-636-gb9f42fead + shift 2 + [[ 1 -gt 0 ]] + case "$1" in + echo 'Unknown option: bash' Unknown option: bash + exit 1 jeffng@dev-secure-jeffng-experiments:/workspace/bitdeerAT$ ./docker_shell.jack --image openroad/orfs-autotuner:26Q1-636-gb9f42fead make DESIGN_CONFIG=designs/asap7/gcd/config.mk metadata + DEFAULT_IMAGE=openroad/orfs:latest + MOUNT_TARGET=/work + USE_USER=false + DRY_RUN=false + PORTS=() + IMAGE= + CMD_ARGS=() + [[ 5 -gt 0 ]] + case "$1" in + IMAGE=openroad/orfs-autotuner:26Q1-636-gb9f42fead + shift 2 + [[ 3 -gt 0 ]] + case "$1" in + echo 'Unknown option: make' Unknown option: make + exit 1 

2. Support interactive bash - not working

jeffng@dev-secure-jeffng-experiments:/workspace/bitdeerAT$ ./docker_shell.jack --image openroad/orfs-autotuner:26Q1-636-gb9f42fead make DESIGN_CONFIG=designs/asap7/gcd/config.mk metadata + DEFAULT_IMAGE=openroad/orfs:latest + MOUNT_TARGET=/work + USE_USER=false + DRY_RUN=false + PORTS=() + IMAGE= + CMD_ARGS=() + [[ 5 -gt 0 ]] + case "$1" in + IMAGE=openroad/orfs-autotuner:26Q1-636-gb9f42fead + shift 2 + [[ 3 -gt 0 ]] + case "$1" in + echo 'Unknown option: make' Unknown option: make + exit 1 jeffng@dev-secure-jeffng-experiments:/workspace/bitdeerAT$ ./docker_shell.jack --image openroad/orfs-autotuner:26Q1-636-gb9f42fead bash + DEFAULT_IMAGE=openroad/orfs:latest + MOUNT_TARGET=/work + USE_USER=false + DRY_RUN=false + PORTS=() + IMAGE= + CMD_ARGS=() + [[ 3 -gt 0 ]] + case "$1" in + IMAGE=openroad/orfs-autotuner:26Q1-636-gb9f42fead + shift 2 + [[ 1 -gt 0 ]] + case "$1" in + echo 'Unknown option: bash' Unknown option: bash + exit 1 

3. support openroad_autotuner - not working

jeffng@dev-secure-jeffng-experiments:/workspace/bitdeerAT$ ./docker_shell.jack --image openroad/orfs-autotuner:26Q1-636-gb9f42fead openroad_autotuner -s /OpenROAD-flow-scripts/flow/designs/asap7/gcd/asap7_gcd_sweep.yaml + DEFAULT_IMAGE=openroad/orfs:latest + MOUNT_TARGET=/work + USE_USER=false + DRY_RUN=false + PORTS=() + IMAGE= + CMD_ARGS=() + [[ 5 -gt 0 ]] + case "$1" in + IMAGE=openroad/orfs-autotuner:26Q1-636-gb9f42fead + shift 2 + [[ 3 -gt 0 ]] + case "$1" in + echo 'Unknown option: openroad_autotuner' Unknown option: openroad_autotuner + exit 1 

[luarss]

Hi Jeff, thanks for raising the bugs. I got it running from the./flow directory - does this mimic your dev flow?
Also I verified using the the below three commands :

# Issue 1: make DESIGN_CONFIG=...  # Run this from ./flow ./util/docker_shell --image openroad/orfs-autotuner:local make DESIGN_CONFIG=designs/asap7/gcd/config.mk metadata # Issue 2: interactive bash  ./util/docker_shell --image openroad/orfs-autotuner:local bash # Issue 3: openroad_autotuner  ./util/docker_shell --image openroad/orfs-autotuner:local openroad_autotuner --help

[jeffng-or]

I think I see the issue. In my environment (and what I'm going to suggest to customers) the initial starting directory in the container should be /OpenROAD-flow-scripts/flow, but the current directory gets mounted as /work. So, we should always be cd'ing into /OptnROAD-flow-scripts/flow.

I can then:

• call openroad_autotuner and reference a YAML file from /work or /OpenROAD-flow-scripts
• call make DESIGN_CONFIG and it finds the ORFS top level makefile. I can even specify a DESIGN_CONFIG mounted in /work
• call with a straight bash interactive shell

Let me know if you have any questions.

[luarss]

@jeffng-or Made some changes and tested with the following - does it work for your use-case?

# 1. Interactive bash — should land in /OpenROAD-flow-scripts/flow ./flow/util/docker_shell bash # 2. make with DESIGN_CONFIG from /work ./flow/util/docker_shell make DESIGN_CONFIG=/work/flow/designs/nangate45/gcd/config.mk # 3. autotuner referencing a yaml in /work ./flow/util/docker_shell --image openroad/orfs-autotuner:6422fcc-orfs-26Q1-325-g333fee363-ray-2.51.2-full openroad_autotuner -s /work/flow/designs/asap7/gcd/asap7_gcd_sweep.yaml

[jeffng-or]

Nope, it looks like it's cd'ing into MOUNT_TARGET "/work", which doesn't work for case #2 and case #1, sort of.

For case 1, I would have expected to cd to ORFS flow, since that's what docker_shell does today.

For case 2, it's semi-related, since we cd to MOUNT_TARGET and we can't find the makefile.

Just for clarity, here's my set up:

1. I have a plain old work directory with no ORFS in it
2. In the work directory, I have two files: docker_shell and AT yaml file (see https://github.com/The-OpenROAD-Project-private/AutoTuner/issues/134 for the YAML file)
3. I have an AT image loaded into my docker cache that was built using the build_autotuner_image.sh script attached. I built it with 6422fcc3c3e34170bb3bd36097e2c2281ba71177

Case 1: ./docker_shell.jack --image openroad/orfs-autotuner:26Q1-636-gb9f42fead bash
bash shell starts fine, but the work dir is /work, not /OpenROAD-flow-scripts/flow

Case 2: ./docker_shell.jack --image openroad/orfs-autotuner:26Q1-636-gb9f42fead make DESIGN_CONFIG=designs/asap7/gcd/config.mk
fails to find makefile, since we're in the wrong directory

make: *** No targets specified and no makefile found. Stop. ``` Case 3: ./docker_shell.jack --image openroad/orfs-autotuner:26Q1-636-gb9f42fead openroad_autotuner -s /work/high_util.yaml works fine 

[luarss]

@jeffng-or Sure. Let us debug case-by-case. First, I tried to copy your setup locally - let me know if that mirrors yours.

➜ jeff-test pwd /tmp/jeff-test ➜ jeff-test ls asap7_gcd_sweep.yaml docker_shell 

Also - please let me know what you see after running ./docker_shell bash (case 1)
Here's mine:

➜ jeff-test ./docker_shell bash + DEFAULT_IMAGE=openroad/orfs:latest + FLOW_DIR=/OpenROAD-flow-scripts/flow + USE_USER=false + DRY_RUN=false + PORTS=() + IMAGE= + CMD_ARGS=() + [[ 1 -gt 0 ]] + case "$1" in + CMD_ARGS=("$@") + break + IMAGE=openroad/orfs:latest + get_container_engine ++ docker --version + local 'DOCKER_VERSION_STRING=Docker version 27.3.1, build ce12230' + [[ Docker version 27.3.1, build ce12230 == *\D\o\c\k\e\r* ]] + container_engine=docker ++ pwd + WORKSPACE=/tmp/jeff-test + YOSYS_EXE=/OpenROAD-flow-scripts/tools/install/yosys/bin/yosys + OPENROAD_EXE=/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad + KLAYOUT_CMD=/usr/bin/klayout + XSOCK=/tmp/.X11-unix + XAUTH=/tmp/.docker.xauth + false + xauth nlist :0 + sed -e 's/^..../ffff/' + xauth -f /tmp/.docker.xauth nmerge - xauth: /tmp/.docker.xauth not writable, changes will be ignored xauth: (argv):1: unable to read any entries from file "(stdin)" + test -t 0 + DOCKER_INTERACTIVE=-ti + false + [[ docker == \p\o\d\m\a\n ]] ++ id -u luars ++ id -g luars + user_args='-u 1027:1028' + user_env= + PORT_ARGS=() + CMD_STR= + [[ 1 -gt 0 ]] + CMD_STR=bash + ENV_SH_CMD='if [ -f ../env.sh ]; then . ../env.sh; fi;' + DOCKER_CMD=("$container_engine" run --rm -i $DOCKER_INTERACTIVE $user_env $user_args -e LIBGL_ALWAYS_SOFTWARE=1 -e QT_X11_NO_MITSHM=1 -e XDG_RUNTIME_DIR=/tmp/xdg-run -e "DISPLAY=${DISPLAY:-:0}" -e QT_XKB_CONFIG_ROOT=/usr/share/X11/xkb -v "$XSOCK:$XSOCK" -v "$XAUTH:$XAUTH" -e "XAUTHORITY=$XAUTH" -e FLOW_HOME=/OpenROAD-flow-scripts/flow/ -e WORK_HOME=/work -e "YOSYS_EXE=$YOSYS_EXE" -e "OPENROAD_EXE=$OPENROAD_EXE" -e "KLAYOUT_CMD=$KLAYOUT_CMD" -v "$WORKSPACE:/work" --network host "${PORT_ARGS[@]}" "$IMAGE" bash -c "set -ex; mkdir /tmp/xdg-run; cd ${FLOW_DIR}; ${ENV_SH_CMD}$CMD_STR") + false + exec docker run --rm -i -ti -u 1027:1028 -e LIBGL_ALWAYS_SOFTWARE=1 -e QT_X11_NO_MITSHM=1 -e XDG_RUNTIME_DIR=/tmp/xdg-run -e DISPLAY=:0 -e QT_XKB_CONFIG_ROOT=/usr/share/X11/xkb -v /tmp/.X11-unix:/tmp/.X11-unix -v /tmp/.docker.xauth:/tmp/.docker.xauth -e XAUTHORITY=/tmp/.docker.xauth -e FLOW_HOME=/OpenROAD-flow-scripts/flow/ -e WORK_HOME=/work -e YOSYS_EXE=/OpenROAD-flow-scripts/tools/install/yosys/bin/yosys -e OPENROAD_EXE=/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad -e KLAYOUT_CMD=/usr/bin/klayout -v /tmp/jeff-test:/work --network host openroad/orfs:latest bash -c 'set -ex; mkdir /tmp/xdg-run; cd /OpenROAD-flow-scripts/flow; if [ -f ../env.sh ]; then . ../env.sh; fi;bash' Unable to find image 'openroad/orfs:latest' locally latest: Pulling from openroad/orfs 96c832531c38: Pull complete 9f9a5e519904: Pull complete ac340c570262: Pull complete 28778c6e2a7e: Pull complete 0c18dcaf792d: Pull complete 4ef2dccff35b: Pull complete 1c04d7e1feb3: Pull complete ceafea6b19cc: Pull complete 758e2ae0b345: Pull complete 203af1be6779: Pull complete ead4ed8ebcb8: Pull complete ed9ddd297dff: Pull complete 64ef0edfb123: Pull complete 8518c3ed068d: Pull complete c729d330d3da: Pull complete f1142e61c10e: Pull complete 6ecbb469fa61: Pull complete 95311e7d7ef4: Pull complete 99d2cf359e02: Pull complete edd4ec2a4260: Pull complete 80819a7067f1: Pull complete Digest: sha256:20ab92d5c43864993b15ee57fa402abf7a9f005c2d9c551585f1608f5f6eb16d Status: Downloaded newer image for openroad/orfs:latest + mkdir /tmp/xdg-run + cd /OpenROAD-flow-scripts/flow + '[' -f ../env.sh ']' + . ../env.sh ++ __setpaths ++ [[ linux-gnu == \d\a\r\w\i\n* ]] ++++ readlink -f ../env.sh +++ dirname /OpenROAD-flow-scripts/env.sh ++ DIR=/OpenROAD-flow-scripts ++ export OPENROAD=/OpenROAD-flow-scripts/tools/OpenROAD ++ OPENROAD=/OpenROAD-flow-scripts/tools/OpenROAD ++ echo 'OPENROAD: /OpenROAD-flow-scripts/tools/OpenROAD' OPENROAD: /OpenROAD-flow-scripts/tools/OpenROAD ++ export PATH=/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/bin/wrapped-cc:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ++ PATH=/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/bin/wrapped-cc:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ++ export PATH=/OpenROAD-flow-scripts/tools/install/yosys/bin:/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/bin/wrapped-cc:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ++ PATH=/OpenROAD-flow-scripts/tools/install/yosys/bin:/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/bin/wrapped-cc:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ++ export PATH=/OpenROAD-flow-scripts/tools/install/kepler-formal/bin:/OpenROAD-flow-scripts/tools/install/yosys/bin:/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/bin/wrapped-cc:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ++ PATH=/OpenROAD-flow-scripts/tools/install/kepler-formal/bin:/OpenROAD-flow-scripts/tools/install/yosys/bin:/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/bin/wrapped-cc:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ++ [[ linux-gnu == \d\a\r\w\i\n* ]] ++ export FLOW_HOME=/OpenROAD-flow-scripts/flow ++ FLOW_HOME=/OpenROAD-flow-scripts/flow ++ unset -f __setpaths + bash groups: cannot find name for group ID 1028 I have no name!@gsoc-jack-1:/OpenROAD-flow-scripts/flow$ 

[jeffng-or]

I think I see one difference. For the bash case, I was running with a docker shell that was built from the AutoTuner Dockerfile. I have a script that runs the build:

https://drive.google.com/file/d/1C6NIYKvyKV8WsdPcp7Ktym-_oNAPKhMB/view?usp=drive_link

With an AutoTuner workspace in the current directory, run:

./build_autotuner_image.sh --release_tag openroad/orfs:26Q1-636-gb9f42fead

[luarss]

Gotcha, this is my setup now:

➜ jeff-test ls AutoTuner asap7_gcd_sweep.yaml build_autotuner_image.sh docker_shell 

AT image builds successfully:

➜ jeff-test docker images REPOSITORY TAG IMAGE ID CREATED SIZE openroad/orfs-autotuner 26Q1-636-gb9f42fead 414448203378 7 minutes ago 7.04GB openroad/orfs 26Q1-636-gb9f42fead c698f8072c04 7 days ago 4.61GB 

Interactive docker_shell works:

➜ jeff-test ./docker_shell --image openroad/orfs-autotuner:26Q1-636-gb9f42fead bash + DEFAULT_IMAGE=openroad/orfs:latest + FLOW_DIR=/OpenROAD-flow-scripts/flow + USE_USER=false + DRY_RUN=false + PORTS=() + IMAGE= + CMD_ARGS=() + [[ 3 -gt 0 ]] + case "$1" in + IMAGE=openroad/orfs-autotuner:26Q1-636-gb9f42fead + shift 2 + [[ 1 -gt 0 ]] + case "$1" in + CMD_ARGS=("$@") + break + IMAGE=openroad/orfs-autotuner:26Q1-636-gb9f42fead + get_container_engine ++ docker --version + local 'DOCKER_VERSION_STRING=Docker version 27.3.1, build ce12230' + [[ Docker version 27.3.1, build ce12230 == *\D\o\c\k\e\r* ]] + container_engine=docker ++ pwd + WORKSPACE=/tmp/jeff-test + YOSYS_EXE=/OpenROAD-flow-scripts/tools/install/yosys/bin/yosys + OPENROAD_EXE=/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad + KLAYOUT_CMD=/usr/bin/klayout + XSOCK=/tmp/.X11-unix + XAUTH=/tmp/.docker.xauth + false + xauth nlist :0 + sed -e 's/^..../ffff/' + xauth -f /tmp/.docker.xauth nmerge - xauth: /tmp/.docker.xauth not writable, changes will be ignored xauth: (argv):1: unable to read any entries from file "(stdin)" + test -t 0 + DOCKER_INTERACTIVE=-ti + false + [[ docker == \p\o\d\m\a\n ]] ++ id -u luars ++ id -g luars + user_args='-u 1027:1028' + user_env= + PORT_ARGS=() + CMD_STR= + [[ 1 -gt 0 ]] + CMD_STR=bash + ENV_SH_CMD='if [ -f ../env.sh ]; then . ../env.sh; fi;' + DOCKER_CMD=("$container_engine" run --rm -i $DOCKER_INTERACTIVE $user_env $user_args -e LIBGL_ALWAYS_SOFTWARE=1 -e QT_X11_NO_MITSHM=1 -e XDG_RUNTIME_DIR=/tmp/xdg-run -e "DISPLAY=${DISPLAY:-:0}" -e QT_XKB_CONFIG_ROOT=/usr/share/X11/xkb -v "$XSOCK:$XSOCK" -v "$XAUTH:$XAUTH" -e "XAUTHORITY=$XAUTH" -e FLOW_HOME=/OpenROAD-flow-scripts/flow/ -e WORK_HOME=/work -e "YOSYS_EXE=$YOSYS_EXE" -e "OPENROAD_EXE=$OPENROAD_EXE" -e "KLAYOUT_CMD=$KLAYOUT_CMD" -v "$WORKSPACE:/work" --network host "${PORT_ARGS[@]}" "$IMAGE" bash -c "set -ex; mkdir /tmp/xdg-run; cd ${FLOW_DIR}; ${ENV_SH_CMD}$CMD_STR") + false + exec docker run --rm -i -ti -u 1027:1028 -e LIBGL_ALWAYS_SOFTWARE=1 -e QT_X11_NO_MITSHM=1 -e XDG_RUNTIME_DIR=/tmp/xdg-run -e DISPLAY=:0 -e QT_XKB_CONFIG_ROOT=/usr/share/X11/xkb -v /tmp/.X11-unix:/tmp/.X11-unix -v /tmp/.docker.xauth:/tmp/.docker.xauth -e XAUTHORITY=/tmp/.docker.xauth -e FLOW_HOME=/OpenROAD-flow-scripts/flow/ -e WORK_HOME=/work -e YOSYS_EXE=/OpenROAD-flow-scripts/tools/install/yosys/bin/yosys -e OPENROAD_EXE=/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad -e KLAYOUT_CMD=/usr/bin/klayout -v /tmp/jeff-test:/work --network host openroad/orfs-autotuner:26Q1-636-gb9f42fead bash -c 'set -ex; mkdir /tmp/xdg-run; cd /OpenROAD-flow-scripts/flow; if [ -f ../env.sh ]; then . ../env.sh; fi;bash' \+ mkdir /tmp/xdg-run + cd /OpenROAD-flow-scripts/flow + '[' -f ../env.sh ']' + bash (base) I have no name!@gsoc-jack-1:/OpenROAD-flow-scripts/flow$ 

[jeffng-or]

OK. just updated again to make sure that I have the latest and it works for all three cases. Sorry for the hassle. Let's move forward on this and the other open PRs. Then, I can work on rolling it out product-wise. Thanks!