From 9384f852e510b153d6d71716fbe66d008bf1f31d Mon Sep 17 00:00:00 2001 From: Aaron Markham Date: Wed, 2 May 2018 11:34:40 -0700 Subject: [PATCH] clarify docs build --- docs/build_version_doc/README.md | 158 ++++++++++++++++++++----------- 1 file changed, 101 insertions(+), 57 deletions(-) diff --git a/docs/build_version_doc/README.md b/docs/build_version_doc/README.md index b297712ebcd0..4fd2c10478ab 100644 --- a/docs/build_version_doc/README.md +++ b/docs/build_version_doc/README.md @@ -7,85 +7,70 @@ This folder contains a variety of scripts to generate the MXNet.io website as we * [AddVersion.py](AddVersion.py) - MXNet.io site data massaging; injects the versions dropdown menu in the navigation bar * [build_site_tag.sh](build_site_tag.sh) - takes version tags as input and generates static html; calls `build_all_version.sh` and `update_all_version.sh` * [build_all_version.sh](build_all_version.sh) - takes version tags as input and builds the basic static html for MXNet.io -* [build_doc.sh](build_doc.sh) - used by the CI system to generate MXNet.io; only triggered by new tags; not meant for manual runs or custom outputs * [Dockerfile](Dockerfile) - has all dependencies needed to build and update MXNet.io's static html * [update_all_version.sh](update_all_version.sh) - takes the output of `build_all_version.sh` then uses `AddVersion.py` and `AddPackageLink.py` to update the static html -## CI Flow (WIP) -* Refer to https://github.com/apache/incubator-mxnet/pull/10485 +## Setting Up a Docs Dev Server -1. Docs build artifacts are deployed to the `asf-site` branch from the [incubator-mxnet-site](https://github.com/apache/incubator-mxnet-site). -2. [MXNet.io](http://mxnet.io) should then show the new content. +For these instructions, you will use an Ubuntu machine. This flow has been tested on a [Deep Learning Base AMI](https://aws.amazon.com/marketplace/pp/B077GCZ4GR), although you may use the full Deep Learning Base AMI or any other Ubuntu 16.04 system with some minor adjustments. -## Manual Generation +**Step 1:** Spin up your Ubuntu server and SSH in. -Use Ubuntu and the setup defined below, or use the Dockerfile provided in this folder to spin up an Ubuntu image with all of the dependencies. Further info on Docker is provided later in this document. For a cloud image, this was tested on [Deep Learning AMI v5](https://aws.amazon.com/marketplace/pp/B077GCH38C?qid=1520359179176). +**Step 2:** Create a Python 2.7 virtual environment (see note). -**Note**: for AMI users or if you already have Conda, you might be stuck with the latest version and the docs build will have a conflict. To fix this, run `/home/ubuntu/anaconda3/bin/pip uninstall sphinx` and follow this with `pip install --user sphinx==1.5.6`. +```bash +sudo apt install virtualenv +virtualenv -p python2.7 mxnet_docs +source mxnet_docs/bin/activate +``` -If you need to build <= v0.12.0, then use a Python 2 environment to avoid errors with `mxdoc.py`. This is a sphinx extension, that was not Python 3 compatible in the old versions. On the Deep Learning AMI, use `source activate mxnet_p27`, and then install the following dependencies. +**Note:** Using a Python 2.7 environment is required to build older versions of the docs that have Python 3 incompatibilities. If you're only building the latest or version 1.0.0+, then you may use a Python 3 environment. +**Step 3:** Clone the repo. -### Dependencies +```bash +git clone --recursive https://github.com/apache/incubator-mxnet.git +``` -These are the dependencies for docs generation for Ubuntu 16.04. +**Step 4:** Install dependencies. -This script is available for you to run directly on Ubuntu from the source repository. -Run `./setup_docs_ubuntu.sh`. +This script will install dependencies for you. +```bash +./incubator-mxnet/docs/build_version_doc/setup_docs_ubuntu.sh ``` -sudo apt-get update -sudo apt-get install -y \ - apt-transport-https \ - ca-certificates \ - curl \ - doxygen \ - git \ - libjemalloc-dev \ - pandoc \ - software-properties-common -# You made need to run `/home/ubuntu/anaconda3/bin/pip uninstall sphinx` -# Recommonmark/Sphinx errors: https://github.com/sphinx-doc/sphinx/issues/3800 -# Recommonmark should be replaced so Sphinx can be upgraded -# For now we remove other versions of Sphinx and pin it to v1.5.6 +**Step 4:** Make the docs. -pip install --user \ - beautifulsoup4 \ - breathe \ - CommonMark==0.5.4 \ - h5py \ - mock==1.0.1 \ - pypandoc \ - recommonmark==0.4.0 \ - sphinx==1.5.6 +Here you have two options: -# Setup scala -echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list -sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823 -sudo apt-get update -sudo apt-get install -y \ - sbt \ - scala +* Build this current version (master) with the following: -# Optionally setup Apache2 -sudo apt-get install -y apache2 -sudo ufw allow 'Apache Full' -# turn on mod_rewrite -sudo a2enmod rewrite +```bash +cd incubator-mxnet +make docs USE_OPENMP=1 +``` -echo 'To enable redirects you need to edit /etc/apache2/apache2.conf ' -echo '--> Change directives for Directory for /var/www/html using the following: ' -echo ' AllowOverride all ' -echo '--> Then restart apache with: ' -echo ' sudo systemctl restart apache2' +* Build all versions as what is seen in the production site. This will have the versions dropdown and any post-build processing that generates site artifacts and other requirements for the production site. -# Cleanup -sudo apt autoremove -y +The following script will build all of the latest versions, set the default site to be `master` and use your dev server's IP or DNS for navigation items. + +```bash +./build_site_tag.sh '1.2.0;1.1.0;1.0.0;0.12.0;0.11.0;master' master http://your-ip-or-dns ``` -### Full Website Build +**Final Step:** Serve and test. + +Refer to [Serving Your Development Version](#serving-your-development-version) for detailed instructions. + + +**Troubleshooting**: for AMI users or if you already have Conda, you might be stuck with the latest version and the docs build will have a conflict. To fix this, run `/home/ubuntu/anaconda3/bin/pip uninstall sphinx` and follow this with `pip install --user sphinx==1.5.6`. + +If you need to build <= v0.12.0, then use a Python 2 environment to avoid errors with `mxdoc.py`. This is a sphinx extension, that was not Python 3 compatible in the old versions. On the Deep Learning AMI, use `source activate mxnet_p27`, and then install the following dependencies. + + +## Full Website Build The following three scripts will help you build multiple version tags and deploy a full site build that with each available API version. If you just want to run master or your current fork's branch you should skip ahead to the [Developer Instructions](#developer-instructions). The full site build scripts can be run stand-alone or in conjunction, but `build_all_version.sh` should be run first. @@ -236,4 +221,63 @@ There are several manual and semi-automatic processes to be aware of, but the bo 1. The root should have the current `.htaccess` file from master in `/docs/`. Make sure you've updated this in master and included the most recent version in your PR. 2. The css file from master `/docs/_static/` will be needed. Be sure that the different versions of the site work. They might need the old version, but the newer version might fix bugs that were in the tags from the legacy versions. 3. Pay attention to `mxdocs.py` as some docs modifications are happening there. -4. Review Any other modifications to the legacy versions can be seen in + + +## Dependencies + +These are the dependencies for docs generation for Ubuntu 16.04. + +This script is available for you to run directly on Ubuntu from the source repository. +Run `./setup_docs_ubuntu.sh`. + +``` +sudo apt-get update +sudo apt-get install -y \ + apt-transport-https \ + ca-certificates \ + curl \ + doxygen \ + git \ + libjemalloc-dev \ + pandoc \ + software-properties-common + +# You made need to run `/home/ubuntu/anaconda3/bin/pip uninstall sphinx` +# Recommonmark/Sphinx errors: https://github.com/sphinx-doc/sphinx/issues/3800 +# Recommonmark should be replaced so Sphinx can be upgraded +# For now we remove other versions of Sphinx and pin it to v1.5.6 + +pip install \ + beautifulsoup4 \ + breathe \ + CommonMark==0.5.4 \ + h5py \ + mock==1.0.1 \ + pypandoc \ + recommonmark==0.4.0 \ + sphinx==1.5.6 + +# Setup scala +echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list +sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823 +sudo apt-get update +sudo apt-get install -y \ + maven \ + sbt \ + scala + +# Optionally setup Apache2 +sudo apt-get install -y apache2 +sudo ufw allow 'Apache Full' +# turn on mod_rewrite +sudo a2enmod rewrite + +echo 'To enable redirects you need to edit /etc/apache2/apache2.conf ' +echo '--> Change directives for Directory for /var/www/html using the following: ' +echo ' AllowOverride all ' +echo '--> Then restart apache with: ' +echo ' sudo systemctl restart apache2' + +# Cleanup +sudo apt autoremove -y +```