manual testing framework

[bob-carpenter]

We aren't testing code that goes into the manual thoroughly and automatically enough. We need a framework where code citations in the manual are automatically extracted from running code and then pasted into the manual.

[betanalpha]

Extend this to example I/O. The introduction is riddled with references to stan_major=1, etc.

[syclik]

I'll try to create a branch that uses the listings latex package.

[syclik]

It looks like there are two main packages that are being used now. There's the listings and minted. I tried with minted because it does some useful things.

With minted, I was able to change the existing code blocks to be processed by minted.
Pros:

• full syntax highlighting; I think @jrnold was behind it!
• will be easy to include from files and specify start lines and end lines
• will be able to remove blank spaces up front easily
• there are options for syntax highlighting. Colors for keywords, etc.

Cons:

• relies on python and the Pygments package, which is straightforward to install
• it's slow. The manual currently builds in ~20 secs. With the minted package, it builds in ~2:20. That's without including files, etc. If we use a simpler lexer (just text instead of Stan), it still builds in ~2:00.

Thoughts?

I'm going to try listings as a comparison.

[syclik]

If anyone wants to try minted, use this branch: feature/issue-319-manual-testing.

To change the style of how the code is rendered, change line 211 of stan-manuals.sty:
\usemintedstyle{friendly}

I tried with \usemintedstyle{bw} and it'll generate the Stan code like it is in the manual now.

[bob-carpenter]

What's the output look like?

You mean it builds end to end from clean-all in 2 mins, or that each run of pdflatex takes 2 mins, so the whole thing takes like 10 mins+?

I'm not keen on dependencies, but then I added the fonts,
so I can hardly object if someone else thinks we need code
highlighting.

I really don't want too much highlighting---otherwise the code looks like an Xmas tree and I find it hard to read. Also, it might not print well if the colors aren't dark and high-contrast (not that I ever print anything). Also, the highlighting has to be correct. Inconsistent or incomplete highlighting is worse than none at all.

[syclik]

With the bw style, it looks like the existing manual.

I checked in a version that'll make it look like a Christmas tree. (I just
did that to demonstrate that we could.)

It builds end to end from clean-all in 2 minutes and change.

On Fri, Sep 2, 2016 at 12:57 PM, Bob Carpenter notifications@github.com
wrote:

What's the output look like?

You mean it builds end to end from clean-all in 2 mins, or that each run
of pdflatex takes 2 mins, so the whole thing takes like 10 mins+?

I'm not keen on dependencies, but then I added the fonts,
so I can hardly object if someone else thinks we need code
highlighting.

I really don't want too much highlighting---otherwise the code looks like
an Xmas tree and I find it hard to read. Also, it might not print well if
the colors aren't dark and high-contrast (not that I ever print anything).
Also, the highlighting has to be correct. Inconsistent or incomplete
highlighting is worse than none at all.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/issues/319#issuecomment-244430457, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F7csHaPLa4JIQdUXC3meBEV7WZnlks5qmFV0gaJpZM4BIN0F
.

[bob-carpenter]

You committed a verison with \usemintedstyle{friendly}.

This is way too slow. It takes over 1s/page on my machine,
and it needs multiple pass, so I estimated over half an hour
and got bored waiting and gave up.

Maybe we could do this for release version, but for
work-in-progress this is going to be a huge drag on my
working on the manual.

I wrote this entire email between pages 110 and 297.

• Bob

On Sep 2, 2016, at 6:54 PM, Daniel Lee notifications@github.com wrote:

If anyone wants to try minted, use this branch: feature/issue-319-manual-testing.

To change the style of how the code is rendered, change line 211 of stan-manuals.sty:
\usemintedstyle{friendly}

I tried with \usemintedstyle{bw} and it'll generate the Stan code like it is in the manual now.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

[bob-carpenter]

What are you running on? Did you have to do something
to juice Python?

• Bob

On Sep 2, 2016, at 7:00 PM, Daniel Lee notifications@github.com wrote:

With the bw style, it looks like the existing manual.

I checked in a version that'll make it look like a Christmas tree. (I just
did that to demonstrate that we could.)

It builds end to end from clean-all in 2 minutes and change.

On Fri, Sep 2, 2016 at 12:57 PM, Bob Carpenter notifications@github.com
wrote:

What's the output look like?

You mean it builds end to end from clean-all in 2 mins, or that each run
of pdflatex takes 2 mins, so the whole thing takes like 10 mins+?

I'm not keen on dependencies, but then I added the fonts,
so I can hardly object if someone else thinks we need code
highlighting.

I really don't want too much highlighting---otherwise the code looks like
an Xmas tree and I find it hard to read. Also, it might not print well if
the colors aren't dark and high-contrast (not that I ever print anything).
Also, the highlighting has to be correct. Inconsistent or incomplete
highlighting is worse than none at all.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/issues/319#issuecomment-244430457, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F7csHaPLa4JIQdUXC3meBEV7WZnlks5qmFV0gaJpZM4BIN0F
.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

[syclik]

No idea. I'm using python 2.7.10 and I just installed Pygments today.

On Sep 2, 2016, at 1:03 PM, Bob Carpenter notifications@github.com wrote:

What are you running on? Did you have to do something
to juice Python?

• Bob

On Sep 2, 2016, at 7:00 PM, Daniel Lee notifications@github.com wrote:

With the bw style, it looks like the existing manual.

I checked in a version that'll make it look like a Christmas tree. (I just
did that to demonstrate that we could.)

It builds end to end from clean-all in 2 minutes and change.

On Fri, Sep 2, 2016 at 12:57 PM, Bob Carpenter notifications@github.com
wrote:

What's the output look like?

You mean it builds end to end from clean-all in 2 mins, or that each run
of pdflatex takes 2 mins, so the whole thing takes like 10 mins+?

I'm not keen on dependencies, but then I added the fonts,
so I can hardly object if someone else thinks we need code
highlighting.

I really don't want too much highlighting---otherwise the code looks like
an Xmas tree and I find it hard to read. Also, it might not print well if
the colors aren't dark and high-contrast (not that I ever print anything).
Also, the highlighting has to be correct. Inconsistent or incomplete
highlighting is worse than none at all.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/issues/319#issuecomment-244430457, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F7csHaPLa4JIQdUXC3meBEV7WZnlks5qmFV0gaJpZM4BIN0F
.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

[bob-carpenter]

I really hate that output. Maybe if all the colors
were darker and less opposite each other and there was
no bold to mess up weight I might be able to learn to
accept it if everyone else really really wants highlighting.

Seems to have failed somehow at the end, but the real problem
is that it takes a long long time.

> make clean-all
> time make manual

...

make[1]: *** No rule to make target `stan-reference.w18'.  Stop.
Latexmk: All targets (stan-reference.pdf) are up-to-date
  building src/docs/stan-reference/stan-functions.txt
cp src/docs/stan-reference/stan-reference.pdf doc/stan-reference-2.11.0.pdf;  cp src/docs/stan-reference/stan-functions.txt doc/stan-functions-2.11.0.txt;

real    17m16.612s
user    15m28.673s
sys 1m39.578s

Running with make -j4 doesn't seem to be any faster.

• Bob

[syclik]

I saw that error too.

Changing the style to bw helps, but yes, the time is a killer. I'll try the listings package out. Less dependencies is better.

On Sep 2, 2016, at 1:23 PM, Bob Carpenter notifications@github.com wrote:

I really hate that output. Maybe if all the colors
were darker and less opposite each other and there was
no bold to mess up weight I might be able to learn to
accept it if everyone else really really wants highlighting.

Seems to have failed somehow at the end, but the real problem
is that it takes a long long time.

> make clean-all
> time make manual

...

make[1]: *** No rule to make target `stan-reference.w18'. Stop.
Latexmk: All targets (stan-reference.pdf) are up-to-date
building src/docs/stan-reference/stan-functions.txt
cp src/docs/stan-reference/stan-reference.pdf doc/stan-reference-2.11.0.pdf; cp src/docs/stan-reference/stan-functions.txt doc/stan-functions-2.11.0.txt;

real  17m16.612s
user  15m28.673s
sys   1m39.578s

Running with make -j4 doesn't seem to be any faster.

• Bob
  —
  You are receiving this because you were assigned.
  Reply to this email directly, view it on GitHub, or mute the thread.

[bob-carpenter]

Why do you want to do this? Is it that people want syntax
highlighting? If it's not subtle (by which I mean you can
barely see it at arm's length), I'm going to object.

• Bob

On Sep 2, 2016, at 7:30 PM, Daniel Lee notifications@github.com wrote:

I saw that error too.

Changing the style to bw helps, but yes, the time is a killer. I'll try the listings package out. Less dependencies is better.

On Sep 2, 2016, at 1:23 PM, Bob Carpenter notifications@github.com wrote:

I really hate that output. Maybe if all the colors
were darker and less opposite each other and there was
no bold to mess up weight I might be able to learn to
accept it if everyone else really really wants highlighting.

Seems to have failed somehow at the end, but the real problem
is that it takes a long long time.

> make clean-all
> time make manual

...

make[1]: *** No rule to make target `stan-reference.w18'. Stop.
Latexmk: All targets (stan-reference.pdf) are up-to-date
building src/docs/stan-reference/stan-functions.txt
cp src/docs/stan-reference/stan-reference.pdf doc/stan-reference-2.11.0.pdf; cp src/docs/stan-reference/stan-functions.txt doc/stan-functions-2.11.0.txt;

real    17m16.612s
user    15m28.673s
sys 1m39.578s

Running with make -j4 doesn't seem to be any faster.

• Bob
  —
  You are receiving this because you were assigned.
  Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.