Loading
Bookmark and Share

RSL to LLVM compiler project is started.

syoyo 15 posts

This is a discussion forum for rsl2llvm, RenderMan Shading Language to LLVM compiler.

rsl2llvm is the project to quickly find what will be the problem when we use LLVM as a shader VM and translate RSL to LLVM IR.

For quickly investigate the problem, rsl2llvm is implemented in Python: simple but powerful and productive language :).

svn tree for rsl2llvm can be accessed at

http://lucille.svn.sourceforge.net/viewvc/lucille/angelina/rsl2llvm/
(for browsing)

svn co http://lucille.svn.sourceforge.net/svnroot/lucille/angelina/rsl2llvm rsl2llvm
(for checking out)

Another discussion of using LLVM for RSL is available at aqsis forum.
http://www.aqsis.org/xoops/modules/newbb/viewtopic.php?viewmode=flat&topic_id=1580&forum=2

Any comments are welcome.

resistor (guest)

Hi. :-)

resistor 7 posts

For what it's worth, I just uploaded my prototype frontend in C++ to GitHub: http://github.com/resistor/rsl-llvm/tree/master

It's still very rudimentary, but this discussion inspired me to beat the parser into shape this evening.

syoyo 15 posts

Hi register, Thanks for uploading your RSL frontend!

I may switch to use your RSL parser/lexer after investigation was done with my rsl2llvm in Python.

resistor 7 posts

I'd appreciate help testing it. I only just got it to parse the reference shaders, so any additional testcases would be welcome. I'll dive into AST construction in the next week or two, but I don't expect that to be too hard.

It could also use a real diagnostic reporting mechanism. I'm hoping to eventually do something spiffy like clang's diagnostics, but for the moment it just asserts with a rather unhelpful message.

c42f 8 posts

Hi guys, fantastic to see such progress already!

I had a quick look at both sourcecode bases out of curiosity, and I'm surprised to see that RSL is apparently simple enough to parse using recursive descent :)

How much dependence on the LLVM codebase is envisaged for the rsl-llvm frontend? If it gets to be better than aqsl (which is quite nasty in some areas), then I'm inclined to write a aqsis shader interpreter backend for it.

It would be interesting to play with the LLVM backend too, though that would require a lot more interface work on my part.

~Chris.

c42f 8 posts

Btw, I'm currently compiling LLVM from source (takes ages ;-) ) and will attempt to get rsl-llvm working after that.

c42f 8 posts

I got llvm installed and tried to compile rsl-llvm, but am running into linking errors. scons builds libRslLLVM fine, but fails when trying to build rslc:

g++ -o rslc -L/usr/local/lib -lpthread -lltdl -lm -lelf -lLLVMCore -lLLVMSupport -lLLVMSystem tools/rslc/rslc.o -Llib -lRslLLVM
tools/rslc/rslc.o: In function `main':
/home/foster/src/rsl-llvm/tools/rslc/rslc.cpp:21: undefined reference to `llvm::cl::ParseCommandLineOptions(int, char**, char const*)'
tools/rslc/rslc.o: In function `basic_parser_impl':
/usr/local/include/llvm/Support/CommandLine.h:490: undefined reference to `vtable for llvm::cl::basic_parser_impl'
tools/rslc/rslc.o: In function `basic_parser':
/usr/local/include/llvm/Support/CommandLine.h:520: undefined reference to `vtable for llvm::cl::basic_parser, std::allocator > >'
tools/rslc/rslc.o: In function `parser':
/usr/local/include/llvm/Support/CommandLine.h:646: undefined reference to `vtable for llvm::cl::parser, std::allocator > >'
tools/rslc/rslc.o: In function `Option':
/usr/local/include/llvm/Support/CommandLine.h:206: undefined reference to `vtable for llvm::cl::Option'

[...snip...]

Any idea what I'm doing wrong? At first sight it looks like a problem with my LLVM installation, but it went off without a hitch.

~Chris.

resistor 7 posts

I'd guess it's an error on my part in the build script. I'm not really an scons expert. I'd suggest toying around with the options in llvm-config to see if you can get a set of flags that work for you.

resistor 7 posts

For what it's worth, it parallelizes well with "make -j4", so if you have multiple processors it doesn't take nearly as long. My quad Xeon machine can compile it in 7 minutes or so.

Yeah, RSL isn't a very complicated language, though I also think that recursive descent doesn't get enough credit sometimes. In my experience, it's a) easier to write for simple languages like RSL and b) easier to insert the necessary "magic" into to parse very complicated languages like C. For instance, the parser in Clang is recursive descent, though with plenty of magic.

Since I was originally writing rsl-llvm as an exercise for myself, and I'm very familiar with the LLVM codebase, I had intended to make extensive use of the facilities it provides. I'd even planned to use Clang's libLex to do preprocessing in-memory.

It would of course be totally awesome if you wanted to make an interpreter based on it. As I said above, I'll probably try to get AST generation going in the next few weeks, though I'm a bit busy w/ school stuff at the moment.

syoyo 15 posts

register,

Using Clang's codebase for RSL seems interesting, especially its expressive diagnostics facility. I'll look at clang's source code for my study.

BTW, I'd like to report following (possibly) rsl-llvm bug when processing show_xyz.sl shader.

$ ./rslc show_xyz.sl
rslc: lib/Lexer.cpp:77: Token Lexer::consume(Token::TokenType): Assertion `tok.type == t && "Unexpected token encountered!"' failed.
Aborted (core dumped)

$ cat show_xyz.sl

surface
show_xyz (float xmin = -1,
ymin = -1,
zmin = -1,
xmax = 1,
ymax = 1,
zmax = 1)
{
uniform point scale, zero;
point objP, cubeP;

/* Check for zero scale components. */
if (xmax == xmin || ymax == ymin || zmax == zmin) {
/*
printf ("bad bounding box %f %f %f %f %f %f in show_xyz()",
xmin, xmax, ymin, ymax, zmin, zmax);
*/
} else {
scale = point (1 / (xmax - xmin), 1 / (ymax - ymin),
1 / (zmax - zmin));
zero = point (xmin, ymin, zmin);

objP = transform ("shader", P);
cubeP = (objP - zero) * scale;
Ci = color (xcomp (cubeP), ycomp(cubeP), zcomp(cubeP) );
}
}

syoyo 15 posts

I'd like to report current status of rsl2llvm.

Now rsl2llvm can parse almost all shaders in the test directoly and also can emit LLVM IR code. But this is just a starting point.

In the next few weeks, I'll try to

- Write simple shader VM runtime written in LLVM with GUI support
- Think about ABI for LLVM code for some intrinsic function(e.g. noise(), texture()), which should be easily specializable.

resistor 7 posts

Did you run it through a preprocessor first? Otherwise, it's probably the comments that are messing it up.

I'm not sure how much of clang's code I'll reuse verbatim versus reusing in spirit. While pulling in the preprocessor code directly might be nice, I suspect that it's error reporting facilities may be too difficult to integrate. So I might write my own that's "inspired" by clang's.

resistor 7 posts

Spiffy! Keep us posted, particularly about ABI work, since I'll need to try to implement something compatible when I have time.

c42f 8 posts

Sounds like you're still making great progress :)

What do you mean by GUI support? Are you thinking about running the shaders in a fake execution environment to test out the VM? This is something I'd like to be able to do for the aqsis shaderVM actually - run it in a test harness which decouples it from the rest of the renderer. I'm not sure if that's quite possible currently, and TBH I don't have much enthusiasm for working on the current mess which is our VM... I can feel a (at least partial) rewrite coming up :-)

How are you going to set the VM up? I understand that the performance considerations required for raytracing are rather different from the SIMD-style VM that's relevant for reyes style rendering... Nevertheless you still have to be able to evaluate numerical derivatives to properly support the RSL intrinsic functions Du(), Dv() etc. I'd be keen to know how you manage that in a non-SIMD context.

It will be interesting to see what you have to do to deal with the ABI issues too.

~Chris.

syoyo 15 posts

You assume semicon in the tail for each formal variable definition.

shader test(float a = 1;) {}

is OK, but

shader test(float a = 1) {}

is BAD for rsl-llvm. RI spec 3.2 allows latter case.

syoyo 15 posts

> What do you mean by GUI support? Are you thinking about running the shaders in a fake execution environment to test out the VM?

Yes. I am trying to build simple shader execution environment with GUI support, which will be implemented using wxWidgets, wxPython or FLTK.

Do you have any preference for GUI toolkit so that we can share the codebase as much as possible?

> How are you going to set the VM up?

LLVM IR itself as VM. This is the reason why I am interesting in using use LLVM infrastructure.
The renderer(shader executor) executes LLVM bitcode by JIT.

But as you mentioned, sometimes we still have to go back from LLVM world to C/C++ to access renderer internal(trace(), Du(), etc).

And there's a couple of ways to map RSL into LLVM.
for example, for SPMD style(reyes) renderer it'd be efficient to emit following LLVM code,

// RSL code
shader myshader(float k)
{
Ci = k * Cs;
}

// pseudo code equivalent to translated LLVM bitcode
shader myshader(param *param, shadingpoint *ps, int n)
{> What do you mean by GUI support? Are you thinking about running the shaders in a fake execution environment to test out the VM?

Yes. I am trying to build simple shader execution environment with GUI support, which will be implemented using wxWidgets, wxPython or FLTK.

Do you have any preference for GUI toolkit so that we can share the codebase as much as possible?

> How are you going to set the VM up?

LLVM IR itself as VM. This is why I am trying to use LLVM infrastructure.
The renderer(shader executor) executes LLVM bitcode by JIT.

But as you mentioned, sometimes we still have to go back from LLVM world to C/C++ to access renderer internal(trace(), Du(), etc).

And there's a couple of ways to map RSL into LLVM.
for example, for SPMD style(reyes) renderer it'd be efficient to emit following LLVM code,

// RSL
shader myshader(float k)
{
Ci = k * Cs;
}

// pseudo code equivalent to LLVM bitcode
shader myshader(param *param, shadingpoint *ps, int n)
{
int i;
for (i = 0; i < n; i++) {
ps->Ci[i] = param->k[i] * ps->Cs[i];
}
}

int i;
for (i = 0; i < n; i++) {
ps->Ci[i] = param->k[i] * ps->Cs[i];
}
}

c42f 8 posts

Hi syoyo,

I have some preference for FLTK, since that's what all of the aqsis GUI programs use so I'm somewhat familiar with it.

Your post seems to have gotten a little messed up somehow. You said there was a couple of ways of mapping RSL to LLVM - I'm familiar with the reyes style since this is what we use in aqsis. What I'm interested in is how any other method could work. A straight SISD method doesn't seem to work since there's no obvious way of evaluating numerical derivatives like Du, or am I missing something?

Also, I'm a bit confused as to whether we're doing SPMD or SIMD here. It seems to me that an reyes-style VM is really a virtual vector processor, so it's probably SIMD...

~Chris.

syoyo 15 posts

> I have some preference for FLTK, since that's what all of the aqsis GUI programs use so I'm somewhat familiar with it.

OK, I'll use FLTK(1.1.9).

> Your post seems to have gotten a little messed up somehow.

To settle your confusin, I am planning to write an architecture proposal for RSL-LLVM integration with figures in this weekend.
Please be patient a while.

> A straight SISD method doesn't seem to work since there's no obvious way of evaluating numerical derivatives like Du

Sometimes derivative can be analytically computed in language analysis level(this is why I need a our own RSL compiler), but anyway, we need to shade 3 or 4 point simultaneously to compute derivatives and LOD(for texturing and so on) numerically for raytracing based renderer.

This fits beam-based ray tracing well, so this is why I am trying to use beam tracing in my renderer.

syoyo 15 posts
> Also, I'm a bit confused as to whether we're doing SPMD or SIMD here. It seems to me that an reyes-style VM is really a virtual vector processor, so it's probably SIMD...

Yes, SPMD is almost equal to SIMD.

When we say SIMD, basically it is referred to as short vector SIMD(4 x float as in x86/SSE and PPC/AltiVec), whereas SPMS stands for SIMD with much longer vector size(e.g. 16, 32 or more).
c42f 8 posts

> To settle your confusin, I am planning to write an architecture proposal for RSL-LLVM integration with figures in this weekend.

Sounds great :-)

Your comments on beam-tracing make lots of sense too.

resistor 7 posts

Fixed now, along with some other issues that your testcase revealed. Feel free to submit more error reports. I don't have a larger body of shaders to test with, and you've already demonstrated that the reference shaders give poor coverage of the language spec.

c42f 8 posts

I've tarred up all the shaders in the aqsis RTS and sent them to you guys - hope they're useful as test cases.

And yes, the reference shaders are very poor for language coverage - they're more aimed at demonstrating one or two simple but useful shaders of each type.

syoyo 15 posts

register, pasrseFormalType() prototype definition is missing in the header.

lib/Parser.cpp: In member function 'void Parser::parseFormals()':
lib/Parser.cpp:39: error: 'parseFormalType' was not declared in this scope
lib/Parser.cpp: At global scope:
lib/Parser.cpp:55: error: no 'void Parser::parseFormalType()' member function declared in class 'Parser'

syoyo 15 posts

Thanks c42f!

syoyo 15 posts

Hi c42f, I've wrote side and uploaded it.

http://www.slideshare.net/syoyo/rsltollvm?src=embed

Hope it helps, and any comments are welcome!

c42f 8 posts

Hey, I've read the slideshow - I think you've got a good outline of the important points there.

Comments:

* You'll need a C interface to DSOs if you want to be compatible with existing compiled shaders. The DSO mechanism could easily call back to the renderer proper (like the trace() function etc) - if somebody is not doing significant computation inside the DSO, then they probably should just write it in SL. If they are doing a decent amount of computation inside the DSO, then the function call overhead should be negligiable.

* The name-mangling scheme looks good; simple and sufficient.

* I don't think incoherency of branching constructs (if/for/while/...) is necessarily a major cause of inefficiency for SPMD execution. (I will admit that the details of the current way it's handled inside the aqsis shaderVM are far from optimal! I have plans to fix this; see http://wiki.aqsis.org/dev/beaker#shadervm_inner_loop_redesign for details). At an instruction level, the aqsis shaderVM still uses conditional jumps which means that it only runs branches which has some of the assicated vector elements currently running.

IMO, we can probably completely avoid emitting scalar code, unless you decide to use LLVM for compiling DSOs as well.

~Chris.

syoyo 15 posts

Sorry for too late reply.

Thanks for the comment, Chris.

I've been working for another coding, for example beam tracing, so I couldn't contribute to rsl2llvm for a while. But now I'm getting back to coding rsl2llvm.
I hope soon I could release a woring demo of rsl2llvm with fake shader runtime engine.

DSOs and scalar or vector is one of the biggest problem we have to solve. Your opinion is helpful for me.