Lance Mindheim's books and

Lance Mindheim's books and website. 

Byron Henderson

Byron Henderson's Layout Vision website

I also second the Lance Mindheim recommendation, of course. 

And if you ever have the opportunity to attend an event where the Layout Design SIG is offering their free consultation sessions, you should go out of your way to get there. I've had my brain reorganized by them several times.

The Journal of Model Railroad Design

Several nice shelf railroads here with 3D renderings to go along.

http://www.jomrd.com/index.php

Books and articles

I've seen lots of them over the years. They can only get you so far though as you have to have an idea of what you want to model and some research into how it should look when done to convey your vision to a viewer.The ones suggested  above plus Ian Rice should be a good place to start.....DaveB

carendt.com

There are a LOT of articles and layout ideas for small shelf layouts at  http://www.carendt.com.

Kalmbach plan books

They have published numerous track plan books over the years, with some nice plans.

101 track plans, 48 Top Notch Track plans, etc.

In addition smaller track plans can be unfolded or a single area on a larger plan "lifted" to be a stand alone switching area.

Some nice bedroom sized plans here

https://appalachianrailroadmodeling.com/category/track-plans/bedroom-sized/

Not sure if you mean a straight shelf, or a shelf around the walls of a small room, etc.

Shelf Layouts

IMHO, shelf layouts work great in small spaces, especially with an urban industrial theme when density is desired. I also think the resulting cityscape would be interesting.

Here are four shelf layouts that I have saved for future reference for use as idea starters. You might find some ideas in them, too. As with almost all track plans, these can be expanded/reduced/modified as needed, but at least you get to start with something that has potential.

I particularly like what the wye in corner brings in the corner brings to the table operationally and visually:

This one could be enhanced nicely if more space is available:

Same can be said for this track plan. One would need to Increase the depth to have any hope of reflecting anything near the visuals portrayed in the artist's rendering:

And last one for now, though you would need to have more than 1' depth to make this one work, I think it has a lot visual interest as well as operational challenge.

All of the above could be enhanced with some form of staging to represent interchange with other railroads, but at least these give you some thought fodder.

Good luck!

Andre

Definition?

I think the term "shelf layout" dates to the days when most layouts were built in the middle of a room.  Not just on 4x8 sheets of plywood either.  Many older layouts that I've visited over the years did not touch the walls.  So arguably a large "around the walls" layout like Allen McClelland's V&O is a type of shelf layout even though it was built on traditional benchwork.

Thus, does a "shelf layout" have to be cantilevered off a wall on shelf brackets?  If it has legs that rest on the floor is it no longer a "shelf layout" even if it's only a foot or two deep?

Scott Chatfield 

Books

There's a whole Kalmbach book about them, both small switching layouts, and the use of shelf construction for parts of much larger, permanent layouts. Also, some of those layouts have a TON of track crammed onto them. Modern designs tend to have less track density, and focus more on realistic operations.