Files/Directories being indexed

[EdenWeb]

I was doing a little link checking on Derek Knight Photography and notice that Google, Yahoo, MSN etc are all indexing parts of the sites such as

Index of /commercial/cfw
Index of /commercial/europa

etc etc

There are no pages there - just views of the directories etc and old pages which are half new design and half old.

Is there a good way to stop this from happening and what are the consequences?
Thanks all!

 

[davidbehan]

remove directory browsing on the server would stop it

 

[mneylon]

You could block most search engines via a robots.txt with explicit DENY rules

 

[EdenWeb]

remove directory browsing on the server would stop it

Thanks Dave. I'll log into the CP and see what my options there are.

 

[Forbairt]

As Michele said ... I'd do the robots thing ...

User-agent: *
Disallow: /directory1/
Disallow: /directory2/
Disallow: /directory3/

for added protection ...

to your .htaccess file you can add

IndexIgnore */*

They should work but I'm feeling sleepy today

 

[davidbehan]

robots.txt would work too but I think directory browsing should be turned off anyway. You never know what someone might stumble across, ya know!

 

[kjt]

Personally I have directory indexing turned off and I pop specific folders I don't want accessed into my robots.txt file