Greetings fellow DLPers!
I just learned that with out of box settings only the first 100K (that's not a typo!) of extracted content is scanned for policy violations. I discovered this when asked why a particular document with violating content was not matched by an EDM policy. This has to do with the Lexer.MaximumNumberOfTokens setting. See TECH233786 EDM detection does not detect content at the end of a file for details.
Saying it does not detect content "at the end of a file" is a bit of a misnomer in my opinion as I was under the impression DLP scanned files up to 30 MB by default and I wouldn't call everything after the first 100K "the end of the file" I'd call it everything except the beginning of the file! If content isn't in the first tiny 100K of a 30 MB file detection will not occur.
To match content in the entire 30 MB of a file would require increasing the Lexer setting from 12000 (default in 12.5) to about 3,600,000 a 300 times increase.
I ran lexer up to 1.2 M in a test environment (which only matched on the first 4.5 MB Of extracted content) with apparently no effect on RAM and CPU just longer detection times but that was just a limited test.
So I was wondering if anyone else has discovered this limitation and how they have dealt with it.
Have you decided if you don't find something in the first 100K that's OK? Or have you increased it just some? What other tuning did you do? How is it working for you?
Thanks in advance!